AI Simulations & Agents as Entertainment

Can AI simulations be a legitimate form of entertainment people watch and enjoy?

The following is a transcript of the AI Agents and Simulations Panel from AI on the Lot 2024. You can watch the full video of the panel discussion on the AI on the Lot YouTube page here.

This panel considered whether AI simulation systems have the potential to be a new medium of storytelling and entertainment. With generative AI enabling near real-time content creation and AI agent frameworks allowing for responsive content, there is an increasing number of artists are moving beyond static formats. What will these dynamic agent-powered stories look like?

This panel brought together:

• Edward Saatchi, CEO of Fable Simulation.

• David Slade, Director (Black Mirror: Bandersnatch)

• Shuying Luo, Co-Founder & CTO of Altera

• Parth Patil, AI Specialist at Office of Reid Hoffman

Panel Transcript

Joanna Popper: We have a fantastic panel lined up for you to talk about AI and agents and simulation. Edward, would you like to give a bit, a short intro and talk about some of the projects we'll show a bit of your video.

Edward Saatchi: Hi, I'm Edward Saatchi, CEO of Fable Simulation and we're building out a simulation of San Francisco, which is SimFrancisco. And attached to that simulation are many television shows. Just as New York might have different genres like Friends and NYPD Blue, we’re going to have the same with San Francisco. And the first show is Exit Valley, which is a vicious but loving satire. Of these AI billionaires who are telling us that they're going to end capitalism, they're going to end money.

And it's just a coincidence that they're going to become incredibly rich in the process. So we think that we should use AI against them and on an industrial scale make fun of them. So that's the first show connected to exit valid connected to San Francisco: Exit Valley.

Joanna Popper: You. All right. And next up. We have David Slade. David Slade is a film director known for Bandersnatch, which I'm sure many of us in here watched. David also worked on Twilight, Breaking Bad, other Black Mirror productions. It’s so great to have you with us. Could give a little bit of an introduction, you know, you've had a amazing career in Hollywood so far, and how are you thinking about working with AI and what's most interesting and exciting to you in the space right now?

David Slade: Yeah, there's a couple of anecdotes, but it'll probably take too long. So I'm going to jump into during the strike, the writer's strike the actor's strike last year or the year before, whenever the hell it was. Because of COVID we have a missing year.

So last year I had been using Mid Journey, I'd been using various other AI tools, and I couldn't work because there was a strike, and the commercial industry also died. And I started training myself. What are the possibilities to make movies and to tell stories with these tools? To tell stories, in fact, that may be impossible to tell using traditional cinematic language. And I started making and working on a series of different things. I did a series of short films. I did a music video. This was before the gen AI programs that enabled easy movement of images, so it was a series of Midjourney sequences that weren't animated. And then I made a little short storyboarded film, which is a little silly noir science fiction takes place in a liminal bar.

I then looked at each shot and I generated, sometimes Midjourney images which would then animate with programs like Pika oand sometimes just sections from Runway. But what I found to be incredibly lonely about the process was I couldn't involve actors. And I think this is why I, you know, I'm a filmmaker coming into this panel about simulated actors.

My drive is to continue to do this, to make these films. And weirdly, I've been spending years and years explaining to people what I want. It seems to work with AI. I seem to get the images. It seems to look like my work, weirdly. Because I'm using, and I'm figuring out which, you know, I'm figuring out my own language—Promptese. But, I really want actors to be involved. Because I can type in the word “he's hysterically happy”, right? And you might get some random noise, but you might get it exactly right. If you talk to an actor, they might know what to do, but most likely you need to use a verb that lets you get them there and that's a language.

And then you collaborate and then whatever idea I had, we made it way better by this actor who has years and years experience. And so that is my, you know, what I'm interested in with regards to this panel is the process by which I can bring people and if not people, entities and figure out how to continue telling stories with them.

Look, at the end of the day the metaphor is we're all sitting around a campfire, we're all telling stories. As a director, I came up with the analogy that, you know, the story is the text, it's written down. If someone tells that story, and it's a really flat voice, everyone goes, that's a crap story, someone tell me a joke.

But if you tell the story evocatively, then you become the voice. And that's one of the definitions of what a director does. And as a director, I am acutely aware that you need other people. It's fantastic to have nobody criticize you, and you can do anything you want in your film, and you can put anything in it, and nobody's going to recut it, and it’s done. But this is a lonely process, which makes me say, “I wish I had some mocap in here. I wish I could plug mocap into this, and I could then workshop the scene with an actor upstairs in my living room until we're done and then we'll shoot it.”

So that's my position. I am a traditional filmmaker. My points of view come from a career of making films and telling stories. Before that I was a writer. I made music videos, make commercials where you tell stories in very short periods of time. I’ve also told stories in two hours.

And so what I'm interested in is two things. One, what are the stories we can't tell with traditional media? And are they interesting and can we make those? I made a series of just proof of concepts which are based on dreams, you know, that I had. I'm very good at that. AI is awesome at making dreams. I can tell AI a story about a dream that I had the night before and I can get pretty much exactly what it felt like and looked like to me. It's great for that. Not so good for realistic things yet, but of course we're on an exponential curve and, you know, recursive exponential curves eventually get there.

So yeah, I guess I probably talked a lot.

*Below is a clip from David Slade’s Black Light Cycle* as well as a music video. *

Joanna Popper: Shuying Luo is the co founder and CTO of Altera. So tell us a little bit about Altera and then we'll show your clip as well.

Shuying Luo: Yeah. Thanks a lot for inviting me on stage. Yeah, I'm the CTO for Altera. Altera is a deep research focused company. Our mission is to really build digital humans that live, love, and grow with us.

What does it mean for digital human beings? What is missing for like the AGI, like all the trajectory, isn't intelligence all we need? So our answer is that for a human, there's some fundamental qualities that are not related to intelligence. They're like emotions, coherence, long term theory of themselves, self-awareness.

All of them are not part of intelligence. At least not part of the language. They're the describable part of human beings. For example, fast processing is not controlled by our intelligence, it’s our prefrontal cortex. In our company, we wanted to build this coherent module of a real human being.

Virtual beings, may not be humans. A virtual being can be living on a machine, and be living in together with humans, and be coworkers, buddies, friends, or whatever, and live together in the society. Here is an example of an AI buddy we built in Minecraft.

Joanna Popper: Thank you. So I'm excited to talk more about, about some of the work you're doing with Minecraft and beyond. We'll circle back to that. Okay, Parth. You work with the office of Reid Hoffman. So talk to us a little bit about how you came to be there. And then some of the work you're doing there, like Reid AI. And then we'll show a clip of Reid AI.

Parth Patil: Cool. In a previous life, I used to work for startups and I was a data analyst. And then I was working at a company a little over a year ago called Clubhouse, it was an app made in the pandemic. So I worked on clubhouse and part of when I was working there was when chatGPT came out in November of 22. I very quickly realized it was the fastest growing topic in every single language in every single country.

And I was also lucky enough to meet people all over the world through the app and learn how they were using chatGPT. And I think there were a couple moments where I realized, I think one, I met a guy and he was like, you need to be programming with this thing. It's like GPT-4 is an incredible programming assistant and it can teach you how to program.

And so then I was like, okay I’ll check that out. And then I met a guy who was a farmer from my home state in India, a state called Maharashtra. And I met this guy and he was using GPT-4 to help him plan his crop cycles. And that's when I realized this is not just for your email. Like for some people, this is actually like a new kind of computer where you wouldn't expect that person to have access to a data science team.

But now they don't need one because they can just talk to the intelligent copilot in their pocket and help and it helps them reason over, you know, decisions they're making that really impact their life. So for me, it was like a realization that we might have discovered one of the, one of the most important technologies that is on like our doorstep, this idea that the abundance of intelligence and that we can access it through natural language, whatever language you speak.

That, that was mind blowing to me. And so then I started working on GPT 4 side projects every single day. I'd have it teach me all kinds of things. I started off, I'd be like, no, teach me how to clone my voice. And then it wrote the first version of that program. And I was like, wait a minute, teach me how to build a GPT powered chat bot.

And then it wrote the first version of that program. And then I was like, wait. Like even sticks and stones don't tell you how to make fire, but intelligence is something else. And we had layoffs and I thought that was the best thing that could ever happen to me. Cause then I was like, now I get to do this full time.

I just spend all my time and money seeing how far this rabbit hole goes. And so I explored all of generative AI that I could get my hands on everything from mid journey to other language models to even runway and Pika labs. And then I started building co pilots for almost everything I could think of.

So from music production to co pilots that I'm working on for game design to just help me, you know, clean my apartment. But the way I see it is if you help you clean your apartment I have a custom GPT and I just aim it. I just take a picture of a mess and then it's prewired to just look at the mess and then suggest ways to clean everything up and then things that I might need to buy to organize my life.

Joanna Popper: Wow. Does the Roomba get jealous?

Parth Patil: The, oh my God, I have a Roomba and I have two cats and I'm pretty sure my cats don't think the Roomba's intelligent, 'cause it gets stuck in the corner. And I'm just sitting there like, why can't I just tell you to to go home like you're done.

You can't do this. I'll do it. But actually I think in the future we're gonna see (especially with the new GPT-4 model as developers start putting it into everyday technology) a lot of inanimate objects interacted with through conversational experiences. The conversational experiences and their constructed personalities will actually make them a lot more useful.

Joanna Popper: All right. So Edward, let's go back to you. Edward was the Obama digital campaign was one of the co founders of Oculus story studios has won Peabody's enemies. Correct? Yes. And it has, is now working on, in addition to this, a number of. A AI cinema.

Edward Saatchi: Yeah. I mean, I love what you were saying about that. And we think about that a lot when making AI movies and, you know, I think that loneliness is gonna be a huge problem.

When you were talking, I was just thinking about Marlon Brando and Bernardo Bertolucci and Last Tango in Paris. It's kind of hard to see how you're going to make a movie, a great movie that doesn't have that conflict between the actor's vision, the director's vision, the DP, the editor contributing something. So we've just on that topic of of it, we've been thinking about AI crew members and how to kind of grow those in simulations so that they have agency and are able to push back.

Cause so much of that push and pull is important to creativity. So let's talk a bit about the, you know, so you also, you, we've created show runner. And so how are you thinking about, you know, you like bringing some of the agents in or are you bringing those into the cinema or are you, are those separate?

Yeah, I think in both cinema and TV, having AI crew members is really exciting. I think what's unique about Showrunner is, you know, people have started saying platitudinously, don't worry, AI is just a tool, it's like VFX. But my hope for Showrunner, eventually, Is that there'll be no humans at all.

And it will just generate amazing content for us from the simulation. On the other hand, I also am very excited about movies that are made by a writer director, by a visionary. But we have to be honest about the difference between content and art. And a lot of us consume a huge amount of content. I re watch lots of shows that I love.

I think it would be very much better to watch some new episodes of shows that I love, or new shows that are in the kind of area that I enjoy. That's content. That's something I do to relax and to kind of, you know, have time. It's a lot of what we actually watch. And then there's art, and I think I'm very excited to see writer directors who, maybe they can't get their movie made.

Maybe they are more like painters or novelists. They like working. Obviously, Julian Schnabel is able to kind of make that transition, but a lot of filmmakers, a lot of people who might make amazing films, we've never seen their movies, because they can't command a set, they don't feel comfortable working with 30 people or 40 people to get their vision out.

David Slade: On that artistic side, I'm excited to see writer directors who can make incredible movies that will really shock us and take us inside of their imagination. Very strange vision of the world are going to revolutionize content. And it's also going to revolutionize art.

As a filmmaker who's worked both in the studio system and independently, there’s one issue that constantly you butt up against. There’s this thing that happens where market forces come to play. And this is really the reason. It's not to do with ambition, or doing your 10,000 hours ala Malcolm Gladwell. Because you get to a certain point, and then you go, yeah, it costs a lot of money to take a thousand people into a desert with twenty cameras and do a massive explosion sequence. And I don't miss that. I think that'll be the first thing to go. I think it's inevitable that there'll be vanity projects for Christopher Nolan. We love Christopher Nolan. I love Christopher Nolan. I love his movies. But he's the only person that can do that now. He's one of the very few people who can do that.

You have to have power over the market forces to shoot film. And then to actually have the intelligence to say, I need a hundred millions worth of prints and advertising so that somebody sees my film and it becomes a financial success. And so these are the tides that push filmmakers right now.

And what I'm looking at, in an optimistic manner, is guaranteed production values. We can make your picture look good, but if you've got tons and tons of experience, you can go in and tweak that and make it look exactly how you want. But the main thing is, we're not gonna go to a Nielsen testing audience to look at your film and tell us whether it's any good or not.

Which is gonna trust you and it'll either be entertaining or it won't. And that's how more films get made. Now, my fear is My great fear is that there will be an AI version of the Nielsen tests and creativity will become a commodity that can be analyzed and quantified. And this is the kind of dream.

This is the edge that we're kind of traversing right now.

Edward Saatchi: To that point on content, which is very different to art, but it's what we actually spend a huge amount of our time consuming. To the point of content, hopefully, eventually generating movies and shows will be as simple as browsing Netflix. Right now you have a delay of three years of Netflix noticing that people really want more movies like Event Horizon. It's kind of amazing. We’re still at the peak of that particular genre and then, you know, you get a glut of these movies. Instead, you might in the future just search, you know, Event Horizon, and it's more like this. You've got some posters, you click on the poster of the thing that looks a bit like it, and you get a movie or a show like that.

So that's, from the showrunner perspective, that's a provocative thing, but I think AI is getting so good so quickly that it's a plausible thing. And then on the artistic side, there are so many people who are just more suited to novels. My mom was a novelist or painting or solitary art. And I'd like to see their movies.

Because right now, I'm not excited about a lot of what's in the cinema. I'd like to see the movies of people who are too strange to operate within the structures of Hollywood. So I think it's going to be terrific for content and what we watch on Netflix on a Thursday evening. And it's going to be terrific for art and the things that actually fuck us up inside.

So I'm excited on both sides.

David Slade: So the other side and the concern that I have to, that the span out that I can throw into the works there is that if you look at, if you have a kid, and I have a 7-year-old, and if you look at market force driven content that has no cultural relevance at all, you can find it. It's on YouTube Kids. So if you go on YouTube Kids, you will find endless skits done by parents pimping out their children and doing skits to make revenue off of YouTube, which I think is a really, a devolution, not an evolution.

Yet at the same time, taking away the cost of making a production and then allowing people who can train and learn is explosive and amazing and incredible.

So it's got the dark side as well. I think this whole aspect of it.

Joanna Popper: So Shuying. Talk us to about the work Altera is doing with AI agents and simulation inside of the game Minecraft.

Shuying Luo: So the reason we choose Minecraft is because Minecraft is like a very open ended, very like composable world. Basically, everything can be crafted. Everything can be. recreated inside the game. And the agent, like any user in it can create everything inside the wall. There's, we can say that like a lot of games, there's kind of like gates here and there where the NPC cannot do this or that, but Minecraft doesn't have it.

That is why we choose Minecraft. What we wanted the users to get out of it. It's they would feel like they're probably playing with another friend they have. That friend is not a superhuman being. It's not like a bot that can craft the world immediately so fast that no human being can do. They probably not even know how to play with Minecraft.

But they would react like a friend would do. They would probably talk converse with you during the gameplay, not about the game itself, care about you. If you play with them for say an hour and come back a week later, they will remember that experience and probably introduce you to some other human being that they've played with or some other agents that have played with and that experience.

And so we are hoping is that when the user come in, they would open up a, like a different form of social interaction. They not only have more fun. playing the game, but they would have more fun finding like some emergent behavior just by interacting with their agents. And they were what we are hoping is that we can bring this like friend, friendship, their.

How we are able to show a little bit of vulnerability, that is how we make friends and form deep relation to make this happen easier in the virtual world.

And it's also, I think it's not only the pandemic, but like it's also just growing up. Yeah. Like when you're getting older, it's harder to show vulnerability. And that is, I think that is part of what I think AI and technology can help to bring this down.

Joanna Popper: So it's also interesting because you said that the AI can also introduce you to both other people as well as other agents in there.

So it can help you draw drive connection both with AI or NPC as well as other humans?

Shuying Luo: Yeah, exactly. So as I part of a very important human quality is that people form long term memories, like the long term experiences really form like their fundamental or different personality characteristic for humans.

So we believe that for the agents to really be kind of. The agent should be having consistent long term memory, and that is how we can make them coherent, and that is how they can help others. Would you hold the mic a little closer to you? Oh, sure. I'm sorry. Going back to the question, sorry.

Oh yeah, we believe that the long term memory is the integral part of a human being. So we wanted to bring that to the agents so that is what can help them to like, help form more social relationships with others. Because they remember you, they have a thinking of you, and that thinking will continue to change as they interact with more people.

Like with other environment or human beings. And then that overall enhances your experience playing Minecraft or whatever game you're inside. Minecraft is our starting playground because it's like very composable. So it's like a very good experiment filled with like many different possible behavior.

And yeah, it can also go beyond Minecraft. It can talk to you in Discord or see your posts on X.

Joanna Popper: So Parth, you Reed's well known for having worked very closely with OpenAI and worked closely with InflectionAI, as well as many other companies, of course. What was it that made you guys decide now is the time to make a Reed AI and release it?

Parth Patil: I think when opening, I released custom GPTs back in November, I was already building, like I had been already building custom GPTs on the API for eight months at that point. And so they just made it really easy and the models have larger context windows. So a year ago we were working with GPT for 8, 000 token context.

And now we're at GPT for one 28 K tokens. And then. You know, the new Gemini model has 2 million context window. So the working conversational memory of these models is getting very large now. And you can imagine we will continue peop They'll keep solving problems to make it possible for theoretically infinite context, is the theory.

But, why now? I guess, we wanted to demonstrate what was possible with commercial grade technology. Custom GPT is, I would say that's your commercial grade AI agent, light. It's not proactive, but there's a number of things that like you can do with agents when you build them even more custom than custom GPTs on open AI.

And then for the video avatar, even that's a consumer application. That's not there. There's more expensive video avatar technology. That's better in some ways, maybe going to be applied in Hollywood, but that's something that you can subscribe to. There's an app called our one, you can make your own avatar.

So we wanted to see what was possible with a pretty small budget, a decent idea on, you know, building an avatar on a guy who's written five books, has given numerous speeches is on a couple of podcasts. Can we create a character? And I think of this, like when you're building agents, I think it's really as someone that designs an agent, I think of it as character design.

And there's like different components to character design from this lens of taking a language model and getting it to pretend to be someone. Referencing primary source data that it has access to using tools. And then like the resulting effect is this like illusion of, you know, a person, I think the bigger takeaway is when you try to replace someone, you discover just how multifaceted they are and just how hard it is to replace a person their actual like I woke up in the middle of the night and had an anxiety attack once because I was like, man, I'm pretty sure chat, GPT is better than me at my job.

And then I, and I talked to my teammate about it and she's part, I'm pretty sure this is better than our CEO. And then I was like, Oh, wow. I think everyone has some of the sentiment of I just need chat you know, me and chat GBD against the world, which is actually a new metagame, like you and your AI, your system of AI tools will allow you to reach into multiple skill sets now that you previously would have had to dedicate a separate career to.

And so I can make a music video using mid journey and runway in two days. You know, I talked to GPT four and it's yeah, you just get this editing tool, here's ideas and it, you know, you have these iterative conversational experience that goes through the idea, makes it, you realize it's real. And then you go make it happen and you realize you have a new ceiling and you're no longer just a data analyst at a startup, like you start visualizing yourself as something else.

And he's not here, but there, there is a guy who he saw what I was working on and he's. James Blevins and he described what I was doing, building agents. He's I think this is actually a new kind of art. And the more I think about it, it is like character design. Like it's what do you, how do you want it to act?

How should it present information? What are its tendencies? And then you know, where does it exist in the world? How does it touch people? Does it leave an impact on them? And more and more, I'm seeing AI agents as like an art form. Then a science, we're going to go to questions as next. So get ready with your questions since this is AI on the lot, you know, how do you what one thing would you want the audience to be thinking about around agents and simulations and storytelling, you know, whether they're creators or IP holders or tech companies I think you're going to see agents of all kinds.

You know, throughout society, they might help you run your business. They might help you solve a creative problem. And I think this is potentially the biggest creative augmentation in human history. Create like the, it's hard for me to compare this to any other thing that people have done, but the version of you.

When you're using tools like this, your creative limit, your potential, it might be hard to visualize, but you should maybe throw this, throw these technologies at things you're actually passionate about. Don't think about it as like a job thing. Think about it as like an exploration, because what you do for fun, what you're passionate about, like no one else has to tell you, and then you might have a high bar for what success looks like.

And when you use these technologies, you realize what they're not good at. And where they fall short, and then you come in as like that grounded person in the real world and you say, Hey here's my opinion. Here's my subjective taste. And I think your ability as a curator is really important because you know, you can generate 500,000 clips, but someone's gonna have to decide what to do.

People end up seeing and that decision is actually really important. It's grounded in your lived experience, the people you've met and like the life you've led and that's unique to you. So keep that because that's actually like the sauce, you know.

Shuying Luo: This is such a hard question. I think that for agent and simulation, what I'm hoping to see is whether there's some emergent behavior that we could observe in this type of simulations, because I think that The machines and compute just open up so much possibility, like so many more possibility in it that like really cannot the amount of compute creates more possible scenario than what we can see in, in just like regular human society.

Really hoping that it would just like from there emerge something that we have never seen and thought about before. On the. So I'm more coming from a tech background, so I think it is a time that This may become controversial but as for people coming from tech backgrounds, I think it is time when something like, I actually talked about that with Parth just before this that things like consciousness.

First, it's just like kind of a more like a philosophical kind of questions, but I think we can start to see whether those things are, can be scientifically benchmarked and evaluated like from a very basic sense and we can build from like the, I think it's a time when like from the ground up we can start building some kind of scientific measurements around that because in, in our human brains and and actually in different animals.

Like their neural circuits of like, how you feel happiness, how you feel about the different basic emotions like fear. It's well studied, but it's not like a t I think it's now a time that we can start experimenting them on like machines and see if we can replicate that with some type of models.

Edward Saatchi: I would just recommend anyone working on agents to try to bring together the light and the dark. And that so much of the tech companies who are trying to build these agents do not, there's not an artistic understanding that an actor might bring to it or that a novelist might bring to it.

We're all very messy people in our heads. And so if we're trying to make real agents, we can't just have them all be very nice to each other. We've got to think about the light and the dark.

David Slade: Just seems to me that actually one of the things to focus on might be just modeling charisma.

Yeah, that's super interesting. Because there are, you know, actors who have this one eyebrow. And that's what they do. And that left eyebrow has got a different agent to the right eyebrow and it makes more money because the eyebrow is the moneymaker, right? But it's not that simple. No, of course it's not.

But it's charisma, which is an unknowable thing. It's what we do when we do casting. It is not magic. It is very much, you know, in the world of objective. It could be objectively quantized but that I think is more interesting than The wide open thing of a person.

Edward Saatchi: If I could say one last thing before you go to questions.

To that very point, David Thompson, who's a brilliant critic, wrote about Cary Grant, that he was the greatest actor of all time, because he had that light and dark. He was unpredictable. He was potentially sadistic, and he was also extremely funny, and lovely. You know, you need that kind of unpredictability in your agents.

Joanna Popper: Thanks all. Now let’s take some questions.

End of Transcript

You can watch the full panel video on the AI on the Lot YouTube channel.