GPT Realtime 2 Can Now Run Your Entire Computer (Just Your Voice)

In the beginning, it was just a command
line.
Then a mouse.
Then a whole desktop.
Then we put it in our pocket.
We even tried talking to it.
>> Sorry, I didn't quite get that.
>> And now
Okay, cut cut cut. This intro is not
working. Hey chat, pause Premiere.
>> Paused.
>> Now uh go back a couple frames.
>> Went back two frames.
>> Yeah, I'm just not loving this intro. I
need to think through this more. Let's
throw on some music to lock in. Open
Spotify.
>> Spotify is open.
>> Nice. Now put on some Herbie Hancock.
>> Putting on Herbie Hancock now.
>> There we go, much better. Now open
Claude Desktop.
>> Claude is open.
>> Nice.
>> [music]
>> Now open my YouTube script project.
>> [music]
>> Your YouTube script project is open.
>> Perfect. And ask Claude to rewrite the
intro.
>> Okay, I'll pass that rewrite request
along to Claude now.
>> [music]
>> Claude responded. This is GPT real time
two in action.
>> All right, nice. I've got it from here.
This is GPT real time two in action and
in this video I'm going to show you
exactly how to build this yourself.
Everything from opening your apps to
fully commanding them just by talking.
You will get a glimpse into the future
of a new kind of operating system when
you can run entirely with your own voice
and the best part, no coding or
technical knowledge is required. All it
takes is a few prompts to Claude code.
So what would it actually take to run
your entire computer with just your
voice? Well, turns out it's way less
than you think. Hey chat, open the real
time two GPT docs from Google.
>> Got it. Pulling up those docs in your
browser now. The search results are open
in your browser now.
>> Thanks. Now, can you click the GBT
real-time 2 model, that first link below
AI overview?
>> Sure. Clicking that first result now.
>> It's crazy. I'm still not used to this.
So, we're going to feed these GBT
real-time 2 docs to Claude code in a
second, but first, just to explain what
GBT real-time 2 actually is. You have
almost certainly talked to ChatGPT's
voice mode before or advanced voice
mode, but this is not that. Voice mode
will chat with you, but it can't reach
out and touch anything else on your
computer. GBT 2 is the same kind of
model, so you can talk with it, it can
reason, it can talk back, but this one
can also call tools, as in fire off real
actions while it's still in conversation
with you. And that really is the whole
unlock of GBT real-time 2. With these
tool calls, they're just little actions
you hand the model, so a small function,
it's a little bit of code that GBT
real-time 2 can trigger on its own. So,
let me show you two quick visualizations
of this. If we go into GBT real-time 2
in OpenAI's playground, so go to What is
this URL?
platform.openai.com/audio/realtime/edit.
I'll go back to that main page here, and
I've already created this test, but what
you can do is just click create and can
start to experiment with this. So, if I
click into test, I have these basic
system instructions. It doesn't really
matter with this test, but I'm saying
you're a fast, friendly voice assistant
that controls the user's computer, yada
yada. Now, the important pieces of this
are the actual functions. So, you can
see I have two functions here, open app
and web search. So, if I just click into
open app, you can see it's just a little
bit of JSON, and this is what tells the
agent to do. So, it instructs the agent
on how to actually open an app. Now,
these are just sample functions, they
won't actually do anything when I hit
start session, but let me show you what
happens when I talk with it. So, we'll
just save this. I'll also show web
search real quick, too. It's similar
kind of JSON, and this actually
instructs the agent on how to search the
web. If I just go and I hit start
session. Hey, chat, can you hear me?
>> I hear you loud and clear.
>> Nice. Uh how's it going?
>> I'm doing well and happy to be here. How
are you?
>> Good. Can you open an app on my
computer? Open Spotify.
But, um, so just like that, you can see,
I don't want to keep talking to it. App
name Spotify. Now, it's not going to
actually open up the app because it
doesn't have actual access to my
computer, and that's exactly what's
happening with the real app that we've
built, where it actually has computer
access. And then the second
visualization I wanted to show are
actually these logs here. So, this is in
terminal. This is actually live in our
app. You can see when I said, "Can you
click the GPT?" It didn't transcribe it
right, but GPT real time two model, it
was able to infer I meant GPT real time
two, and it actually clicked. You can
see the link. That's the tool call
there. It clicked the link position
first. And then above, when I said,
"Open the real time two GPT docs from
Google," got it, and then you can just
see the web search call, query real time
two docs. And that's essentially what
we're doing. We're just mapping out in
an easy-to-understand way exactly how to
do things on my computer, like open
apps, like do web search, like open
notes. So, now let me show you how to
actually set this up yourself. Of
course, it's actually really easy to do,
as you might imagine. All we need to do
is go over to Claude code, and we're
just going to hand it those docs that we
were looking at a second ago, the GPT
real time two docs, these ones here. So,
we'll just go here, copy this, go over
to Cursor, and open up Claude in a new
session. We're going to type in this
prompt. The prompt doesn't really
matter, but if you want, you can pause
your screen and grab this. And I'll
quickly gloss over this as well. So, I
say, "Here are the GPT real time two
docs. Build me a simple voice assistant
for my Mac using GPT real time two. It
should open a live web socket connection
to the model." I'll explain that in a
second. "Capture my microphone. Let me
control my computer through voice." And
then we're just going to start with one
tool call, and we'll keep track of the
logs by printing every tool call to the
terminal as it fires. So, it's really as
simple as that. Or you could just not
say that and say, "Here are the docs.
Set this up for me." And so, what Claude
just do the coding here. This is a
relatively simple build. What Claude's
doing is opening a live connection to
GPT real time two over what's called a
web socket. It's going to hook up my
microphone in a second, and actually
I'll give permissions there. And then
it's wiring up all the tool calls we
just talked about as well. It's just
starting with that one tool call, but
we'll slowly start to stack in more
complicated ones as we go through this
build. One thing of course you need as
well is just an open AI API key, which
you can get at platform.openai.com.
This is exactly where we were in the
playground when we were testing the
audio model, so just go to API keys and
then create a new key, and then just
drop it into a .env.local or just give
it to your agent. And I'll talk about
API costs here in a second. The normal
way where it's streaming to the cloud
the whole time, costs will add up very
quickly, but the method I'll detail once
we get this wired up keeps it super
efficient to where each call ends up
costing like pennies on the dollar. So
let's go back over to Claude, and in
just a couple minutes, like literally 2
and 1/2 minutes, it looks like
everything is set up. So it just needs
to do some sort of NPM install. I also
need to add my OpenAI key, so I didn't
give it that yet, so we'll do that.
Okay, OpenAI API key is in .env.local
now. Go ahead and run this build
entirely and then just do a quick test.
And Claude ran a couple tests, so it
looks like everything is working. Now we
just need to do NPM start, and we need
to get microphone capture set up. So
Claude needs to connect to my
microphone, so I'm just going to tell
Claude, please do NPM start and wire up
my microphone so I can talk to it.
>> change our absolute notification logic
for
>> It just started.
>> If you launch it, it's crashing, try
force quit with command plus option plus
escape. Tell me the app name and I can
open it. Tell me which one to
>> Okay, I need to stop it. So what's
happening is I was talking to Claude in
another session cuz I'm also doing five
things at once, so I was just talking to
Claude about something else and then it
just started listening and then picking
up my conversation with Claude
separately. So this is this is one of
the problems of having real-time
streaming, just constantly listening. So
we need first to set up some sort of
wake word
>> Before we set up a wake word, let's just
have it do something to show that it is
in fact working. Hey chat, can you hear
me?
>> Yep, I can hear you.
>> What's uh what's going on?
>> Just hanging out with you. What's up on
your side?
>> Nothing. Can you open Safari?
>> Safari opened.
>> Boom, look at that. So just like that,
you saw exactly what happened in the
terminal. When I called it
>> Yep.
>> Oh. Okay, okay, okay, got it, got it.
Okay, turn off, stop, turn off. All
right.
>> Okay, I'll keep it brief and leave
things idle now. Got it. I'll stay quiet
and wait.
>> Okay, now that that's off, you can see
how easy that was to set up, right? It
was just a couple back and forth with
Claude actually get this set up and
really the only problem was it just
listening nonstop, which is annoying. Of
course, we don't want to do that. So,
what we're going to do is set up a
couple different things. So, you might
have noticed that in my original demo, I
had that little waveform displaying when
I was talking to it, but right now in
the version that we just built, it's
always listening. So, it's streaming my
voice to the cloud nonstop. It's
constantly transcribing everything I say
while it waits for a command. Not to
mention, it's just sitting there every
time I say something, it's just
interrupting me cuz it thinks I'm
talking to it. So, here's what we're
actually going to do. One hot key, I'll
hold it down and that waveform will pop
up exactly like you saw in the demo, and
then I talk to it, I let it go, and then
it acts. So, nothing's listening,
nothing's streaming to the cloud until I
press that key. And to do that, we're
just going to go back to our agent and
set it up with a very simple prompt.
Here's the prompt too, you can
screenshot this if you like, but it's
relatively straightforward. It's just
saying, "Add push-to-talk to my voice
assistant. Set up a global hot key.
While I hold the key down, capture my
microphone and show a clean on-screen
HUD so I know when it's listening. While
I release it, stop listening and then
have the model act on what I said."
Okay, so we've got our push-to-talk set
up. Just hold command Q and let's see if
this thing works. Hey chat, can you hear
me?
>> Yep, I can hear you loud and clear.
>> Cool. Uh, what's going on?
>> Hey, I'm here and ready when you are.
>> Okay, the only thing that's just
annoying me, I I just want to fix this.
The looks like the speaking, like the
waveform doesn't show up in real time,
so I'm going to just tell Claude that.
This is a very minor UI thing, but I
know I'm going to be bugged by that as
we move through this demo, so I'm just
going to tell Claude. The only thing is
when it's speaking, so I see thinking
and then it's weird to have thinking be
a waveform, right? Can we just like not
do that? I don't know if you just need
like a loading indicator, just do it in
the same HUD. And then when it's
speaking, it just kind of like either
lags with the waveform or just kind of
like is stuck. I don't know if we can
get like a real-time streaming of the
waveform generation showing up when
real-time 2 is talking, but that would
be tight.
Hey chat, can you hear me?
>> Yep, I can hear you.
>> Nice. All right, there we go. That
real-time streaming waveform looks
great. So, now we've got real-time 2
built out and fully configured. It can
trigger with a hotkey. Now, it is time
for our first real test. We're going to
be searching the web. So, to do that,
all you need to do is go back to Claude
and ask it to add three new tools. One
that will open a Chrome tab, one that
types a query into the search bar and
hits enter, and then one that can use
its vision to click a link. So, again,
really easy to do. We're going to go
back to Claude and just type in this
prompt. Add web search to my voice
assistant. Three tools. One, open a new
Chrome. Two, type a query into the
search bar. Hit enter. Three, click a
link using vision. Take a screenshot of
the page using the model's vision to
find the link I describe and click it.
When asked it search yada yada. So, I
give the actual example of that, but
it's really that simple. So, you just
describe what we want to do and then
Claude can reason how to actually set
this up for the real-time 2 assistant.
Okay, and we've got this set up with a
couple different caveats for you to know
where you'll just need to grant
permissions. So, the minute I ask chat
to open up something here in a second,
it will open up my accessibility
settings and I just need to connect some
stuff for screen recording for the
actual automation with controlling
Chrome. And once I grant permissions
once, it should be able to just run
automatically. I don't need to give
permissions every single time. Let's run
this thing and see if we can do a Google
search. Search for the best ramen in
Brooklyn.
>> Got it. I'll trade out. Search is open.
>> Okay, you didn't see that cuz I was just
in another window, but it immediately
searched that. Like it was so fast. Let
me see if I can actually show this on my
monitor. Maybe I just need to try it
again. So, if I say, "Search for the
best salad in Brooklyn."
>> Okay.
>> Look how fast that is. That's crazy.
Good job, Chat. Now, click the first
link.
>> Okay. I'll open the first link on the
page.
>> Okay, so this is where the permissions
come up. As you can see, all right,
we're going to turn on Okay, try
clicking now.
>> Which link should I click? Say something
like the first result or the link's
name.
>> You can see how fast this is, too. It's
crazy. Uh just click the first link.
>> Clicked the first result.
>> All right, so it's clicking the images.
Click the first link that's a URL, like
a website URL.
>> Click the first URL link.
>> Look at that. That is cool. Now, okay,
searching the web, that's relatively
easy. Let's raise the stakes a little
bit by having it open and interact with
applications. So, what if I wanted to, I
don't know, save these GPT real-time 2
docs somewhere? Or, I don't know, save
any website URL for that matter. What if
I could just save this straight into
Obsidian? Well, we can do exactly that.
Let's go ahead and set this up in Claude
to where we're now training the agent on
how to actually open and access and
interact with our applications. And by
train, I just mean literally showing the
agent how to access this application. In
this case, Obsidian, like a lot of apps
you'd be using, has a well-documented
MCP server, which means just by
connecting this, it'll give our
real-time 2 agent basically full access
to my Obsidian vault. So, reading,
writing, creating, searching notes, all
of that. Now, I'm not going to get too
into the weeds, but it's genuinely easy
to do this. We're just going to go over
to Obsidian, go to settings, click
community plugins on the left-hand side,
go browse, then search for MCP, and it's
the local res API with MCP. We're going
to install this, then we'll enable it,
and then we'll click options. And then
we've got an API key here, which I'm
blurring out, and we also have the MCP
endpoint, which we need to give the
agent, which is this 127.0.0.1
URL here. And then I'll just take this,
I'll go back to Claude code and tell it,
"Connect my Obsidian MCP server to the
voice assistant. It's running at that
URL." And then I said the API key is in
.env.local, wired up so I can read,
write, create, search notes in vault by
voice. Okay, and in 6 minutes, we have
our Obsidian app connected to real-time
2. So, I can just go ahead and talk to
the agent, trigger it with control Q.
Hey chat, open Obsidian.
>> Obsidian is open.
>> Cool. Now, create a new note.
>> Got it. I'll create a new note in
Obsidian. New note created.
>> Sweet. Uh can you open up the GPT
real-time 2 docs via Google search, copy
all of those docs, and then paste that
into this quick note?
>> GPT real-time 2 documentation. Then
we'll handle what's possible with it. I
opened the OpenAI search. I can't copy
all their content into your note.
>> Yeah, you accidentally clicked AI mode
in
Google. Go back.
>> I'm that AI overview and return to the
regular results. I reopened the search
results.
>> Yeah, now I just need you to click that
first link there, GPT Real-Time 2.
>> Got it. I'll try that first link again
directly. Done. It's open now.
>> Cool. Now, can you take these docs, copy
them, and then paste it into Obsidian,
into that note that you just created?
>> Okay, grabbing the page and dropping it
into that note. Done. Full page saved
into ClickUp.
>> Crazy. So, we just go back to Obsidian.
There it is. Wild. And then you can just
feed this into an agent. Really freaking
cool. Of course, obviously, like you you
can there's easier ways to access docs
that the agent can just do web fetch,
but the idea is you can just copy and
paste anything online and paste it into
Obsidian for whatever reason. Okay, so
we've got Obsidian connected, and
remember, this isn't just for Obsidian.
You can apply this same setup for any
application, any SAS with an MCP server
or an API. Just give that to Real-Time
2, and it will be able to access that
programmatically. So, to recap, our
Real-Time 2 agent can now interact with
your browser via Google Search setup,
and it can interact with tons of
applications and SAS via an MCP server.
This covers almost all computer use, but
the kicker is, what if an application
doesn't have an MCP or an API? What if
it's an application like, say, Premiere
Pro? Well, in that case, we have
officially reached the highest
difficulty level for GPT Real-Time 2.
But, this is actually really easy to set
up as well. We can pull this off with a
combination of two things. The first is
what's called an accessibility tree.
Every operating system has a feature
that's built for people who are blind or
have low vision called a screen reader,
which will read everything on screen out
loud. And for that to work, each
application is required to hand the
computer a structured map of itself.
This is the accessibility tree, and we
can have GPT Real-Time 2 leverage this
same thing as well. Now, having said
that, that map is just information on
its own. It doesn't really do anything.
So, that is where this second piece
comes in. And it is an open-source repo
called Agent Desktop. Shout-out to this
guy, Lafir. I'll put his X profile on
the screen. But, what Agent Desktop will
do is it will read that map for the
agent, and then it gives that to the
agent. The agent can actually drive the
app, so it can click the buttons, it can
press play, it can move forward, it can
go into all these different menus. So,
the accessibility tree tells the agent
what's there, and then Agent Desktop is
the actual hands that operate this. By
the way, all of this is going to be
included in the GitHub repo down below,
including my fork of Agent Desktop. But,
for this example, we're just going to go
here, we're going to copy this Agent
Desktop GitHub, and I'm just going to
feed this into Claude. And so, if you
wanted to just use Lefeuvre's Agent
Desktop, you can do that as well. And
here in this prompt, I'm just saying,
"Add the desktop app control to my voice
assistant using Agent Desktop." I give
it the GitHub repo there, explaining it
should read the macOS accessibility tree
so the agent can see any apps, buttons,
fields, menus, act on them, and then
wiring it in as tools as it can control
apps by my voice. Leverage Agent Desktop
to actually give your agent hands and
start controlling applications.
Hopefully, I'm not getting too in the
weeds with those explanations, but I
wanted to at least give people context
on how this is actually being built.
But, if you're lazy like I am, you can
just feed the GitHub repo that I have
included to your agent. Controls are now
live in Premiere. Looks like everything
is set up, so I'm just going to talk to
the agent and test this out. We'll open
up Premiere. So, I'm in the middle of
editing this actual video that you are
currently watching, and let's have the
agent do something. Hey, can you move
three frames right in Premiere?
>> Stepping forward a few frames. Moved
three frames forward.
>> Few frames forward. Let's uh let's play
this.
Can you pause? Can you pause? Oh, it's
Oh, it's playing my program monitor
right now. Okay, that's why. Um into
Let's quit out of this track, I'm
assuming. Okay, there we go. Let's just
bring this program monitor here so you
can see what's playing right now, and I
will have Real-Time 2 pause this. So,
let's hit play.
Can you hit pause on Premiere?
Boom. Look at that.
Go back three frames.
That's sick. Can you cut the timeline? I
don't know what the term is.
>> Cut the play head.
>> To actually do it, that's crazy.
Look at that. Look at that cut. Boom. So
it's working. Okay, now mark in.
>> Mark in.
>> Okay, move the play head four frames to
the right and then mark out.
>> Got it. Stepping in, sitting in. Move
four frames out.
>> That's crazy. Now ripple delete.
>> Ripple delete, done.
>> Jeez. All right, I don't want it to mess
with my timeline. I think you get the
idea. All right, so now you have an
ability to via GPT real time 2 have an
agent basically control your entire
computer with just your voice. You know
how to open Chrome and search the web.
You can work with applications that have
an MCP server like Obsidian and you can
build out the ability for your agent to
handle even the most complicated
applications like Premiere Pro. But
before I let you go, let's run through a
few honest caveats. First, it doesn't
work perfectly on every app. The web and
most standard apps are easy, but some
apps just won't expose many of their
controls. When that happens, it takes a
bit more back and forth with your coding
agent to get to the bottom of it or you
have to duct tape something together.
But between the GitHub repo I'm giving
you an agent desktop, you should be able
to with an agent figure out how to
access and control a lot of these
applications. Second, there is a real
cost. It's cheap. It's a few pennies per
command, but it's not free and of course
if you leave it streaming to the cloud
all day, which is that initial version
that we did how to doing, which I highly
recommend you don't do, but those costs
are just going to balloon really quick.
And then third, this is where it gets a
little less realistic. It is not
instant. I tried to show in real time as
much as possible how long this takes,
but there is a little bit of latency.
And so if you're used to, you know,
working in a GUI and just clicking
things and having it be automatic, it's
going to be a little bit weird to have
an agent do this. But I think it's
important to emphasize you can see in a
very near future where this thing is
almost in real time. And I for one since
I hate clicking in applications cuz it
wastes so much time, I'm excited for
that future. Anyway, uh let me know if
you're experimenting with GPT real time
2. If you have any questions, happy to
help and uh I'll see you in the next
one.
>> Mhm.