No More Hustleporn: this clip from the gpt-4 demo was massively slept on

By Yiren Lu — Mar 21, 2023

Tweet by Alex
https://twitter.com/alexalbert__

@alexalbert__:

this clip from the gpt-4 demo was massively slept on

gpt-4 can "see" your screen and describe the user interface of the application you are looking at

here's

@gdb

getting gpt-4 to describe a screenshot of a discord server in painstaking detail

@alexalbert__:

my take is that text generation will not be the main value prop of LLMs very soon

instead, it will be their ability to operate the tools we already use

in this example, gpt-4 proves that it has a near-human level of understanding of discord's UI through just one screenshot

@alexalbert__:

i've said it before and it's been hinted at by OAI employees

to everyone on AI twitter this may seem obvious but it's worth reiterating: chatGPT is NOT the final product here... it will look like a toy soon enough

twitter.com/alexalbert__/s…

@alexalbert__:

prediction:
the GPT-4 iphone is going to be an app that uses the model's multimodal abilities to control your computer for you in a self-driving fashion

the discord screenshot example in the gpt4 demo was just too obvious that it's within its current capabilities

twitter.com/rapha_gl/status/1636041957029060608

@alexalbert__:

OpenAI still has a few hurdles to solve like speed, cost, and reliability

but once these issues are ironed out, expect to see Microsoft's Edge transform into a full copilot-like system with Bing Chat being the portal that you guide it through

@alexalbert__:

some are already starting to get this to work

in this example, GPT-4 breaks down complex browser-based tasks into actionable steps and navigates the web by evaluating custom browser-driving code that it itself generates

youtube.com/watch?v=Gndk9P…

@alexalbert__:

the best part about that example is that it doesn’t even use gpt-4's visual abilities

I can’t wait to see how much more powerful it will be when that is incorporated