Josh Bickett
@josh_bickett
Followers
7,281
Following
1,042
Media
758
Statuses
5,359
New dad | Engineer @hyperwriteai @othersideai | On the side - experimenting with VLMs playing games
Bend, Oregon
Joined August 2016
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#Eurovision2024
• 1443880 Tweets
América
• 499930 Tweets
#母の日
• 178533 Tweets
#MothersDay
• 79192 Tweets
ヴィクトリアマイル
• 69792 Tweets
ストフェス
• 34224 Tweets
Pachuca
• 33408 Tweets
Tatum
• 32866 Tweets
Barco
• 31953 Tweets
カーネーション
• 31647 Tweets
Chivas
• 29422 Tweets
Newwiee Honey Teerak🩷
• 27193 Tweets
Roger Corman
• 24228 Tweets
Saint MSG Insan
• 24216 Tweets
Igor
• 24081 Tweets
カグヤ様
• 20636 Tweets
ティアキン
• 16434 Tweets
Garland
• 16287 Tweets
ナミュール
• 15785 Tweets
Buckley
• 11944 Tweets
Derrick Lewis
• 11770 Tweets
Toluca
• 11740 Tweets
Hannibal Lecter
• 10997 Tweets
Last Seen Profiles
Pinned Tweet
𝗪𝗲 𝗮𝗿𝗲 𝗲𝘅𝗰𝗶𝘁𝗲𝗱 𝘁𝗼 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝘁𝗵𝗲 𝗦𝗲𝗹𝗳-𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝘂𝘁𝗲𝗿 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 𝘁𝗵𝗮𝘁 𝗲𝗻𝗮𝗯𝗹𝗲𝘀 𝗺𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗺𝗼𝗱𝗲𝗹𝘀, 𝗶𝗻𝗰𝗹𝘂𝗱𝗶𝗻𝗴 𝗚𝗣𝗧-𝟰-𝗩𝗶𝘀𝗶𝗼𝗻 𝘁𝗼 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗲 𝗵𝘂𝗺𝗮𝗻-𝗹𝗶𝗸𝗲 𝗺𝗼𝘂𝘀𝗲…
111
413
3K
@elonmusk
@WalterIsaacson
Perfect. He does a great job. Assume that means you are no longer writing your own?
30
31
1K
@elonmusk
@ajtourville
@Erdayastronaut
@SpaceX
So it will be determined which to cut off based on data available right after relight?
5
16
1K
𝗢𝘂𝗿 𝘁𝗲𝗮𝗺 𝗱𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝗲𝗱 𝘆𝗼𝘂 𝗰𝗮𝗻 𝘂𝘀𝗲 𝗚𝗣𝗧-𝟰𝗩𝗶𝘀𝗶𝗼𝗻 𝘁𝗼 𝗰𝗿𝗲𝗮𝘁𝗲 𝗮 𝘀𝗲𝗹𝗳-𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗰𝗼𝗺𝗽𝘂𝘁𝗲𝗿.
By looking at the user interface, GPT-4 decides which series of click or type events are required to accomplish an objective.…
48
111
954
We're working on a model that can click precisely by X & Y coordinates. I've plugged in an iteration of the model into the Self-Operating Computer Framework.
Here it is, autonomously controlling my computer to post on Twitter:
33
37
328
𝗝𝘂𝘀𝘁 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗮 𝗖𝗼𝗹𝗮𝗯 𝗻𝗼𝘁𝗲𝗯𝗼𝗼𝗸 𝘁𝗼 𝘀𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗲 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗼𝗳 𝗟𝗹𝗮𝗺𝗮 𝟮 𝟳𝗯 𝘂𝘀𝗶𝗻𝗴 𝗛𝘂𝗴𝗴𝗶𝗻𝗴 𝗙𝗮𝗰𝗲'𝘀 𝗔𝘂𝘁𝗼𝗧𝗿𝗮𝗶𝗻. 𝗦𝗶𝗺𝗽𝗹𝘆:
1. Upload your dataset as `train.csv`.
2. Input your Hugging Face…
10
26
276
Can GPT-4-Vision Play Texas Hold'em Poker?
I used Multimodal Gamer to let GPT-4-Vision control my mouse and browser to play.
After a few hands, GPT-4-Vision's pot was up, and it had a couple of wins.
19
26
226
The Self-Operating Computer Framework now supports the OCR-based approach for click decisions, by far surpassing previous methods attempted in the project. It's now the default - just type `operate`.
Discovering the capability shown in the demo is a bit stunning to me. This…
The Self-Operating Computer Framework now supports key commands like 'open terminal' (cmd+t) and 'focus on address bar' (cmd+l). It now performs multiple tasks in a single cycle, speeding up operations.
Here's a demo at (8x speed) of it opening a new terminal window and…
6
13
88
8
29
204
@Erdayastronaut
@SpacePadreIsle
Looks like the scene from Interstellar on Miller’s planet (the water planet)
2
0
191
The Self-Operating Computer Framework now supports Claude-3 thanks to
@RoyWei
.
Compare Claude-3-Vision, GPT-4-Vision, and Gemini-Vision at each's ability to operate a computer.
For Claude-3, simply type:
```
operate -m claude-3
```
The Self-Operating Computer Framework now supports the OCR-based approach for click decisions, by far surpassing previous methods attempted in the project. It's now the default - just type `operate`.
Discovering the capability shown in the demo is a bit stunning to me. This…
8
29
204
3
33
178
Excited to share we're going to open-source the
@HyperWriteAI
Self-Operating Computer framework.
Digital interfaces were built for humans. We believe new AI systems will be able to use them even more effectively.
Come help us build it!
Open-sourcing soon — Stay tuned!
𝗢𝘂𝗿 𝘁𝗲𝗮𝗺 𝗱𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝗲𝗱 𝘆𝗼𝘂 𝗰𝗮𝗻 𝘂𝘀𝗲 𝗚𝗣𝗧-𝟰𝗩𝗶𝘀𝗶𝗼𝗻 𝘁𝗼 𝗰𝗿𝗲𝗮𝘁𝗲 𝗮 𝘀𝗲𝗹𝗳-𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗰𝗼𝗺𝗽𝘂𝘁𝗲𝗿.
By looking at the user interface, GPT-4 decides which series of click or type events are required to accomplish an objective.…
48
111
954
11
21
173
𝐂𝐚𝐧 𝐆𝐏𝐓-𝟒-𝐕𝐢𝐬𝐢𝐨𝐧 𝐨𝐫 𝐂𝐥𝐚𝐮𝐝𝐞-𝟑-𝐕𝐢𝐬𝐢𝐨𝐧 𝐚𝐮𝐭𝐨𝐧𝐨𝐦𝐨𝐮𝐬𝐥𝐲 𝐏𝐥𝐚𝐲 𝐂𝐡𝐞𝐬𝐬?
I let each VLM control my keyboard and browser for a game. GPT-4-Vision managed to capture a piece from a real online opponent.
Though both models make invalid moves,…
12
28
167
The Self-Operating Computer Framework now supports Set-of-Mark Prompting.
Type: `operate -m gpt-4-with-som`
Set-of-Mark (SoM) Prompting is a new visual prompting method to unleash the visual grounding abilities of large multimodal models. Learn more in the arXiv paper:…
The Self-Operating Computer Framework now supports the new `gemini-pro-vision` model thanks to GitHub user @ linusaltacc
1
1
19
7
23
165
𝗝𝘂𝘀𝘁 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗮 𝗻𝗲𝘄 𝗖𝗼𝗹𝗮𝗯 𝗻𝗼𝘁𝗲𝗯𝗼𝗼𝗸 𝘁𝗼 𝘀𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗲 𝗿𝘂𝗻𝗻𝗶𝗻𝗴 𝗠𝗶𝘀𝘁𝗿𝗮𝗹 𝟳𝗕 𝗜𝗻𝘀𝘁𝗿𝘂𝗰𝘁!
1. Open the Notebook
2. Hit 𝗦𝗵𝗶𝗳𝘁 + 𝗘𝗻𝘁𝗲𝗿 keys on code cells
3. Input your instructions
Start running inference in under 5…
3
14
156
Can GPT-4-Vision Play Super Mario 64?
I created 'Multimodal Gamer,' a framework enabling multi-modal models (combining text and visual inputs) to play games.
Check out my video overview below and let me know your thoughts!
13
16
147
Is anyone serving OpenAI's Whisper over an API? Interested in trying it out for a new project I'm working on.
20
8
146
@Erdayastronaut
This should be the intro of the Mars documentary that will be made some day. It’s very cinematic
0
1
134
Computers are learning to operate themselves.
hi! Open Interpreter 0.2.0—The New Computer Update—is out today.
everything's new.
- OS Mode lets vision models operate your computer
- We included a new model for precise GUI control
- We're launching a Computer API for LLMs
↓
115
291
2K
2
10
112
4
0
110
I'm excited to share a sneak peek of what we're working on at HyperWrite. We're building an Auto-GPT like agent, but the difference is that you can collaborate with it directly in your browser.
I can't wait for people to try it out! Here's an agent posting a joke:
7
18
86
Adding Claude Vision to the Self-Operating Computer Framework this week which will let it operate a computer like a human (screen + mouse/keyboard)
Community member just made a PR. Just need to review and merge.
Anthropic cofounder Jared Kaplan says AI systems will soon be able to understand and control your computer screen
46
76
378
4
5
90
The Self-Operating Computer Framework now supports key commands like 'open terminal' (cmd+t) and 'focus on address bar' (cmd+l). It now performs multiple tasks in a single cycle, speeding up operations.
Here's a demo at (8x speed) of it opening a new terminal window and…
The Self-Operating Computer Framework now supports Set-of-Mark Prompting.
Type: `operate -m gpt-4-with-som`
Set-of-Mark (SoM) Prompting is a new visual prompting method to unleash the visual grounding abilities of large multimodal models. Learn more in the arXiv paper:…
7
23
165
6
13
88
The beginning of the real space age.
2
3
80
@ParikPatelCFA
Shorts 🩳 are worn traditionally in warmer climates and are sold at retail stores.
Let us know if there are more questions Dr. Parik Parel, BA, CFA, ACCA Esq
0
0
65
@paulg
Many towns could become the darling of their region by just turning a few streets to pedestrians only spaces and adding trees.
Especially in America, people are craving walkable spaces
1
2
71
Try - A simple game where you find an AI-generated image out of a list of real images.
See if you can get a score above 10.
I am guessing the game will get harder as images improve in the near future
AI images from
14
13
69
Notice the bounding boxes.
It appears an agent model is a fundamental part of their new computer. If I understand the product overview and vision correctly an agent operating at the OS level will do things for you. AI assistants can help with many problems by function calling…
6
2
66
Ah! Sounds like a fun challenge.
Weekend project - Who wants to see if I can get GPT-4-vision to play Mario 64?
11
7
63
@PPathole
@elonmusk
@Erdayastronaut
@Twitter
@TwitterComms
@verified
@jack
Like imagine if the government only gave you valid IDs if you were a celebrity. It’s dumb.
1
1
58
Welcomed a baby girl into the world a few days ago and stepped into fatherhood. Looking forward to all the joys this new stage of life will bring.
13
0
62
The Self-Operating Computer Framework now supports Voice thanks to
@didntdrinkwater
Here's a demo of navigating to YouTube for holiday music 🌲
The Self-Operating Computer Framework now supports Linux thanks to
@michaelhhogue
@ shubhexists
1
1
20
1
12
58
@TinaBoca
Any sufficiently advanced technology is indistinguishable from magic
Arthur c. Clarke
0
0
60
@Austen
What’s great is AWS basically gives away $5000 in credits to any company that can make a website and has an idea.
Most companies only actually need to pay for AWS if they scale. By that time, they aren’t moving.
0
1
58
Astronauts are in space from American soil again for the first time in 9 years.
#LaunchAmerica
1
7
51
@goodside
After reading a bit of Life 3.0 I can’t help but create this meme. Doesn’t mean I believe it, but can’t help but think of it.
0
5
58
@flcnhvy
@elonmusk
@boringcompany
True, a lot of parking burdens may be solved by FSD.
Parking may move outside of cities as electricity for car to drive to a remote parking lot will be minimal compared to real estate savings in cities like NYC.
1
0
56
The Self-Operating Computer Framework now supports Windows thanks to
@RonNachum
!
𝗪𝗲 𝗮𝗿𝗲 𝗲𝘅𝗰𝗶𝘁𝗲𝗱 𝘁𝗼 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝘁𝗵𝗲 𝗦𝗲𝗹𝗳-𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝘂𝘁𝗲𝗿 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 𝘁𝗵𝗮𝘁 𝗲𝗻𝗮𝗯𝗹𝗲𝘀 𝗺𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗺𝗼𝗱𝗲𝗹𝘀, 𝗶𝗻𝗰𝗹𝘂𝗱𝗶𝗻𝗴 𝗚𝗣𝗧-𝟰-𝗩𝗶𝘀𝗶𝗼𝗻 𝘁𝗼 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗲 𝗵𝘂𝗺𝗮𝗻-𝗹𝗶𝗸𝗲 𝗺𝗼𝘂𝘀𝗲…
111
413
3K
1
11
48
@lexfridman
It’s important to note, that a lot of that biomass is domesticated by humans though.
“Humans account for about 36 percent of the biomass of all mammals. Domesticated livestock, mostly cows and pigs, account for 60 percent, and wild mammals for only 4 percent.”
Source below
3
1
49
1
1
46
@Erdayastronaut
@SpaceX
Ha, I’m secretly happy about the delay because the sunset launches are awesome on the west coast
4
0
46
0
12
41
Wonder how many understand the reference to Whole Earth Catalog
“Drop the “Catalog”. Just “Whole Mars”. It’s cleaner”
37
3
250
10
1
44
Meta plans to make Llama 3 𝐦𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 in the near future, with longer context and improved reasoning and coding capabilities.
The craziest LLaMA 3 reveal:
The 400B+ version of the model is **on par with Claude 3 Opus**, and it's still training.
Soon, we'll have a better-than-Opus, fully open-source model.
The implications are huge.
120
395
3K
1
8
41
@AlexAndBooks_
Biographies that aren’t designed to be self help books are the best self help books.
2
5
39
2
0
39
@latestinspace
Imagine asking someone where they live and they append Earth to the end of the address.
“San Diego, USA, Earth”
Someday I hope
1
2
37
Life is like riding a bicycle. To keep your balance you must keep moving.
- Albert Einstein
3
10
37
@mckaywrigley
@ID_AA_Carmack
When I’m really confused about a problem I’m guessing to be a dumb human error I’ve found prompting an LLM with “what’s wrong with this code?” is surprisingly simple and effective.
3
0
36
There are many questions about the future of AI, but one of my favorites is - will it have a sense of humor?
Find out - beta is open to new users
Powered by OpenAI’s GPT-3, the app creates memes based on your input
8
9
37
@levelsio
Same, I often find myself two or three messages into a thread and get a poor output and am like ???, then I look up and I got switched back to 3.5. It is the most annoying part of ChatGPT.
I wasn't sure if I accidentally switched it at first, but had it happen enough that I…
5
0
42
@FredericLambert
@elonmusk
@ElectrekCo
What we really need to learn from Elon is how fast it can brake 60-0 mph with a trunk full of Boring Co. candy?
1
0
32
Our Chrome Extension hit 100k last week, excited to add another zero (or two) this year.
5
6
35
I understand this Open-Interpreter device to be the first of its kind.
Instead of purely a function calling agent or an LLM web agent, it is a vision-based agent that operates a computer’s mouse & keyboard remotely and the user interfaces with this agent purely by speech.…
Introducing the 01 Developer Preview.
Order or build your own today:
The 01 Light is a portable voice interface that controls your home computer. It can see your screen, use your apps, and learn new skills.
This is only the beginning for 01— the…
609
1K
6K
3
7
33
@WholeMarsBlog
The trading App Public gets it. I moved there. Perfect chance to become the a lead broker
4
7
33
I wonder if there are observable improvements with GPT-4-Vision's ability to operate a computer (via Self-Operating Computer Framework).
I'll try this week and share thoughts.
GPT-4 Turbo with Vision is now generally available in the API. Vision requests can now also use JSON mode and function calling.
Below are some great ways developers are building with vision. Drop yours in a reply 🧵
120
459
2K
4
1
33
1
0
31
We integrated LlaVA into the Self-Operating Computer Framework, but it struggles to open Chrome.
Meanwhile, GPT-4-Vision can open Chrome, navigate to Google Maps, click on the input field, and get directions from point A to point B without too much trouble.
Looking forward to…
Llama-3 is closing the gap with GPT-4, but multimodal models gotta catch up. Vision capabilities of open models like LlaVA are far, far behind GPT-4V. Video models are even worse. They hallucinate all the time and fail to give detailed descriptions of complex scenes and actions.…
46
113
863
3
4
32
