Yuntian Deng
@yuntiandeng
Followers
8K
Following
9K
Media
53
Statuses
866
Assistant Professor @UWaterloo | Visiting Professor @NVIDIA | Associate @Harvard | Faculty Affiliate @VectorInst | Former Postdoc @ai2_mosaic | PhD @Harvard
Joined September 2016
Can we build an operating system entirely powered by neural networks?. Introducing NeuralOS: towards a generative OS that directly predicts screen images from user inputs. Try it live: Paper: Inspired by @karpathy's vision. 1/5
"Chatting" with LLM feels like using an 80s computer terminal. The GUI hasn't been invented, yet but imo some properties of it can start to be predicted. 1 it will be visual (like GUIs of the past) because vision (pictures, charts, animations, not so much reading) is the 10-lane
6
30
151
RT @SyntellectAi: 🤖 Future of HCI is here! Generative interfaces powered by LLMs & multimodal AI are making human-computer interaction natu….
0
1
0
RT @webagentlab: NeuralOS: Towards Simulating Operating Systems via Neural Generative Models. NeuralOS is a neural framework that simulates….
0
1
0
RT @webagentlab: 🚨#GUIAgent Papers of the Week: . ◾️NeuralOS.◾️LaSM.◾️SGV. check it out 👉
0
1
0
RT @TheGrizztronic: This is awesome. Can't wait til my computer can read my mind too. Just saying, it's coming.
0
1
0
What happens when NeuralOS visits itself? NeuralOS-ception!🤯.Try it yourself:
Can we build an operating system entirely powered by neural networks?. Introducing NeuralOS: towards a generative OS that directly predicts screen images from user inputs. Try it live: Paper: Inspired by @karpathy's vision. 1/5
1
1
10
Just released a @huggingface Space demo of NeuralOS, so you can easily duplicate it, run it yourself, or build on top of it:. (Recommended GPU: at least L40, ideally A100-large. Our demo at used H100 GPUs.).
1
0
0
This work was led by my amazing student @RivardLuke, together with fantastic collaborators Sun Sun, @Harry_HongyuGuo, and @WenhuChen. Code: Demo code:
1
1
6
What made this work:.– RNN Pretraining: we first supervised RNN to predict screen images with MSE loss.– Scheduled Sampling: mitigates error accumulation over long sequences.– Finetuning: once deployed, we continuously finetuned with real user data ( 4/5.
1
0
3
NeuralOS is trained to predict the next screen image conditioned on previous frames and user inputs (mouse, keyboard). We use a training dataset of Ubuntu XFCE desktop recordings, collected via automated random exploration and demonstrations by humans and AI agents. 3/5
1
1
5
NeuralOS comprises two primary components:.– A recurrent neural network (RNN) for modeling persistent OS state, similar to an OS kernel. – A diffusion model for rendering screen images, analogous to a graphical interface. 2/5
1
0
4
RT @WenhuChen: Thanks for sharing the work!.@yuntiandeng and his student did a great job to simulate real operating system with generative….
0
2
0
RT @_akhaliq: NeuralOS. Towards Simulating Operating Systems via Neural Generative Models. a generative OS that predicts screen images from….
0
121
0
RT @sama: we planned to launch our open-weight model next week. we are delaying it; we need time to run additional safety tests and review….
0
1K
0
RT @brandondamos: Excited to release AlgoTune!! It's a benchmark and coding agent for optimizing the runtime of numerical code. 🚀 https://t….
0
34
0
RT @kchonyc: to cap the weekend of vibe coding, i asked Gemini to create a web app version, and it did it perfectly in one go. https://t.co….
0
1
0
RT @ZhitingHu: 🔥🔥Super excited to launch Mirage: a real-time interactive world model for video games! . Play and create your own GTA with t….
0
21
0