John Schulman
@johnschulman2
Followers
69K
Following
3K
Media
6
Statuses
140
Recently started @thinkymachines. Interested in reinforcement learning, alignment, birds, jazz music
Joined May 2021
Happy to share a new paper! Designing model behavior is hard -- desirable values often pull in opposite directions. Jifan's approach systematically generates scenarios where values conflict, helping us see where specs are missing coverage and how different models balance
New research paper with Anthropic and Thinking Machines AI companies use model specifications to define desirable behaviors during training. Are model specs clearly expressing what we want models to do? And do different frontier models have different personalities? We generated
13
46
610
I am super excited to share a new AI tool, Refine. Refine thoroughly studies research papers like a referee and finds issues with correctness, clarity, and consistency. In my own papers, it regularly catches problems that my coauthors and I missed. 1/
36
198
1K
We're happy to support the Human Centered LLMs course, on topics close to our hearts. We'd like to support more classes with free credits for students to use on assignments and projects. If you're an instructor interested in using Tinker in your course, please reach out to
15
52
613
Fine-tuning APIs are becoming more powerful and widespread, but they're harder to safeguard against misuse than fixed-weight sampling APIs. Excited to share a new paper: Detecting Adversarial Fine-tuning with Auditing Agents ( https://t.co/NqMeGSCQIF). Auditing agents search
arxiv.org
Large Language Model (LLM) providers expose fine-tuning APIs that let end users fine-tune their frontier LLMs. Unfortunately, it has been shown that an adversary with fine-tuning access to an LLM...
10
48
462
Excited to release new repo: nanochat! (it's among the most unhinged I've written). Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single,
663
3K
24K
Great to see an open source backend in the works for the Tinker API. If Tinker is going to power open science and open software, it shouldn’t depend on a single proprietary implementation.
The Tinker API recently released by Thinking Machines will have a big impact on how people think about post-training and inference systems. To allow more people to experiment with Tinker like systems and run it on their own hardware, we started SkyRL tx 🧸, an open source project
1
24
380
Even if I've tested a result extensively, it's hard to know how well it'll generalize to different experimental setups and software stacks
2
0
57
Really happy to see people reproducing the result that LoRA rank=1 closely matches full fine-tuning on many RL fine-tuning problems. Here are a couple nice ones: https://t.co/x7hcgNL3Bd
https://t.co/5JyKuKd9wS
much more convinced after getting my own results: LoRA with rank=1 learns (and generalizes) as well as full-tuning while saving 43% vRAM usage! allows me to RL bigger models with limited resources😆 script: https://t.co/p6IIiBQA6c
13
86
955
Tinker provides an abstraction layer that is the right one for post-training R&D -- it's the infrastructure I've always wanted. I'm excited to see what people build with it. "Civilization advances by extending the number of important operations which we can perform without
Introducing Tinker: a flexible API for fine-tuning language models. Write training loops in Python on your laptop; we'll run them on distributed GPUs. Private beta starts today. We can't wait to see what researchers and developers build with cutting-edge open models!
49
115
1K
Big fan of Jeremy’s work on optimization—great to see his first Thinking Machines post!
Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.
12
29
527
good question... thinking back to pre-LLM interviews I experienced (before 2019)… they were all in-person on-site, no chance of ''llm cheating,'' very different across places, and somehow way more memorable. > old deepmind had brutal ''quizzes'' -- 2-hour marathons with 100+
At which of these places did you have the coolest interview in your career? I know it's an ill-posed poll, but what am i gonna do with only 4 options?! I tried grouping them by interview similarity to the best of my knowledge. Comment if "other". Might make a second round.
21
123
2K
I'm more annoyed at whoever named us homo sapiens sapiens
34
7
469
Thinking Machines Lab exists to empower humanity through advancing collaborative general intelligence. We're building multimodal AI that works with how you naturally interact with the world - through conversation, through sight, through the messy way we collaborate. We're
649
700
8K
For people who don't like Claude's behavior here (and I think it's totally valid to disagree with it), I encourage you to describe your own recommended policy for agentic models should do when users ask them to help commit heinous crimes. Your options are (1) actively try to
123
41
716
A research project related to sycophancy: define explicit features like "does the response agree with the user" as in https://t.co/Ev5Q2PrpjK, and then construct a preference function that subtracts out their effect, as in https://t.co/kEaBgqar9V. I.e., remove some bad causal
8
21
278
Whether to collect preferences ("do you prefer response A or B?") from the same person who wrote the prompt, or a different person, is important and understudied. Highlighted this question in a recent talk https://t.co/7fcGmvG1Kd. Sycophancy probably results when you have the
This is serious, and we should make sure to prevent sycophantism as much as possible... Related: have we tried using other humans' feedback for RLHF instead of the original prompter's? This might somewhat help with debiasing 🤔
13
35
379
Excited to build a new AI research lab with some of my favorite former colleagues and some great new ones. Looking forward to sharing more in the coming weeks.
Today, we are excited to announce Thinking Machines Lab ( https://t.co/gD5QlPMfWw), an artificial intelligence research and product company. We are scientists, engineers, and builders behind some of the most widely used AI products and libraries, including ChatGPT,
41
48
1K