This week's blog post explores methods for incorporating longer-term context in transformers! Featuring 6 unique approaches: - Sparse Transformers - Adaptive Span Transformers - Transformer-XL - Compressive Transformers - Reformer - Routing Transformer https://t.co/xzl1yrrdyx

Leaving Meta and PyTorch I'm stepping down from PyTorch and leaving Meta on November 17th. tl;dr: Didn't want to be doing PyTorch forever, seemed like the perfect time to transition right after I got back from a long leave and the project built itself around me. Eleven years

We're hiring in NYC! DM if you're interested in training SoTA OCR models, and helping thousands of customers (including tier 1 AI labs) work with documents.

Any good OSS options for translating LLM token usage numbers to $? Looking for an option that properly accounts for cached input pricing and has some auto-update mechanism to pull latest prices.

Beyond just text quality! We're introducing #DocReward, a model that evaluates and improves the visual structure and style of documents. In our tests, DOCREWARD achieved a 60.8% win rate in generating human-preferred documents, compared to GPT-5's 37.7%. https://t.co/ChY9z0X05f

It's a strange time to be a programmer—easier than ever to get started, but easier to let AI steer you into frustration. We've got an antidote that we've been using ourselves with 1000 preview users for the last year: "solveit" Now you can join us.🧵 https://t.co/GLKm0woI8b

Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.

I'm ready to accept a definition of "agent" that I think is widely-enough agreed upon to be useful: An LLM agent runs tools in a loop to achieve a goal This is a big piece of personal character development for me! I've been dismissing the term as hopelessly ambiguous for years

One thing I beg @charliermarsh to build: https://t.co/j1VgMoaA0V for python. - "uv docs": build docs for given python packages - A https://t.co/LIyi0S181w website: similar to https://t.co/j1VgMoaA0V, auto build after every version updates. - Markdown native support with runnable

The amount of alpha in this post is off the charts https://t.co/hpEomeSoZY

So pleased to announce the Humanloop team is joining @AnthropicAI! I couldn't imagine a better home for the team. Everyone I've interacted with at Anthropic has been incredibly talented, high-trust and conscious of the stakes at play. Enormous gratitude to our customers and

why aren’t there more research papers like this?

Scaling up RL is all the rage right now, I had a chat with a friend about it yesterday. I'm fairly certain RL will continue to yield more intermediate gains, but I also don't expect it to be the full story. RL is basically "hey this happened to go well (/poorly), let me slightly

I really like this diagram from @_jasonwei and @hwchung27 about how to view the bitter lesson: It's a mistake not to add structure now, it's a mistake to not remove that structure later. We're at the precipice of setting up a huge, powerful RL training run that will define the

The race for LLM "cognitive core" - a few billion param model that maximally sacrifices encyclopedic knowledge for capability. It lives always-on and by default on every computer as the kernel of LLM personal computing. Its features are slowly crystalizing: - Natively multimodal

+1 for "context engineering" over "prompt engineering". People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window

yes to queuing, and how about a kanban board for your whole fleet of agents?

Nice - my AI startup school talk is now up! Chapters: 0:00 Imo fair to say that software is changing quite fundamentally again. LLMs are a new kind of computer, and you program them *in English*. Hence I think they are well deserving of a major version upgrade in terms of

I suggest a new metric: Pass@1/K. For a given "K" You only get a point if all "K" attempts were successful. So it's a continuation of the Pass@K graph to the left hand site and intuitively measures robustness / confidence.

i want to be able to queue messages in cursor. (e.g. send a follow up while it's still thinking about the previous message) cc/ @ryolu_

excited to announce i'm leaving my PhD to join @OpenAI! i'll be working on memory + personality for AGI and ChatGPT memory will fundamentally change our relationship to machine intelligence, and i plan to work extraordinarily hard to make sure we get this right for humanity