We’re excited to announce that Tilde completed an $8M seed round earlier this year, led by Khosla Ventures. Understanding model intelligence is the most important problem in the world, and the key to actualizing the promise that ASI can offer. 🧵 A thread on our approach:

Applications closing today! 👻 If you need an extension to finish up, reach out to us directly, and we can give you a few days.

5 days left! 🎃

:)

Today we're very happy to announce that we’re launching the Tilde Fellowship Program to support research in a mechanistic understanding of pre-training science (arch, optimizers, learning dynamics, etc.). Much of modern ML progress has come from scaling models and empirically

Apply here https://t.co/RvvrjAgScS with: - Proposed duration and scope - Compute needs (ballpark GPU ask is fine) - Any teammates (or interest in matching) - Background and motivations Applications are reviewed on a rolling basis, and we’ll get back to you quickly.

We’ll work side-by-side with fellows, providing compute, mentorship, and direct collaboration with our technical team, along with community support - with the outcome of open-source science and release. Whether you are a graduate student/postdoc with an existing moonshot project

Today we're very happy to announce that we’re launching the Tilde Fellowship Program to support research in a mechanistic understanding of pre-training science (arch, optimizers, learning dynamics, etc.). Much of modern ML progress has come from scaling models and empirically

Read the full post here: https://t.co/9f2qvFlLNu

~6/6~ Experiments demonstrate that our relaxed optimizers maintain improved conditioning over Adam, while offering slightly more flexibility than the Stiefel manifold.

~5/6~ These variants are special cases of a single, unifying framework. By applying different self-adjoint projectors (P) to the Gram matrix constraint, one can systematically generate families of manifolds—including the Stiefel manifold and our two proposed constraints— and

~4/6~ This perspective yields two alternative manifold constraints: DGram-Muon: Enforces orthogonality while allowing column norms to vary. Oblique-Muon: Enforces unit-norm columns (like Stiefel manifold) while allowing non-orthogonality (i.e. off-diagonal entries can be

~3/6~ We frame the optimization problem in Gram space. By operating on the Gram matrix (W^TW), which encodes the inner products between weight vectors, we can directly manipulate key geometric properties like vector norms and orthogonality.

~2/6~ Our work extends Manifold Muon (by @thinkymachines ), an optimizer that constrains weights to the Stiefel manifold. Stiefel requires perfect orthonormality (W^TW=I) → a condition number of 1. We investigate whether this constraint can be productively relaxed.

Modern optimizers can struggle with unstable training. Building off of Manifold Muon, we explore more lenient mechanisms for constraining the geometry of a neural network's weights directly through their Gram matrix 🧠 A 🧵… ~1/6~

Check out @nathancgy4's awesome Deltaformer PR and stay tuned for a post on the architecture soon!

~4/4~ Here’s the post: https://t.co/5nA2IkBmKr If you’re interested in the intersection of TCS and applied ML, reach out. We’re releasing vignettes to share brief-form thoughts from researchers. Stay tuned for the next one → 😶‍🌫️

~3/4~ Fundamental lower and upper bounds on transformer expressivity! We derive from first principles that even a restricted transformer (softmax at zero-temp) can solve problems beyond the circuit class AC⁰ (originally from Merrill et al.) More recent work (e.g. on DeltaNet)

~2/4~ Circuits, not Turing machines Another lens on computation is Boolean circuits - DAGs of AND/OR/NOT gates. They’re key to understanding parallel vs serial computation (proving lower bounds here→ P?=NP). It turn we can also use them to analyze transformers!

~1/4~ P?=NP 👉 P = problems we can solve efficiently. 👉 NP = problems we can check efficiently once given a solution. The (literally) well-known million-dollar question: are these really the same set of problems? We give an introduction to the P ?= NP problem through the

Vignette #2 is here! Join @AlecDewulf to: Learn about circuit complexity theory Derive theoretical capabilities and limitations of transformers Discuss the future of theoretical computer science in architecture design A thread 🧵