As someone very early in their AI learning journey, reading @kenneth0stanley and @joelbot3000’s “Why Greatness Cannot Be Planned” has been a truly foundational philosophical text. Would highly recommend to those who live their life thinking a clear objective function is needed to

I like the relative anonymity of X. <11 months ago I didn’t know what LLM meant, I thought it meant “large learning model”. I decided I was going to spend a year levelling up my life and I fell in love with learning for the first time in a very long time. Next is AI for MatSci.

In SF for the next 1 1/2 weeks. Didn’t post my AI journey because I was working silently. As an update I got into @GoogleDeepMind in London and will be joining soon! Anyone at @Techweek_ doing cool stuff in AI for Science or AI for energy / compute please reach out!

…everyone sees Meta AI icons in WhatsApp and instagram. The deep expenditure in VR around 2022, and the pushing of Meta glasses may provide an edge in being able to train on human-like visual data. Connecting what we see what what our brain thinks would be a huge leap forward

…the more important thing to note is the change in the architecture from dense transformers to MoE architectures in line with the latest frontier models. Most compute efficient, better for multi-model native models…

Day 3/4: I watched @dwarkesh_sp’s podcast with Mark Zuckerberg. Many people have commented that Zuck’s comments about benchmark indicate him ceding the race to AGI vs Deepmind, OpenAI, X and Anthropic + Chinese Labs. However I think it was a smokescreen while Llama catches up…

New Paper: Continuous Thought Machines 🧠 Neurons in brains use timing and synchronization in the way that they compute, but this is largely ignored in modern neural nets. We believe neural timing is key for the flexibility and adaptability of biological intelligence. We

Following on from this. Assuming models can edit and update their own policies and value functions, could they “prune” themselves too? Pruning is the removal of less important parts of the model and in the search of reward models don’t need safety policies right?

Need to research how value functions and policies are actually set

…And David Silver. What strikes me, is how important sandboxing is, as @demishassabis mentioned. Is the most important thing for safe AGI, now understanding how “pseudo-labels” are generated. If a model can recursively change its own policy can it change its own value function?

Day 2 of diving into AI. Spent the day reading how we’ve moved from SFT + RLHF to traditional scaling laws for pre-training models may be plateauing but through test-time compute for reasoning we are seeing scaling laws 2.0. Now reading Era of Experience from @RichardSSutton…

Logging my journey so far, been reading a lot for an interview at Deepmind (and generally). Read some of the seminal papers from Attention is all you need to recent MoE vs Dense transformer posts comparing the change from Llama 3 to 4. Love the AI for Materials Science space too