Imo this is along the lines of how talking to an LLM via text is like typing into a DOS Terminal and "GUI hasn't been invented yet" of some of my earlier posts. The GUI is an intelligent canvas.

VAMOOOOOOOS!!!! 🇪🇸

Our TPUs are headed to space!  Inspired by our history of moonshots, from quantum computing to autonomous driving, Project Suncatcher is exploring how we could one day build scalable ML compute systems in space, harnessing more of the sun’s power (which emits more power than 100

Hear from Michel Devoret, our Chief Scientist of Quantum Hardware, on our latest breakthrough algorithm: Quantum Echoes. His early work on superconducting artificial atoms laid the foundation for the Willow chip, enabling verifiable quantum advantage.

Here's my 6 hour conversation with @dhh, a legendary programmer, creator of Ruby on Rails, author, and race car driver. This was a fun and inspiring conversation on everything from the future of programming & AI to the nature of happiness & productivity to the value of family,

It's yours Carlitos 🏆 #RolandGarros

🎉🎉🎉

Gosh luckily, I thought I was the only weirdo in the room

Such a wild ride, here’s to 20 more years of it 🍩

That was actually fun 🚀🚀🚀

Here's my conversation with Pieter Levels (@levelsio), self-taught developer and entrepreneur who designed, programmed, shipped, and ran over 40 startups, many of which are hugely successful. In most cases, he did it all by himself, while living the digital nomad life in over 40

Every time I diversify I lose money

Interestingly, recently released Gemma 2 https://t.co/58xdfKhHIl seems to have a combination of sliding window attention mechanism as well: "We alternate between a local sliding window attention (Beltagy et al., 2020a,b) and global attention (Luong et al., 2015) in every other

Sources: - Build a Large Language Model (From Scratch)

Sample code: https://t.co/RxW2MVRkrL

Same implementation as to Causal Attention ( https://t.co/5MMWZISvXZ) instead: 1. Adds window_size param to constructor, determines sliding window size 2. Main change is in how we create and apply the mask 3. Rest remains almost the same 4. Also removed dropout layer for

While playing around with simple self attention mechanisms, I went curious about different types of self attention implementations that I have not seen in https://t.co/DZDtm4jfqL Came across the Sliding Window attention which turns out to be a simple variation but yet powerful

Sources: - Build a Large Language Model (From Scratch) https://t.co/DZDtm4jfqL

Sample code: https://t.co/5MMWZISvXZ

Implementing dropouts 1. Add new Dropout layer at the end, before computing values - Randomly sets a fraction of input units to 0 at each update during training time 2. Exact placement of dropout can vary in different implementations - Some might apply dropout to the