single file version open sourced here: https://t.co/98bopwSYkA

my attempts at implementing both flash attention papers 1 & 2 from when I was learning now that I look at it, it could use a lot more optimizations and it’s time I revisit this weekend

CUDA brain dump:

being confused is a good sign you’re learning

reading this right now: https://t.co/dya2GZQPGq

tfw you find a good cuda blog that you can actually follow along and reproduce the results because it’s optimized for your gpu arch

uhh same

guess I can’t hide from inline ptx kernels anymore xD because if optimized well the speedup you can get from this seems promising https://t.co/2vPz6DERBy

there isn’t much info out there on how to write inline PTX, or what PTX instructions to use. but I found a few blog posts and some snippets from nvidia’s docs and was able to piece it together to write a GEMM kernel using mma instructions, at warp level

forever a city girl !

knowing the limitations for this gpu saved me a lot of debugging time. I’ve made this mistake before where I try running an example code from the docs, not realizing it’s meant to be run on hopper architecture, or needs more tensor cores etc.

you should be spending more time getting familiar with your gpu architecture if you want to write efficient kernels. im writing mma kernels using tensor cores, on a 4050 that has > 80 tensor cores > 20 SMs > 4 tensor cores per sm > 4 warp schedulers per sm > 256 fused multiply

it’s that time of the year!

my muse for this week 🫠 i will call it a successful week if I can internalize these fragment layouts from tensor core mma

looking at building cuda/c++ extensions in pytorch so I can call custom kernels to sit in my inference stack. worth mentioning, for weeks i could not include <torch/extension.h> into my stack without errors and couldn’t figure out why. I fixed it after installing libtorch,

this is awesome, and they have a whole section for infra, gpu, kernel stuff i know what im doing this weekend yay

my only fall activity this year is to wake up before the sun and lock in 🍁 you can expect a blog/worklog from me on llm optimizations soon

I try to clear up most of my week and spare time for side project maxxing. I’m a full time student at uni but I make sure to spend as little time and mindshare possible on homework, tests and pointless electives once a week i speed run all my uni work and free up the rest of my

if we’re talking about consciousness as simply being aware, I thought the closest we got to giving LLMs some form of consciousness was through RL

it’s quite cool that you can just see your entire inference file run and identify exactly what’s slowing you down