RT @upperwal: EKA-Eval : A Comprehensive Evaluation Framework for Large Language Models in Indian Languages. First paper and open source co….

It is nothing short of robbery in plain sight by the insurance companies to torture both patients and doctors alike.

RT @upperwal: We are building an open source training framework (COOM) inspired by HAI-LLM from @deepseek_ai . Doing an intro call on 26th….

I am in that camp (work in ML with a stats background), and lot of ideas we see in ML have strong roots in statistics.

Compressive Learning

6.3/6. Can we directly burn the model onto silicon? That is, can we take a Boolean MLP specified in PyTorch and create HDL files using the latest and greatest in VLSI technology? See hls4ml project.

6.2/n.Modeling for an approach or the more recent BOLD: Boolean Logic Deep Learning where networks are trained based on a new mathematical principle of Boolean Variation with considerations for chip architecture, memory hierarchy, dataflow and arithmetic precision.

6.1/n.At the heart of it, it is the matrix multiplication that needs to be done efficiently. What if networks have no explicit MatMul operations - see Scalable MatMul-free LLMs or Boolean Logic Deep Learning.

6/n.Hardware Accelerators. We are seeing a surge in hardware accelerators (and downstream toolchains including compilers, and hardware-software co-design) to make both training and inference faster. GPUs are the backbone of compute infra to train Deep Neural Nets.

5.2/n.If all the weights are sampled from {-1,0,+1} they require log2⁡(3)=1.58 bits. The optimal Boolean MLPs are precisely 1 bit networks, but it is easy see that, the fully specified and overparametrized Boolean MLPs are 1.58 bits, where 0 weight models skip connections.

5.1/n.Pruning and 1-bit LLMsIn the context LLMs, we see large models with parameters in the order of billions are successfully getting compressed/quantized without much degradation in the performance. See for example, The Era of 1-bit LLMs. All LLMs are 1.58 bits.

@karpathy 4.2/n.Essentially a look up table. And a dense MLP would have implemented a soft version of the look-up table (this essentially shows again that all models are K-Nearest neighbors). The Boolean MLP might be an interesting model to probe the Lottery Ticket Hypothesis further.

@karpathy 4.1/n.The Lottery Ticket Hypothesis conjectured that, when networks are densely and randomly intialized, some sub-networks would reach test accuracy comparable to original network. In the discrete case, it would mean that some inputs are exactly mapped to the output.

@karpathy 3/n. Initialization:.Draw the weights with mean around +1/-1 for weights and +/-1/2 for bias terms.

2/n.Based @karpathy's micrograd, I implemented BoolGrad, a proof-of-concept implementation of basic ideas introduced in BOLD. The implications for Deep Learning and LLMs are huge, if this scales out well.

1/n.BOLD: Boolean Logic Deep Learning is an interesting paper where they developed a framework to define gradients for mixed data types including Booleans.

RT @AI4Code: I am delighted to announce an exciting new course ✨ CIS 7000: Large Language Models ✨ I am teaching this semester: https://t.c….

RT @YiMaTweets: We will prepare entirely new manuscripts, lecture notes, and teaching materials and will publicize them as soon as they ar….

RT @docmilanfar: There’s a single formula that makes all of your diffusion models possible: Tweedie's. Say 𝐱 is a noisy version of 𝐮 with 𝐞….

RT @docmilanfar: The history of Tweedie’s formula is fascinating. He sent the result privately in a letter to Robbins, who published it in….