Prompt caching lowers inference costs but can leak private information from timing differences. Our audits found 7 API providers with potential leakage of user data. Caching can even leak architecture info—OpenAI's embedding model is likely a decoder-only Transformer!.🧵1/9

Induction heads are commonly associated with in-context learning, but are they the primary driver of ICL at scale?. We find that recently discovered "function vector" heads, which encode the ICL task, are the actual primary drivers of few-shot ICL. 🧵

One Token to Fool LLM-as-a-Judge. Watch out for this one, devs!. Semantically empty tokens, like “Thought process:”, “Solution”, or even just a colon “:”, can consistently trick models into giving false positive rewards. Here are my notes:

Along this point, there's a long tail of issues that cause an LLM to choke:.- "Context rot", where models become distracted by long+irrelevant contexts (especially from long conversations). See You need to open a new chat often. This effect is worsened if.

Kimi K2 is number one trending on HF, congrats!

A Survey of LLMs for Healthcare. This looks like a nice comprehensive overview of LLMs applied to the healthcare domain.

Surprisingly, a Large Language Model trained on health systems data did a better job predicting patient outcomes than traditional machine learning methods. “we show that it is possible to use LLMs as universal prediction engines for a wide range of medical predictive tasks.”