Stella Biderman
@BlancheMinerva
Followers
17K
Following
11K
Media
638
Statuses
13K
Open source LLMs and interpretability research at @AiEleuther. She/her
Joined May 2019
Two years in the making, we finally have 8 TB of openly licensed data with document-level metadata for authorship attribution, licensing details, links to original copies, and more. Hugely proud of the entire team.
Can you train a performant language models without using unlicensed text?. We are thrilled to announce the Common Pile v0.1, an 8TB dataset of openly licensed and public domain text. We train 7B models for 1T and 2T tokens and match the performance similar models like LLaMA 1&2
18
69
572
Here's our abstract and you can more on OpenReview: We specifically didn't solicit "new benchmarks" since there's a while NeurIPS track for that, and decided to focus on what's currently under incentivized: best practices and real-world impact.
1
0
7
Generate videos in just a few seconds. Try Grok Imagine, free for a limited time.
507
957
4K
This workshop seems great and I'm sure it'll be a good time, but it's extremely notable that a substantially similar workshop with a focus on evaluating the use and impacts of these technologies (instead of "capability benchmarks") was rejected for being not technical enough.
We are happy to announce our @NeurIPSConf workshop on LLM evaluations! . Mastering LLM evaluation is no longer optional -- it's fundamental to building reliable models. We'll tackle the field's most pressing evaluation challenges. For details: 1/3
4
7
65
How did you learn to present code? Are there resources that you recommend using to help teach people?.
1
0
3
RT @alz_zyd_: Back when LLMs sucked at math, a bunch of people wrote papers about why the technical structure of LLMs made it impossible fo….
0
220
0
What do you call those units of semantic text the LLM compresses English and German into when you brag about the compression rate? It's not UTF-8 bytes. there's a word for it, maybe starts with a a T?.
Introducing two new tokenizer-free LLM checkpoints from our research lab: TFree-HAT 7B. Built on our Hierarchical Autoregressive Transformer (HAT) architecture, these models achieve top-tier German and English performance while processing text on a UTF-8 byte level.
8
7
79
RT @ErnestRyu: This is really exciting and impressive, and this stuff is in my area of mathematics research (convex optimization). I have….
0
311
0
RT @StephenLCasper: Some good thoughts on our paper from @jackclarkSF in his newsletter. I'll share a couple of thoughts on this here 🧵🧵.ht….
0
2
0
RT @StephenLCasper: Here are a couple of slides that I presented yesterday at #aitechgov about open-weight model risk management. https://t….
0
5
0
I'm trying to use GPT-5 for lit review and in particular to survey what the decision-making factors leading people to use Pythia in their research is. Unfortunately I have to keep reminding it that I know the opinions of myself and my collaborators 🙃
6
1
41
RT @sharongoldman: NEW: Thanks to @BlancheMinerva for speaking to me about Deep Ignorance, the new paper from @AiEleuther & the UK AISI. Bo….
fortune.com
New research shows that scrubbing risky material from AI training data can build safeguards that are harder to bypass — and one author calls out tech giants for keeping such work under wraps.
0
2
0
RT @soundboy: I am keen to see more work on AI security that starts from a "open-first" perspective as @BlancheMinerva puts it. Great to se….
0
10
0
Paper: Project page: EAI Blog post: Artifacts: Enjoy! I can't wait to see what y'all do with these new toys.
huggingface.co
2
4
38
It was a joy to work with @AISecurityInst and all of my wonderful co-authors on this project: Kyle O’Brien, @StephenLCasper, @QuentinAnthon15, @tomekkorbak, @_robertkirk, @alxndrdavies, Ishan Mishra, @geoffreyirving, and @yaringal.
1
1
26
Also, if you're interested in machine unlearning and have been saying "well I can't compare my unlearned model to what I would have gotten had I not trained on the data because pretraining is too expensive". well, now you don't have an excuse.
1
0
22
See the paper for more results such as defenses-in-depth, negative results when training on corrupted data, contrast between our results and the wonderful "Safety Pretraining" by @pratyushmaini et al., and some issues with the WMDP-Bio benchmark itself.
arxiv.org
As large language models (LLMs) are increasingly deployed in high-stakes settings, the risk of generating harmful or toxic content remains a central challenge. Post-hoc alignment methods are...
1
2
42
"But wait," the skeptic cries. "Surely this is infeasible for frontier models! Their datasets are far too large to expect companies to meaningfully understand or document!". Actually our methodology is extremely cheap, mostly runs on CPU, and adds an overhead of less than 1%.
1
2
39
While this is a real limitation, we believe that there are genuine contexts in which this intervention is meaningful such as when the primary aim is to prevent models from sharing certain types of information with people who lack it.
2
0
22
Preventing a model from going into an interaction with relevant knowledge isn't a panacea though: filtered models see minimal performance loss when it comes to reasoning about information provided in-context. The models are still smart, they just don't know the concepts innately.
3
1
26
Our results are competitive with circuit-breaking when it comes to out-of-the-box performance and substantially more robust to both adversarial and benign finetuning. These results lead us to believe that we are genuinely impairing the knowledge of LLMs, not just suppressing it
2
0
28