#NLProc.AI Co-Scientists 🤖 can generate ideas, but can they spot mistakes? (not yet! 🚫). In my recent paper, we introduce SPOT, a dataset of STEM manuscripts (math, materials science, chemistry, physics, etc), annotated with real errors. SOTA models like o3, gemini-2.5-pro

For more details,.paper: project: I'm planning follow-up works, so let me know if you are interested! 🔥.

Imagine you’re collaborating with an AI co-scientist: you ask it to proofread your manuscript and flag any errors. Which LLM would you choose? 🤔. We evaluated the new Claude 4 models on SPOT. It looks like o3 is still the best model for this.

RT @BlancheMinerva: People are really eager to use AIs "to accelerate science" (whatever that means). Designing meaningful tests tailored t….

Last but not least, I’d like to thank all coauthors for their help 👍👍👍. @jiwoohong98 @Void13950782 @hazel_heejeong @cartinoe__5930 @sngwonlim @jinyeop_song @GoncaloSPaulo @YoungjaeYu3 @stella.

🔥 SPOT drives home a crucial point – verification must catch up with generation if AI co-scientists are to earn our trust. It’s time to build smarter error detectors before we rely on AI in labs 🛠️. Check out the paper for more details!.

Our deep-dive case studies show typical AI blind spots: models struggle with long-tail knowledge absent from web data and extremely long contexts; without fully spelled-out derivations, they misinterpret calculations and ignore domain-specific conventions, leading to student-like

Models even struggle to be consistent. They rarely rediscover the same error across 8 trials and assign near-zero confidence, making automated review unreliable 🤷

We benchmarked 10 top models from closed to open source. Results are sobering – recall under 21 percent and precision under 6 percent highlight a massive shortfall 📉

To make sure these errors are genuine, we only include those that have been acknowledged by the original authors themselves!

1️⃣ SPOT comprises 83 research papers and 91 author-validated errors across 10 STEM fields and 6 error types – equation/proof, figure duplication, Data inconsistency, Statistical reporting, Reagent identity, and Experiment setup. 🧮. 2️⃣ The dataset spans long documents (avg 12,000

I'll be presenting KMMLU, the-most used korean benchmark by Korean big techs at the moment with @seungonekim today, at 2pm!.

RT @seungonekim: @naaclmeeting I'll also be presenting our KMMLU paper with @gson_AI! It is one of the most widely adopted benchmarks used….

RT @TrelisResearch: + GRPO is Poor and for the GPU-Rich +.-------------------------------. *A specific GRPO vs SFT video will be out next w….

RT @lifan__yuan: lessons learned: (1) *capable* (small) base models are good enough to start rl, where (2) reasoning patterns *tailored to….

RT @TheTuringPost: 10 Free Comprehensive Datasets for Supervised Fine-Tuning:. ��️ Awesome ChatGPT Prompts.▪️ FineWeb from @huggingface.▪️ F….

RT @seungonekim: #NLProc .Just because GPT-4o is 17 times more expensive than GPT-4o-mini, does that mean it generates synthetic data 17 ti….

Link to Dataset: 📷

Are you fascinated by O1, QwQ, and Deepseek-R1? Why not try training your own⁉️ I’m sharing QwQ-LongCoT-130K, an SFT-style dataset for training O1-like language models in Apache 2.0 License. 🔥 Feel free to use and let me know what you think about it!