1/ When is an algorithm an improvement over an existing (e.g., human) decision policy? Our #ICML2024 work offers an approach for recovering tighter relative performance intervals under unmeasured confounding by isolating comparison-related uncertainty. https://t.co/NzGjUv6BpM

📄 https://t.co/dSUMntqCnL This work was in collaboration with the amazing team @dev7saxena (co-first author), @snchancellor, @zstevenwu, and @d19fe8 Thank you for making my first adventure into qualitative research a delightful experience :)

Our paper offers design implications to support this, such as: - Protocols to help data scientists identify minimum standards for validity and other criteria, tailored to their application context - Tools to help data scientists identify and apply strategies more effectively

The challenge for HCI, CSCW, and ML is not to *replace* these bricolage practices with rigid top-down planning, but to develop scaffolding that enhances the rigor of bricolage while preserving creativity and adaptability

Yet from urban planning to software engineering, history is rife with examples where rigid top-down interventions have failed while bottom-up alternatives designed to better scaffold *existing* practices succeeded

In light of these findings, we might be tempted to more stringently enforce a rigid "top-down planning approach" to measurement, in which data scientists more carefully define construct → design operationalization → collect data

How do data scientists evaluate validity? They treat their target variable as a tangible object to be scrutinized. They "poke holes" in their definition then "patch" them. They apply "spot checks" to reconcile their theoretical understanding of a concept with observed labels

Data scientists navigate this balancing act by adaptively applying (re)formulation strategies For example, they use "swapping" to change target variables when the first has unanticipated challenges, or "composing" to capture complementary dimensions of a concept

While engaging in bricolage, data scientists balance the validity of their target variable with other criteria, such as: 💡Simplicity ⚙️ Resource requirements 🎯 Predictive performance 🌎 Portability

We find that target variable construction is a *bricolage practice*, in which data scientists creatively "make do" with the limited resources at hand

To explore this tension, we interviewed 15 data scientists from education and healthcare sectors to understand their practices, challenges, and perceived opportunities for target variable construction in predictive modeling

Traditional measurement theory assumes a top-down workflow, where data is collected to fit a study's goals (define construct → design operationalization → collect data) In contrast, data scientists are often forced to reconcile their measurement goals with *existing* data

A subtle aspect of predictive modeling is target variable construction: translating an unobservable concept like "healthcare need" into a prediction target But how does target variable construction unfold in practice, and how can we better support it going forward? #CSCW2025🧵

💡Can we trust synthetic data for statistical inference? We show that synthetic data (e.g. LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moments of synthetic data and those of real data

1/10. In a new paper with @didaoh and Jon Kleinberg, we mapped the family trees of 1.86 million AI models on Hugging Face — the largest open-model ecosystem in the world. AI evolution looks kind of like biology, but with some strange twists. 🧬🤖

Excited to share our #FAccT25 translation tutorial, where we'll explore how to reconceptualize AI measurement as a stakeholder-engaged design practice 🙋🔍🖥️ Next week Thurs 6/26 at 3:15 pm (last day and session - please don't leave the conference early!) 🧵

Check out the camera-ready version of our @icmlconf position paper ("Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge") to learn more!!!

In @techreview, I wrote about the crisis in AI evaluations — and why a new focus on validity could be the best way forward

✨How can we help communities collaboratively shape policies that impact them? In our #CHI2025 paper, we present PolicyCraft, a system that supports collaborative policy design through case-grounded deliberation. (🧵/11)

Remember this @NeurIPSConf workshop paper? We spent the past month writing a newer, better, longer version!!! You can find it online here:

I'm recruiting a postdoc via the CBI Fellowship Program ( https://t.co/0c1WTuhy40). Interested in designing & evaluating interactive systems for participatory AI, augmented and collective intelligence, responsible AI, or related areas? Please apply and reach out to me via email!