This was a great retreat put together by the @UCBEPIC team. As expected, applications of LLMs were central to many talks and discussions. Identifying the core problems invariant to the particularities of the next big LLM is key for research ROI in these applications.

Can you explore the space using LLMs - but do it in a way that is efficient? How do we find the high error regions?

It’s hard to robustly test edge cases in a model and make user defined concepts explicit

Fereshte Khani from Microsoft describes how to collaboratively develop NLP models, ensuring alignment and safety

A new system they are working on is Humboldt for data discovery. You shouldn’t have to ask experts about what data you should explore!

Alex Bauerle from Sigma Computing tells us about what’s hard when building a spreadsheet for cloud data warehouses

Can you fuse structural understanding of API programs with LLM techniques? Naman provides a way! Parametric templates for the win!

LLMs by themselves are insufficient for this task - brittle and hard to control

Naman Jain explores how to summarize data transformation scripts using a template-based approach, informed by LLMs

Flor allows users to travel back in time to help debug ML training. You can also inspect and “jump into” another user’s training history. Time travel and shapeshifting!

Rolando Garcia @rogarcia_sanz describes the next generation of Flor, a tool for rapid iteration during ML training via a live notebook demo!

Haotian leverages large language models to identify visualization intent (variants of BERT) and prior work on automatically translating visualization intent into actual visualizations (eg Lux).

Haotian Li describes how to support conversation with data via visualization - why write code when you can just talk to your data!

Can we check extensional equality (ie two programs have similar outputs) for constrained domains like biology? So that we can automatically rewrite and make code more performant — component by component?

There is a trade off between easy to understand code (eg one that loops through arrays) and those that are performant (eg one that manipulates arrays in NumPy)

Biologists, like many other non computer scientists, struggle to write performant code, especially on large datasets, such as genome sequences

Justin Lubin @jplubin embedded himself in a “wet lab” biology group to identify their programming challenges

Yet more challenges in Machine Learning - operationalizing, explaining and trusting it.

More open challenges in helping novice users through the data science workflow - so that one can go from “zero to hero”

Open challenges in data prep - even with sophisticated GUI tools, users often want to inspect and tweak underlying scripts - in tandem. Current tools don’t support seamless transitions and sensemaking