Very excited to share that I have been awarded @NSF CAREER award @plfmse @IllinoisCS. At the Intelligent CAT lab, we have already made awesome progress on some of the proposed ideas and looking forward to exploring other proposed ideas in future.

By my amazing PhD students, Changshu Liu (the lead author) and Yang Chen.

Paper: https://t.co/P4JFOTcdhU Code*: https://t.co/tdhtfgF89c * We have dockerized the artifacts to make it easy to build on top of them. But don't hesitate to create issues if you are interested in assessing a new model or extending the evaluation to new benchmarks (8/8)

Frontier LLMs perform well on bug prediction, localization, and repair in HumanEval, but with about 40% correct and coherent reasoning. This is the elephant in the room: without proper reasoning, we observe success in benchmarks that cannot be generalized to new benchmarks (7/8)

CES is 𝗯𝗲𝘆𝗼𝗻𝗱 𝘀𝗼𝗹𝗲𝗹𝘆 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗰𝗼𝗱𝗲 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴, and can be used to determine 𝘀𝘂𝘀𝗽𝗶𝗰𝗶𝗼𝘂𝘀 success in other programming tasks that inherently require code execution understanding (6/8)

Even when LLMs correctly and coherently reason about the code execution, their reasoning across different tests is inconsistent, mostly random (48.87%) or weak (45.37%) (5/8)

Frontier models, while good at predicting the output, may do so through 𝗻𝗮𝘁𝘂𝗿𝗮𝗹 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝘀𝗵𝗼𝗿𝘁𝗰𝘂𝘁𝘀 or 𝗵𝗮𝗹𝗹𝘂𝗰𝗶𝗻𝗮𝘁𝗶𝗼𝗻. Yet recent models keep claiming victory in code reasoning by relying solely on output (and input) prediction (4/8)

The coherence metric allows you to determine (1) where the execution reasoning diverges from the ground truth for incorrect output prediction or (2) flag suspiciously correct output predictions (despite the divergence of execution reasoning from the ground truth) (3/8)

𝗰𝗼𝗵𝗲𝗿𝗲𝗻𝗰𝗲 metric determines whether the simulation complies with commonsense execution logic, and 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 determines the degree to which LLMs correctly reason about different execution paths of the same code (2/8)

Announcing "Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models" (@ICSEconf'26): the Code Execution Simulation (CES) task to evaluate LLMs for 𝗰𝗼𝗱𝗲 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 concerning 𝗰𝗼𝗵𝗲𝗿𝗲𝗻𝗰𝗲 and 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 (🧵1/8)

If you do research related to code translation, transformation, and modernization, consider submitting your research to ReCode workshop and attend it (colocated with @ICSEconf in Brazil)! Check these tweets and workshop website for more details.

I am looking to fill a Research Associate/Assistant position on an exciting project at the intersection of program analysis and GenAI. Candidates should be able to start soon (in the next few months). If you are interested, please get in touch ASAP! https://t.co/KR3xHHPQjm

#ILLINOIS CS professor Lingming Zhang has been inducted into the 2024 class of ACM Distinguished Members. This year, ACM has named 56 Distinguished Members for their technical achievements as well as volunteer service to their professional community. ▶️ https://t.co/1RosNdsH1j

Are you graduating with a PhD?  Do you work on program analysis, formal verification, software correctness, or AI?  Apply to join the UMass LASER lab as a postdoctoral researcher, a vibrant team focused on using the latest NLP technology to ensure software correctness!

The experience that changed my life.

While the program mostly targets undergrads interested in perusing MS/PhD degree, some of us (e.g., Darko and I) also work with high school students. Please reach out to individual faculty before applying, if you are a high school student.

We have opened the call for application to UIUC+ Summer Research Program for undergrads: https://t.co/1zXuXapw6V. Please apply or spread the words. The mentors are 12 UIUC PLSE professors/alumni. The (soft) deadline is February 28th, where we start reviewing the applications.

I keep forgetting to post @IllinoisCDS Systems Research Seminar ( https://t.co/KnzdF4Kmcc). Xudong (@xu_dong_sun) kicked off the seminar this semester, with his practice talk -- "Improving Reliability of Modern Cloud Systems with Verification in the Loop." Interview him if you

IEEE Fellow, as of today! I love all my students and collaborators. @IllinoisCDS @uofigrainger #IEEE https://t.co/YXYXInvaCw

🚨Attention Software Engineering researchers! 🚨 @UCFComputerSci is hiring an Assistant Professor this year in the area of SE + systems! If you would like to join me at @UCFCECS in beautiful Orlando, check out the job ad below and please reach out! Our department is set to

Xudong Sun is on the faculty job market. He is truly brilliant, and has been doing exciting research on System Verification and Software Testing towards the vision of provably correct cloud infra and systems. His materials can be found at, https://t.co/cXdlm4HzTT Interview him;