✨ Excited to be at @NeurIPSConf in San Diego next week with the team at @poolsideai ⛱️ Booth #913! If you’re building, researching, or thinking about the next wave of AI systems, especially around RL and agentic workflows, I’d love to connect. Ping me if you’re around! ✨

Better late than ever, finally, code is available🙂 💻 https://t.co/bsgZsbq5Oh

Research Scientist (permanent) positions just opened in our NLP team in London - Huawei Noah's Ark Lab! Looking for experienced researchers to help us tackle some interesting (and persistent) questions :) Take a look and apply if interested: https://t.co/Q4JajEMFLQ

Delighted to present our #naacl2024 paper, #HumanRankEval: Automatic Evaluation of #LMs as Conversational Assistants with @glampouras_NLP and @iiacobacNLP providing fast, reliable and private evaluation of instruction-tuned #LLMs. A truly new eval paradigm, no #gpt4-as-a-judge :)

Checkout our latest work!

So happy to be back to MCR for this talk! Thanks again for the invitation!

Looking to grow our NLP team in London - Huawei Noah's Ark Lab! We have positions for Research Scientists (permanents) and Engineers (contractors), where they will conduct academic and applied research in NLP and ML. Details below: https://t.co/dDxE97BgRl https://t.co/LGN9snl3yb

https://t.co/5QagPC5fjx

If you are in Malta 🇲🇹 (hi!👋), come join me today at the Generation session (Marie Louise 2 - Radisson Blu Hotel) 14:00 - 15:30 local time, where I will present our work "Text-to-Code Generation with Modality-relative Pre-training"! 🎉

If you are attending @eaclmeeting, join #UncertaiNLP workshop! I will give my keynote discussing uncertainty quantification & evaluation and —spoiler alert— conformal prediction. If you are interested in uncertainty (if not, maybe you should :)) #UncertaiNLP is the place to be!

Many thanks to all the wonderful reviewers, area chairs & senior area chairs that support our reviewing process. Here's a special shout out to the *best reviewers* for October cycle. https://t.co/wbQoMireaV @ReviewAcl #NLProc

See you in Malta! 🇲🇹

By analysing embeddings of code-related words, we find (as expected) that in the "code" space similar operations come close together but in the "NL" space they are further apart, depending on their semantics (e.g. open-close).

We experiment with 2 LMs, 2 model scales and 4 training objectives, testing on HumanEval & MBPP. Our findings suggest that embedding separation, as a 2nd training stage, can improve code generation mostly on p@10 and p@100, indicating that it helps increase model expressiveness.

To granually measure performance, we propose a new variation of the classic functional correctness, pass@k metric: incremental pass@k. This includes code completion eval, models are prompted with partial code solutions, incrementing each successive prompt one code line at a time.

This is realised via 2 stategies: partial separation (only separating embeddings based on programming language keywords) and full separation (separate all embeddings).

We posit a simple idea: Code keywords have strictly defined semantics, hence they need to be treated as a different *modality* from natural language. We start from a pre-trained CodeLM and associate distinct embeddings for each token depending on its modality, focusing on Python.

Happy to share our work on "Text2Code Generation with Modality-relative Pre-training" w/ @gcsanity & @glampouras_NLP, accepted at #EACL2024! 🎉 We propose to treat code and natural language as different modalities. 📜 https://t.co/fOYsUARpve 💻coming soon (pending int. review)

Very happy to be co-organizing this workshop and shared task with the fine folks at @TCD and @AdaptCentre! Please help us spread the word of our research track and shared task: Paper deadline: Dec 18th ARR commitment deadline: Jan 17th Shard task deadline: Jan 20th

There is a beautiful story that just happened in AI so let me share it for a lighter tone weekend post among all the doom stories in our AI field this week. It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI