Khanh Nguyen
@khanhxuannguyen
Followers
2K
Following
847
Media
87
Statuses
1K
Postdoc at CHAI Berkeley with Prof. Stuart Russell, Prev. Postdoc at Princeton NLP, PhD @umdcs, Human-AI Communication, Interactive Learning, NLP.
Joined September 2014
The RLHF page of HuggingFace ( misses many important citations. Here are some classical RLHF papers that you should cite and why.
4
55
325
📢 Excited to announce our new paper . Language-guided world models: A model-based approach to AI control. • We develop LWMs: world models that can read texts to capture new environment dynamics.• These models enable humans to efficiently control agents by providing language
4
46
200
😠It is still ridiculous to me how much money/time was wasted simply because people don't read some old papers. 💡If you want to know why REINFORCE/A2C is better than Actor-Critic, read our paper: We have identified all of the common issues for you:.-
7
26
172
The objective mismatch issue raised in John Schumann's ICML talk was already foreseen by our paper ( 6 years ago. Sadly it wasn't cited nearly enough. Biased opinion: our paper deserves more readers and acknowledgement. In fact, not OpenAI's papers, but.
1
18
142
Why RL-tuning hurts calibration of LLMs? RL objective can be written as a reverse KL divergence which encourages mode-seeking behavior (i.e. peaky distribution). RL+translation has studied this phenomenon a long time ago ( .
4
11
90
Working on calibration/uncertainty for LLMs, which papers should I cite? Guo et al. ( is pretty popular but it is about classification tasks. Calibration on sequences comes with distinct challenges.
3
13
90
I have finally graduated, thanks to tremendous support from my research mentors (@haldaume3 @brendan642 @debadeepta, Dipendra Misra, and others), my family, and friends. I will be a postdoc @princeton_nlp and later @CHAI_Berkeley. Looking for opportunities to give talks :P.
14
5
80
This paper is so awesome!!! Debunking the “emergent capabilities” myth is the key to break the LLM recipe.
2
13
70
Our HANNA paper on Visual Navigation with Natural Multimodal Assistance has been accepted to #emnlp2019. New task/dataset/model/learning algorithm for leveraging vision-and-language human assistance in object-finding tasks in photo-realistic environments! (with @haldaume3)
2
14
62
After a wonderfullll year at Princeton, I am excited to join CHAI Berkeley, working Prof. Russell and Prof. Dragan to continue my effort to make AI communicate more effectively with humans. Connect with me if you are interested in learning from language feedback, learning to ask.
9
2
59
🚀 Dive into the untold story of Alignment via Human Feedback from an NLP perspective! This paper brilliantly encapsulates the epoch often overlooked in surveys written by RL groups. An absolute must-read for newcomers in the field! 📚.
0
20
56
This great work confirms my intuition: people have rediscovered problems of RLHF that was observed and documented many years ago when the method was first tried on machine translation. The finding in this paper is similar to People, especially.
"Less (tuning) is more for alignment" is an intriguing hypothesis. Is alignment tuning really that “superficial”⁉️ 🤔 If so, how so? 🤔 Can any straightforward analysis explain this? 🤔 What if I tell you “no tuning can also be great for alignment”? 🫢 😉 If you’re interested in
2
8
53
Maybe it's time to move beyond rewards and start 𝘁𝗮𝗹𝗸𝗶𝗻𝗴 properly to our ML agents! .Our ILIAD #ICML2021 paper formulates a learning framework where natural language is the only communication medium used by the teacher. Blog:
1
20
50
Happy to introduce 𝗚𝗹𝗼𝗯𝗮𝗹 𝗩𝗼𝗶𝗰𝗲𝘀, an evaluation dataset for multilingual and cross-lingual summarization in 15 languages (w. @haldaume3). New materials for studying translation quality in downstream task, zero-shot learning, etc. #NLProc #summarization #multilingual
1
9
47
Passing false-belief tests = model HAS theory of mind .Passing false-belief tests ≠ model USES theory of mind to perform tasks .Our #ACL2023 paper: formulates 𝑻𝒂𝒔𝒌-𝑶𝒓𝒊𝒆𝒏𝒕𝒆𝒅 cognitive capabilities, which are used to perform tasks.
1
10
45
Very delighted to receive an Outstanding paper award at @tom_icml2023. It is a great honor to be acknowledged by experts in the domain you have just recently ventured into :).
Passing false-belief tests = model HAS theory of mind .Passing false-belief tests ≠ model USES theory of mind to perform tasks .Our #ACL2023 paper: formulates 𝑻𝒂𝒔𝒌-𝑶𝒓𝒊𝒆𝒏𝒕𝒆𝒅 cognitive capabilities, which are used to perform tasks.
4
7
40
HANNA: Visual Navigation with Multimodal Natural Assistance is online.Our agent finds objects in photo-realistic environments by learning to query simulated humans for instructions. Paper: Github:
1
11
39
I wrote a thought piece showing RLHF = variational inference on Bayesian cognitive model (generalized RSA). I hope that realizing this connection can help better understand recent developments on LLMs and inspire future research.
0
6
31
When a language model guides a human, giving false instructions can frustrate them or even put them in danger. We propose a cost-effective method for detecting hallucinations in navigation instructions. More about our #EMNLP2023 findings paper⬇️ (1/n)
2
5
27
📢Internship at CHAI Berkeley. Apply by Nov 13. Opportunity to work with a group of leading experts in AI safety. I am particularly looking for students who are interested in learning from language feedback, and learning to ask questions.
0
4
23
Do language-to-world models like OpenAI SORA excite you? We are too! In this recent paper, we lay out a vision for this type of models. Not just video-creating tool, they will enable humans to collaborate safely and control AI easily. The code has been released. Check it out!.
📢 Excited to announce our new paper . Language-guided world models: A model-based approach to AI control. • We develop LWMs: world models that can read texts to capture new environment dynamics.• These models enable humans to efficiently control agents by providing language
0
2
20
Watch our agent, HANNA, find objects by asking for directions along the way. Full demo on Youtube: Paper: Github:
0
7
18
The hardest paper I have ever been a part of, both in terms of arguments, experimental setup, and technical depth. Could not achieve without help from amazing co-authors, and the open-minded reviewers. Learning from language is challenging but (to me) it is the future of AI!.
0
2
16
First time co-organize a workshop at a major conference. Great interactive audience, wonderful talks and discussions about #interactiveNLP. Simultaneous interpretation still awkward. Everyone seemed to be happy. Thank you all for contributing to this experience :D
1
2
14
Nguyen and O'Connor ( and Kuleshov and Liang ( are the first papers on calibration for sequences. They formulate and discuss challenges to this problem. Consider reading and citing these papers if you work on this topic :).
0
0
10
No offense to my Chinese friends. But if you are speaking to a general audience and you are unsure that they are all from China, use the term "𝗟𝘂𝗻𝗮𝗿 𝗡𝗲𝘄 𝗬𝗲𝗮𝗿". In Vietnam, we call it "Tết Nguyên Đán" (if anyone cares about inclusiveness).
1
0
11
This is great! It might imply that we have been doing actor-critic the wrong way the whole time? Actor critic seems like coordiante descent but the problem is that the coordinates are correlated?.
🔥Major Breakthrough in #RLHF! Traditional approaches fall short in characterizing policy-driven data dependency. Introducing PARL: a Unified Stochastic Bilevel Formulation. One of the FIRST provable solutions to #Alignment. 🚀 Essential for ethical AI! 📄
3
2
10
Non-OpenAI papers on the slide:. [Ranzanto et al.] [Bahdanau et al.] [Sokolov et al.] [Nguyen et al.] [Kreutzer et al.] [Kirk et al.] (survey).
0
3
10
Model uncertainty must be correct to be useful. Posterior calibration for NLP models http://t.co/5NJUQXhxWE
#nlproc #mlearning #datascience.
2
4
9
@umdclip @ml_umd @umdcs students presenting their work at EMNLP'19 in Hong Kong. A memorable event: first EMNLP paper for @swetaagrawal20 and last for @yogarshi and Weiwei Yang as PhD candidates.
1
3
7
(5/7) Julia Kreutzer is a veteran on this topic. She authors so many papers that analyze the feasbility of learning translation systems from human feedback (those with Sokolov, and .
1
0
8
@aahmadian_ @chriscremer_ @mgalle @mziizm @KreutzerJulia @ahmetustun89 @sarahookr this is the comparison I have been looking for! in fact, all of the early work on RLHF for text generation employed simple algorithms like A2C and REINFORCE and they worked fine.
1
0
8
By the way, @a1zhang is on the PhD market this year. He is smart, diligent, and productive, and is experienced with vision&language research. Grab him while you can 😃.
1
2
9
@fhuszar “Enough” does not mean “efficient”. A two-layer neural network with sufficient width can approximate any function. But the width could grow exponentially with the complexity of the function. Deep nets are more efficient function appriximators.
0
1
8
(1/7) In terms of RL for text gen, cite Ranzato+15 ( and Shen+ ( who pioneer training text generators to optimize rewards, and Bahdanau+17 ( who attempt the first actor-critic solution.
1
0
7
Come to our #emnlp2017 poster at 10.30am today (Sep 10 GMT+2) on Reinforcement Learning for Neural MT with Simulated Ratings. #nlproc
1
1
7
@QuanquanGu @iampanxu Indeed, all the early RLHF papers on text generation use REINFORCE and A2C.
0
0
8
I wonder if there has been work that compares DPO, PPO with simpler RL algo like A2C or even REINFORCE in fine-tuning LLMs. DPO can be interpreted as actor-critic with a cool math trick to obtain a reliable critic for free (i.e. use the policy itself as critic). It also has a.
4
1
8
(6/7) All of these works happened before or around the time of Christiano+17 ( who introduce the now well-known method for learning from rankings, and Stiennon+20 ( who apply the method with real humans on text summarization.
1
0
6
(4/7) Our 2017 paper ( is first to present and simulate the risk of using user ratings for training text generators. People have different opinions; one's opinion varies over time. We show RL is robust to granularity, skew in rewards but not variance.
1
0
6
(7/7) I hope this tweet conveys a better snapshot of the history of RLHF. Thanks for reading :).
1
0
6
(0/7) To some people, RLHF means "learn a reward model from human rankings and RL on it". But the term literally conveys a much broader meaning: any RL method that can learn from any type of human scalar feedback.
1
0
6
@StephenLCasper Nice survey but missing key citations. Please see this tweet for a deeper history of RLHF
The RLHF page of HuggingFace ( misses many important citations. Here are some classical RLHF papers that you should cite and why.
2
0
6
Finally had time to write some introduction about my research on calibration #NLP #calibration #machinelearning.
0
2
4
@ShunyuYao12 @tedsumers @karthik_r_n @cocosci_lab @princeton_nlp @PrincetonCS Share many of the opinions <3 In I was also thinking of a two-system architecture because inference with the rigorous reasoning could be slow.
0
0
6
The discussion on VLN reminds me of our motivation for creating VLNA (. The first thing we changed was to replace initial detailed instructions with high-level instructions, essentially removing the assumption that the requester knows the task solutions. .
The need for open data & benchmarks in modern ML research has led to an outpouring of #NLProc data creation. But @harm_devries, @DBahdanau & I suggest the low ecological validity of most of this data undermines the resulting research. Comments welcome!
1
2
6
@DrJimFan We did Sora+Genie but at a much more humble scale :p Still we realize that the problem of grounding language to dynamics is extremely difficult. With immense data, maybe you will generalize in distribution well, but achieving true compositional.
0
1
5
(2/7) In those works, rewards given to the model were dense and computed automatically (BLEU). Sokolov+15,16,17 ( is one of the first to really think about learning from human ratings, modeling the problem as bandit learning.
2
0
5
@DrJimFan @yoavgo @johnschulman2 yeah, the (learned) reward function may be still imperfect but the (unconfirmed) hypothesis is that evaluation is easier than generation so the reward function may still be of higher quality than a policy learned with the same amount of labeling effort.
2
0
6
The theoretical fact that RL = Reverse KL optimization is pretty well-known and has been re-discovered multiple times (e.g., .
Why RL-tuning hurts calibration of LLMs? RL objective can be written as a reverse KL divergence which encourages mode-seeking behavior (i.e. peaky distribution). RL+translation has studied this phenomenon a long time ago ( .
0
2
5
(3/7) "Bandit" is important because naturally you could only ask a human to give one rating for a whole text. Sokolov formulation characterizes how difficult the problem is compared to video-game dense-reward RL problems.
1
0
4
@yoavgo @johnschulman2 i think the viewing of llm having a fixed knowledge graph is slightly misleading, by instruct-tune you also add knowledge and modify the knowledge graph. the issue to me is overgeneralization: instead of learning just the taught knowledge, llm also learns hallucination behavior.
2
0
5