[6/6] Using the GSM-Symbolic dataset (derived from GSM8K templates with transformed conditions/results), we verified that additional intervention along LiReF directions boosts generalization performance on the reasoning tasks and lessens dependence on memory recall.

[5/6] Furthermore, we found that by intervening model residual stream along the LiReF directions, we can intentionally modulate the switch between reasoning and memory recall mechanisms. This helps the model overcome issues such as insufficient memory recall or overthinking.

[4/6] Through causal intervention, we find that activating LiReFs during LLM's inference significantly enhances model performance on reasoning tasks. Conversely, suppressing LiReFs encourages memory recall, thereby improving performance on memory-intensive tasks.

[3/6] LiReF directions exhibit a gradient structure in representation space. Representations at extremes indicate pure Reasoning or Memory recall, respectively. Activation values near zero suggest the corresponding problem likely requires both memory and reasoning abilities.

[2/6] We identify Linear Reasoning Features (LiReFs) in LLM residual stream that linearly separate Reasoning-vs-Memory task representations. Experiments (4 LLMs, 6 datasets) confirm LiReFs broadly explain and influence reasoning across languages, domains, and tasks.

[1/6] LLMs excel on reasoning benchmarks. However, research shows they often generalize poorly to unseen problems, possibly due to over-reliance on memorized training data. Further, the mechanisms governing how/when LLMs switch between reasoning and memory recall remain unclear.

📎 https://t.co/M1CdaqMWK9 💻 https://t.co/ovqPJ5kxxA Thanks to lead Yihuai Hong @YihuaiH91773, incoming PhD student @NYU_Courant, advisors Zhijin Jing @ZhijingJin & Lei Yu @jade_lei_yu (UToronto) and collaborators Meng Cao @Meng_0209 (McGill) & Dian Zhou (UIUC)

🤔How do LLMs perform reasoning and recall memorized knowledge? How similar are their underlying mechanisms? We reveal their inherent distinction within LLMs' representations, and identify linear features that mediate model switch between genuine reasoning and memory recall.

@littlefish3625 We believe this is a good example of how the analysis of internal mechanisms and representations of AI models can yield actionable insights and make models better and safer! Paper at: https://t.co/gBbEBCfB8D Work done with @gini_do @mahnerak and @nicola_cancedda at @AIatMeta

@littlefish3625 3) We propose a new adversarial training method, ReFAT (Refusal Feature Adversarial Training) that is both more effective against adversarial attacks and much less computationally expensive than known alternatives

@littlefish3625 2) The ablation of the refusal feature is a strong approximation to the worst-case adversarial degradation used in state-of-the-art adversarial training methods aimed at protecting against jailbreaking.

@littlefish3625 1) The most effective jailbreaking methods all work by suppressing the representation along the refusal feature, to the point that they become ineffective when such feature is restored at inference time after the attack

@littlefish3625 observed that the decision to refuse a request is mediated by a single direction, and highlighted a connection between adversarial attacks and refusal direction alterations: https://t.co/0mDRqaCoy0

New paper! 🎊 We are delighted to announce our new paper "Robust LLM Safeguarding via Refusal Feature Adversarial Training"! There is a common mechanism behind LLM jailbreaking, and it can be leveraged to make models safer!