Self-fulfilling alignment? (image credit: @QuintinPope5) https://t.co/VAho38TDgR

We had a great time on @natolambert 's new podcast talking about our new Alignment Pre-Training Research Agenda, where we're focusing on compute intensive interventions through end-to-end training. https://t.co/Jsy5ssXL8c

https://t.co/3kRPnYDEDc

p(simulation) is going up

See here for the full post:

We show this phenomenon persists despite filtering our dataset with methods commonly used in modern deliberative alignment training pipelines.

Generalisation hacking is a process by which a model generates outputs via reasoning such that training on these reasoning–output pairs leads to a specific behaviour on a separate distribution.

We show how deliberative alignment training can be undermined through a process we coin as 𝘨𝘦𝘯𝘦𝘳𝘢𝘭𝘪𝘴𝘢𝘵𝘪𝘰𝘯 𝘩𝘢𝘤𝘬𝘪𝘯𝘨.

Find the full work here:

Open-weight labs apply pressure to the CoT in a variety of ways: • Intense Cold Start SFT for reasoning • Readability Incentives • Preference training 𝙖𝙛𝙩𝙚𝙧 reasoning training is completed

What can we learn from open-weight training practices? Maybe a lot -- we've just released a blog post on common open-weight training practices across 6 model families and report implications for chain-of-thought monitorability.

TL;DR Open-weight labs apply pressure to the CoT in a variety of ways: • Intense Cold Start SFT for reasoning • Readability Incentives • Preference training 𝙖𝙛𝙩𝙚𝙧 reasoning training is completed

See https://t.co/1coPiQPc3f for more information on our current projects.

Apart will provide institutional support to our four cohorts built through MARS 3.0. These projects include work on establishing clear metrics for chain-of-thought health, novel AI control protocols, and fine-tuning procedures that increase the monitorability of LLM reasoning.

We’re partnering with Apart Research due to their track record in facilitating high-quality AI Safety research in their Apart Studio and Fellowship programs. https://t.co/4dBjNQ9IgH.

Geodesic Research would like to announce its partnership with @apartresearch to assist in the development of Geodesic’s fellowship programs!