Wrote my congressman and senators today to ask them to speak up against the sudden and capricious cancellation of study sections at the NIH. Delaying funding for essential research into human health is idiotic.

Based on these findings, we propose that dopamine release depends on the "total valence" of a stimulus, or S = reward + aversion.

We also found that omissions of expected stimuli do not trigger release and that several stimuli with little or no valence lead to minimal release even when they are engaging, showing that PFC dopamine is not simply a surprise signal.

Dopamine release in the prefrontal cortex (PFC) is fascinating because it occurs following both reward and aversion. Here we tested what happens when you mix reward and aversion and found that amount of release seemed to depend on the sum of the rewarding and aversive components.

Proud to present work from my graduate student Yating Yang that just posted on BioRxiv.

One way of interpreting my paper is that, if the “handful of synapses” in Larkum’s formulation are selected via a comparison of pre and postsynaptic activity (which I call the “match and control principle” in the paper), you can implement transformer-like attention.

. while a handful of synapses several orders of magnitude less in number but with explosive impact dictate the firing of the neuron”.

When this project was in its infancy, a paper by Matthew Larkum, “Are dendrites conceptually useful” wrote of layer 2/3 pyramidal neurons, “The emerging picture is one where thousands of inputs combine to clamp the neuron at a particular subthreshold value, . .

In this paper, the attention computation is performed fully in parallel, using the large number of spines on apical dendrites to perform the many comparisons. The only limitation is that the number of spines on the dendrite caps the number of comparisons to hundreds.

The queries are then compared, one at a time, with the set of keys (N steps for N queries). This computation is linear in the number of keys and queries. It’s a great approach for sequence processing, but relatively slow if all the keys and queries must be processed at once.

Most “transformers in the brain” articles have tried to avoid the quadratic time computation of transformers (N keys compared with N queries requiring N^2 comparison) by first storing the keys in a neural network (N steps for N keys).

My article on how apical dendrites of pyramidal neurons can implement a computation similar to transformer attention is now published in PLOS Computational Biology

Tiny headsets for mouse VR! Great fun collaborating with the Schaffer-Nishimura lab on this project. Lead authors Matt David Isaacson and Hongyu Chang put in incredible effort to design, build and thoroughly test this system!.

It's still unclear if transformers have relevance to the brain, but in this paper I show that short-term Hebbian synaptic potentiation is enough to implement transformer-like attention: