RT @yulu_qin: Does vision training change how language is represented and used in meaningful ways?🤔 The answer is a nuanced yes! Comparing….

RT @najoungkim: Here's work where we try to take analogies quite seriously, plus our takes about LLM experiments serving to raise the bar f….

What do you want to know?.

I will be at #EMNLP2024 with some awesome tinlab people. Contact me if you would like to chat (esp. about 1. the similarities and differences between Connectionist/neural and Classical/symbolic computation, and 2. what compositionality, productivity, and systematicity are).

RT @najoungkim: tinlab at Boston University (with a new logo! 🪄) is recruiting PhD students for F25 and/or a postdoc! Our interests include….

I am at @COLM_conf till tomorrow night. Let me know if you would like to chat!.

These results suggest that statistical learners can use semantic training signals to achieve hierarchical generalization. Here is a link to the preprint!

We also find that grokking of hierarchical generalization from form alone depends on the presence of agreement in the training data (sheding light on the source of hierarchical grokking), while models trained on form and meaning are robust to this.

Why do some Transformers not generalize hierarchically? Maybe we aren't training them long enough! Recent work shows that (form only) models trained for longer generalize better (this is called 'grokking': see . We find that yes, longer training helps!

We find that Transformers trained on mapping form to meaning do show a stronger preference for hierarchical generalization than models trained on form alone.

Caveat: though children do not directly get such logical representations, we take these representations to be a conservative upper bound to what children can recover from context (see our paper for more discussion).

Would learners without a hierarchical bias (e.g., Transformers) generalize hierarchically when trained to map form to meaning?. We take meaning to be logical representations of the sort used by semanticists (linguists who study meaning)

But these works focus training on form alone (words and their order), while children presumably also get signals about the meaning of the sentences they hear that help them map the form onto the corresponding mental representations of meaning.

We found that various neural networks generalize linearly when trained on form alone, unlike children.

Prior (and also my computational work with @RTomMcCoy @tallinzen and @bob_frank !) suggests that hierarchical generalization requires stronger inductive biases than modern neural networks have (e.g., a hierarchical bias).

What is hierarchical gen and why is it an important/interesting problem?. Though kids often don't get direct evidence ruling out linear rules (e.g., for English yes/no questions), they always choose hierarchical rules. This generalization might need an innate hierarchical bias

NEW PAPER! We (@najoungkim and I) find that training on mapping from form to meaning leads to improved hierarchical generalization.

I am helping organize BUCLD this year! Please consider submitting your abstracts and symposium ideas.

I’ll be at the LSA annual meeting this week, and happy to chat! Message me if you would like to.

RT @RTomMcCoy: 🌲Interested in language acquisition and/or neural networks? Check out our poster today at #acl2023nlp! Session 4 posters, 11….

NEW PREPRINT. Excited to release my first first-author paper! We investigate if neural network learners (LSTMs and Transformers) generalize to the hierarchical structure of language when trained on the amount of data children receive. Paper: