RT @glnmario: How should computational (psycho)linguists properly apply token-level language models to the field’s inherently character-lev….

Excited for tomorrow to present our work on LLM response stability in the context of political bias assessment at @COLM_conf! Stop by between 9 and 11 at poster #39. Joint work w/@j_vamvas and @LenaAJaeger. 🔗

Join millions who have switched to Grok.

But why do LMs emulate readers with low rather than high vocabulary size? We answer this question and discuss the implications of our results for the cognitive mechanisms of language processing in our paper: 📄 Thanks for reading!.

We find that LM surprisal is more predictive of reading time (RT) for readers scoring low in verbal intelligence and vocabulary size. LM entropy on the other hand predicts RT more accurately for readers with high memory updating and operation span scores.

(2) For each test, we assess the difference in predictive power of surprisal and entropy on reading times for the group or readers that performed above average vs the group of readers that performed below average.

We find that for most tests, readers that obtained higher scores show smaller surprisal effects (see Table) as well as smaller entropy effects.

(1) For each psychometric score, we fit a linear-mixed regression model with reading time as target variable and word length, lexical frequency, surprisal, entropy, psychometric score, and either surprisal x psychometric score or entropy x psychometric score as predictors.

💡in this paper, we test (1) whether readers exhibit surprisal and entropy effects relative to their cognitive capacities assessed via psychometric scores and (2) whether surprisal/entropy has higher predictive power on reading times for certain groups of readers.

👀📖 Excited to share our #ACL2024 findings paper on how language models emulate readers with specific cognitive capacities. See 🧵 . We'll present this work next week in Bangkok during the findings poster session 4 and at CMCL. w/@lsbolliger and @LenaAJaeger

RT @DiLi_lab: 📖👀 We're excited to present PoTeC -- The Potsdam Textbook Corpus! PoTeC is a German naturalistic eye-tracking-while-reading c….

RT @j_vamvas: Really excited about our new preprint: Generating high-quality machine translations can be accelerate….

RT @tannonk: 🔍Looking for some #multilingual #LLM reading for the holidays or just that last minute stocking filler? 🎅 . 👀Look no further!….

RT @lsbolliger: 📢 Interested in #eyemovements for #NLP? Or wanna chat about cognitive enhancement and interpretability of #LMs? .💃 Come by….

RT @noeminaepli: how do you evaluate systems that generate non-standardized #dialects? .check out our WMT23 paper 📜.

RT @lsbolliger: Check out the pre-print of our #EMNLP2023 paper on ScanDL, a diffusion model that generates human-like eye movements on tex….

2⃣ We should investigate phenomena that have not been at the center of current theories of individual-level sentence processing so far. For more details, check out the preprint!.📄

This finding is particularly important for theories of sentence processing that generalize to the individual level!.1⃣ We might have to revisit theories of individual-level sentence processing incorporating effects that are stable on the group level.

Our results show that individual sensitivities to word length are stable across experimental sessions and methodologies. However, most syntactic predictors and predictability effects have neither been found to be temporally nor cross-methodologically test-retest reliable.

Therefore, in our work, we assess the test-retest measurement reliability of individual differences for several psycholinguistic predictors: dependency distance, number of left dependents, surprisal, lexical frequency, and word length.

Importantly, the test-retest measurement reliability of individual-level effects cannot be taken as a given due to the "reliability paradox"! It states that measurement reliability at the individual level is lower for manipulations with high between-subjects reliability.