Ian Hussey
@ianhussey
Followers
2,812
Following
1,790
Media
668
Statuses
11,397
Meta-scientist @unibern . “Jumped-up punk who hasn’t earned his stripes.” All views a product of my learning history. Chief recommender at @error_reviews .
Switzerland
Joined February 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
LA ROCHE MELA B3 x FOURTH
• 98673 Tweets
HAPPY 3RD BINIVERSARY
• 74894 Tweets
エアコン
• 31522 Tweets
白血球さん
• 28205 Tweets
Mela B3 With NEWWIEE
• 25836 Tweets
LAROCHE x JIMMYSEA
• 20457 Tweets
ライエモ
• 14912 Tweets
永野芽郁
• 13548 Tweets
ジンくん
• 13336 Tweets
細胞実写化
• 12263 Tweets
#サッカー日本代表
• 11025 Tweets
血小板ちゃん
• 10710 Tweets
#GiveMeThat1stWin
• 10518 Tweets
Pinned Tweet
It's not just p-values that get hacked.
We added a 3rd large dataset to our preprint: APA PsycTests.
67512 Cronbach's alphas from 60491 scales again show excesses at the .70 rule of thumb threshold.
@maltoesermalte
@rubenarslan
@TaymAlsalti
@frankbosco
7
68
210
How to configure Zotero to use Sci-Hub to automatically download full-text PDFs
16
519
2K
Amazing
@zotero
feature I just discovered:
If you highlight text in a pdf, then copy-paste it into another document, it pastes as a quote with a reference and page number!
17
159
1K
Working on a meta analysis.
Either researchers are *phenomenally* good at choosing sample sizes to find effects barely p < .05...
Or p-hacking/publication bias have turned the literature to mush.
Can anyone look at this and still think preregistration wouldn't help?
22
156
654
Today is the last day of my postdoc & I don’t have a prof or industry job lined up to sing about.
I don’t see many threads like this, but it must be happening to many given the current academic job market.
I honestly don’t know what I’ll do next, but I’m happy.
Ciao for now.
32
32
625
SciHub now contains 69% of all published articles and can fulfil 99% of article requests, may be unstoppable
16
359
549
Today in desk rejections of replications:
Original paper: X is the case. Published in top journal. Hundreds of citations.
Our paper: X is not the case. Failed replication, 7 extension studies showing severity of problem. Desk rejected, told to apply to a specialist journal.
15
76
538
New
#preprint
: Completing a Race IAT increases the racial
#implicitbias
it seeks to assess. 4 preregistered studies (N=940) including a direct self replication
#AcademicTwitter
#openscience
13
121
309
I just withdrew an accepted symposium from a conference that will now be online
- I would still have to pay to register
- Others would have to pay to view
- Organisation would own the content & restrict access
Why would I agree to this? Esp when preprints & YouTube are free?
4
33
306
My dad has abstained from voting for 10 years, but this morning we both voted
#Yes
to make Ireland a safer and more compassionate place for women and girls
#IvotedYes
#RepealThe8th
#Referendum2018
#TogetherForYes
#HomeToVoteYes
3
21
256
The minimally honest title here would be “leveraging climate change for neuroscience research”.
Our new paper outlines how neuroscience can be used to:
(1) study the impact of climate change on the brain
(2) identify ways to adapt
(3) study the neural substrates of environmental decisions & outcomes
(4) create better communication & interventions
11
64
247
12
32
282
Are citations in the psychology literature accurate? Did the work cited actually support the claim made in the citing work?
Across 3347 citations in 8 leading psych journals, 9.3% of citations were partially inaccurate & 9.5% were entirely false.
BUT:
6
107
332
Our manuscript 'Hidden invalidity among fifteen commonly used measures in social and personality psychology' was just accepted at Advances in Methods and Practices in Psychological Science! w/
@seanhughes102
18
90
244
Many universities require researchers to pay out of pocket and be reimbursed later. Grants are meaningless if you can't afford to pay out of pocket up front - and fragile employment means this is the case for many.
9
87
225
New study confirms that data is NOT available on request.
93% of authors with “data available on request” statements in their manuscript either did not respond or declined to share their data.
11
86
217
New preprint in which we show that there is an excess of Cronbach’s alpha values at common rule of thumb cutoffs (.70) in the psychology and I/O literatures.
@maltoesermalte
@rubenarslan
@TaymAlsalti
10
65
200
Manuscript just accepted
@Meta_Psy
!
p-hacking - the updating or adjusting of data or analyses in light of prior beliefs about hypotheses - is a cornerstone of psychology research.
I propose a way to streamline it: simply generate random numbers < .049
8
42
170
New preprint: Many psychology self-report scales may have hidden invalidity. Large
#dataset
of 150k participants and 15 scales
#replicability
#AcademicTwitter
#openscience
#preprint
8
60
167
Kahneman pronounces behavioral priming ‘effectively dead’.
More importantly, he tells us about the effective philosophy of science of our field:
It’s dead because it’s ‘not a wise career move’ not because it’s false.
truth < jobs
Daniel Kahneman, towering figure of social psychology, declares behavioral priming - which he himself was instrumental in making popular - "effectively dead".
68
1K
5K
5
20
172
1
19
143
What's the largest p value you've ever seen described as marginally significant?
I just found "p = .47" described as “marginal superiority" of one condition over another.
10
9
164
"Fraud is bad BUT" bingo card
Thanks for the input everyone!
5
25
156
I have a tenure joke but very few people get it.
I have a scientific publishing joke, please purchase PDF for $39 to access this content
41
757
5K
3
11
146
If you’ve ever completed an IAT online, you’ll know it gives you feedback on your level of bias.
@psycholojamie
and I thought it was weird that no one has ever put confidence intervals on IAT scores, so we did it.
It turns out the IAT can say very little about individuals…
The individual-level precision of implicit measures
0
4
14
4
43
151
A journal has rejected my paper on the grounds that it is 'already published' on a preprint server (
@PsyArXiv
) that that this 'prevents the ability to transfer ownership to the publisher'.
It's 2022. 🙃
Does anyone have resources I can send them?
11
17
140
Overheard in academia today:
Early career researcher: Any advice on how to deal with power dynamics?
Full prof: There are no power issues in academia, and ECRs should feel grateful to have the opportunity to learn from seniors.
🤡
8
8
125
THREAD
I'm psyched to finally announce this. We've curated a very large dataset on implicit & explicit attitudes, beliefs, and identity that we're making available. Journals including
@PsychScience
JPSP,
@CollabraOA
& others have agreed to take
@RegReports
based on it.
6
67
121
I just came across
@MatthewBJane
's Shiny app illustrating the impact of biases and artifacts (reliability attenuation, range restriction, & scale coarseness) on observed effect sizes - very cool learning tool
3
33
120
After you defend for a psychology PhD they take you into a dark room and tell you the truth. "The stuff that’s true is mostly unsurprising and the stuff that’s surprising mostly isn’t true. We just made this all up for grant money."
3
15
119
“Power analysis is just something you do to get your manuscript through the editor. You just have to report *a* power analysis, not the one corresponding to your tests.” - coauthor who I don’t see myself working with again
11
5
109
It’s not just p values that are hacked. The distribution of published AUC values also show distortions at the rules of thumb cut offs.
Nice work
@aidybarnett
and co!
Also applies Cronbach’s alphas:
Our paper on prediction models shows that many researchers are willing to fudge their models to get a "good" AUC. Have you ever been asked to nudge an AUC from just under to just over 0.8? . With
@nicolem_white
,
@RexParsons8
,
@GSCollins
0
25
68
1
30
109
I've changed my mind, Bem was right, we can feel the future.
It's the only explanation for how many authors I have contacted with a data sharing request have "recently wiped their laptop and lost that data".
Data wiping reliably precedes data requests: precognition is real
3
23
107
“one runs the risk of becoming a slave to the concepts that have been generated. Many researchers have taken terms … from everyday language and expect this linguistic categorization to somehow map to identifiable mechanisms in the brain”
organisms, environments and cognitive systems.
‘No one knows what attention is’ (Hommel et al., 2019)
1
8
38
2
22
98
Could someone please tell me under what circumstances further research is *not* needed?
17
3
100
Plane from Brussels is stuffed full of Irish people!
#hometoveote
#Repealthe8th
We are making this journey so that no person will have to make this journey in the future. They will get the proper medical attention they need right here in their own country, free from shame and stigma
#repealthe8th
3
58
261
0
7
79
🚨🚨🚨Shameless little bullies alert 🚨🚨🚨
Without my prior permission, a cell of self appointed data police have unfairly scrutinised my seminal work, in an unfair and bullying fashion. They claim to have found an "error".
2
18
101
"Crowdsourcing hypothesis tests: Making transparent how design choices shape research results" was just accepted at Psych Bulletin!
This was my first very large authorship study: I am 106th of 193 authors! Cool experience, cool paper.
THREAD:
1
13
83
I am very happy to announce that I will be moving to Switzerland to work with
@maltoesermalte
at the University of Bern!
7
4
90
I know
#rstats
very well, so why is it so hard to learn
#Python
/
#JuliaLang
on top of it?
- Different IDEs throw me for a loop
- Especially lack of Rstudio’s package help! Where the hell do I get documentation if not here?
- Not being able to run code line by line
Suggestions?
34
11
87
Remember early career peeps: it’s your mindset that’s the problem, definitely not your material conditions, job security, work hours, etc.
[growth mindset intensifies]
The two most common causes of hardship in PhD students are an inability to accept failure and choosing this career path for the prestige, rather than out of any real interest in research.
358
358
876
3
9
81
Job/Tenure application: I alone can design, run, analysis, and report studies of this brilliance, you must hire me to move science forward
After fraud is detected: I have never even been in the same room as the data, I do not read the manuscripts I coauthor, a rogue RA did it.
1
10
81
You can now rip another classic study out of your psych 101 textbooks. This is becoming an almost weekly thing.
New---and pretty substantially different from the textbook take---findings on split brain patients. The standard story always did seem a bit odd to me
3
52
162
6
18
59
I’m increasingly worried that much of psychology is analogous to Chomsky’s “colourless green ideas sleep furiously”: it has all the syntax of science and you can’t immediately dismiss it out of hand as not being a sentence/science, but it’s meaningless on inspection.
2
10
62
Speaking only for myself and not my coauthors, this replication project taught me more about the behaviour of researchers and their beliefs than it did about the effect being studied. And what I learned was pretty depressing.
🧵
Our RRR of Evaluative Conditioning in the absence of awareness is finally accepted at
@PsychScience
!
w/
@TalMoran3
@seanhughes102
+ many others below
Preprint:
1
12
34
2
23
66
Delighted to say I’ll soon be starting a 3-year postdoc working on this exciting meta science project with
@maltoesermalte
,
@rubenarslan
&
@TaymAlsalti
!
(the good kind of meta, not the zucc)
Very grateful to
@dfg_public
to approving the grant proposal "Standardization of Behavior Research Methods" led by
@rubenarslan
and myself as part of the priority program
@meta_rep
. What are our plans for the next three years, besides finding better soccer metaphors? 1/10
2
8
63
9
2
67
Psychological research needs to worry when *video game speed runs* are given more statistical scrutiny than most published articles.
This investigation into a recent Minecraft run contains a good discussion on sampling bias, p-hacking & optional stopping
6
10
67
Publication bias and hidden moderators: A quick simulation
Imagine a population effect of Cohen's d = 0.2
Simulate 100 studies with Ns of 50 to 150 sampling from this distribution. Meta analysis recovers the population value pretty well: d = .19, no heterogeneity
1
19
61
For any early career researchers wondering if the job market is sane: I’ve published in the journals (psych science, inquiry, and bulletin, JEP:G, AMPPS), won >250k in funding, moved countries.
2
5
61
Either it’s a massive coincidence that
@RegReports
produce way more null results, or our traditional publication system serves to pump out huge numbers of false positives
Any researcher who has ever had an article rejected because of null results will appreciate this analysis by
@ChrisPGAllen
&
@neuroccino
now in press at
@PLOSBiology
If you want quality rather than luck to shape your academic career then use
@RegReports
3
181
341
3
22
58
Career trajectories in psychology PhDs (N=1195):
Pipers - start & stay in TT - 17%
Droppers - start & leave TT - 34%
Nevers - never enter TT - 49%
Hoppers - start in industry but later enter TT - 11%
I've seen questions/speculations that many academics are leaving for industry.
I have a paper that follows the career trajectories of ~9k PhDs in STEM who graduated 2000-2008 that looks at exactly this.
Link below, highlights in thread.
7
108
332
2
14
60
"Data is available upon request" and other lies we tell ourselves
1
7
61
Just accepted at JCBS:
- power analyses suggest the IRAP literature is severely underpowered.
- much worse than social and personality psych & not improving
- 1% of the IRAP literatures meets minimum power (.80) to detect the *average* effect size in psych (r=.2)
Sample sizes in studies using the Implicit Relational Assessment Procedure are small. How small? Really, really small.
This has negative implications for statistical power and therefore replicability of claims in the literature as a whole.
2
16
55
6
17
59
To add:
So you're thinking of doing a PhD! Yay! Here's my best advice:
Seriously consider not doing it.
Get a full understanding of the long run economic impact of forgoing good salary and pension contributions for your 20s, and whether it will actually get you a better job.
So you're going to do a PhD! Yay! Here's my best advice:
1) Don't work nights/weekends unless you have a hard deadline. Work 40hrs, no more. My most cherished grad school experiences were hanging out with my friends. Do not waste your youth providing unpaid labor. Don't do it!!
343
6K
53K
2
4
51
Me: can we get accurate estimates of effect size?
Mom: we have estimates at home, we don't need preregistration, it constricts creativity.
Effect sizes at home:
()
1
11
50
Pinker calls a publisher’s choice not to publish a book a “banning” of that book.
TIL I have been banned from several prestigious journals.
1
6
51
Remembering the time when, after presenting results from multiple failed replications, the speaker said they’d keep trying as the original finding must be true “because it was published in Science”
4
5
56
Same. I’ve spent 6 years doing open science/methods reform work, and received a huge amount of hostility and criticism for genuine efforts to help.
It has often impacted my mental health, it’s hardened me and made me cynical. I miss being excited about science.
1
4
55
@aeriELLE_allen
I know many professors that would benefit massively from being more willing to admit ignorance, ask questions and change behaviour in light of new knowledge. Keep doing what you’re doing.
0
2
53
Sample sizes in studies using the Implicit Relational Assessment Procedure are small. How small? Really, really small.
This has negative implications for statistical power and therefore replicability of claims in the literature as a whole.
2
16
55
Introducing the Unregistered Unreport journal submission format:
- Conclusions only.
- Introduction, methods, results, limitations, code, and data ‘available upon request’.
5
8
46
I was asked to guest AE for a journal I’ve published in.
Said I was happy to, but that I’d likely ask for power analyses and cautious interpretation of interactions in ANOVAs (which are historically rare in this type of work)
Editor stopped replying a week ago.
5
0
46
I just got reviews back on a paper. One reviewer signed their review and one did not. Can you guess whether it was the anonymous or named reviewer who was oddly and unnecessarily insulting?
I continue to be pro-signing reviews as it makes us accountable.
4
5
49
@NAChristakis
@lessig
How could the university have caved to internet outage if it acted months before data colada published their findings online? There’s a basic issue of causality here. Did they feel the future?
1
0
49
Interactive publications are the future
There is a multiverse of possible ways data in a paper could have been transformed, analysed, or reported. Instead of a stale old PDF, what if you could change these options to inspect that multiverse? 2/7
10
247
617
0
18
51
Recently I experienced an entirely novel mode of snark on Twitter that genuinely made me laugh for its inventive pettiness:
Someone added me to a list called “People I will never collaborate with”
I added them to a list called “Likewise”
4
2
51
For anyone who is confident that "Science is self-correcting":
It is surprisingly hard to get a journal to retract *your own paper* even when you are explicit that you are explicit that you have detected data fabrication.
4
7
50
Research findings that are probably wrong cited far more than robust ones
2
17
51
I’m surprised more academics aren’t intolerable NFT guys, publishing makes just as little sense
3
3
48
We are so fucking back, let’s fucking goooOOOOOOO
Me: [chanting] psi, psi, PSI
Bem: [pounding his clipboard] PSI, PSI, PSI!
New preprint claims people can "telepathically" know who's calling them on the phone Big hmmmm for this one...
@EJWagenmakers
@BorsboomDenny
8
6
43
3
4
45
I am available for hire as an adjunct professor on a without labour basis. Employers must understand that there will no work provided for this position.
@UCAFT_UCLA
u interested?
2
6
42
Opening Twitter to see why
@RetractionWatch
is talking about my work
“Among articles stating that data was available upon request, only 17% shared data upon request.”
1
28
60
1
2
46
If you search for "p hacking" on google scholar you will find an astronomer by that name, and I think that is just wonderful.
BRB changing my name
3
3
46
My feed is full of people leaving philanthropy for industry. I’m not saying being an astronaut cowboy millionaire is perfect, but I love being one and think it can be a wonderful career option. A thread on why ⬇️
3
3
42
“You know, I know this effect doesnt replicate. I know that when I publish it in a journal, the Matrix is telling my brain that it is impactful & gets funding. After nine years, you know what I realize? Ignorance is bliss.
I dont want to remember nothin. Nothing. You understand?”
2
6
45
I had to go back and look at the individual scale items for an analysis, and now I'm wondering why anyone should be surprised that they're all associated.
#schmeasurement
3
6
42
I was disappointed to read that "Better Accuracy for Better Science...Through Random Conclusions" is a novel statistical method rather than a call for us to simply make random conclusions, which would be far funnier and more efficient.
2
4
40
M A T E R I A L C O N D I T I O N S
“Society’s understanding of mental health issues locates the problem inside the person – and ignores the politics of their distress”
0
5
42
Dan Ariely claims authorship order shields him from blame; speculates that a low-level envelope stuffer committed the fraud
6
21
91
1
6
41
A review of 107 institutions’ promotion and tenure policies across 7 countries shows “open and responsible
research practices are minimally rewarded and problematic practices of quantification [such as publication quantity] continue to dominate”
Indicators of research quality, quantity, openness and responsibility in institutional promotion, review and tenure policies across seven countries
1
5
20
0
16
40
@saul
This person has either misunderstood or is suffering from internalised capitalism or both.
0
0
38
New preprint w/
@ChadEDrake
Meta-analyses of published & unpublished studies (N = 1650) suggest the Implicit Relational Assessment Procedure has poor measurement properties:
Internal consistency = .52
Test-retest reliability = .18
@myACBS
2
10
36
Another step closer to my prediction that psych 101 textbooks containing only replicable effects will eventually be possible to write on the back of a bus ticket
TIL.
The "history of psychology" part of my lecture course is slowly becoming a 90 minute epic on the Stroop task.
4
34
172
0
7
39
@lakens
I made the mistake of forgetting that science uses the same “yes and” rules as comedy improv
2
3
40
When my department offers Early Career Researchers yoga sessions but not job stability
THERAPIST: I think stress is causing you to feel all this weight on your shoulders
ATLAS: Or
4
61
256
2
4
37
Very cool item content analysis of different measures of psychological flexibility by
@ongclarissaw
@eeericleee
Very little content overlap!
Raises further questions about how well we are measuring PF - or possibly even whether we're measuring it at all
1
9
39
If a given psychological measure is used in X publications, how relatively popular is that measure?
Some additional information from the APA PsycTests database that we couldn't get into the paper are available at
New paper with
@ianhussey
,
@TaymAlsalti
and
@rubenarslan
. We show that most measures are only used once or twice, with no tendency to agree on gold standards. This is a serious barrier to cumulative science, and leads to lots of papers without meaningful progress.
10
68
185
1
6
38
Deepfakes are both impressive and scary.
We were surprised there was no literature looking at their efficacy in changing peoples evaluations, and decided to change that.
New preprint and 🧵 - with links to our deepfakes
4
11
37
Open Science: Not even once.
0
2
37
@adamndsmith
@shaun_vids
I think the stronger version of this argument is that even households make use of debt and debt is very useful - buying a house is the most expensive thing most people buy and it’s almost always via a mortgage
3
1
35