Joe Simmons
@jpsimmon
Followers
7K
Following
1K
Media
27
Statuses
599
Prof. at Wharton. I research/teach decision making & research methods. Blog: https://t.co/XOI7aJXW1d ; Easy Pre-registration: https://t.co/qQwukhfPZm
Joined November 2012
Gino's case against us has been dismissed. Scientists cannot effectively sue other scientists for exposing fraud/errors in their work. Those who work to correct the scientific record can sleep better tonight. Those who don’t want it corrected, well, I don’t care how they sleep.
57
468
2K
I used to teach this finding to my MBA students. It has more than 400 citations on Google Scholar. It's about dishonesty. Turns out it's fraudulent. "Evidence of Fraud in an Influential Field Experiment About Dishonesty"
27
258
987
In 2021, a team of us sent a report to Harvard University detailing evidence of fraud in 4 papers (spanning 11 yrs) co-authored by HBS professor Francesca Gino. This is the 1st in a series of Data Colada posts describing that evidence.
13
129
549
Fraud has so many victims, and we don’t see or hear from most of them. You can’t compete with fraud in a system that doesn’t stop it. And so we have to stop it.
9
58
336
Thanks to everyone who supported us, both emotionally and financially. Thanks to our schools for the generous financial support. And thanks to our amazing lawyer, Jeffrey Pyle. We never felt alone in this, and we never felt like we had to stay silent. Thank you.
5
15
314
Hypothetical: Imagine a world in which some scientists cheat and some don’t and that the cheaters can’t/don’t get caught bc nobody checks. Who’s more likely to get papers/jobs/editorial positions: cheaters or non-cheaters? Who’s more likely to have to quit?.
16
52
285
New Data Colada post. Harvard's Gino Report allows us to re-construct what Harvard says is one of Gino's "original" data sets. We can compare that to the posted data to see how the data were altered
7
61
260
In the file that Ariely sent to coauthor Nina Mazar back in 2011 (3 yrs after the study), the effect was in the wrong direction. When Nina said so, Dan told her that he had accidentally flipped the conditions when preparing the file. See Footnote 14 in
Ariely, in a statement, now says: "Getting the data file was the extent of my involvement with the data."
2
26
196
Doesn’t look like any of the pre-registered effects are significant in the analyses that correct for multiple comparisons. The biggest effect - which was also not sig. after the correction - was not pre-registered. Great the authors reported everything. But the evidence is weak.
2
15
135
Part 2 of Data Falsificada is now up on Data Colada: "My Class Year Is Harvard".
7
32
110
An update on Gino's lawsuit against us (Data Colada). We had a hearing about our motion to dismiss, and we've learned some legal things along the way. No big news yet, but some details:
2
18
114
VERY excited to announce ResearchBox, a new and easy way to share data, code, pre-regs, & materials. I love using it, both as a researcher and a reader. We hope you do too.
1
30
106
Part 3 of Data Falsificada is now up on Data Colada: "The Cheaters Are Out of Order".
4
35
105
Open science isn’t just about posting materials/data. It's about allowing actual humans to *easily* access, comprehend, and reproduce them.
1
32
103
In response to some unpleasant accusations following the @susandominus NYT piece, I've decided to continue to be a rigorous scientist.
5
22
103
Participants were not randomly assigned to condition. n = 12 in the morning condition.n = 20 in the afternoon condition. All significant p-values were between .02 and .05. I'm going to keep running in the morning.
Depending who you are, exercising in the afternoon may be better than in the morning.
5
8
91
Reviewers cannot know what actually happened in a study w/o having access to the study materials. Going forward, I will respond to any review request by asking for the materials & data. If not provided (w/o a good reason), I will decline to review. I encourage everyone to do this.
9
24
92
In my MBA class, I ask my students to generate a random integer from 1 to 100. In every year, I find that numbers ending in 3 or 7 are the most common. So, when people are trying to type in numbers that look random, they disproportionately type in numbers that end in 3 or 7.
4
10
84
Fraud has its visible victims. But there are countless invisible ones.
2
8
83
The insurance company says they didn't fake the data (see Data Colada [98]). They say they ran the study in 2007-2008 (with way fewer observations) and found nothing. Ariely was already publicly saying the study worked in July 2008. See minute 38:20:
SCOOP: For years, Dan Ariely has suggested that the fabricated data in one of his most famous studies was doctored before he got it. This week, the insurance company he collaborated with came back swinging -- with lots of details:
2
11
82
In my MBA class yesterday, a student asked, "Aren't scientists required to make their data available?" In response I just started screaming.
1
5
76
My dept at Wharton is hiring Assistant Professors in the area of decision-making, defined very broadly, including psych, marketing, management, econ, computational social science. Please apply ASAP and definitely by Oct 13.
0
31
68
Part 4 (of 4) of Data Falsificada is now up on Data Colada: "Forgetting The Words"
0
12
67
I do think that giving (enough) cash to poor mothers is very likely to have beneficial behavioral and developmental effects. And I think those effects should be studied. Just not convinced of this one piece of evidence, or that brain activity is what we should be focused on.
3
3
55
The fourth and final installment of the Data Falsificada series will, god willing, be out later this week, probably Friday. The delay is due to miscellaneous life chaos and (my own) personal shortcomings. (These are also the reasons why I haven't responded to your email.).
2
3
57
That self-appointed data police vigilante is Steve Haroz (@sharoz). VERY much appreciate his catching this mistake and notifying us.
Some vigilante from the self appointed data police downloaded the posted data, tried to reproduce our figure and found an error. Arrow was pointing at the observation in wrong condition for 'harvard' lower case. We have corrected it. . Very sorry.
3
7
61
New post showing how meta-analytic averages of dishonesty research include (1) erroneous effect sizes, (2) a study of nuns, and (3) 48 variations of the same study. Averages of invalid estimates are invalid. Averages of disparate studies are meaningless.
3
11
59
Without preregistering it is very hard for a researcher to not p-hack. Me included. P-hacking is not fraud. No academic researcher has ever been fired or asked to resign for p-hacking. Researchers who are asked to resign have been caught engaging in intentional misconduct.
6
8
57
New retraction at OBHDP. Gino was an author. She had nothing to do with the anomalies. Study 1B had data "that were unlikely to have occurred naturally". The study was done at Wharton but "they do not know who handled the data."
2
13
59
Finally got around to reading this. It is excellent. Informative, useful, easy to read. This article should be assigned in PhD methods courses. Highly recommend.
Happy and excited to see this article appear online! I hope it will help correct common misconceptions in how outliers should be removed, and ultimately lead to lower false-positive rates in the literature.
1
11
49
If you use R, and you like it when your code works, then you should use groundhog. It's pretty amazing. Check out Uri's post about it here:
2
8
49
Sorry if someone has said this already, but the incorrect statistics in the recent Ariely expression of concern may simply be a mistake of reporting t-values as F-values. E.g., whereas F(1,84)=2.41 implies p=.124, t(84)=2.41 implies p=.018, as reported in the original article.
2
3
43
A recent Journal of Marketing Research paper finds that consumers find "progression" weight loss ads (like the one on the right) more credible than before/after ads (like the one on the left). We tried to replicate this finding.
1
5
42
“this means that as a default, authors will be required to post their data, materials, and code to a trusted repository before accepted papers are published.” Hell. Yes.
Check out Lucas's awesome editorial:
1
4
37
There were three measures of performance. Two were non-significant and a third was p = .023 (and it looks like there were a few different ways to score that one). The evidence here isn't overwhelming.
Coffee is a mind-enhancing drug that actually works!. A randomized, placebo-controlled, between-subject, double-blind study shows the caffeine equivalent of one cup of ☕️ increases problem-solving ability, especially for problems requiring insight.
1
3
35
Those accused of p-hacking often act as if they have been accused of murder. But p-hacking = not knowing exactly how to analyze your data and so trying more than one way. Not quite murder. Good (and bad) people are p-hackers. If I don't preregister my studies, I will p-hack them.
4
5
27
This is a such an impressive effort and a great way to audit a literature. I strongly recommend reading this paper. I also recommend watching Leif's Data Colada Seminar presentation, during which he describes this work:
In 2019, Leif Nelson and I co-taught a class on open science and replication. Everyone in class replicated a published finding examining the psychological effects of scarcity. The results were published today in PNAS
1
4
29
It is awesome that so many scientists elect to be transparent. But we can't rely on that in the long term. Being a scientist means showing your work so others can check it. (It also means taking the time to check others' work.) Transparency should be a requirement, not a choice.
0
5
28
It was great to talk with two such incredibly thoughtful people about a topic that has not been talked about enough.
Episode 73 of Two Psychologists Four Beers is live! Check out “We Need to Talk About Fraud (with Joe Simmons)” at
1
6
29
Research assistants need oversight.
New (Gino-related) blog post: Attack of the 50-foot research assistants.
1
0
27
Devastating loss for Wharton. Killer acquisition for Yale. Deb is an amazing advisor, researcher, colleague, and human. Determined to be happy for Deb (and Yale) instead of sad for me (and Wharton).
Some personal news: Today is my first day on the faculty at Yale!.
0
1
28
The "Evil Genius" retraction cites "only" Experiment 4. I would bet my house/leg/whatever that the problems go beyond just Experiment 4. But even if it were "only" Experiment 4, this is the right model. ALL papers containing fake data should be retracted.
1
5
28
I had forgotten (or never knew) this detail about Festinger & Carlsmith’s (1959) classic cognitive dissonance experiment.
1
2
24
Some failures to replicate are due to differences in design between the original and the replication attempt. But *many* failures to replicate occur simply because the original finding is a false-positive.
0
4
24
I'm reflecting on this: Just bc there are no good arguments against commonsense calls for change (e.g., materials/data posting) doesn't mean that change will happen. "Let's keep talking about this" is how those who control our journals prevent change. They are still talking. .
1
5
27
Impossible question. But 50 years after publication, Kahneman & Tversky's "Belief in the Law of Small Numbers" is still so widely applicable. So many errors & biases arise from the fact that we underestimate sampling error and the power of random chance.
What’s the best academic paper you’ve ever read and why?.
1
3
26
My favorite thing about Microsoft Word is how it defaults to assuming that I want my footnotes to be in a different font than the font I am using in the rest of the paper.
1
0
25
Another threat to the validity of scientific findings: hidden confounds. Until we require authors to post original materials, to show what they actually did in their studies, we aren't doing science. We are playing dress-up. And rewarding the wrong things.
0
10
23
I want this book, but I don't want it used. Should I buy the hardcover or the paperback?
3
0
25
I always tell my students: If you want your article to have impact, make sure its publication gets announced on the weekend before Christmas. Well done, guys.
0
3
19
Nice work. And it is time to start ensuring that all pre-registrations are thoroughly peer reviewed. If a pre-registration is vague/incomplete/not-adhered-to, the author should not be able to claim that the study was pre-registered.
"The Curious Case of the Convenient Outliers: A Twitter Thread". A recent paper in a leading psych. journal reports a pre-registered experiment with significant results: The "Predicted High" condition is significantly different from the two "Predicted Low" conditions.
2
5
22
Powerful new paper showing that disfluent fonts probably don't do anything, including things Leif and I had claimed. http://t.co/mkCT9eguFH.
2
13
19
@morewedge Some folks are trying to get a process in place. In the mean time, co-authors should feel free to reach out.
0
0
20
This paper by @minah_jung, Alice Moon, and Leif Nelson is simply awesome. Every study - and there are *many* - taught me something interesting that I didn't know before. Psychological science at its best.
0
1
17
New on Data Colada: Don't Trust Internal Meta-Analysis. Arguably our most important post. I super wish IMA was a good idea, but it is catastrophic, dramatically increasing false-positive rates. {Insert frowny face here}.
1
9
14
It is also worth noting that the person (@giladfeldman) who reported these erroneous stats to the journal was careful to point out they he did successfully replicate the results of Study 1 of this paper.
2
2
16
@StuartJRitchie They find that left v right leads to a 211-calorie reduction from a base of 865 calories (24% effect). But a big RCT in a similar setting finds that NO labels vs. calorie labels (i.e., a bigger intervention) = much smaller effect (3%). 24% is too big.
1
2
15
How to evaluate replication results: A new, thoughtful, and *much* better approach from @uri_sohn. http://t.co/MTqQ4MZqTw.
0
8
13
New on Data Colada. Meta-analytic means are (very) often uninformative or misleading, as it does not make sense to average invalid results together with valid results, or to average across studies with different manipulations or measures.
0
5
13
What if the NY Times reported every HUMAN-caused traffic fatality on its front page?
0
3
9
New on the Data Colada - MTurk vs. The Lab: Either Way We Need Big Samples http://t.co/S9h5zb6lzB.
1
9
14
Yeah, but psychology is 220% complete.
0
1
12
The ad hominem argument: A good sign someone is on the wrong side of the truth MT @yorl @BrianNosek
http://t.co/j42kPjNXjG.
2
15
12
New on Data Colada: Descriptions of serious and unexplainable data anomalies in a Psych Science article. In my opinion, an "expression of concern" is not enough in this circumstance; the article should have been retracted.
0
2
11
Why are people's confidence intervals too narrow? Don Moore (@donandrewmoore) is going to help us figure it out tomorrow, in the Data Colada seminar series (12 pm Eastern). Very excited. To get the link to this seminar (and others), subscribe here:
0
2
12
It is (selfishly) smart, and now super easy, for researchers to pre-register their studies:
0
6
12
New on Data Colada: Leif describes why it is usually impossible to compute an average effect size.
0
2
10
Agree 100%. Also, I can tell you from experience that when you go through the "hassle" of making your data and materials available, you catch errors before you submit your paper. That's an important positive side effect of this requirement.
0
2
10
Jerry Nelson, Leif Nelson's dad, recently passed away. He was a great scientist and a great man.
0
3
11
We ran another replication of a study that was recently published in the Journal of Consumer Research.
0
1
12
1. Looking for methodological solutions that fix *every* problem will be unproductive. Different problems require different solutions.
1
1
7
@MetaMethodsPH I can't speak to everything in this thread, but I have looked into this a little, and it seems like the standard deviations reported in the tables must really be standard *errors.*.
1
0
10
Does it make sense to average the effects of (1) a “dish-of-the-day” label and (2) reminders to go to bed at a decent hour? No. But this is something meta-analysts do all the time. And readers accept these averages as truth.
0
3
10
New on Data Colada: Pilot-Dropping Backfires (So Daryl Bem Probably Did Not Do It)
0
2
7
@I_Evangelidis Agreed: some p-hacking is intentional. But (1) I still believe that most of it is unintentional, even now; I still catch myself doing it sometimes and I've been on the warpath against it for a decade; and (2) even if intentional, you can't prove it or fire someone for it.
1
0
10
Excellent episode! And an excuse for me to plug "Pre-registration: Why and How," a recent paper with Leif and @uri_sohn, in which we discuss many of the issues that Yoel and Alexa discuss here, though in undoubtedly a less fun way.
Episode 76 of Two Psychologists Four Beers is live! Check out “Preregistration (What is it Good For)” at
0
2
8
When Does Making Detailed Predictions Make Predictions Worse? My forthcoming paper with Theresa Kelly.
0
1
9
Universities and journal editors act solely in their own self-interest. If you want them to do the right thing you have to make it in their interest. The only way to do that is to go public (or threaten it) but that puts you at risk and it may not work. It’s the Wild West.
Eighteen months ago, I tried reporting him to his institution. A year ago, I tried asking editors for retractions. Today, he’s publishing faster than ever.
1
2
9
@lakens @donandrewmoore @PNASNews Doesn’t answer the question. What of consequence isn’t preregistered here? If you’re going to impugn these specific pre-regs as totally worthless you should be able to answer that. Where is the p-hacking that another template would’ve caught?.
1
0
8