Register:
📍 Toronto (Apr 29, www.surveymonkey.ca/r/Second_AI_...)
📍 Cambridge (May 11, www.surveymonkey.ca/r/Second_AI_...)
📍 UCL (May 13, www.surveymonkey.ca/r/BGMTY3G)
📍 KCL (May 14, www.surveymonkey.ca/r/BGMF357)
We hope many of you will join us again 🙌
If you participated before—or are new and curious—this is a great time to get involved. Only one day of work and you get to coauthor our 2nd AI meta paper.
Who can participate?
Social scientists (econ, pol sci, psych)
-last year undergrad students
-master and phd students
-faculty
-researchers with a PhD
Why this matters:
The first paper gave us a foundation.
This new design lets us explore who benefits most from AI—and where.
Think of it as mapping the full “AI productivity gradient.”
This new round builds directly on the previous AI Games project—but with an important twist.
We now want to understand how AI affects performance:
• Across disciplines (horizontal)
• Across experience levels (vertical: undergrad → grad → faculty)
🚨 New AI Games round is live 🚨
Our first AI Games paper is now out at a top general interest journal (see preprint here:
www.econstor.eu/bitstream/10...
So… we’re launching the next stage 👇
Virtual Event April 16 // 1 pm ET NEW EVIDENCE ON REPRODUCIBILITY ACROSS SOCIAL AND BEHAVIORAL RESEARCH Moderator: Tim Errington Speakers: Katrin Auspurg, Abel Brodeur, and Andrew Tyner
What can large-scale studies tell us about reproducibility? In our webinar on April 16, researchers from COS, I4R, and META-REP will discuss findings from three papers—one from the recently published SCORE effort—and insights on reproducibility, transparency, and credibility
cos-io.zoom.us/webin...
This is how it started on my 50th birthday in Melbourne 😇
Photos show some of my 350+ co-authors 😍
A massive seven-year project exploring 3,900 social-science papers has ended with a disturbing finding
go.nature.com/4bZ9k0W
11/ Research briefing here: doi.org/10.1038/d415...
And a few other articles on reproducibility worth checking: Link to April 2nd issue: www.nature.com/nature/volum...
10/ Bottom line:
Reproducibility in social science is higher than often claimed, but robustness remains a key challenge.
More open data, more robustness, and more scrutiny can only help.
Link to article: www.nature.com/articles/s41...
Preprint: ideas.repec.org/p/zbw/i4rdps...
9/ This project is ongoing. We will shortly release an updated version with 250 papers reproduced. This follow-up project will test which methods and subfields are the most/least robust.
8/ Importantly, this is a best-case sample:
These are top journals with strong data & code policies.
So these results likely represent an upper bound on reproducibility.
7/ Additionally, six independent research teams examined 12 pre-specified hypotheses about determinants of robustness. Reproducers with more experience found lower levels of robustness, and robustness correlated with neither author characteristics nor data availability.
6/ We also find:
Coding errors in ~25% of papers. Major coding errors in about 10% of studies, ranging from duplicates to conducting a simple difference instead of a difference-in-differences.
5/ Effect sizes tell an interesting story:
Median effect ≈ 99% of original. This result suggests that robustness checks impact the standard errors rather than the magnitude of the coefficients.
4/ But reproducibility ≠ robustness.
When we re-analyze the same data using reasonable alternative choices:
72% of statistically significant results remain significant (same direction).
3/ First result: good news
85% of results are computationally reproducible
That means independent researchers can run the original code and recover the published findings in most cases.
2/ We reproduced 110 papers from top journals (2022–2023), all with mandatory data & code sharing.
Goal: test computational reproducibility and robustness at scale.
🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science.
Here’s what we found 👇
We are happy to announce the Best Article Award for 2025. This paper examines the impact of papers meant to help reassess previously published papers. These findings are important for our discipline to understand. Congrats to @jrgptrs.bsky.social, Nathan, and Florian! doi.org/10.1111/ecin...
We're thrilled to open registration for the Utrecht Replication Games. The event will be at the at the University of Utrecht on June 4th. Psych, public health, pol sci and econ studies will be reproduced!
Register here: www.surveymonkey.ca/r/Replicatio...
This blog post explores the idea that science might work better if papers functioned more like software: modular, versioned and continuously improved.
Check it out here: i4replication.org/what-will-th...
We have a new blogpost on What will the paper of the future look like?
What if research papers stopped being static PDFs and became closer to software?
The most downloaded paper on EconStor in Feb. 2026 was:
"Briggs, Ryan C.; Mellon, Jonathan; Arel-Bundock, Vincent (2026) : It must be very hard to publish null results, I4R Discussion Paper Series, No. 281, Institute for Replication (I4R), s.l." hdl.handle.net/10419/336819
@i4replication.bsky.social
Almost at the end of a very intense 3-day CBS Replication Games! Co-organised by @odissei.bsky.social and @i4replication.bsky.social, 19 researchers (6 teams) came together to replicate 6 papers using Dutch admin data from top economics journal
@fialalenka.bsky.social @jackfitzgerald.bsky.social
For now, the figure raises a simple question:
Is economics shifting away from formal, journal-based critique — even as concerns about credibility move center stage?
Full blog post here: i4replication.org/the-vanishin...
So is this a supply problem (fewer comments submitted)?
Or a demand problem (fewer accepted)?
We don’t know.
Answering that would require:
• Submission & acceptance data
• Referee reports
• A survey of economists on incentives to write comments
Why?
In their paper, surveyed AER editors did not report an explicit policy change on comments.
Possible explanations they mentioned:
• Comments are tedious to referee
• Tenure incentives
• Prefer writing new papers
• Longer papers leave less room to critique
What makes this striking:
Concerns about credibility & reproducibility have increased.
And the AER itself published influential evidence on p-hacking & publication bias (Andrews & Kasy, 2019; Brodeur et al. 2020, 2023).
Yet formal comments keep declining.