A couple of years ago, I stream-of-consciously wrote this guide for how I structure academic papers. We discussed writing in my graduate class today, and I remembered it. Here is the link below if it might be helpful (or give you a different perspective)!
billchopik.wordpress.com/wp-content/u...
We have a new preprint that underscores some key claims here: even if one *can* design an agent that gets through a survey fine, it doesn't follow that such agents are undetectable or common. We find that they are far from common! Preprint link in thread👇
**At my funeral**
Student: "Will this be on the exam?"
A guy asking ChatGPT to review a series of fart sound effects and getting a serious kiss ass response that calls it atmospheric
I can't stop laughing at this post. It's perfect.
imo, an underappreciated threat of LLMs to social science research is data faking. it would be very easy for someone to claim their survey/task samples online volunteer participants when in reality the researcher just prompts gpt to complete the task 200 times and confirm their hypothesis.
New preprint! w/Tessa Charlesworth & @williambrady.bsky.social:
The Psychology of Algorithmic Bias
We introduce a psychology-centered framework to specify mechanisms through which human behavior interacts dynamically with AI systems to produce algorithmic bias.
osf.io/preprints/psyarxiv/rxu37_v1
New preprint out today (osf.io/preprints/ps...). We tested whether AI agents are actually infiltrating online surveys.
Spoiler alert: they aren't
Thread 🧵
[1/9]
The p-curve would be a great name for a finisher
Scroll all the way down. ❤️
Queen's University in Kingston is hiring a Queen's University National Scholar (QNS) Senior Faculty Position in Social and/or Affective Neuroscience, Department of Psychology
Closing date: 2026/01/30
CPA Career Ads and Resources page: cpa.ca/careers/
My dept (Queen's Psych) is hiring! We are searching for an Associate or Full Prof in Social/Affective Neuroscience to begin July 1 2026. Applications due Jan 30. Full job ad in link below. Feel free to send to anyone you think might be interested!
www.queensu.ca/psychology/s...
The Sora AI disinfo nightmare is here
For more like this:
tiktok.com/@drewharwell
instagram.com/bydrewharwell
When I was a younger faculty, I kept wavering on whether I should apply for the Rising Star every year. And then when I finally worked up the courage to apply, I was no longer eligible. Lesson here is that you should not miss your shot and just apply! Don’t pray for someone to secretly nominate you!
Excited to share that I’ll be the incoming Editor of AMPPS. My first priority is building a diverse team of Associate Editors and Editorial Board members. If you’re interested, DM me or add your name via this super simple survey.
cornell.ca1.qualtrics.com/jfe/form/SV_...
Please share!
Can large language models stand in for human participants?
Many social scientists seem to think so, and are already using "silicon samples" in research.
One problem: depending on the analytic decisions made, you can basically get these samples to show any effect you want.
THREAD 🧵
Summary of design and results from our three studies. (A: Design) Each study used a similar experimental design, measuring both positive and negative demand in an online experiment, with three commonly-used task types (dictator game, vignette, intervention). Our experiments had ns ≈ 250 per cell. (B: Results) Observed demand effects were statistically indistinguishable from zero. The plot shows means and 95% confidence intervals for standardized mean differences derived from frequentist analyses of each experiment and an inverse variance-weighted fixed-effect estimator pooling all experiments (solid bars). Prior measurements of experimenter demand from a previous dictator game experiment (de Quidt et al., 2018; standardized mean difference from regression coefficient) and a meta-analysis primarily including small-sample, in-person studies (Coles et al., 2025; Hedge’s g statistic) are also shown for comparison (striped bars). The main text includes Bayesian analyses that quantify our uncertainty.
We often hear from reviewers: "what about demand effects?" So we developed a method to eliminate them. Something weird happened during testing: We couldn’t detect demand effects in the first place! (1/8)
I’ve noticed this too. If you are a Qualtrics user, you can discourage this by disabling copy+paste. Details below
bsky.app/profile/dave...
instructors! today is a good day to ask yourself this question again
In a new paper, my colleagues and I set out to demonstrate how method biases can create spurious findings in relationship science, by using a seemingly meaningless scale (e.g., "My relationship has very good Saturn") to predict relationship outcomes. journals.sagepub.com/doi/10.1177/...
Kingston folks, cute lemonade stand alert now (Mon Aug 18) on Napier between Brock and Johnson. Kids raising money for refugees. Show your support!
Are there reasons this would be a bad idea? Rather than tell me, give me the time to figure them out on my own, it’s the best way for me to learn
Caveats:
-must disable "new survey taking experience"
-survey layout must be flat, modern, or classic
-you should probably tell Ps you are disabling copy & paste just in case they are planning on drafting their responses in a word doc and pasting it in later
To prevent Ps from pasting (GPT-generated) responses to your text entry Qualtrics question, disable pasting.
Add this code to the OnReady section of your question's javascript:
jQuery("#"+this.questionId+" .InputText").on("cut copy paste",function(e) {
e.preventDefault();
});
Enjoy!
ChatGPT shows signs of the same biases that arise in audit studies of human beings.
When you give ChatGPT resumes, it's biased in how it evaluates minorities.
When you ask ChatGPT to generate resumes for women & minorities, it generates systematically different types of resumes
Cover page for the manuscript: Morey, R. D., & Davis-Stober, C. P. (2025). On the poor statistical properties of the P-curve meta-analytic procedure. Journal of the American Statistical Association, 1–19. https://doi.org/10.1080/01621459.2025.2544397
Abstract for the paper: The P-curve (Simonsohn, Nelson, & Simmons, 2014; Simonsohn, Simmons, & Nelson, 2015) is a widely-used suite of meta-analytic tests advertised for detecting problems in sets of studies. They are based on nonparametric combinations of p values (e.g., Marden, 1985) across significant (p < .05) studies and are variously claimed to detect “evidential value”, “lack of evidential value”, and “left skew” in p values. We show that these tests do not have the properties ascribed to them. Moreover, they fail basic desiderata for tests, including admissibility and monotonicity. In light of these serious problems, we recommend against the use of the P-curve tests.
Paper drop, for anyone interested in #metascience, #statistics, or #metaanalysis! @clintin.bsky.social and I show in a new paper in JASA that the P-curve, a popular forensic meta-analysis method, has deeply undesirable statistical properties. www.tandfonline.com/doi/full/10.... 1/?
FIGURE 7: LIFECYCLE PROFILE: FULL-TIME EMPLOYMENT AMONG PHD GRADUATES
FIGURE A3: SHARE OF PHDS WORKING AS PROFESSORS
FIGURE A5: TOTAL INCOME OVER THE LIFE CYCLE, BY HIGHEST DEGREE EARNED
TABLE B1: DESCRIPTIVE STATISTICS ON PHD COHORT: EARNINGS DURING THE PHD (BY GENDER, CITIZENSHIP, AND FIELD)
Is a #PhD worth it?
In data from #Canada, doctoral grads earned less at first because (A) they entered the job market later, but they surpassed others if they (B) got academic jobs and (B) kept them.
So a PhD's value is waning as B and C become harder.
econpapers.repec.org...
We’re hiring a tenure track assistant professor in our amazing Vassar cognitive science department! If you have any questions, let me know. I’ll be at Cog Sci in San Francisco all week if you’d like to chat in person
The word “Research” is doing way too much work. We need separate words for “creating new verifiable knowledge” and “looking shit up on the internet”
Lack Of Concrete Dinner Plans Leaves Power Vacuum Filled By Radical Pro-Tapas Fanatics
Lack Of Concrete Dinner Plans Leaves Power Vacuum Filled By Radical Pro-Tapas Fanatics theonion.com/lack-of...
