Recently, I got quite frustrated listening to a large group of economists discussing preregistration and pre analysis plans. So, I've channelled that into an argument for why they should be preregistering their work whenever performing confirmatory research
kdoroc.substack.com/p/arguing-wi...
Our Executive Committee expresses its support for @apsa.bsky.social in light of the proposed drastic cuts to research funding in the USA:
👉 Statement of Support: www.dvpw.de/informatione...
👉 APSA Statement on the Proposed Elimination of the SBE Directorate at the NSF: www.dvpw.de/fileadmin/do...
It would be interesting to know whether the tighter restrictions are based on an evaluation of past resubmissions that showed that resubmissions are usually not successful (which can be for multiple reasons, only one being quality).
I am undecided about the move. What would be your alternative?
I do not agree with this reasoning, this is not the way!
🔥🔥🔥🔥🔥Regine Riphahn "Unsere Forscher bekommen eine Lupe, wo man international mit Mikroskopen arbeitet. Unsere Forscher bekommen ein Fernglas, wo international Observatorien eingesetzt werden." 🫳🎤
Science is a cumulative and collaborative process that requires human connection and interaction. I find it interesting that the 'lone genius' idea still gets romanticised in academia.
making all the p-hackers look like rookies
The big problem is simple immorality:
Reviewers not being honest about their own incompetence (in some fields) and in most, being biased against ideas other than their own.
Vetoing in secret peer-review, with very few reviewers, can kill scientific advancement for generations never mind decades.
Wenn die Gesamtzahl der Ministerien unabhängig von der Sitzverteilung ist, dann nicht, oder? Bei einer ungeraden Anzahl kann das "Law" nicht aufgehen bei nahezu Stimmengleichheit. Ich würde aber bezweifeln, dass Unabhängigkeit vorliegt, weil Özdemir so der CDU entgegenkommt.
I see your point, but how much would you bet on this being known by empirical researchers? Asking this not because empirical researchers are stupid, but because it takes time, sometimes a lot of time, before "new" methods insights diffuse.
Fascinating. How is this set up? Is this based on Claude somehow? (of course, these days my first idea that came to mind)
#polisky share far and wide: @apsa.bsky.social Statement on the Proposed Elimination of the Social, Behavioral, and Economic Sciences Directorate at the National Science Foundation
apsanet.org/wp-content/u...
I was consulted recently by an editor of an interdisciplinary journal who was ready to retract a paper based on hallucinated citations. It’s a real research integrity problem, and it’s arriving in humanities and social sciences publishing. (It’s already been a problem in STEM.)
Late to the conversation, but I'm kind of unbothered by this lack of replication. If the study is not internally valid to begin with, then being able to replicate it only means replicating systematic biases. And something tells me this is likely to be a bigger problem in the literature.
The repo for the reproducibility paper includes three repro packages, each with a manuscript. I assume that this corresponds to the first and second submission and the accepted paper. If so, one could track differences now, but still not the same as being open during the process.
I may have missed it, but I have to say I was surprised how close the SCORE program was. The versions of record were the first papers that were circulated. No OSF preprints, no panels at conferences, nothing, right? If so, I find this approach hard to understand.
The real metascience nightmare will be when journals start using AI to require that authors actually followed their preregistration.
Personally, I can’t wait. So many people having to eat the silly “our preregistered study” claims slapped all over their papers regardless how much they deviated.
I agree it makes it complicated. There is an article behind the blog post that may address this. My understanding is that one does not want to publish too many false-positives (accepted low-quality papers, however defined), and that journal space may be too scarce to publish all split-review papers.
of controversial high-quality papers. It may work out like this, but it could also be that reviewers predisposed against controversial, innovative papers review them more negatively to prevent them from entering the lottery.
Does anyone know of a journal/venue that adopted this approach? 2/
Going through old posts: To ensure the quality of peer reviewed research introduce randomness
blogs.lse.ac.uk/impactofsoci...
One suggestion is to select papers with mixed reviews randomly for publication. This would lead to acceptance of low-quality papers, but potentially also fewer rejections 1/
Got it, thanks (quotes did not work on Deck Blue). Great term, will keep it in mind.
Batman breaking a firearm and saying, "This is the journal of the enemy. We do not need it. We will not use it."
how I feel when open science initiatives choose to publish in Nature
be that researchers would submit even more to second-tier journals, making them adopt constraints too, leading to a cascade. Which may be what some want who propose a research-bound submission/publication gap. I am still not convinced that such a constraint is the best way forward. 3/
A cap on submissions may raise the bar even more. The one try that you make must be really good, so more work is put into one paper. I don't think this would be a good development because interesting seconday insights may get lost in monograph-like papers.
Another effect of submission cap could 2/
What is meant by "better papers"? I am not convinced that published papers aren't good enough, judged by relevance and rigor.
A cap on publications in a top journal propably wouldn't change much because most researchers don't publish more than once/year in the same top journal.
1/
This is a cataclysm for social science in the United States. SBE helped to foster the field of Science and Technology Studies in the country (it supported the development of #STS training programs at UCSD and my alma mater, Cornell, in the 1990s).
Unfortunately, Vannevar Bush would love this part.
What is DOPRA?
I also started looking into one OSF repo out of curiosity and fully agree that it is more difficult to navigate than I thought. Maybe it is the best way given the size and duration of the project, but it is not userfriendly.
A screenshot showing: Introduction These are notes for my class on probability models. In these notes, I walk through the concepts and computation that support modern probability modeling in political science using both maximum likelihood and Bayesian approaches. The Goal There are many excellent books on probability models. But I felt the need to write my own. Why? I saw three problems. First, some classes assign a huge textbook. It might be possible for the strongest and most motivated students to become familiar with the range of topics covered in these textbook, but impossible to master. Instead, these textbooks seem like references, something you’re supposed to constantly be referring back to throughout your career. I know this because many of these books have instructors’ guides that suggest what should be covered in a single semester, what should be skipped, and how one might jump around. Instead, I want a book that students can work through beginning to end and master each idea. Second, some classes assign a variety of sections from several books and a collection of articles. But then the story told in the readings isn’t coherent. The styles are changing, the author’s tastes are changing, and the notation is changing. Switching among authors can feel like whiplash when learning a difficult subject. Instead, I want a book that tells a continuous story with consistent style, tastes, and notation. Third, some classes assign readings that support the lecture material, without exact alignment between the two. For better or worse, the content covered by the instructor in class feels like the most important material. Thus, I want a book that exactly aligns with the material I cover in class.
You guys @carlislerainey.bsky.social has a free textbook online and it seems really useful pos5747.github.io/notes/
In case you've seen results going around today about a paper on reproducibility in the social sciences in Nature and it seems like reactions differ, you could be forgiven for being confused because there are actually *FOUR* such studies all published the same day, with somewhat differing results.