On the performance of the Neyman Allocation with small pilots The Neyman Allocation is used in many papers on experimental design, which typically assume that researchers have access to large pilot studies. This …

#statstab #489 On the performance of the Neyman Allocation with small pilots

Thoughts: If you know your treatment condition will have larger variance you can optimise your sample size.

#nhst #samplesize #neynan #heterogeneity #welch #variance #pilot #se

www.sciencedirect.com/science/arti...

More than meets the ITT: A guide for anticipating and investigating nonsignificant results in survey experiments | Journal of Experimental Political Science | Cambridge Core More than meets the ITT: A guide for anticipating and investigating nonsignificant results in survey experiments - Volume 12 Issue 1

#statstab #487 More than meets the ITT: A guide for anticipating and investigating nonsignificant results in survey experiments

Thoughts: I see a lot of papers that make at least one of the 7 errors for "no effect".

#survey #nhst #nulleffects #nonsignificant #pvalue #power

doi.org/10.1017/XPS....

18  Experimental : causal – Research Design in the Social Sciences

#statstab #476 Experimental : causal

Thoughts: Randomized experiments are the gold standard for inference for a reason. But they are hard to design.

#design #r #statistics #methods #experiment #tutorial #pedagogy #education #hypothesis #nhst #causal #ancova

book.declaredesign.org/library/expe...

Redirecting

#statstab #467 Replication, statistical consistency, and publication bias

Thoughts: Is the replication of a a finding the "gold standard" for scientific discovery? Maybe not.

#replication #metascience #metapsychology #statistics #bias #QRPs #nhst

doi.org/10.1016/j.jm...

#statstab #463 One-and-a-half sided test

Thoughts: Alberson has a more different take on the issues from #461 and #462: human behaviour.

(excerpt from the book)

#pvalue #onesided #NHST #directional #hypothesis #testing #logic

www.routledge.com/Statistics-A...

The paradox of one-sided vs. two-sided tests of significance | OneSided.org A lot of people find as paradoxical the claim that a one-sided test of significance at a given p-value offers the same type I error guarantees as a two-sided test that produced the same p-value. Here ...

#statstab #462 The paradox of one-sided vs. two-sided tests of significance

Thoughts: A solution to Royall's paradox from #461. The "null" is not one thing.

#pvalue #Royall #paradox #onesided #nhst #null #hypothesis #logic

www.onesided.org/articles/the...

There is still only one test In 2011 I wrote an article called "There is Only One Test" , where I explained that all hypothesis tests are based on the same framework, wh...

#statstab #458 There is still only one test

Thoughts: Downey make a good point that all/any testing ensemble does essentially the same thing.

#randomization #testing #nhst #pvalue

allendowney.blogspot.com/2016/06/ther...

Significance tests, p-values, and falsificationism This thread takes its inspiration from the recent discussions in social science and statistics about significance tests, what they’re good for, whether p-values should be banned, and what all of that ...

#statstab #469 Significance tests, p-values, and falsificationism

Thoughts: An "interesting" thread of philosophy of science and debating.

#discussion #debate #pvalue #falsification #nhst

discourse.datamethods.org/t/significan...

#statstab #468 Why Summaries of Research on Psychological Theories are Often Uninterpretable

Thoughts: psychology uses significance testing backwards relative to hard sciences.

#meehl #theory #criticism #falsification #popper #methods #nhst #psychology

scienceplusplus.org/metascience/...

Significance tests, p-values, and falsificationism This thread takes its inspiration from the recent discussions in social science and statistics about significance tests, what they’re good for, whether p-values should be banned, and what all of that ...

#statstab #449 Significance tests, p-values, and falsificationism

Thoughts: A statistician and a philosopher debate p-values (not the setup to a joke). Good thread.

#pvalues #significance #nhst #fisher #greenland #epistemology #statistics

discourse.datamethods.org/t/significan...

Original post on fediscience.org

In an hour and a half (12:30 UTC / 14:30 CET) I'll give a lightning talk at the online BTS conference about the tool we are putting together for standardizing hypotheses.

Free registration: https://bigteamscienceconference.github.io/
Tool: https://tinyurl.com/hypothesizer
Slides […]

A Vast Graveyard of Undead Theories - Christopher J. Ferguson, Moritz Heene, 2012 Publication bias remains a controversial issue in psychological science. The tendency of psychological science to avoid publishing null results produces a situa...

#statstab #429 A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null

Thoughts: Ignoring null findings keeps bad theories alive.

#nulleffects #nhst #falsification #theory #critique #reproducibility #pvalue

journals.sagepub.com/doi/full/10....

Screenshot of bibliography. Open link in post for full (and updated) text.

We're working on a tool for standardizing hypothesis formulation. We've put together a bibliography of previous work on the topic. Are we missing any important papers?

Feel free to edit or comment […]

[Original post on fediscience.org]

#statstab #413 Counternull Sets in Randomized Experiments

Thoughts: The counternull is a lost statistic, that is woefully underused when teaching stats.

#counternull #nhst #pvalue #effectsize #teaching #randomization #rosenthal

www.tandfonline.com/doi/pdf/10.1...

The National HIV Story Trust has an exciting opportunity for someone to join the HIVstory project based at #TheLondonArchives on a paid archives placement.

Full details on the role and how to apply are below 👇

#NHST #HIVstory #Archives

www.nhst.org.uk/archive-plac...

A big thank you to @jisc.bsky.social, @britishlibrary.bsky.social and The National Archives for such a fantastic conference. It was the perfect space to promote the work of the #NHST team, their project, and their incredible collection held by us at TLA. Visit us and take a look yourselves!
🧵 4/4

#NHST have deposited 103 full-length recordings with #TheLondonArchives, documenting the lived experience of the epidemic, including people living with HIV, carers, partners, relatives, friends, and healthcare practitioners. As part of the #PositiveHistory Project, these can be viewed at TLA.
🧵 2/4

We made a trip to @durham-university.bsky.social for #DCDC25. Among the many panels was an incredible keynote speech by the co-founders of the National HIV Story Trust, set up to film, record and preserve the history of those affected by the HIV/AIDS epidemic of the 1980s and 90s. #NHST 🧵 1/4

Unfortunately, when we do null hypothesis significance testing #NHST with #p-values and the like, which is still the default approach in most disciplines, we do not know whether a lack of statistical significance means no true effect, or there is simply not enough evidence. 2/3

When to use Fisher versus Neyman-Pearson framework? I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections, when ...

#statstab #384 When to use Fisher versus Neyman-Pearson framework?

Thoughts: 13y old post, still a good read today. Uni season is almost upon us, so it's good to learn this stuff.

#NHST #pvalues #RAFisher #NeymanPearson #Fisher #forum
#statsexchange

stats.stackexchange.com/questions/23...

A Pragmatic Approach to Statistical Testing and Estimation (PASTE) The p-value has dominated research in education and related fields and a statistically non-significant p-value is quite commonly interpreted as ‘confirming’ the null hypothesis (H0) of ‘equivalence’. ...

#statstab #359 A Pragmatic Approach to Statistical Testing and Estimation (PASTE)

Thought: A (basic) guide to some alternatives to p-values: bayesian posterior intervals, Bayes Factors, and AIC.

#NHST #pvalues #TOST #BayesFactor #AIC #modelcomparison

doi.org/10.1016/j.hp...

The Abuse of Power It is well known that statistical power calculations can be valuable in planning an experiment. There is also a large literature advocating that power calculations be made whenever one performs a s...

#statstab #353 The Abuse of Power; The Pervasive Fallacy of Power Calculations for Data Analysis

Thoughts: An seminal paper on "post hoc" power calculations.

#power #QRPs #NHST #posthoc #samplesize #effectsize

www.tandfonline.com/doi/abs/10.1...

Frontiers | Confidence intervals and tests are two sides of the same research question A commentary on The new statistics: why and how by Cumming, G. (2014). The new statistics: why and how. Psychol. Sci. 25(1), 7-29. doi: 10.1177/0956797613504...

#statstab #337 Confidence intervals and tests are two sides of the same research question

Thoughts: Comment describing the connection between NHST p-values/test and Confidence Intervals (CI).

#NHST #ConfidenceIntervals #pvalues #frequentist #estimation

doi.org/10.3389/fpsy...

#statstab #335 Bayesian New Statistics

Thoughts: An influential paper with a great overview of different approaches to research.

#bayesian #nhst #nhbt #estimation #testing #frequentist
link.springer.com/content/pdf/...

#statstab #327 Effect Sizes for Equivalence Testing: Incorporating the Equivalence Interval

Thoughts: Not sure what value this approach has yet, but sharing for feedback.

#equivalencetests #sesoi #nhst #noeffect #nonsignificant #practicalsignificance

osf.io/preprints/ps...

Mark Rubin - One-Sided Tests In this paper (Rubin, 2022), I make two related points: (1) researchers should halve two-sided p values if they use them to make directional claims, and (2) researchers should not halve their alpha le...

#statstab #314 One-sided significance tests

Thoughts: Directional tests are more complex than would initially seem. Here is one view on their proper use.

#onetail #directional #NHST #significance

sites.google.com/site/markrub...

Improving Your Statistical Inferences - 5  Asking Statistical Questions This open educational resource contains information to improve statistical inferences, design better experiments, and report scientific research more transparently.

#statstab #310 Directional (One-Sided) versus Non-Directional (Two-Sided) Tests

Thoughts: This topic is more nuanced than just power, but useful to know that you save on sample size *if* you're right!

#hypothesis #nhst #power #onetail #directionaltest

lakens.github.io/statistical_...

The file drawer problem in social science survey experiments | PNAS The file drawer problem—often operationalized in terms of statistically significant results being published and statistically insignificant not bei...

Good news. The file drawer problem, of insignificant results not getting written up and submitted, is shrinking. #openscience #reproducibility #NHST #metascience 🧪 www.pnas.org/doi/full/10....

#statstab #295 The Fallacy of the Null-Hypothesis Significance Test

Thoughts: "the [..] aim of a scientific experiment is not to precipitate decisions, but to make an appropriate adjustment in the degree to which one accepts, or believes, the hypothesis"

#NHST #Bayes

stats.org.uk/statistical-...

Sage Journals: Discover world-class research Subscription and open access journals from Sage, the world's leading independent academic publisher.

#statstab #290 An Evaluation of Four Solutions to the Forking Paths Problem

Thoughts: Multiverse analysis & the forking paths problem must be considered methodologically but also logically: based on one' stat philosophy.

#metascience #forkingpaths #NHST

doi.org/10.1037/gpr0...