Appraising ML prediction models @maartenvsmeden.bsky.social
1) Share model and code
2) Predictors and language
3) Class imbalance & calibration
4) Model validation sample size
5) Is the model clinically useful in practice?
www.jclinepi.com/article/S089...

Signal or noise? Evaluating commonly used attribution methods for explaining deep neural networks in electrocardiogram classification AbstractAims. Attribution-based explainability methods are widely used in electrocardiogram (ECG) analysis to interpret predictions from ‘black-box’ deep n

Fantastic new paper casting doubt on explainability of explainable AI. To explain complex machine learning algorithms you need reproducibility of the explanation at a minimum academic.oup.com/ehjdh/articl... #machinelearning #Statistics #StatsSky @maartenvsmeden.bsky.social

Happy to see this in print!

doi 10.1146/annurev-statistics-042324-123749
@maartenvsmeden.bsky.social @laurewynants.bsky.social @vanamsterdam.bsky.social and Ewout Steyerberg

Cluster analysis will always give you clusters

I have got the data
and I figured out exactly how to do my cluster analysis
all I need is a relevant question that my cluster analysis is going to answer

What if you combine open datasets with AI? Apparently, a 3 fold increase in low quality research papers, mass-produced by paper mills.

Interesting study in @jclinepi.bsky.social #academicsky #episky #medsky #Skystats

Thanks to @maartenvsmeden.bsky.social for initially posting this on Linkedin!

Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance Numerous measures have been proposed to illustrate the performance of predictive artificial intelligence (AI) models. Selecting appropriate performance measures is essential for predictive AI models i...

Our guidance regarding performance measures for medical AI models is finally out!

- Stop bashing AUROC, although it does not settle things
- Calibration and clinical utility are key
- Show risk distributions
- Classification statistics (e.g. F1) are improper

www.thelancet.com/journals/lan...

NEW PAPER

The use of explainable AI in healthcare evaluated using the well known Explain, Predict and Describe taxonomy by Galit Shmueli

link.springer.com/article/10.1...

🧐

this is one of my favourite observations about sample size calculations. (afaik first articulated by Miettinen in 1985)

Ha! I did not know I quoted Miettinen :). Thanks for the reference

For some research studies the optimal sample size should be estimated at 0

“Data available upon reasonable request” is academic language for you can get my data OVER MY DEAD BODY

I take version control very seriously

Manuscript_Final_Version_actualFINALcopy_version9b_USETHISONE.docx

Prediction models that are used to guide medical decisions are usually regulated under medical device regulation. This means, putting a calculator out there to promote the use your new prediction model is likely to break some rules.

The lasso works really well in particular settings and for particular purposes. If you are after high prediction performance alone and you have a rather large sample size, it can be an excellent choice indeed. But most analytical goals are not only about prediction

Kind reminder: data driven variable selection (e.g. forward/stepwise/univariable screening) makes things *worse* for most analytical goals

Vacancy — PhD position on AI methodology for prediction of patient outcomes using organoid models Are you passionate about bringing personalized medicine to the next level and make real impact in healthcare? Join our team and develop novel AI methodology to improve predictions of relevant patient ...

NEW FULLY FUNDED PHD POSITION

Looking for a motivated PhD candidate to join our team. Together with Danya Muilwijk, Jeffrey Beekman and I, you will explore opportunities and limitations of AI in the context of organoids

For more info and for applying 👉
www.careersatumcutrecht.com/vacancies/sc...

Interpretable "AI" is just a distraction from safe and useful "AI"

This is right tho. Let’s therefore call them sensitivity positive predictive value curves bsky.app/profile/laur...

Performance evaluation of predictive AI models to support medical decisions: Overview and guidance A myriad of measures to illustrate performance of predictive artificial intelligence (AI) models have been proposed in the literature. Selecting appropriate performance measures is essential for predi...

For details: arxiv.org/abs/2412.10288

No.

I wonder who those people are who come here dying to know what GenAI has done with some prompt you put in

If you think AI is cool, wait until you learn about regression analysis

TL;DR: Explainable AI models often don't do a good job explaining. They can be very useful for description. We should be really careful when using Explainable AI in clinical decision making, and even when judging face validity of AI models

Excellently led by @alcarriero.bsky.social

NEW PREPRINT

Explainable AI refers to an extremely popular group of approaches that aim to open "black box" AI models. But what can we see when we open the black AI box? We use Galit Shmueli's framework (to describe, predict or explain) to evaluate

arxiv.org/abs/2508.05753

Guidelines for Reporting Observational Research in Urology: The Importance of Clear Reference to Causality - PubMed Observational studies often dance around the issue of causality. We propose guidelines to ensure that papers refer to whether or not the study aim is to investigate causality, and suggest language to ...

This is, however, not clever or safe writing, it is a bad collective habit that needs to stop. Not by avoiding references to causality but by clear referencing to it

pubmed.ncbi.nlm.nih.gov/37286459/

The healthcare literature is filled with "risk factors". This word combination makes research findings sound important by implying causality, while avoiding direct claims of having identified causal associations that are easily critiqued.

And taking this analogy one step further: it gives genuine phone repair shops a bad name