Two conferences on science quality this week
Worth looking through the tags for the International Research Integrity Conference #IRICSydney and the Association for Interdisciplinary Meta-research & Open Science Conference #AIMOS2025
Great and detailed live posting from bsky.app/profile/elis... from #IRICSydney the International Research Integrity Conference, 16-18 November 2025, Sydney, Australia
#ResearchIntegrity #ResearchMisconduct
#PublicationEthics
Thank you, @simongandevia.bsky.social, David Vaux, @jabyrnesci.bsky.social and Bronwyn Chapman, for organizing a great conference with passionate speakers and great discussions.
It was perfectly run on time - I also loved Simon's bell, Bronwyn's amazing organization, and the pastries.
#IRICSydney
View from the rooftop of the museum. In the foreground, a very large bent spoon (artpiece by Ricky Swallow). Harbour Bridge and Opera House in the distance. Blue sky, views of water, some white clouds.
Same view as the first photo, but now taken at night. You still see the bent spoon in the foreground, and the bridge on the left, opera house on the right, but now with dark sky and city lights.
Here are some photos from last night's conference dinner at the Museum of Contemporary Art in Sydney, with amazing views of the Harbour Bridge and the Opera House.
In the foreground: part of the Bent Forms #1–#4 exhibit by Ricky Swallow.
#IRICSydney
Simon Gandevia @simongandevia.bsky.social thanks Bronwyn Chapman
We have come to the end of this conference.
Simon Gandevia @simongandevia.bsky.social thanks Bronwyn Chapman for all her fantastic help with the organization of this conference. He promises: We will be back in two years!
EB: Here ends my reporting of the IRIC. Hope y'all liked it.
#IRICSydney
* Legislation *is* possible, e.g social media ban for kids, if the public broadly supports it.
* We have only seen anecdotal evidence - no proof of a structural problem, need to do research to know the numbers.
* Need a clear simple point to convey.
#IRICSydney
* People know what they should be doing - how effective is education? Can we reward research integrity with incentives (instead of only punishing fraud)?
* Reward scientists for mentoring, not only focus on publications.
#IRICSydney
* Personal communications are very important; knowing whom to talk to, but also choosing the right moment - maybe right after a breaking story.
* How will a future ORI relate to current structures?
(the word "stakeholders" is mentioned a lot - cheers! in my virtual drinking game)
#IRICSydney
The three moderators of the discussion
Slide with lots of text - What change do we want? Behaviorial goal, barriers and enablers
Discussion:
What is needed – the Sydney Statement
Led by Annalese Bolton (Behavioural Economics Team of the Australian Government), Jenny Byrne (University of Sydney) and David Vaux (Wehi)
We don't want a document that goes nowhere - it should really make a statement, be practical.
#IRICSydney
Table showing ORI Findings Over Time
IO: The US Office of Research Integrity typically releases ~10 cases per year, but only 2 cases this year so far.
The NSF research integrity office has subpoena power, which gives them much more teeth.
These are some pressure points. What will *you* do?
#IRICSydney
IO: Very few cases of criminal sanctions for scientific misconduct (Macchiarini).
Case Study: pubmed.ncbi.nlm.nih.gov/25541134/
Tragedy might lead to actual action.
#IRICSydney
IO: More penalties for misconduct: People can lose grants, jobs. False Claims Act: whistleblowers can get rewards if they find evidence of false claims to the US government.
FTC won case against OMICS (but never got the money) for false advertising.
#IRICSydney
IO: Pressure points are individual reputation damage, financial pressure, career damage, civil liability, criminal charges.
Scientists are not penalized if they have a retraction.
Commercial pressure: Wiley's stock price dropped with the Hindawi affair.
www.nature.com/articles/sre...
#IRICSydney
IO: Most institutions are just as flawed as most people. Reaching across the aisle to leverage shared interest. Strategy is important.
The Center for Scientific Integrity includes several projects, including Retraction Watch, Medical Evidence Project.
centerforscientificintegrity.org
#IRICSydney
IO: We need to acknowledge that if you are inside a system, that system will enforce itself.
But let's talk about the Theory of Change.
Anger might result in focus, or are we yelling at whoever answers the phone?
Retraction Watch does not have the authority to retract papers.
#IRICSydney
Title slide: What to do next. Ivan "Pure Evil"* Oransky Co-founder Retraction Watch (social media handles) "99.4% evil, 0.6% Zinfandel, but the (faked) error bars are quite large
The speaker
After the tea break, we continue with Session 7: 'Publication problems', co-chaired by @jabyrnesci.bsky.social and David Vaux.
Ivan Oransky @retractionwatch.com will talk about 'Summing up: What to do next?'
If a headline is a question, the answer is no!
#IRICSydney
EV: Rejection rate now 57% (48% desk reject by RI team). We use an AI model that can recognize paper mill papers based on 15 data points (red flags/trust markers), trained on a 64K dataset.
Benchmarking AIRA with other tools - but not much overlap. Still learning from this.
#IRICSydney
Speaker and title slide
Elena Vicario @frontiersin.bsky.social - we now have 65 team members in our research integrity team.
We check every submitted article and developed an in-house tool, AIRA. It flags things, but a human takes the decision. We also look at patterns in submissions and peer reviewers.
#IRICSydney
Speaker and title slide
Yan Peng with 'Using artificial intelligence to assess the trustworthiness of RCTs' - human assessment is slow. Can AI help?
PDF-> LLM extracts data -> puts them into CSV tables.
We are working on automatic integrity reports and send emails to journal editors.
AI can help to check data.
#IRICSydney
Sports ethics oversight / testing agency slide
MH: Cycling has a watchdog, but after more pressure, an independent testing agency was started.
The ITA is non-profit, independent, active - should be a great example for how to set up science integrity testing agency.
We can learn a lot from systems that work!
#IRICSydney
Speaker
Mark Hooper @markhooper.bsky.social: Science and professional cycling share a problem: cheaters.
They make the workplace unsafe, they are incentivised by the system.
But doping in cycling has been tackled more than 20 years ago. No laws were broken but it ruined the sport.
#IRICSydney
PYN: Out of 388 outcomes, about half had a discrepant outcome, but only a third included a rational for the change.
ITS are evidence generators, serve as sources for RCTs/systematic reviews
#IRICSydney
Speaker and one of her slides
Phi-Yen (Phoebe) Nguyen, with 'An investigation of selective outcome reporting in interrupted time series studies: a methodological study'
ITS: no need for randomizations, compare before/after intervention. We investigate selective reporting of outcomes.
#IRICSydney
VM @vincentmourik.bsky.social - In Feb 2025, Microsoft unveiled the world's first quantum processor - linking to a paper that was not related. Note that the company has gotten public funding for this research.
We should keep on debunking, hashing raw data, novel publishing models.
#IRICSydney
Speaker with title slide
Vincent Mourik with 'Fake it till you make it: Microsoft’s topological qubit claims' - there is also fraud in physics!
This new quantum computing chip is based on Majorana fermion - based on a now-retracted paper. Some data was spliced out/left out. Similar problems in other groups.
#IRICSydney
IH: Sometimes studies including in meta-analysis/systematic reviews are not even randomized clinical trials!
We randomly selected 100 systematic reviews in orthopedics - and looked at their 917 included RCTs. 32% of SRs included non-RCTs.
#IRICSydney
The speaker at the lectern
Title slide
After the lunch break, we will have several short presentations, chaired by Annie Butler
First, Ian Harris with 'The varying use of randomisation in “randomised” trials included in systematic reviews in orthopaedic surgery'
Lots of clinical studies are done very poorly.
#IRICSydney
* Why do reporters write about pseudoscience and conspiracy theories?
* It is very difficult to debunk those stories
* Ten minutes of talking to a genAI which takes an opposite view can start to change your mind - this is scary!
#IRICSydney
Moderator and the two speakers near the lectern, laughing
A new discussion session, 'What is needed?'
Discussion led by Michael Green:
* We are not against the p-value - it should just slowly left to wither.
* Discussion about how to make science 'sexy' - a word that might be controversial.
* If you use 'significant' - know what it means.
#IRICSydney
Speaker and slide with title "Way forward?" Text summarized in my post
JR: Way forward?
* Institutional support (publishers coud hire journalists; independent science integrity watchdog; protect whistleblowers)
* Tools and training (PubPeer, German Scence Media Centre)
* Fix science? That one is yours (meaning the audience), sorry!
#IRICSydney
