Use as directed? A comparison of software tools intended to check rigor and transparency of published work The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison.

Eckmann et al* compare software tools for checking rigor and transparency in published work.

*incl. @aidybarnett.bsky.social @gcabanac.cpesr.fr @kaitlynhair.bsky.social @skmccann.bsky.social @martijnroelandse.dev @colbyvorland.bsky.social @tweissgerber.bsky.social

journals.plos.org/plosone/arti...

I’m working on a small, experimental tool that creates narrative, multi-signal overviews for scholarly books (citations, usage, attention).

Looking for a few (university) presses (and others) to test an MVP and give candid feedback.

Open data. No black boxes. DM me.

#scholarlypublishing #books

Imagine a future driven by your ideas! The STM Association invites fearless innovators to present their pitches. Don’t miss out on the chance to make your mark — register by next Tuesday, September 30!

eLife and SciScore Partner to Enhance Rigor and Reproducibility in Scientific Publishing By providing authors with tools to enhance their reporting standards, we collectively advance the reliability and trustworthiness of scientific literature.

1/ We’ve partnered with @sciscore.com to help bolster the reproducibility and transparency of research published in eLife. Authors will have access to SciScore's comprehensive rigour and reproducibility analysis at no extra cost. buff.ly/PSKpQrk

SciScore and bioRxiv Launch New Key Resource Tables for Preprints, Pioneering New Era of Reproducibility & Transparency Surfacing reagents and resources

SciScore and @biorxivpreprint.bsky.social Launch New Key Resource Tables for Preprints, Pioneering a New Era of Reproducibility and Transparency in Life Science Research world.einnews.com/article/7829... cc @chanzuckerberg.bsky.social

Addendum: here’s an example of an old post (complete with old Twitter handle tagged for @whysharksmatter.bsky.social)

bsky.app/profile/jero...

If interested; one method for porting your items: chromewebstore.google.com/detail/porto...

Hat tip: @martijnroelandse.dev

With a Chrome plugin called Porto it is dead easy, just follow the steps. Only generating the Twitter dump and uploading in BlueSky takes quite some time before it's ready but you don't (always) have to wait for that.

Sorry folks, I've imported my tweets using Porto here in BlueSky althou it looks like I've been posting 2k Skeets overnight.... 😅

Yeah, she's great, also instrumental in the Bookmetrix development back in the days! I've imported all my tweets into BlueSky hence the throwback time 😊

5 years SciScore - end of year YouTube video by martijnroelandse_dev

This year, SciScore celebrated five years of advancing research rigor and transparency. Many thanks for your trust and feedback—they helped shape SciScore into what it is today.

Here’s to another year of innovation, community, and advancing research rigor—one score at a time. youtu.be/XFSdsyWdL_g

Drop us a note if you want to be included as well! go.bsky.app/AaaV3mY #scholarlypublishing #tools

Of course, totally doing this to see how fast these citations are picked up by @altmetric.com (where I also learned how to tweak my handle 😎)

Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality Background: Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature; however, the assessment of measures of transparency tends to be v...

And here: doi.org/10.2196/37324

Assessing the Effectiveness of SciScore in Supporting the Reproducibility of Scientific Research - Science Editor Abstract It is critical for researchers and grantees alike to adhere to rigor and transparency criteria to ensure their contributions to scientific research are sufficiently transparent so they can be...

Some of the underlying data can be found in our recent papers here: doi.org/10.36591/SE-...

For those attending the #stmweek innovation fair, you can download the @sciscore.bsky.social handouts with the 'lessons learned' here: www.scicrunch.com/s/SciScore-V...
#researchintegrity #innovation

Sure was, I much enjoyed the conference!

@rrids.bsky.social just posted a new blog clarifying how to use RRIDs and RORs for facilities. If you wonder how each of these persistent identifiers can help you, please see www.rrids.org/news/2024/11...

Next week, I’ll be in London for the #STM Innovation & Integrity Days! I’m excited to showcase SciScore on behalf of SciCrunch Inc. and support the community as a proud sponsor of the #VesaliusInnovationAward through Park 56.

Vrijburg verwelkomt ds. Rosaliene Israël als nieuwe dominee per 1 januari 2025 -

mailchi.mp/ef031676b7f5/5…

After many months of work by the @SciCrunch team headed by @Anitabandrowski an all new version of SciScore has been released!

x.com/sciscore/statu…

"We're going to make biology programmable, that's the biggest opportunity for AI" says @JellePrins fromhe may have a point 😉
#tnwconference

cradle.bio

Anna Koivuniemi from @GoogleDeepMind on ChatGTP: We [Google] release when our product is ready and do this responsibly as these tools also impact society. That's more important than beating competition.
#tnwconference

Absolute pleasure attending #cziopenscience in Boston on @SciCrunch' behalf. Learned a lot, many thanks to the organisers!

x.com/cziscience/sta…

Next week I'll be at the 2024 Chan Zuckerberg Initiative Open Science Meeting in Boston, representing @SciScore. It is an interactive, invite-only event with attendees from universities, foundations and tech startups. CU there!

#CZIOpenSci @cziscience #rigor #reproducibility

Congrats to all the finalists! 🥳🥳
Looking forward to the pitches in December 🙌

x.com/KargerPublishe…

Research veracity/integrity: SciScore an AI tool for rigor and reproducibility - presented by Dr. Martijn Roelandse and Dr. Anita Bandrowski | DataSeer: checking Open Science - presented by Dr. Tim Vines | For reproducibility, we need the methods behind the data (an intro to protocols.io) - presented by Dr. Lenny Teytelman Reproducibility checklists are very useful things to remind us of the things that we should remember when publishing our results. They ar...

I'll be presenting @sciscore at AI Tool-Up Tuesdays! on 05 Sep

cassyni.com/events/9bkRZ9n…

Why AI writing detectors don’t work Can AI writing detectors be trusted? We dig into the theory behind them.

Interesting article (and examples!

arstechnica.com/information-te…

Woohoo, very pleased to see my next first-author paper (after 20 years) published!

x.com/CScienceEditor…

After having missed it in 2022 due to some virus (...) looking forward to this year's edition! If you'd like to chat and learn more about the services we offer at @park56nl, do drop me a line. CU there!

#tnwconference #tnw #networking #tnw2023

earlybird.thenextweb.com/tnw2023/share/…

Zoo, @krabbetv nieuwe voiture om de mensen naar het nieuwe huis te brengen? Très bien! Fijne uitzending trouwens, goed begin 👍🏻 #kopenzonderkijken