New Changes to Badging Terminology

The ACM Publications Board recently approved a change to the terminology used for its artifact review badges to ensure that our current definitions of “reproducibility” and “replication” are consistent with definitions used by other research communities outside the field of Computing. Following discussions with the National Information Standards Organization (NISO), it was recommended that ACM harmonize its terminology and definitions with those used in the broader scientific research community, and as a result the ACM Publications Board voted to interchange the definitions of “Results Replicated” and “Results Reproduced” to adopt the NISO standard.

In the coming days, we will be updating the Artifact Review and Badging web page and providing revised badges that should be used for all artifacts judged to meet the revised standards of reproducibility or replication. ACM will also be making corrections to existing papers in the Digital Library that have replication or reproducibility badges. ACM will notify all authors of affected papers/artifacts.

The changes being implemented are relatively minor and involve swapping the terms “reproducibility” and “replication” with the existing definitions used by ACM as part of its artifact review and badging initiative.

As updated, the definitions are:

  • Results Replicated (Different team, same experimental setup)

    The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts.

  • Results Reproduced (Different team, different experimental setup)

    The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

All other definitions remain the same.

Lastly, there are many ongoing efforts in the computing research community on reproducibility and replication of scientific results. The ACM Publications Board wishes to thank the many volunteers who have been involved in these efforts. Everyone has been gratified to see these efforts expand and grow. Indeed, other organizations are adopting the badges that ACM volunteers developed and are using them to indicate works that have been independently reproduced or replicated.

Publish with ACM

ACM's prestigious conferences and journals are seeking top-quality papers in all areas of computing and IT. It is now easier than ever to find the most appropriate venue for your research and publish with ACM.

Publish your work

Why I Belong to ACM

Hear from Bryan Cantrill, vice president of engineering at Joyent, Ben Fried chief information officer at Google, and Theo Schlossnagle, OmniTI founder on why they are members of ACM.