New Changes to Badging Terminology

Document last revised August 24, 2020

The ACM Publications Board recently approved a change to the terminology used for its artifact review badges to ensure that our current definitions of “reproducibility” and “replication” are consistent with definitions used by other research communities outside the field of Computing. Following discussions with the National Information Standards Organization (NISO), it was recommended that ACM harmonize its terminology and definitions with those used in the broader scientific research community, and as a result the ACM Publications Board voted to interchange the definitions of “Results Replicated” and “Results Reproduced” to adopt the NISO standard.

ACM will be implementing a versioning system for artifact badging definitions. Badges issued until May 14, 2020 will be considered Version 1.0. Please visit the Badging Version 1.0 webpage for complete description and badge definitions. Badges issued after May 15, 2020 will be badged under Version 1.1 with the version noted on the badge awarded. Please visit the Badging Version 1.1 webpage for complete description and current badge definitions. New versions will be identified on the badge itself and documented by updating the version pages. We will be updating citation pages for all badged items in the DL to reflect the badging history with links to version definitions. We will not make any changes to the originally badged documents. 

The changes being implemented are relatively minor and involve swapping the terms “reproducibility” and “replication” with the existing definitions used by ACM as part of its artifact review and badging initiative.

As updated, the definitions are:

  • Results Reproduced ( Different team, same experimental setup)

The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

  • Results Replicated ( Different team, different experimental setup)

The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts.

All other definitions remain the same.

Lastly, there are many ongoing efforts in the computing research community on reproducibility and replication of scientific results. The ACM Publications Board wishes to thank the many volunteers who have been involved in these efforts. Everyone has been gratified to see these efforts expand and grow. Indeed, other organizations are adopting the badges that ACM volunteers developed and are using them to indicate works that have been independently reproduced or replicated.

ACM Case Studies

Written by leading domain experts for software engineers, ACM Case Studies provide an in-depth look at how software teams overcome specific challenges by implementing new technologies, adopting new practices, or a combination of both. Often through first-hand accounts, these pieces explore what the challenges were, the tools and techniques that were used to combat them, and the solution that was achieved.

The DevOps Phenomenon

ACM Queue’s “Research for Practice” serves up expert-curated guides to the best of computing research, and relates these breakthroughs to the challenges that software engineers face every day. This installment, “The DevOps Phenomenon” by Anna Wiedemann, Nicole Forsgren, Manuel Wiesche, Heiko Gewald and Helmut Krcmar, gives an overview of stories from across the industry about software organizations overcoming early hurdles of adopting DevOps practices, and coming out on the other side with tighter integration between software and operations teams, faster delivery times for new software features, and achieving higher levels of stability.

Why I Belong to ACM

Hear from Bryan Cantrill, vice president of engineering at Joyent, Ben Fried chief information officer at Google, and Theo Schlossnagle, OmniTI founder on why they are members of ACM.