Citation: Bruce R. Childers, Panos K. Chrysanthis (2017) Artifact Evaluation: Is it a Real Incentive?. International Conference on e-Science (RSS)
DOI (original publisher): 10.1109/escience.2017.79
Semantic Scholar (metadata): 10.1109/escience.2017.79
Sci-Hub (fulltext): 10.1109/escience.2017.79
Internet Archive Scholar (search for fulltext): Artifact Evaluation: Is it a Real Incentive?
Download: https://doi.org/10.1109/eScience.2017.79
Tagged: Computer Science (RSS) academia (RSS), academic software (RSS)

Summary

AE is correlated with higher citation count. We cannot establish causality, however.

Background

Artifact evaluation := optionally, after acceptance, authors can submit code and data artifacts to the journal which will be published alongside their article. The article bears a badge declaring the artifacts, and this badge is a small incentive to submit artifacts.
This studies ACM's implementation of AE
Author participation stable at 40% in 2011.
ACM's AE does not require open-source. This is makes quiets objections relating to IP rights, especially from industry labs.
Review is done independently from the program committee, to avoid putting more burden on the PCs.
Artifacts are evaluated by senior PhD students and post docs, who are more aware of the latest software methods and more often working with code.
Artifacts are evaluated after acceptance, so it means fewer artifacts to review.
Artifact evaluation does not influence paper acceptance.

AE does correlate with greater citation count.
- Can't say if causal.
- Citation count has many biases, but it is a proxy for research impact.
Google Scholar to get citation count.

This study shows correlation, not necessarily causation.
Nevertheless, if causation is true, that would incentivize AE.
A stronger incentive would be for grant-funding agencies to mandate or recommend for reproducibility engineering.