Artifact Evaluation: Is it a Real Incentive?

From AcaWiki
Jump to: navigation, search

Citation: Bruce R. Childers, Panos K. Chrysanthis (2017) Artifact Evaluation: Is it a Real Incentive?. International Conference on e-Science (RSS)
DOI (original publisher): 10.1109/escience.2017.79
Semantic Scholar (metadata): 10.1109/escience.2017.79
Sci-Hub (fulltext): 10.1109/escience.2017.79
Internet Archive Scholar (search for fulltext): Artifact Evaluation: Is it a Real Incentive?
Tagged: Computer Science (RSS) academia (RSS), academic software (RSS)


AE is correlated with higher citation count. We cannot establish causality, however.


  • Artifact evaluation := optionally, after acceptance, authors can submit code and data artifacts to the journal which will be published alongside their article. The article bears a badge declaring the artifacts, and this badge is a small incentive to submit artifacts.
  • This studies ACM's implementation of AE
  • Author participation stable at 40% in 2011.
  • ACM's AE does not require open-source. This is makes quiets objections relating to IP rights, especially from industry labs.
  • Review is done independently from the program committee, to avoid putting more burden on the PCs.
  • Artifacts are evaluated by senior PhD students and post docs, who are more aware of the latest software methods and more often working with code.
  • Artifacts are evaluated after acceptance, so it means fewer artifacts to review.
  • Artifact evaluation does not influence paper acceptance.


  • AE does correlate with greater citation count.
    • Can't say if causal.
    • Citation count has many biases, but it is a proxy for research impact.
  • Google Scholar to get citation count.


  • This study shows correlation, not necessarily causation.
  • Nevertheless, if causation is true, that would incentivize AE.
  • A stronger incentive would be for grant-funding agencies to mandate or recommend for reproducibility engineering.