Artifact Evaluation: Is it a Real Incentive?
Citation: Bruce R. Childers, Panos K. Chrysanthis (2017) Artifact Evaluation: Is it a Real Incentive?. International Conference on e-Science (RSS)
DOI (original publisher): 10.1109/escience.2017.79
Semantic Scholar (metadata): 10.1109/escience.2017.79
Sci-Hub (fulltext): 10.1109/escience.2017.79
Internet Archive Scholar (search for fulltext): Artifact Evaluation: Is it a Real Incentive?
Tagged: Computer Science (RSS) academia (RSS), academic software (RSS)
AE is correlated with higher citation count. We cannot establish causality, however.
- Artifact evaluation := optionally, after acceptance, authors can submit code and data artifacts to the journal which will be published alongside their article. The article bears a badge declaring the artifacts, and this badge is a small incentive to submit artifacts.
- This studies ACM's implementation of AE
- Author participation stable at 40% in 2011.
- ACM's AE does not require open-source. This is makes quiets objections relating to IP rights, especially from industry labs.
- Review is done independently from the program committee, to avoid putting more burden on the PCs.
- Artifacts are evaluated by senior PhD students and post docs, who are more aware of the latest software methods and more often working with code.
- Artifacts are evaluated after acceptance, so it means fewer artifacts to review.
- Artifact evaluation does not influence paper acceptance.
- AE does correlate with greater citation count.
- Can't say if causal.
- Citation count has many biases, but it is a proxy for research impact.
- Google Scholar to get citation count.
- This study shows correlation, not necessarily causation.
- Nevertheless, if causation is true, that would incentivize AE.
- A stronger incentive would be for grant-funding agencies to mandate or recommend for reproducibility engineering.