AcaWiki - User contributions [en]

Reproducibility vs. Replicability: A Brief History of a Confused Terminology

2023-05-19T17:09:52Z

Charmonium:

{{Summary
|title=Reproducibility vs. Replicability: A Brief History of a Confused Terminology
|authors=Hans E. Plesser
|url=https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5778115/
|tags=reproducibility
|journal=Frontiers in Neuroinformatics
|pub_date=2017
|doi=10.3389/fninf.2017.00076
|summary=
* Jon Claerbout defines ''reproducibility'' as running same software on same inputs, getting same outputs in [[Electronic documents give reproducible research a new meaning]].
* Pre-existing scientific tradition defines
** ''Repeatability'': the exact same procedure and exact same results.
** ''Reproducibility'': is testing the same hypothesis with variations in the unspecified part of the method.
* ACM adopted:
** ''Repeatability'': Same team, same experimental setup
** ''Replicability'': Different team, same experimental setup
** ''Reproducibility'': Different team, different experimental setup
* Three years after this was published, ACM changed their tune to be more consistent with metrology [https://www.acm.org/publications/policies/artifact-review-and-badging-current]:
** ''Reproducibility'': Different team, same experimental setup
** ''Replicability'': Different team, different experimental setup
* Goodman lexicon resolves the ambiguity.
}}

{| class="wikitable"
! Goodman !! Claerbout !! ACM (unknown date <2017) !! ACM 2020
|-
| || || Repeatability || Repeatability
|-
| Methods reproducibility || Reproducibility || Replicability || Reproducibility
|-
| Results reproducibility || Replicability || Reproducibility || Replicability
|-
| Inferential reproducibility || || ||
|}

Software Engineering reading list

2022-04-07T17:38:03Z

Charmonium: /* Software Testing */

= Human factors =
* [[What Predicts Software Developers' Productivity?]]
* [[Software Engineering at Google]]

= Technical factors =
* [[CoDeSe: fast deserialization via code generation]]

= Software Testing =
* [[SRRTA: Regression Testing Acceleration via State Reuse]]
* [[Practical regression test selection with dynamic file dependencies]]
* [[Regression testing minimization, selection and prioritization: a survey]]

= Related Lists =
* [[Computer Science reading list]]
* [[Research Software Engineering reading list]]

Regression testing minimization, selection and prioritization: a survey

2022-04-07T17:37:47Z

Charmonium: Created page with "{{Summary |title=Regression testing minimization, selection and prioritization: a survey |authors=S. Yoo, M. Harman |url=https://onlinelibrary.wiley.com/doi/epdf/10.1002/stvr...."

{{Summary
|title=Regression testing minimization, selection and prioritization: a survey
|authors=S. Yoo, M. Harman
|url=https://onlinelibrary.wiley.com/doi/epdf/10.1002/stvr.430
|tags=software engineering
|summary=* Test Suite Minimization Problem
** GE: Greedily select tests that test a set of requirements, until all requirements are covered.
** GRE: remove all redundant test cases in the test suite, then run GE.
** the dual of the minimal hittingset problem, i.e. the set cover problem
** Integer linear programming
** More
** Observations:
*** The reduction in size is greater intest suites with a higher block coverage in most cases.
*** The fault-detection effectiveness was decreasedby test case reduction, but the overall decrease in fault-detection effectiveness is not excessive
**** Not in all studies
* Test Case Selection Problem
** Integer Programming
** Dataflow
*** Not all deps might be captured
** Symbolic execution
*** Slow
*** Pointer aliasing
** Slice
*** Relevant slice >= execution slice >= dynamic slice
*** If CFG changes, not complete
** Control Dependence Graph
*** Could use CFG, but that might not be complete due to data dep
*** TestTube system adds other program entities such as defns and macros
** Firewall based on module modification
*** Conservative
** CFG cluster identification
*** Sounds like CDG
** Design-based
*** Why not use “empirical” design (aka static analysis)
* Test Case Prioritization Problem
** Prioiritize tests with high coverage (branches, conditions, statements, mutant-killing) x (additional, total)
*** Mutant-killing <= statement coverage
*** “approaches with coarser granularity would produce lower APFD values, which was confirmed statistically”
*** Newer tests more likelu to reveal faults
*** Greedy algorithm is generally efficient [128]
** Interaction-based prio/coverage
** Distribution-based Approach: reduce clusters of tests which have similar traces
|journal=Software Testing, Verification and Reliability
|pub_date=2012
|doi=10.1002/stvr.430
|subject=Computer Science
}}

Practical regression test selection with dynamic file dependencies

2022-04-07T16:55:43Z

Charmonium:

{{Summary
|title=Practical regression test selection with dynamic file dependencies
|authors=Milos Gligoric, Lamyaa Eloussi, Darko Marinov
|url=https://dl.acm.org/doi/10.1145/2771783.2771784
|tags=software engineering
|summary=* Google TAP can only preform RTS _across_ projects.
** I don't think this is true anymore. Perhaps they are talking about libraries that compile down to one file.
* Try to build an RTS usable in practice
** Use Maven (33%), use JUnit (78%),
* Total runtime = Analysis + Execution + Collection
* In Ekstazi, granularity is file/class instead of method.
** File/class is always superset of method (has extra).
** Method-granularity often select classes anyway, since the class could override a parent method, and could have final fields.
** For external libs, "collecting only the classes is not safe, and hence Ekstazi uses files as dependencies". This is not explained.
** Don't use strace because:
*** Multiple tests could be run on the same JVM process
*** Multiple claseses bundled in one `.jar` file.
*** Instead dynamically instrument the bytecode and monitors the execution
*** Checksum
**** From bytecode, not source; robust to syntactic sugar.
**** Omit debug info
*** In the future, use DB for dep graph.
|journal=ISTA 2015
|pub_date=13/07/2015
|doi=10.1145/2771783.2771784
|subject=Computer Science
}}

= Elsewhere =
* [https://pubpeer.com/publications/B4E71C3A06D84A90FF654F0FC4FA4B PubPeer]
* [http://ekstazi.org/ Code]

Software Engineering reading list

2022-04-07T16:54:17Z

Charmonium: /* Software Testing */

= Human factors =
* [[What Predicts Software Developers' Productivity?]]
* [[Software Engineering at Google]]

= Technical factors =
* [[CoDeSe: fast deserialization via code generation]]

= Software Testing =
* [[SRRTA: Regression Testing Acceleration via State Reuse]]
* [[Practical regression test selection with dynamic file dependencies]]

= Related Lists =
* [[Computer Science reading list]]
* [[Research Software Engineering reading list]]

Practical regression test selection with dynamic file dependencies

2022-04-07T16:53:54Z

Charmonium: Created page with "{{Summary |title=Practical regression test selection with dynamic file dependencies |authors=Milos Gligoric, Lamyaa Eloussi, Darko Marinov |url=https://dl.acm.org/doi/10.1145/..."

What Predicts Software Developers' Productivity?

2022-04-07T16:03:55Z

Charmonium:

{{Summary
|title=What Predicts Software Developers' Productivity?
|authors=Emerson Murphy-Hill, Ciera Jaspan, Caitlin Sadowski, David Shepherd, Michael Phillips, Collin Winter, Andrea Knight, Edward Smith, Matthew Jorde
|url=https://doi.org/10.1109/TSE.2019.2900308
|tags=software engineering
|summary=The authors use survey developers on their productivity and other activities the authors believe might affect productivity in a variety of different companies. The authors have settled on a couple of factors they believe correlate the most with productivity. Of which, the most important are: job enthusiasm, peer support for new ideas, useful feedback about job performance.
|relevance=This is one of the most complete and large-scale estimations of productivity factors. New research on developer productivity often starts from this study.
|journal=IEEE Transactions on Software Engineering
|pub_date=2019/02/19
|doi=10.1109/TSE.2019.2900308
|subject=Computer Science
}}

= Research Questions =
* What factors most strongly predict developers' self-rated productivity?
* How do these factors differ across companies?
* What predicts developer self-rated productivity, in particular, compared to other knowledge workers?

= Companies Studied =
[[File:Screenshot from 2021-08-15 15-31-09.png|800px]]

= Methodology =
== Questions ==

* Dependent factor: self-rating of "productivity"
** Wanted the question to be a fixed-point of reference, with a high ceiling, that focuses on frequency and intensity of productivity.
** Rated on [https://en.wikipedia.org/Articles/Likert_Scale Likert Scale] "I regularly reach a high level of productivity."
* Independent factors
** Subjective survey factors: based on survey questions from prior literature
** Demographic factors: gender, tenure, seniority.
** Attention question: to make sure people are reading the survey carefully "Respond with 'Somewhat disagree' to this item"
* Compare to software developers to other knowledge workers, to find unique differences. For "other knowledge workers," use Google data analysts.

== Analysis ==
* Hold demographic factors constant for regression.
* Use Benjamini-Hochberg method to correct p-value over multiple linear models.
* For causality, look at how prior work relates the survey factors with productivity. This study itself does not attempt to establish that.

= Results =
* Significant positive correlation for both, CL merges more strongly, but there is still much unexplained variance.
* Google had a much higher response-rate than NI and ABB.
* Between 5 and 20% of the responses failed the attention question.

== Demographic correlations ==
* Only at ABB was gender statistically significant.
** Female respondants statistically significantly rated their productivity greater than their male counterparts, and custom genders reported greater than females.
* Tenure at position was statistically significant only at ABB.

== Research Questions ==

* RQ1: What factors most strongly predict developers' self-rated productivity?
** Job enthusiasm, peer support for new ideas, useful feedback about job performance
*** What makes software developers enthusiastic about their job? What accounts for differences in levels of enthusiasm between developers? What interventions can increase enthusiasm? This work can extend existing work on developer happiness [24] and motivation [25].
*** What kinds of new ideas are commonly expressed in software development practice? What actions influence developers' feelings of support for those ideas? What interventions can increase support for new ideas, while maintaining current commitments?
*** What kinds of job feedback do software engineers receive, and what makes it useful? What kinds of feedback is not useful? What interventions can increase the regularity and usefulness of feedback?
** COCOMO factors were not as predictive. Either COCOMO is missing important factors or COCOMO measures something different than this kind of productivity.
* RQ2: How do these factors differ across companies?
** Most consistent-between-companies: Use of remote work to concentrate, Useful feedback about job performance, Peer support for new ideas
*** These are all social/environemntal, not technical.
** Most variant-betwee-companies: Use of best tools and practices, code reuse, accuracy of incoming information
*** Use of best tools/practices only matters in big codebases. Google has this, NI does not.
*** Code reuse is easier in a monorepo. Google has this, ABB does not.
*** Accurate information mattered more at NI than ABB. ABB has more support teams and layers at which to correct such information.
* RQ3: What predicts developer self-rated productivity, in particular, compared to other knowledge workers?
** Analysts' productivity correlates uniquely with positive feelings about their teammates and time management autonomy.
** Software developers' productivity correlates uniquely with doing a variety of tasks and working effectively way from their desk.

== Future work ==

* Systematic literature review
* Consider more factors, such as demogrpahics
* Multiple dimensions and metrics of productivity
* Determine costs of changing factors which affect productivity.

== Threats to validity ==

* They use self-rated productivity rather than objective metrics.
** Prior literature seems to validate this choice.
* Their question had no time-window (productivity this year vs productivity today).
* They couldn't put too many questions, so they had to group things together. E.g. Q14 "Do you use best practices?", doesn't distinguish *which* best practices.
* They rely on prior work to establish causality and just measure correlation.
** Maybe higher productivity causes higher enthusiasm, not the other way around.
* Maybe companies are not representative.
** They tried to pick an archetype of each "kind" of company, new big tech (Google), old big tech (ABB), small tech (NI).
** But the results are not weighted by prevalence of these kinds of companies. Maybe it is best to zoom in to the data which match your kind of company.
* Maybe analysts are not representative of other knowledge workers like lawyers.
* Maybe two factors are linked; no covariance analysis.
* Respondants might not be totally blind if they could guess the study's objectives from the survey questions.
* Had to reword the questions from software developers to suit analysts.

= See Also =
* [https://news.ycombinator.com/item?id=23766987 Hackernews discussion]

Software Engineering reading list

2022-04-07T16:01:55Z

Charmonium:

= Human factors =
* [[What Predicts Software Developers' Productivity?]]
* [[Software Engineering at Google]]

= Technical factors =
* [[CoDeSe: fast deserialization via code generation]]

= Software Testing =
* [[SRRTA: Regression Testing Acceleration via State Reuse]]

= Related Lists =
* [[Computer Science reading list]]
* [[Research Software Engineering reading list]]

Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science

2022-01-31T05:30:07Z

Charmonium:

{{Summary
|title=Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science
|authors=Reed Milewicz, Miranda Mundt
|url=https://arxiv.org/abs/2201.04007
|summary=
* Software engineering researchers (SE researchers) and computational science/engineering (CSE) have had a symbiotic relationship.
* But few formal interactions between DOE laboratories (CSE) and SE researchers.
** Only few papers coauthored by DOE lab staff.
* Opportunities:
** Take SE research to CSE domain.
** Conduct novel SE research in CSE domain.
*** Currently CSE problems is underrepresented in SE research.
** Make DOE care about SE research.
|journal=arXiv
|pub_date=2022/01/11
|arxiv=2201.04007
|subject=Computer Science
}}

Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science

2022-01-31T05:22:54Z

Charmonium:

{{Summary
|title=Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science
|authors=Reed Milewicz, Miranda Mundt
|url=https://arxiv.org/abs/2201.04007
|summary=
* Software engineering researchers (SE researchers) and computational science/engineering (CSE) have had a symbiotic relationship.
* But few formal interactions between DOE laboratories (CSE) and SE researchers.
** Only few papers coauthored by DOE lab staff.

Placeholder
|journal=arXiv
|pub_date=2022/01/11
|arxiv=2201.04007
|subject=Computer Science
}}

SRRTA: Regression Testing Acceleration via State Reuse

2022-01-31T05:13:03Z

Charmonium:

{{Summary
|title=SRRTA: Regression Testing Acceleration via State Reuse
|authors=Jinhao Dong, Yiling Lou, Dan Hao
|url=https://ieeexplore.ieee.org/abstract/document/9286088
|summary=Placeholder
|journal=ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering
|pub_date=2020/12
|doi=10.1145/3324884.3418928
|subject=Computer Science
}}

Research Software Engineering reading list

2022-01-26T17:49:32Z

Charmonium: /* RSE Departments, Organizations, and Institutes */

= RSE Departments, Organizations, and Institutes =
* [[Addressing Research Software Sustainability via Institutes]]
* [[Research Software Development & Management in Universities: Case Studies from Manchester's RSDS Group, Illinois' NCSA, and Notre Dame's CRC]]
* [[Research, Develop, Deploy: Building a Full Spectrum Software Engineering and Research Department]]
* [[Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science]]

= Current Practices =
* Practices which can be improved
** [[Technical Debt in Computational Science]]
** [[Troubling Trends in Scientific Software Use]] (ecology)
* Pure observations
** [https://zenodo.org/record/14809#.YNdUKDpOkUE UK Research Software Survey (dataset)]
** [[Vertical Integration]]
** [[Developers Perception of Peer Code Review in Research Software Development]]
* Improved practices
** [[The Research Software Engineer]]
** [[Ten Simple Rules for the Open Development of Scientific Software]]
** [[Reducing Technical Debt with Reproducible Containers]]
** [[A Workflow for Increasing the Quality of Scientific Software (in Computational Science and Engineering)]]
** [[Mining Development Data to Understand and Improve Software Engineering Processes in HPC Projects]]
** [[Easing the burden of code review]]
** [[Software Engineering Challenges and Best Practices for Multi-Institutional Scientific Software Development]]
** [[Some Simple Guidelines for Effective Data Management]]

= Famous Bugs =
* [[A Scientist's Nightmare: Software Problem Leads to Five Retractions]]

= Related Lists =
* [[Open Academia reading list]]
* [[Reproducibility reading list]]
* [[Software Engineering reading list]]
* [[Computer Science reading list]]

Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science

2022-01-26T17:46:12Z

Charmonium: Created page with "{{Summary |title=Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science |authors=Reed Milewicz, Miranda Mundt |url=https://a..."

{{Summary
|title=Building Bridges: Establishing a Dialogue Between Software Engineering Research and Computational Science
|authors=Reed Milewicz, Miranda Mundt
|url=https://arxiv.org/abs/2201.04007
|summary=Placeholder
|journal=arXiv
|pub_date=2022/01/11
|arxiv=2201.04007
|subject=Computer Science
}}

Research Software Engineering reading list

2022-01-26T16:39:05Z

Charmonium: /* Institutions and Incentives */

= RSE Departments, Organizations, and Institutes =
* [[Addressing Research Software Sustainability via Institutes]]
* [[Research Software Development & Management in Universities: Case Studies from Manchester's RSDS Group, Illinois' NCSA, and Notre Dame's CRC]]
* [[Research, Develop, Deploy: Building a Full Spectrum Software Engineering and Research Department]]

= Current Practices =
* Practices which can be improved
** [[Technical Debt in Computational Science]]
** [[Troubling Trends in Scientific Software Use]] (ecology)
* Pure observations
** [https://zenodo.org/record/14809#.YNdUKDpOkUE UK Research Software Survey (dataset)]
** [[Vertical Integration]]
** [[Developers Perception of Peer Code Review in Research Software Development]]
* Improved practices
** [[The Research Software Engineer]]
** [[Ten Simple Rules for the Open Development of Scientific Software]]
** [[Reducing Technical Debt with Reproducible Containers]]
** [[A Workflow for Increasing the Quality of Scientific Software (in Computational Science and Engineering)]]
** [[Mining Development Data to Understand and Improve Software Engineering Processes in HPC Projects]]
** [[Easing the burden of code review]]
** [[Software Engineering Challenges and Best Practices for Multi-Institutional Scientific Software Development]]
** [[Some Simple Guidelines for Effective Data Management]]

= Famous Bugs =
* [[A Scientist's Nightmare: Software Problem Leads to Five Retractions]]

= Related Lists =
* [[Open Academia reading list]]
* [[Reproducibility reading list]]
* [[Software Engineering reading list]]
* [[Computer Science reading list]]

Research Software Engineering reading list

2022-01-26T05:53:34Z

Charmonium: /* Institutions and Incentives */

= Institutions and Incentives =
* [[Addressing Research Software Sustainability via Institutes]]
* [[Research Software Development & Management in Universities: Case Studies from Manchester's RSDS Group, Illinois' NCSA, and Notre Dame's CRC]]
* [[Research, Develop, Deploy: Building a Full Spectrum Software Engineering and Research Department]]

= Current Practices =
* Practices which can be improved
** [[Technical Debt in Computational Science]]
** [[Troubling Trends in Scientific Software Use]] (ecology)
* Pure observations
** [https://zenodo.org/record/14809#.YNdUKDpOkUE UK Research Software Survey (dataset)]
** [[Vertical Integration]]
** [[Developers Perception of Peer Code Review in Research Software Development]]
* Improved practices
** [[The Research Software Engineer]]
** [[Ten Simple Rules for the Open Development of Scientific Software]]
** [[Reducing Technical Debt with Reproducible Containers]]
** [[A Workflow for Increasing the Quality of Scientific Software (in Computational Science and Engineering)]]
** [[Mining Development Data to Understand and Improve Software Engineering Processes in HPC Projects]]
** [[Easing the burden of code review]]
** [[Software Engineering Challenges and Best Practices for Multi-Institutional Scientific Software Development]]
** [[Some Simple Guidelines for Effective Data Management]]

= Famous Bugs =
* [[A Scientist's Nightmare: Software Problem Leads to Five Retractions]]

= Related Lists =
* [[Open Academia reading list]]
* [[Reproducibility reading list]]
* [[Software Engineering reading list]]
* [[Computer Science reading list]]

Research, Develop, Deploy: Building a Full Spectrum Software Engineering and Research Department

2022-01-26T05:53:04Z

Charmonium: Created page with "{{Summary |title=Research, Develop, Deploy: Building a Full Spectrum Software Engineering and Research Department |authors=Reed Milewicz, James Willenbring, Dena Vigil |url=ht..."

{{Summary
|title=Research, Develop, Deploy: Building a Full Spectrum Software Engineering and Research Department
|authors=Reed Milewicz, James Willenbring, Dena Vigil
|url=https://arxiv.org/abs/2010.04660
|summary=* Description of the Software Engineering and Research Department at Sandia National Laboratories
** Used to be Software Engineering, Maintenance, Support, but this was not an official department.
*** Lack of departmental status made it difficult to hire, retain, and advance RSEs.
** Uses [https://www.metaltoad.com/blog/beware-matrix-model Matrix-management] rather than hierarchical management (reporting to multiple rather than one member).
** See also, University of Manchester's Research Software and Data Science (RSDS) group.
* Authors believe placing RSEs in inter-disciplinary (cross-functional) teams makes the team more productive.
* RSEs engage in three activities: Research, Develop, Deploy, with a focus on develop.
** Research: Applied research in software engineering
** Develop: Embeded development, maintenance, and support for scientific software
** Deploy: maintain systems (e.g. Jenkins build/test farms)
|journal=arXiv
|pub_date=2020/10/09
|arxiv=2010.04660
|subject=Computer Science
}}

Introducing distributed dynamic data-intensive (D3) science: Understanding applications and infrastructure

2022-01-22T01:17:07Z

Charmonium:

{{Summary
|title=Introducing distributed dynamic data-intensive (D3) science: Understanding applications and infrastructure
|authors=Shantenu Jha, Daniel S. Katz, Andre Luckow, Neil Chue Hong, Omer Rana, Yogesh Simmhan
|url=https://onlinelibrary.wiley.com/doi/10.1002/cpe.4032
|summary=* Traditional application := program run by one group written to find answer to scientific question.
* Infrastructure application := a program written in multiple stages run by different groups.
* Big data
* Distributed := presence of data in different physical or logical locations. This could because the data comes from different sensors, it could be too big to be processed by a single node on a timely manner, it could be because you want more reliability given by redundancy and load-balancing, it could be for privacy or policy reasons.
** Replicated
** Partitioning
** Streaming
* Dynamic := an application with spatiotemporal variability.

== Examples ==
* Next Generation Sequencing (NGS) := map/align short reads to a reference genome.
** Application type: traditional
** Data: terrabyte scale data of DNA sequences
** Distribution: the problem can be distributed, but it is unclear how to get optimal performance. Few workflow systems natively manage distribution.
** Dynamic: the data itself is not dynamic, but properties of the running program are (when tasks complete).
* ATLAS := Analyze experimental physics data (pleasingly parallel)
** Application type: infrastructure; data generation and processing are controlled by different people. Scientists submit requests to run certain analyses on the data.
** Data: 20Tb per day of serialized C++ objects
** Distribution: 250,000 cores over 140 sites.
** Dynamic: data streams in continuously, and applications is run 2 or 3 times per year.
* Large Synoptic Survey Telescope (LSST) := find and study moving objects using a telescope.
** Application type: infrastructure; the data gets used by others downstream.
** Data: tens of TB per day of FITS images
** Distributional: talks to other telescopes, compute resources, and storage resources
** Dynamic: Data streams in. The system has to decide whether or not to interrupt its existing observing program to get another look at an anomalous object.
* SOA Astronomy := uses the data from LSST
** Application type: Infrastructure
** Data: 1Gb images
** Distribution: Data exists on different servers and is processed in a distributed cluster.
** Dynamic: Source data is constantly in flux.
* ... others
|journal=Concurrency and Computation: Practice and Experience
|pub_date=2017/02/02
|doi=10.1002/cpe.4032
|arxiv=1609.03647
|subject=Computer Science
}}

Introducing distributed dynamic data-intensive (D3) science: Understanding applications and infrastructure

2022-01-21T23:07:00Z

Charmonium: Created page with "{{Summary |title=Introducing distributed dynamic data-intensive (D3) science: Understanding applications and infrastructure |authors=Shantenu Jha, Daniel S. Katz, Andre Luckow..."

SRRTA: Regression Testing Acceleration via State Reuse

2022-01-21T21:13:26Z

Charmonium: Created page with "{{Summary |title=SRRTA: Regression Testing Acceleration via State Reuse |authors=Jinhao Dong, Yiling Lou, Dan Hao |url=https://ieeexplore.ieee.org/abstract/document/9286088 |s..."

{{Summary
|title=SRRTA: Regression Testing Acceleration via State Reuse
|authors=Jinhao Dong, Yiling Lou, Dan Hao
|url=https://ieeexplore.ieee.org/abstract/document/9286088
|summary=Regression testing := Running tests
|journal=ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering
|pub_date=2020/12
|doi=10.1145/3324884.3418928
|subject=Computer Science
}}

What's the use of factor contents?

2022-01-19T05:24:23Z

Charmonium:

{{Summary
|title=What's the use of factor contents?
|authors=Edward E. Leamer
|url=https://www.sciencedirect.com/science/article/pii/S0022199699000045
|journal=Journal of International Economics
|pub_date=2000/02
|doi=10.1016/S0022-1996(99)00004-5
|subject=Economics
|journal_volume=50
}}

What's the use of factor contents?

2022-01-19T05:23:27Z

Charmonium:

{{Summary
|title=What's the use of factor contents?
|authors=Leamer, Edward E.
|journal=Journal of International Economics
|pub_date=2000
|journal_volume=50
}}

Software Engineering reading list

2022-01-19T05:12:25Z

Charmonium:

Compilers reading list

2022-01-19T05:11:47Z

Charmonium:

== Core ==

* [[Compilers: Principles, Techniques, and Tools]] ([https://www.worldcat.org/title/compilers-principles-techniques-and-tools/oclc/1022369647 WorldCat])
* [[A data locality optimizing algorithm]]
* [[Parameterized object sensitivity for points-to analysis for Java]] (needs work)
* [[Code Generation Schema for Modulo Scheduled Loops]]

== Internal Organization ==
* [[An Overview of the PL.8 Compiler]]
* [[LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation]]

== Dataflow Analysis ==
* [[Global Data Flow Analysis and Iterative Algorithms]]
* [[Engineering a Compiler]] ([https://www.worldcat.org/title/engineering-a-compiler/oclc/1153004203 WorldCat])

== Single Static Assignment ==
* [[Efficiently Computing Static Single Assignment Form and the Control Dependence Graph]]

== Interprocedural Analysis ==
* [[Program Analysis via Graph Reachability]]

== Pointer analysis ==
* [[The Ant and the Grasshopper: Fast and Accurate Pointer Analysis for Millions of Lines of Code]]

== Vectorization ==
* [[Exploiting Superword Level Parallelism with Multimedia Instruction Sets]]

== Program Synthesis ==
* [[A Fast Fourier Transform Compiler]]
* [[A Comparison of Empirical and Model-Driven Optimization]]

== JIT ==
* [[CoDeSe: fast deserialization via code generation]]

== Dynamic Analysis ==
* [[Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation]]
* [[Trace-based Just-in-Time Type Specialization for Dynamic Languages]]

== Native Code Generation ==
* [[Improvements to Graph Coloring Register Allocation]]
* [[Automatic Generation of Peephole Superoptimizers]]

== Correctness ==
* [[Automatic Predicate Abstraction of C Programs]]
* [[Saturn: A Scalable Framework for Error Detection Using Boolean Satisfiability]]
* [[ABCD: Eliminating Array Bounds Checks on Demand]]

== Static Analysis ==
* [[Lessons from Building Static Analysis Tools at Google]]
* [[A few billion lines of code later: using static analysis to find bugs in the real world]]
* [[Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale]]

== Related fields ==
* [[Computer Science reading list]]

CoDeSe: fast deserialization via code generation

2022-01-19T05:10:53Z

Charmonium:

{{Summary
|title=CoDeSe: fast deserialization via code generation
|authors=Milos Gligoric, Darko Marinov, Sam Kamin
|url=https://dl.acm.org/doi/abs/10.1145/2001420.2001456
|summary=* Serialization/marshalling := encoding the state of an object into a stream of bytes.
* Deserialization/unmarshalling := restoring the state of an object from that stream of serialized bytes.
* Traditional de/serialization maps data elements to bytes repeatedly.
* On the other hand, CoDeSe (COde-based DEserialization and SErialization) serialization writes executable code into the bytestream, while deserialization just executes that code.
|journal=ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
|pub_date=2011/07
|doi=10.1145/2001420.2001456
|subject=Computer Science
}}

= See Also =

* [http://mir.cs.illinois.edu/codese Code repository]

CoDeSe: fast deserialization via code generation

2022-01-19T05:10:30Z

Charmonium:

CoDeSe: fast deserialization via code generation

2022-01-19T05:04:46Z

Charmonium: Created page with "{{Summary |title=CoDeSe: fast deserialization via code generation |authors=Milos Gligoric, Darko Marinov, Sam Kamin |url=https://dl.acm.org/doi/abs/10.1145/2001420.2001456 |jo..."

{{Summary
|title=CoDeSe: fast deserialization via code generation
|authors=Milos Gligoric, Darko Marinov, Sam Kamin
|url=https://dl.acm.org/doi/abs/10.1145/2001420.2001456
|journal=ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
|pub_date=2011/07
|doi=10.1145/2001420.2001456
|subject=Computer Science
}}

Template:Summary

2022-01-18T19:27:25Z

Charmonium:

<noinclude>
This is the 'Summary' template.
It should be called in the following format:
<pre>
{{Summary
|authors=
|journal=
|journal_volume=
|url=
|pub_date=
|issn=
|doi=
|arxiv=
|subject=
|tags=
|summary=
|relevance=
|pub_open_access=
|wikidata=
}}
</pre>

Edit the page to see the template text.
</noinclude><includeonly>
[[Category:Summary]][[title::{{PAGENAME}}| ]][[type::article| ]]

<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rfr_id=info:sid/acawiki.org:summary&rft.genre=article&rft_id=info:doi/{{{doi}}}&rft.atitle={{urlencode:{{#if:{{{title|}}}|{{{title}}}|{{PAGENAME}}}}}}&rft.jtitle={{urlencode:{{{journal}}}}}&rft.date={{urlencode:{{{pub_date}}}}}{{#arraymap:{{{authors|}}}|,|xxx|&rft.au={{urlencode:xxx}}| }}"> __NOFACTBOX__ __NOTOC__ '''Citation:''' ''{{#arraymap:{{{authors|}}}|,|xxx|[[Author::xxx]]|,  }} {{#if:{{{pub_date|}}}| ([[Date::{{{pub_date|}}}]]) | }} {{PAGENAME}}. {{#if:{{{journal|}}}|
[{{fullurl:Special:BrowseData/Summary|Journal={{anchorencode:{{{journal|}}}}}}} {{{journal|}}}] [[Journal::{{{journal|}}}| ]] {{#if:{{{journal_volume|}}}| (Volume [[Volume::{{{journal_volume|}}}]]) | }} ({{#ask:[[Journal::{{{journal|}}}]]| rsstitle={{{journal|}}} | rssdescription=All summaries from {{{journal|}}} | format=rss }})
| }}'' <nowiki>
</nowiki>{{#if:{{{doi|}}}|DOI (original publisher): [http://dx.doi.org/{{{doi|}}} {{{doi|}}}] [[DOI::{{{doi|}}}| ]] | }}<nowiki>
</nowiki>{{#if:{{{arxiv|}}}|arXiv (preprint): [http://arxiv.org/{{{arxiv|}}} arXiv:{{{arxiv|}}}] | }}<nowiki>
</nowiki>{{#if:{{{doi|}}}|Semantic Scholar (metadata): [https://api.semanticscholar.org/{{{doi|}}} {{{doi|}}}] | }}<nowiki>
</nowiki>{{#if:{{{doi|}}}|Sci-Hub (fulltext): [https://sci-hub.se/{{{doi}}} {{{doi}}}] | }}<nowiki>
</nowiki>Internet Archive Scholar (search for fulltext): [https://scholar.archive.org/search?q={{urlencode:{{PAGENAME}}}} {{PAGENAME}}] <nowiki>
</nowiki>{{#if:{{{wikidata|}}}|Wikidata (metadata): [https://www.wikidata.org/wiki/{{{wikidata|}}} {{{wikidata|}}}] [[Wikidata::{{{wikidata|}}}| ]] | }}<nowiki>
</nowiki>{{#if:{{{issn|}}}|'''issn:''' [[ISSN::{{{issn|}}}| ]] | }} {{#if:{{{url|}}}| '''Download:''' {{#arraymap:{{{url}}}|,|xxx|[[URL::xxx]]|,  }} | }}<nowiki>
</nowiki>'''Tagged:''' {{#if:{{{subject|}}}| [{{fullurl:Special:BrowseData/Summary|Subject={{anchorencode:{{{subject|}}}}}}} {{{subject|}}}] [[Subject::{{{subject|}}}| ]]
({{#ask:[[Subject::{{{subject|}}}]][[Category:Summary]]| rsstitle={{{subject|}}} | rssdescription=All summaries with this subject {{{subject|}}} | format=rss }})
| }} {{#if:{{{tags|}}}|
{{#arraymap:{{{tags|}}}|,|xxx|[{{fullurl:Special:BrowseData/Summary|Tag={{anchorencode:xxx}}}} xxx] [[Tag::xxx| ]]({{#ask:[[Tag::xxx]][[Category:Summary]]| rsstitle=xxx | rssdescription=All summaries tagged with xxx | format=rss }})|,  }}
| }}
{{#if:{{{summary|}}}|
= Summary =
{{{summary|}}}
| }}
{{#if:{{{relevance|}}}|
== Theoretical and Practical Relevance ==
{{{relevance|}}}
| }}
{{#if:{{{pub_open_access|}}}|
<div>
[[Published in an Open Access journal::{{{pub_open_access|}}}| ]]
{{{{{pub_open_access|}}} Published in an Open Access journal}}
</div>
| }} </includeonly>

Compilers reading list

2021-10-26T18:42:33Z

Charmonium: /* Static Analysis */

== Core ==

* [[Compilers: Principles, Techniques, and Tools]] ([https://www.worldcat.org/title/compilers-principles-techniques-and-tools/oclc/1022369647 WorldCat])
* [[A data locality optimizing algorithm]]
* [[Parameterized object sensitivity for points-to analysis for Java]] (needs work)
* [[Code Generation Schema for Modulo Scheduled Loops]]

== Internal Organization ==
* [[An Overview of the PL.8 Compiler]]
* [[LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation]]

== Dataflow Analysis ==
* [[Global Data Flow Analysis and Iterative Algorithms]]
* [[Engineering a Compiler]] ([https://www.worldcat.org/title/engineering-a-compiler/oclc/1153004203 WorldCat])

== Single Static Assignment ==
* [[Efficiently Computing Static Single Assignment Form and the Control Dependence Graph]]

== Interprocedural Analysis ==
* [[Program Analysis via Graph Reachability]]

== Pointer analysis ==
* [[The Ant and the Grasshopper: Fast and Accurate Pointer Analysis for Millions of Lines of Code]]

== Vectorization ==
* [[Exploiting Superword Level Parallelism with Multimedia Instruction Sets]]

== Program Synthesis ==
* [[A Fast Fourier Transform Compiler]]
* [[A Comparison of Empirical and Model-Driven Optimization]]

== Dynamic Analysis ==
* [[Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation]]
* [[Trace-based Just-in-Time Type Specialization for Dynamic Languages]]

== Native Code Generation ==
* [[Improvements to Graph Coloring Register Allocation]]
* [[Automatic Generation of Peephole Superoptimizers]]

== Correctness ==
* [[Automatic Predicate Abstraction of C Programs]]
* [[Saturn: A Scalable Framework for Error Detection Using Boolean Satisfiability]]
* [[ABCD: Eliminating Array Bounds Checks on Demand]]

== Static Analysis ==
* [[Lessons from Building Static Analysis Tools at Google]]
* [[A few billion lines of code later: using static analysis to find bugs in the real world]]
* [[Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale]]

== Related fields ==
* [[Computer Science reading list]]

Compilers reading list

2021-10-26T18:42:18Z

Charmonium:

== Core ==

* [[Compilers: Principles, Techniques, and Tools]] ([https://www.worldcat.org/title/compilers-principles-techniques-and-tools/oclc/1022369647 WorldCat])
* [[A data locality optimizing algorithm]]
* [[Parameterized object sensitivity for points-to analysis for Java]] (needs work)
* [[Code Generation Schema for Modulo Scheduled Loops]]

== Internal Organization ==
* [[An Overview of the PL.8 Compiler]]
* [[LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation]]

== Dataflow Analysis ==
* [[Global Data Flow Analysis and Iterative Algorithms]]
* [[Engineering a Compiler]] ([https://www.worldcat.org/title/engineering-a-compiler/oclc/1153004203 WorldCat])

== Single Static Assignment ==
* [[Efficiently Computing Static Single Assignment Form and the Control Dependence Graph]]

== Interprocedural Analysis ==
* [[Program Analysis via Graph Reachability]]

== Pointer analysis ==
* [[The Ant and the Grasshopper: Fast and Accurate Pointer Analysis for Millions of Lines of Code]]

== Vectorization ==
* [[Exploiting Superword Level Parallelism with Multimedia Instruction Sets]]

== Program Synthesis ==
* [[A Fast Fourier Transform Compiler]]
* [[A Comparison of Empirical and Model-Driven Optimization]]

== Dynamic Analysis ==
* [[Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation]]
* [[Trace-based Just-in-Time Type Specialization for Dynamic Languages]]

== Native Code Generation ==
* [[Improvements to Graph Coloring Register Allocation]]
* [[Automatic Generation of Peephole Superoptimizers]]

== Correctness ==
* [[Automatic Predicate Abstraction of C Programs]]
* [[Saturn: A Scalable Framework for Error Detection Using Boolean Satisfiability]]
* [[ABCD: Eliminating Array Bounds Checks on Demand]]

== Static Analysis ==
* [Lessons from Building Static Analysis Tools at Google]
* [A few billion lines of code later: using static analysis to find bugs in the real world]
* [Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale]

== Related fields ==
* [[Computer Science reading list]]

Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale

2021-10-26T18:39:33Z

Charmonium:

{{Summary
|title=Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale
|authors=Yechan Bae, Youngsuk Kim, Ammar Askar, Jungwon Lim, Taesoo Kim
|url=https://dl.acm.org/doi/10.1145/3477132.3483570
|tags=programming languages
|journal=SOSP '21: Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles
|pub_date=10/26/2021
|doi=10.1145/3477132.3483570
|subject=Computer Science
|summary=
Rust has a safe and unsafe mode; the former guarantees memory safety but the latter gives the programmer more control over the system, often needed for high-performance. For example, the standard library uses unsafe code internally. The authors build Rudra, a static analyzer for unsafe Rust. They find a large amount of bugs in real-world unsafe Rust code.
}}

= Problem =
* There is a tradeoff between safety and control. Giving programmer low-level control makes it harder for compiler to check safety.
* [https://www.rust-lang.org/ Rust language] balances this with safe and unsafe rust.
** In unsafe rust, programmer is responsible for memory safety, but has full control.
** Safety of a program depends on safety of all unsafe code.
** Unsafe code either exposed directly in API or wrapped in a safe API. Bugs in the wrapper are serious problems, because user can cause memory safety bug without typing <code>unsafe</code>.

= Solution =
* Rudra: Static analyzer for unsafe Rust.
** Checks for these bugs:
*** Panic-unsafety: incorrect resource deallocation in compiler-inserted invisible code paths.
*** Higher-order invariant bug: unchecked assumptions on user-provided higher-order values. E.g., assuming comparator is reflexive, anti-symmetric, and transitive.
*** Send/sync variance bug: incorrect condition for manual thread safety assertions.
** Challenges:
*** Incomplete definitions (e.g. higher-order invariants are rarely specified).
*** Some information is not available in later compilation stages (e.g. types get dropped after checking).
** Use high-level IR to find unsafe blocks and mid-level IR to find callgraph and other things.
** Unsafe-dataflow checker checks panic-unsafety and higher-order invariant bugs; send/sync variance bug requires its own checker.

= Evaluation =
* Run on all Rust crates.
* 2 bugs in Rust standard library.
* 112 Rust security advisories.
* 76 CVEs.
* Compare with dynamic fuzzers, Fuzzers and Miri, and other static analyzers, UAFChecker.
** None found bugs that Rudra found.
** Miri found bugs that Rudra didn't find.

= Limitations
* Rudra is not exhaustive.
* Rudra has a high false-positive rate (50% for high precision, 80% for high scalability).
* Rudra finds Bugs at the definition site: pro and con.
** Is the buggy code-path in API definition actually invoked by callers?

= See also =
* [https://www.youtube.com/watch?v=7pI9GfYEu-s short video]
* [https://www.youtube.com/watch?v=Hfl6EQquUU0 long video]

Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale

2021-10-26T18:32:30Z

Charmonium:

Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale

2021-10-26T18:14:13Z

Charmonium: Created page with "{{Summary |title=Rudra: Finding Memory Safety Bugs in Rust at the Ecosystem Scale |authors=Yechan Bae, Youngsuk Kim, Ammar Askar, Jungwon Lim, Taesoo Kim |url=https://dl.acm.o..."

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-17T18:17:03Z

Charmonium: /* Solution: IncPy */

{{Summary
|title=Using automatic persistent memoization to facilitate data analysis scripting
|authors=Philip J Guo, Dawson Engler
|url=https://dl.acm.org/doi/abs/10.1145/2001420.2001455
|tags=software engineering
|journal=ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
|pub_date=2011/07
|doi=10.1145/2001420.2001455
|subject=Computer Science
}}

= Problem =
* Programmers across a wide range of disciplines write scripts to transform, process, or analyze data in Python, their scripts are divided into stages with intermediate results, and they develop iteratively.
* Data-analysis is often ad hoc and exploratory.
* Data-analysts have a wide range of backgrounds; not strictly CS/engineering.
* They primarily use Python, Perl, or Ruby.
* Saving intermediate outputs by-hand is time-consuming and error-prone. Users forget to invalidate datasets.
* How to speed up their programs by saving intermediate outputs without being stale or confusing?

= Solution: IncPy =
IncPy is a modified Python interpreter that automatically memoizes functions. It invalidates a function's cache if its source code changes, and invalidates an entry if a file it depends on changes.

* Benefits:
** Less code (no de/serialization or cache testing)
** Automated data management
** Faster iteration times
* [https://github.com/pajju/IncPy Source code]
* Goals:
** No code changes (change interpreter instead)
** Low run-time overhead
** Compatible with 3rd party libraries.

== Implementation ==
* Each entry contains: function name, arguments, return, stdout, stderr, global vars dependencies, files read, files written, transitive code dependencies (name and bytecode).
* Modify interpreter's file IO calls to record file accesses to all functions on call stack.
* Each entry is a file named with a hash of the inputs. This has to be written atomically, so multiple threads/processes can share the same cache.
* Entries are written eagerly; if interpreter crashes, still have entries.
* Which calls to memoize: Only pure, time-consuming calls.
** When an impurity is detected (mutation of {closure, global, or argument}, source of non-determinism {PRNG, stdin, time}) all functions on call stack.
*** Functions that are pure on some paths but impure on others can still be memoized for those pure paths, on which we have IncPy has dynamically proven purity.
*** File IO does not count as impurity since these are captured in the entry.
** Heuristic: If the call is shorter than 1 second, overhead of memoization (disk read/write) is often not worthwhile. Otherwise, memoize.
*** Heuristic doesn't work if returned data is very large.
* Reachability analysis determines if global is mutated (impure) or if global is read (add dependency).
** Each object has two more fields: global name and func start.
*** Global name is the global this object is reachable from.
*** Func start is the time the variable was created.
*** If there is a read or write to a variable whose func start predates the current function, it must be marked as a dependency or impurity, respectively.

== Generalizability ==
* For dynamic languages, interpose the interpreter's handlers for function calls, value accesses, and file I/O (as does IncPy)
* For static languages, source-to-source translator or binary rewriter that inserts callback.
** Could augment with static purity, escape, and callgraph analysis.
* For whole workflows, use ptrace to identify file read/writes.
* Perhaps could be used to accelerate regression tests.

= Related Work =
* [https://en.wikipedia.org/wiki/Make_(software) Make], but Make requires explicit dependency information and only works between processes.
* Self-adjusting computation, but IncPy is more general.
* JIT, although it is focused on program micro-optimizations, not memoization, and these don't outlive the process.

= Future Work =
* Provenance browsing
* Network‐aware/database‐aware caching
* Lightweight programmer annotations
* Finer‐grained tracking within functions

= See Also =
* [https://www.usenix.org/legacy/event/tapp10/tech/slides/guo.pdf Usenix presentation]

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-17T17:37:02Z

Charmonium:

{{Summary
|title=Using automatic persistent memoization to facilitate data analysis scripting
|authors=Philip J Guo, Dawson Engler
|url=https://dl.acm.org/doi/abs/10.1145/2001420.2001455
|tags=software engineering
|journal=ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
|pub_date=2011/07
|doi=10.1145/2001420.2001455
|subject=Computer Science
}}

= Problem =
* Programmers across a wide range of disciplines write scripts to transform, process, or analyze data in Python, their scripts are divided into stages with intermediate results, and they develop iteratively.
* Data-analysis is often ad hoc and exploratory.
* Data-analysts have a wide range of backgrounds; not strictly CS/engineering.
* They primarily use Python, Perl, or Ruby.
* Saving intermediate outputs by-hand is time-consuming and error-prone. Users forget to invalidate datasets.
* How to speed up their programs by saving intermediate outputs without being stale or confusing?

= Solution: IncPy =
IncPy is a modified Python interpreter that automatically memoizes functions. It invalidates a function's cache if its source code changes, and invalidates an entry if a file it depends on changes.

* Benefits:
** Less code (no de/serialization or cache testing)
** Automated data management
** Faster iteration times
* [https://github.com/pajju/IncPy Source code]
* Goals:
** No code changes (change interpreter instead)
** Low run-time overhead
** Compatible with 3rd party libraries.

== Implementation ==
* Each entry contains: function name, arguments, return, stdout, stderr, global vars dependencies, files read, files written, transitive code dependencies (name and bytecode).
* Modify interpreter's file IO calls to record file accesses to all functions on call stack.
* Each entry is a file named with a hash of the inputs. This has to be written atomically, so multiple threads/processes can share the same cache.
* Entries are written eagerly; if interpreter crashes, still have entries.
* Which calls to memoize: Only pure, time-consuming calls.
** When an impurity is detected (mutation of {closure, global, or argument}, source of non-determinism {PRNG, stdin, time}) all functions on call stack.
*** Functions that are pure on some paths but impure on others can still be memoized for those pure paths, on which we have IncPy has dynamically proven purity.
*** File IO does not count as impurity since these are captured in the entry.
** Heuristic: If the call is shorter than 1 second, overhead of memoization (disk read/write) is often not worthwhile. Otherwise, memoize.
*** Heuristic doesn't work if returned data is very large.
* Reachability analysis determines if global is mutated (impure) or if global is read (add dependency).
** Each object has two more fields: global name and func start.
*** Global name is the global this object is reachable from.
*** Func start is the time the variable was created.
*** If there is a read or write to a variable whose func start predates the current function, it must be marked as a dependency or impurity, respectively.

= Future Work =
* Provenance browsing
* Network‐aware/database‐aware caching
* Lightweight programmer annotations
* Finer‐grained tracking within functions

= See Also =
* [https://www.usenix.org/legacy/event/tapp10/tech/slides/guo.pdf Usenix presentation]

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-17T07:00:59Z

Charmonium: /* Implementation */

{{Summary
|title=Using automatic persistent memoization to facilitate data analysis scripting
|authors=Philip J Guo, Dawson Engler
|url=https://dl.acm.org/doi/abs/10.1145/2001420.2001455
|tags=software engineering
|journal=ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
|pub_date=2011/07
|doi=10.1145/2001420.2001455
|subject=Computer Science
}}

= Problem =
* Programmers across a wide range of disciplines write scripts to transform, process, or analyze data in Python, their scripts are divided into stages with intermediate results, and they develop iteratively.
* Data-analysis is often ad hoc and exploratory.
* Data-analysts have a wide range of backgrounds; not strictly CS/engineering.
* They primarily use Python, Perl, or Ruby.
* Saving intermediate outputs by-hand is time-consuming and error-prone. Users forget to invalidate datasets.
* How to speed up their programs by saving intermediate outputs without being stale or confusing?

= Solution: IncPy =
IncPy is a modified Python interpreter that automatically memoizes functions. It invalidates a function's cache if its source code changes, and invalidates an entry if a file it depends on changes.

* Benefits:
** Less code (no de/serialization or cache testing)
** Automated data management
** Faster iteration times
* [https://github.com/pajju/IncPy Source code]
* Goals:
** No code changes (change interpreter instead)
** Low run-time overhead
** Compatible with 3rd party libraries.

== Implementation ==
* Each entry contains: function name, arguments, return, stdout, stderr, global vars dependencies, files read, files written, transitive code dependencies (name and bytecode).
* Modify interpreter's file IO calls to record file accesses to all functions on call stack.
* Each entry is a file named with a hash of the inputs. This has to be written atomically, so multiple threads/processes can share the same cache.
* Entries are written eagerly; if interpreter crashes, still have entries.
* Which calls to memoize: Only pure, time-consuming calls.
** When an impurity is detected (mutation of {closure, global, or argument}, source of non-determinism {PRNG, stdin, time}) all functions on call stack.
*** Functions that are pure on some paths but impure on others can still be memoized for those pure paths, on which we have IncPy has dynamically proven purity.
*** File IO does not count as impurity since these are captured in the entry.
** Heuristic: If the call is shorter than 1 second, overhead of memoization (disk read/write) is often not worthwhile. Otherwise, memoize.
*** Heuristic doesn't work if returned data is very large.

= Future Work =
* Provenance browsing
* Network‐aware/database‐aware caching
* Lightweight programmer annotations
* Finer‐grained tracking within functions

= See Also =
* [https://www.usenix.org/legacy/event/tapp10/tech/slides/guo.pdf Usenix presentation]

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-17T06:40:44Z

Charmonium: /* Solution: IncPy */

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-17T05:42:26Z

Charmonium: /* Solution: IncPy */

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-17T05:26:41Z

Charmonium:

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-11T21:35:21Z

Charmonium:

Using automatic persistent memoization to facilitate data analysis scripting

2021-10-11T20:41:13Z

Charmonium: Created page with "{{Summary |title=Using automatic persistent memoization to facilitate data analysis scripting |authors=Philip J Guo, Dawson Engler |url=https://dl.acm.org/doi/abs/10.1145/2001..."

A few billion lines of code later: using static analysis to find bugs in the real world

2021-10-11T20:13:21Z

Charmonium:

{{Summary
|title=A few billion lines of code later: using static analysis to find bugs in the real world
|authors=Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, Dawson Engler
|url=https://dl.acm.org/doi/abs/10.1145/1646353.1646374
|tags=software engineering
|summary=Authors built a static bug-finding tool, Coverity, and apply it in practice.
|journal=Communications of the ACM, Volume 53, Issue 2
|pub_date=2010/02
|doi=10.1145/1646353.1646374
|subject=Computer Science
}}

* Coverity has false-positives (coverity flags code that is not erroneous) and false-negatives (some errors are not flagged).
* "Circa 2000, unsoundness [having false-negatives] was controversial in the research community, though it has since become almost a de facto tool bias for commercial products and many research projects."
* Sales strategy: Send an engineer and salesperson to the client, run the tool on their codebase, the engineer helps with "unique" client configurations and helps educate the client. This is a tough hurdle for the system, because no time to cherry-pick results and massage configuration.
* Educating users is difficult:
** Initially the tool used the output of Make to learn how to compile source-code, and where the source-code was.
*** Clients have bespoke build systems and might not even know about Make.
**** Later on, the tool intercepted syscalls to learn the compiler invocation and context. But this needs the commandline.
***** Client developers don't necessarily build from the commandline.
* Clients are often risk-averse to change, so you have to work around broken software instead of fixing it.
* Compilers deviate from language standard intentionally and otherwise.
* Often clients want to buy their tool, but restrict their source code.
* Some clients don't believe that bugs the tool finds are real bugs. They often depend on non-standard behavior.
** Some clients try to argue with you, often emotionally. It's best not to argue; try to make a meeting with their peers.
* Upgrading tool to catch more bugs negatively effects metrics for managers.
* Determinism is more important to users than finding more bugs.
* Deep analysis can catch bugs, but those are hard to explain to users (e.g. races).
* Checking for trivial bugs is still useful. Given enough code, they will occur.

A few billion lines of code later: using static analysis to find bugs in the real world

2021-10-11T18:12:31Z

Charmonium:

A few billion lines of code later: using static analysis to find bugs in the real world

2021-10-09T06:05:52Z

Charmonium:

A few billion lines of code later: using static analysis to find bugs in the real world

2021-10-09T01:01:58Z

Charmonium:

{{Summary
|title=A few billion lines of code later: using static analysis to find bugs in the real world
|authors=Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, Dawson Engler
|url=https://dl.acm.org/doi/abs/10.1145/1646353.1646374
|tags=software engineering
|summary=Authors built a static bug-finding tool, Coverity, and apply it in practice.
|journal=Communications of the ACM, Volume 53, Issue 2
|pub_date=2010/02
|doi=10.1145/1646353.1646374
|subject=Computer Science
}}

* Coverity has false-positives (coverity flags code that is not erroneous) and false-negatives (some errors are not flagged).
* "Circa 2000, unsoundness [having false-negatives] was controversial in the research community, though it has since become almost a de facto tool bias for commercial products and many research projects."
* Sales strategy: Send an engineer and salesperson to the client, run the tool on their codebase, the engineer helps with "unique" client configurations and helps educate the client. This is a tough hurdle for the system, because no time to cherry-pick results and massage configuration.
* Educating users is difficult:
** Initially the tool used the output of Make to learn how to compile source-code, and where the source-code was.
*** Clients have bespoke build systems and might not even know about Make.
**** Later on, the tool intercepted syscalls to learn the compiler invocation and context. But this needs the commandline.
****** Client developers don't necessarily build from the commandline.
* Clients are often risk-averse to change, so you have to work around broken software instead of fixing it.

A few billion lines of code later: using static analysis to find bugs in the real world

2021-10-08T20:37:55Z

Charmonium: Created page with "{{Summary |title=A few billion lines of code later: using static analysis to find bugs in the real world |authors=Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Set..."

Code Sharing Is Associated with Research Impact in Image Processing

2021-09-30T19:30:48Z

Charmonium:

{{Summary
|title=Code Sharing Is Associated with Research Impact in Image Processing
|authors=Patrick Vandewalle
|url=10.1109/MCSE.2012.63
|tags=academic software
|summary=As the title reads, the author found that code sharing correlates with research impact in their field by sampling papers from mainstream journals in the mid-2000s.
|journal=Computing in Science & Engineering
|pub_date=2012/07
|doi=10.1109/MCSE.2012.63
|subject=Computer Science
}}

= Background =
* Most papers in signal processing do not release source code. This makes it hard to reproduce.
* It is widely agreed upon that this _shouldn't_ be the case (according to the author's), but it is.
* Academics have little explicit incentive to release their source code.
* There are implicit incentives: notoriety through downloads, feedback from users, ease of collaboration, usage by other researchers, and '''citations'''.
* '''Thesis:''' correlation between releasing code and citations.
** Causality can be determined by a "controlled experiment" which is left as future work.
* Releasing source code is neither necessary nor sufficient for reproducibility. Open source can be unreproducible if it depends on the system; reproducible work which does not require code or describes it very carefully does not to release code.
** But they are correlated.
* See also:
** Peace research: N.P. Gleditsch and H. Strand, “Posting Your Data: Will You Be Scooped or Will You Be Famous?” Int’l Studies Perspectives, vol. 4, no. 1, 2003, pp. 89–97.
** Cancer research: H.A. Piwowar, R.S. Day, and D.B. Fridsma, “Sharing Detailed Research Data Is Associated with Increased Citation Rate,” PLoS ONE, vol. 2, no. 3, 2007, p. e308; [https://www.plosone.org/article/info:doi/10.1371/journal.pone.0000308]
** Astronomy: E.A. Henneken and A. Accomazzi, “Linking to Data—Effect on Citation Rates in Astronomy,” Proc. Astronomical Data Analysis Software and Systems, Astronomical Soc. of the Pacific, 2011; [https://arxiv.org/pdf/1111.3618v1.pdf]
** Open access: S. Lawrence, “Free Online Availability Substantially Increases a Paper’s Impact,” Nature, vol. 411, no. 6837, 2001, p. 521; [https://www.nature.com/nature/journal/v411/n6837/full/411521a0.html]

= Methodology =
== First study ==
* IEEE Transactions on Image Processing 2004 -- 2006, 645 papers
* Searched for source by hand.
** Roughly 10% had source.
* Use Google Scholar for citation numbers. Web of Science tends to be more selective when counting citations.
** Long-tail of rarely cited papers, so median is better than mean.
* Look for difference in median (median citation count no source = 25, with source = 76 in 2004).
* [https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test Mann-Whitney U-test] says the difference is statistically significant.
** The null hypothesis is still rejected even if one removes half of the papers with source code from consideration; result is robust to ignoring some the "superstar" reproducible papers.
* '''Conclusion:''' Publications with source are more often cited.

== Second study ==
* Only best-cited papers from 2004 -- 2008, IEEE Transactions on Image Processing 2004 (TIP), IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), and IEEE Transactions on Signal Processing (TSP)
* Count those which have source code available.
** Roughly 90% had source with the exception of TSP, due to its theoretical nature.
* TIP and TPAMI have a much greater proportion of their best-cited papers with source code than TSP.
* '''Conclusion:''' Best-cited papers release their source code

= Other considerations =
* The lifetime of source repo can be cut short if the website host is phased out.
* Industry research may have problems releasing source; they should still make studies reproducible internally.

Developers Perception of Peer Code Review in Research Software Development

2021-09-30T19:16:57Z

Charmonium:

{{Summary
|title=Developers Perception of Peer Code Review in Research Software Development
|authors=Nasir U. Eisty, Jeffrey C. Carver
|url=https://arxiv.org/abs/2109.10971
|tags=software engineering
|journal=arXiv
|pub_date=2021/09/22
|arxiv=2109.10971
|subject=Computer Science
}}
= Background =
* Software is important for research.
* Research software engineers should follow standard software practices.
* However, these practices differ from industry.
** Risks due to exploration.
** Constantly changing requirements.
** Complex communication or I/O patterns.
** Need highly-specialized knowledge
** Larger scale of single executions
** Complex software due to modeling complex phenomena
** Different goals, knowledge, skills than industry devs
* Tests are hard because no oracle, large number of parameters, and legacy code.
== Solution ==
* On the other hand, peer code review could work
** Numerous benefits
*** Reviewers suggest comments that improve code quality
*** Authors more likely to make code readable
*** Spreads out knowledge of the change
*** Community building
** Google developers expect four key themes from peer code review: education, maintaining norms, gatekeeping, and accident prevention.
** Microsoft dev spends 15--25% of time reviewing.
** 25% of comments on improve core functionality
= Study =
* RQ1: How do research software developers perform peer code review?
* RQ2: What effect does peer code review have on research software?
* RQ3: What difficulties do research software developers face with peer code review?
* RQ4: What improvements to the peer code review process do research software developers need?
= Methodology =
== Survey Design ==
* Questions from prior literature on peer code review.
== Pilot Interviews ==
* Pilot interviews suggested ways of revising the questions, and develop multiple-choice answers.
* Interview audience: 13 from NCSA, 9 from Einstein Toolkit Workshop.
* [https://en.wikipedia.org/wiki/Convenience_sampling "Convenience sampling"]
== Survey ==
* Solicitation:
** Emailed mailing listsprojects associated with interviewers
** [https://se4science.github.io/SE-CODESE17/ 2017 International Workshop on Software Engineering for High Performance Computing in Computational and Data-Enabled Science and Engineering],
** Emailed mailing list for UK RSEs,
** Pinged RSE Slack channel,
** Advertised in [http://bssw.io Better Scientific Software] newsletter,
** All excluding the pilot interviewees.
== Data Analysis ==
* Valid response := answer all quantitative and at least one qualitative
** What motivates requiring one qualitative answer?
* Coded these to qualitative answers individually, and then merged codes, resolving differences case-by-case.
= Results =
* Most respondents are financially compensated for their participation, have been on both sides of code review, and more than five years of experience.
== RQ1: Code review details ==
* Most respondents spend less than 5 hours per review, half spend 1 to 5 hours.
* Most requests get a response within 3 days, 40% within 1 day.
* Most commits goes through review.
* Most of reviews are resolved within a month, half within a week.
* Number of LoC and number of reviewers varies widely.
* Common criteria when deciding reviews: coding standards, domain knowledge are roughly tied followed by functionality, correctness, time, tests, documentation, always-accept
* Common mistakes corrected during review: code mistakes, design, style, testing, documentation, performance, readability, maintainability.
== Positive experiences about code review ==
* Knowledge sharing, improved code quality, helpful feedback, positive feeling, problems identified
* "In a big project it is rare that anyone understands the whole picture... It [code review] can lead to more complete understanding of the task."
* "It [code review] leads to design discussions happening that would not have happened otherwise."
* "It makes the team more knowledgeable about what work is."
* "People found mistakes in code that I wrote, that I would have missed and only found out about much further on the validation process."
* Code review results in "much better code and a better understanding of different parts of the code."
== Negative experiences about code review ==
* Takes too long, requestors misunderstand criticism, disagreements, bottleneck, hard to find reviewers, difficult task, unresponsive author
* "it [peer code review] can be long and time consuming for very small changes, as the process must be followed for even a single character change if it affects results."
* There are also problems when the "review process gets stalled while nit-picking irrelevant details."
* "Sometimes people get annoyed when they get feedback especially if they think they are experts"
== RQ2: Impact of code review on research software ==
* By a large margin, respondents strongly agreed code review is important for their project.
** This could be due to selection bias.
* Impacts: improves code quality followed by knowledge sharing
* Why does code review improve code quality: correctness followed by a tie between improves readability, more eyes, and better maintainability.
* On correctness: "If you’ve written code yourself, it’s hard to see the assumptions you’ve made. Others can spot these and ask you to clarify, also spot your mistakes"
* On readability: "make[ing] the codebase more uniform and improves the quality of the code"
== RQ3: Difficulties research software developers face with code review ==
* Difficulties: understanding code, understanding system, administrative issues
* Barriers: Finding time followed distantly by phrasing comments, finding the right people, participation, developer egos, takes too long
== RQ4: What improvements do research software developers need? ==
* Formalizing process, followed by tooling, more people, better incentives, more training, more time
* Formalizing process: "a more formal structure of at least one science review followed by one technical review. It’s currently a bit of a free-for-all"
* Tooling: branching VCS and automatic analysis
= Threats to validity =
* Participants might not know what certain terms mean, but authors think they do.
* Human-perception can be wrong, but there is no better source of truth.
* Perhaps the sample is not representative of the population.
** Those wliling to answer a survey on code review are more likely to be aware of it.
* Participants may have misunderstood questions, but authors tried to be clear.
= Conclusion =
* Similar results to commercial software engineering, despite differences in research context.
* Code review largely beneficial, but could benefit from explicit process.
* Authors plan to raise awareness of code review, its flaws, and its benefits, within community.
== References ==
* [https://figshare.com/articles/dataset/_/14736468/0 Raw data], possibly dead link?
* [https://figshare.com/articles/journal_contribution/Contemporary_Peer_Code_Review_in_Research_Software/7761989 Slides]

Research Software Engineering reading list

2021-09-28T23:06:37Z

Charmonium:

= Institutions and Incentives =
* [[Addressing Research Software Sustainability via Institutes]]
* [[Research Software Development & Management in Universities: Case Studies from Manchester's RSDS Group, Illinois' NCSA, and Notre Dame's CRC]]

= Current Practices =
* Practices which can be improved
** [[Technical Debt in Computational Science]]
** [[Troubling Trends in Scientific Software Use]] (ecology)
* Pure observations
** [https://zenodo.org/record/14809#.YNdUKDpOkUE UK Research Software Survey (dataset)]
** [[Vertical Integration]]
** [[Developers Perception of Peer Code Review in Research Software Development]]
* Improved practices
** [[The Research Software Engineer]]
** [[Ten Simple Rules for the Open Development of Scientific Software]]
** [[Reducing Technical Debt with Reproducible Containers]]
** [[A Workflow for Increasing the Quality of Scientific Software (in Computational Science and Engineering)]]
** [[Mining Development Data to Understand and Improve Software Engineering Processes in HPC Projects]]
** [[Easing the burden of code review]]
** [[Software Engineering Challenges and Best Practices for Multi-Institutional Scientific Software Development]]
** [[Some Simple Guidelines for Effective Data Management]]

= Famous Bugs =
* [[A Scientist's Nightmare: Software Problem Leads to Five Retractions]]

= Related Lists =
* [[Open Academia reading list]]
* [[Reproducibility reading list]]
* [[Software Engineering reading list]]
* [[Computer Science reading list]]

Some Simple Guidelines for Effective Data Management

2021-09-28T23:05:22Z

Charmonium: Created page with "{{Summary |title=Some Simple Guidelines for Effective Data Management |authors=Elizabeth T. Borer, Eric W. Seabloom, Matthew B. Jones, Mark Schildhauer |url=https://esajournal..."

{{Summary
|title=Some Simple Guidelines for Effective Data Management
|authors=Elizabeth T. Borer, Eric W. Seabloom, Matthew B. Jones, Mark Schildhauer
|url=https://esajournals.onlinelibrary.wiley.com/doi/full/10.1890/0012-9623-90.2.205
|tags=data management
|summary=# Use a scripted program for analysis.
#* NB: use a workflow management system to run your scripts. This balances speed of computation with reproducibility.
#* "GUI-driven" analysis is harder to scrutinize and reproduce.
# Store data in non-proprietary software formats (e.g., comma delimited text file, .csv).
#* NB: CSV is not space-efficient for large amounts of numeric data. HDF5, still non-proprietary, may be more appropriate in that case.
# Store data in non-proprietary hardware formats.
#* NB: Archival websites are even better than physical storage.
# Store an uncorrected data; make corrections within a scripted language.
#* Make it a read-only file.
#* This way, you can revert mistakes in the analysis.
# Use descriptive names for your data files
#* NB: Other authors recommend not parsing filenames to get metadata. Often there is too much metadata to fit in a filename, and that metadata is not necessarily unique.
#* No spaces in filenames.
# Include a “header” line that describes the variables as the first line in the table.
#* NB: The first row becomes the "name" of the column in Pandas, so if you use a descriptive short-name there, write a sentence describing each column in the second row.
# Use plain ASCII text for your file names, variable names, and data values.
#* NB: I consider this a little out-of-date: UTF-8 is the new de facto standard, and it is backwards compatible with ASCII, so UTF-8-unaware programs will render most of the text properly and emojibake over special characters.
# When you add data to a database, try not to add columns; rather, design your tables so that you add only rows.
#* NB: I believe this is more commonly known as [https://en.wikipedia.org/wiki/First_normal_form First Normal-Form].
# All cells within each column should contain only one type of information (i.e., either text, numerical, etc.).
# Record a single piece of data (unique measurement) only once
#* NB: This is [https://en.wikipedia.org/wiki/Third_normal_form Third Normal-Form].
# Record full information about taxonomic names.
# Record full dates, using standardized formats.
#* [https://en.wikipedia.org/wiki/ISO_8601 ISO 8601]
# Always maintain effective metadata.
#* NB: This metadata should be machine-readable, e.g. in YAML.
|journal=Ecological Society of America Bulletin
|pub_date=2009/05/01
|doi=10.1890/0012-9623-90.2.205
}}

Reproducibility reading list

2021-09-27T21:53:46Z

Charmonium: /* Reproducibility in software */

== Reproducibility in general ==
* [[Why Most Published Research Findings Are False]] (medicine)
* [[Reproducible Research Practices and Transparency across the Biomedical Literature]] (biomed)
* [[Estimating the reproducibility of psychological science]] (psychology)

== Reproducibility in software ==
* [[Repeatability in computer systems research]]
* [[ReScience C: A Journal for Reproducible Replications in Computational Science]]
* [[Reproducible and User-Controlled Software Environments in HPC with Guix]]
* [[Artifact Evaluation: Is it a Real Incentive?]]
* [[Sciunits: Reusable Research Objects]]
* [[Reproducibility vs. Replicability: A Brief History of a Confused Terminology]]
* [[Reproducibility PI Manifesto]]

== Related fields ==
* [[Open Academia reading list]]
* [[Research Software Engineering reading list]]