A critical, but oft-neglected, piece of the scientific method is reproducibility, the confirmation of past results through repeating experiments. As science has grown larger and more technically complex, these reproductions become more difficult and expensive, particularly with rising usage of big data and high-performance computing.

With a $1.5 million grant from the National Science Foundation (NSF), University of Chicago scientists will lead a new research coordination network to incentivize reproducibility in computer science. Called REPETO, the esperanto word for “repeat,” the new network’s leadership also includes the University of California, Santa Cruz and New York University.

Kate Keahey, Senior Scientist at The University of Chicago Consortium for Advanced Science and Engineering (UChicago CASE) and Argonne National Laboratory.

UChicago CS affiliates on the grant include lead principal investigator Kate Keahey, Senior Scientist at The University of Chicago Consortium for Advanced Science and Engineering (UChicago CASE) and Argonne National Laboratory, and Haryadi Gunawi, associate professor of computer science.

Keahey emphasizes that the project will focus on promoting “practical reproducibility,” new technologies and incentives that encourage rerunning experiments for education and research.

“For many years, a huge barrier to repeating somebody else’s experiment was that you might not have the same level of access to the same type of hardware that they did their experiment on” Keahey said. “The fact that the NSF now funds open testbeds supporting a broad range of computer science experiments means that this obstacle goes away – but we still face barriers in the shape of cost of repeating an experiment and lack of community practices. Our objective is to make reproducibility practical by both lowering its cost and aligning it with mainstream research and education activities.”

NSF-funded platforms such as Keahey’s Chameleon, a cloud computing testbed, FABRIC, and PAWR now provide researchers with the resources they need to reproduce large-scale computational experiments. Many computer science conferences and publications now also require scientists to package their research – including data, methods, and the computing environment – for easy replication by outside observers.

But incentivizing the use of those resources in a field that rewards novelty over verification requires new strategies. The REPETO steering committee will gather representatives from NSF testbeds, conference organizers, and educators to promote the use of shared infrastructure and electronic artifacts, create new educational methodologies and programs, and work with existing conferences and workshops to facilitate opportunities for creating reproducible research.

One of these programs includes a “Summer of Reproducibility,” scheduled to start in 2023 and inspired by Google’s popular Summer of Code, but one where students would work on activities that reproduce published research. The project will also organize hackathons around reproducibility, and create new curricula that draw upon testbed resources and make it possible for students to recreate large-scale experiments in their coursework.

“This approach will be a much more engaging way to learn and to explore existing research, as it draws you into the ongoing scientific debate,” Keahey said. “We’re going to produce examples on how to teach using reproducibility, create sets of best practices and methodology guides, and repositories of experiments packaged for reproducibility.”

The REPETO grant was funded as part of the NSF Findable Accessible Interoperable Reusable (FAIR) Open Science Research Coordination Networks (FAIROS RCN) initiative. The $12.5 million program funded 10 new project groups that “advance the means by which investigators can share information and ideas, coordinate ongoing or planned research activities, foster synthesis and new collaborations, develop community standards, and in other ways advance science and education through communication and sharing of research products.”

Related News

More UChicago CS stories from this research area.
In the News

UChicago CS Researchers Share in Special Prize on COVID-19 Research

Dec 01, 2022
UChicago CS News

UChicago’s Parsl Project Pivots to Sustainability and Community with New Grants

Nov 17, 2022
UChicago CS News

UChicago/Argonne Computer Scientist Ian Foster Receives ACM/IEEE Ken Kennedy Award

Sep 07, 2022
UChicago CS News

UChicago/Argonne Researchers Will Cultivate AI Model “Gardens” With $3.5M NSF Grant

Aug 30, 2022
UChicago CS News

EPiQC Post-Doc Pens Op-Ed on Potential of Quantum Computing for Chemistry

Jun 24, 2022
UChicago CS News

Two Incoming UChicago CS PhD Students Receive Department of Energy Fellowship

Jun 16, 2022
UChicago CS News

DSI Summer Lab Returns In-Person With 49 Students From Across the U.S.

Jun 14, 2022
In the News

Chameleon Cloud Testbed Hosts HPC Competition, Gains New Associates and Users

Mar 14, 2022
UChicago CS News

$3.25m DOE Grant Funds UChicago/Argonne Research on AI Models of Physics Simulations

Sep 27, 2021
UChicago CS News

Scientists Use Artificial Intelligence to Detect Gravitational Waves

Jul 16, 2021
UChicago CS News

C3.ai Digital Transformation Institute Announces CFP to Advance AI for Energy and Climate Security

Feb 02, 2021
UChicago CS News

UChicago CS Professor Rick Stevens Named ACM Fellow

Jan 16, 2021
arrow-down-largearrow-left-largearrow-right-large-greyarrow-right-large-yellowarrow-right-largearrow-right-smallbutton-arrowclosedocumentfacebookfacet-arrow-down-whitefacet-arrow-downPage 1CheckedCheckedicon-apple-t5backgroundLayer 1icon-google-t5icon-office365-t5icon-outlook-t5backgroundLayer 1icon-outlookcom-t5backgroundLayer 1icon-yahoo-t5backgroundLayer 1internal-yellowinternalintranetlinkedinlinkoutpauseplaypresentationsearch-bluesearchshareslider-arrow-nextslider-arrow-prevtwittervideoyoutube