A critical, but oft-neglected, piece of the scientific method is reproducibility, the confirmation of past results through repeating experiments. As science has grown larger and more technically complex, these reproductions become more difficult and expensive, particularly with rising usage of big data and high-performance computing.

With a $1.5 million grant from the National Science Foundation (NSF), University of Chicago scientists will lead a new research coordination network to incentivize reproducibility in computer science. Called REPETO, the esperanto word for “repeat,” the new network’s leadership also includes the University of California, Santa Cruz and New York University.

Kate Keahey, Senior Scientist at The University of Chicago Consortium for Advanced Science and Engineering (UChicago CASE) and Argonne National Laboratory.

UChicago CS affiliates on the grant include lead principal investigator Kate Keahey, Senior Scientist at The University of Chicago Consortium for Advanced Science and Engineering (UChicago CASE) and Argonne National Laboratory, and Haryadi Gunawi, associate professor of computer science.

Keahey emphasizes that the project will focus on promoting “practical reproducibility,” new technologies and incentives that encourage rerunning experiments for education and research.

“For many years, a huge barrier to repeating somebody else’s experiment was that you might not have the same level of access to the same type of hardware that they did their experiment on” Keahey said. “The fact that the NSF now funds open testbeds supporting a broad range of computer science experiments means that this obstacle goes away – but we still face barriers in the shape of cost of repeating an experiment and lack of community practices. Our objective is to make reproducibility practical by both lowering its cost and aligning it with mainstream research and education activities.”

NSF-funded platforms such as Keahey’s Chameleon, a cloud computing testbed, FABRIC, and PAWR now provide researchers with the resources they need to reproduce large-scale computational experiments. Many computer science conferences and publications now also require scientists to package their research – including data, methods, and the computing environment – for easy replication by outside observers.

But incentivizing the use of those resources in a field that rewards novelty over verification requires new strategies. The REPETO steering committee will gather representatives from NSF testbeds, conference organizers, and educators to promote the use of shared infrastructure and electronic artifacts, create new educational methodologies and programs, and work with existing conferences and workshops to facilitate opportunities for creating reproducible research.

One of these programs includes a “Summer of Reproducibility,” scheduled to start in 2023 and inspired by Google’s popular Summer of Code, but one where students would work on activities that reproduce published research. The project will also organize hackathons around reproducibility, and create new curricula that draw upon testbed resources and make it possible for students to recreate large-scale experiments in their coursework.

“This approach will be a much more engaging way to learn and to explore existing research, as it draws you into the ongoing scientific debate,” Keahey said. “We’re going to produce examples on how to teach using reproducibility, create sets of best practices and methodology guides, and repositories of experiments packaged for reproducibility.”

The REPETO grant was funded as part of the NSF Findable Accessible Interoperable Reusable (FAIR) Open Science Research Coordination Networks (FAIROS RCN) initiative. The $12.5 million program funded 10 new project groups that “advance the means by which investigators can share information and ideas, coordinate ongoing or planned research activities, foster synthesis and new collaborations, develop community standards, and in other ways advance science and education through communication and sharing of research products.”

Related News

More UChicago CS stories from this research area.
No Name

Five UChicago CS students named to Siebel Scholars Class of 2024

Oct 02, 2023
No Name

UChicago Scientists Make New Discovery Proving Entanglement Is Responsible for Computational Hardness In Quantum Systems

Jul 25, 2023
No Name

Chicago Magazine on Aurora, “The Computer That Will Change Everything”

Feb 01, 2023
Video

Ian Foster – Better Information Faster: Programming the Continuum

Jan 06, 2023
No Name

Q&A: Ian Foster on Receiving the 2023 IEEE Internet Award

Jan 06, 2023
No Name

UChicago CS Researchers Share in Special Prize on COVID-19 Research

Dec 01, 2022
No Name

UChicago’s Parsl Project Pivots to Sustainability and Community with New Grants

Nov 17, 2022
No Name

UChicago/Argonne Computer Scientist Ian Foster Receives ACM/IEEE Ken Kennedy Award

Sep 07, 2022
No Name

UChicago/Argonne Researchers Will Cultivate AI Model “Gardens” With $3.5M NSF Grant

Aug 30, 2022
No Name

EPiQC Post-Doc Pens Op-Ed on Potential of Quantum Computing for Chemistry

Jun 24, 2022
No Name

Two Incoming UChicago CS PhD Students Receive Department of Energy Fellowship

Jun 16, 2022
No Name

DSI Summer Lab Returns In-Person With 49 Students From Across the U.S.

Jun 14, 2022
arrow-down-largearrow-left-largearrow-right-large-greyarrow-right-large-yellowarrow-right-largearrow-right-smallbutton-arrowclosedocumentfacebookfacet-arrow-down-whitefacet-arrow-downPage 1CheckedCheckedicon-apple-t5backgroundLayer 1icon-google-t5icon-office365-t5icon-outlook-t5backgroundLayer 1icon-outlookcom-t5backgroundLayer 1icon-yahoo-t5backgroundLayer 1internal-yellowinternalintranetlinkedinlinkoutpauseplaypresentationsearch-bluesearchshareslider-arrow-nextslider-arrow-prevtwittervideoyoutube