NSF Grant Backs funcX — A Smart, Automated Delegator for Computational Research

Computational scientific research is no longer one-size-fits-all. The massive datasets created by today’s cutting-edge instruments and experiments — telescopes, particle accelerators, sensor networks and molecular simulations — aren’t best processed and analyzed by a single type of machine. For faster and more efficient discovery, data can be chopped up and shipped to specialized resources, including supercomputers, campus clusters, cloud data centers, and “accelerators” optimized for specific tasks such as machine learning or visualization.

But delegating chunks of data and analysis functions to their ideal destination isn’t trivial. A team led by UChicago CS researchers Ian Foster and Kyle Chard and Daniel S. Katz of the National Center for Supercomputing Applications at the University of Illinois seeks to streamline the process with funcX, a new distributed “function-as-a-service” (FaaS) platform that makes it easier for researchers to easily and automatically delegate their computational workload. With a pair of grants to the Universities of Chicago and Illinois from the National Science Foundation totalling $3.14 million, the team will work with several large science projects and cyberinfrastructure partners to build and test this new system for “computationally fluid” research.

“Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences,” said Ian Foster, Director of Argonne’s Data Science and Learning Division, Argonne Senior Scientist and Distinguished Fellow and the Arthur Holly Compton Distinguished Service Professor of Computer Science at the University of Chicago. “funcX makes it easy for scientists to run computations wherever it makes the most sense and move computations between resources. It will enable new science, reduce barriers and democratize access to advanced cyberinfrastructure, and enable researchers to compute wherever is most efficient.”

The scientific software used to process and extract discoveries from experimental data is typically made up of tens to thousands of smaller functions, blocks of code that handle individual jobs in the long pipeline of data analysis. These programs can be run in their entirety on a single system — be it a laptop, a campus cluster, or a supercomputer — but that uniform approach may not be optimal. Some complex tasks may need to run on high-performance computing resources, but some specialized functions may be better served by GPU accelerators, and the more routine jobs could be tackled by small, energy-efficient computers. 

Many obstacles prevent scientists from splitting their applications in this manner. Software is usually designed to run on a single type of machine or system, even if in parallel, and it is difficult to write code that splits off functions and sends them to different destinations. Beyond the programming, additional barriers exist for moving data to the desired computational endpoint, obtaining the proper security authorizations, and scheduling usage between different remote resources based on availability. 

funcX solves this problem by adapting the serverless “function-as-a-service” (FaaS) model for the specialized needs and resources of the science and engineering research community. Many of today’s mobile apps and Internet of Things devices use this approach for their sporadic computational demand — offloading more intensive tasks that can’t be handled on their smaller, local computers. 

“The research ecosystem comprises a range of existing systems: from HPC clusters to clouds and supercomputers,” said Chard, Research Assistant Professor at UChicago CS. “Researchers have varied workloads, large amounts of distributed data, and dynamic collaborations, so there isn't a one-size-fits-all mapping from workloads to systems. funcX will make it easy to distribute these functions to different computing systems without needing to think about their differences, and will integrate with the scientific ecosystem for authentication, data management, and heterogeneous computing infrastructure.”

The funcX platform builds upon two existing research technologies: Globus, a research data management platform created at the University of Chicago and Argonne National Laboratory, and Parsl, a Python library for executing parallel workflows created at the University of Chicago, Argonne National Laboratory, and the University of Illinois. funcX will also use Amazon Web Services for hosting management services, and integrate cyberinfrastructure from campuses and national laboratories, such as Blue Waters, Stampede2, and XSEDE

The project will work with a wide range of scientific partners studying urban science, materials science, quantum chemistry, neuroanatomy, cosmology, and other data-intensive subjects. These partners will provide use cases and testbeds for the funcX platform, working with the designers to create powerful, flexible features and an ecosystem of users and computational endpoints. 

“We've got an amazing group of collaborators from different science domains, research computing centers and national cyberinfrastructure providers, and NSF software institutes that will help shape this project,” Chard said. “We see this engagement as vital for not only understanding needs across a range of domains but also for piloting the system and demonstrating value in their domain. This will help us ensure that funcX is not only valuable for our partners but also more broadly to all researchers.”

This project is a collaboration between the Universities of Chicago and Illinois, following their successful partnership in developing Parsl. 

“Since its creation by NSF and the state of Illinois in 1986, NCSA has been a leader in working with and supporting scientific and engineering communities through the development, deployment, and use of new computing and software technologies,” said Katz, Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications (NCSA) at the University of Illinois. “This project exemplifies our effort to develop leading-edge software that enables new research on our systems.”

Follow the project online at funcx.org.

Related News

More UChicago CS stories from this research area.
In the News

Professors Rebecca Willett and Ben Zhao Discuss the Future of AI on Public Radio

Jan 26, 2023
UChicago CS News

UChicago Launches Transform Accelerator for Data Science & Emerging AI Startups

Jan 19, 2023
UChicago CS News

Professor Heather Zheng Named ACM Fellow

Jan 18, 2023
Two students looking at a wearable device
UChicago CS News

High School Students Find Their Place in Computing Through Wearables Workshop

Jan 13, 2023

Ian Foster – Better Information Faster: Programming the Continuum

Jan 06, 2023
UChicago CS News

Q&A: Ian Foster on Receiving the 2023 IEEE Internet Award

Jan 06, 2023
UChicago CS News

Assistant Professor Robert Rand Receives Air Force Young Investigator Grant

Dec 19, 2022
Professor Fred Chong advising students
UChicago CS News

Prof. Fred Chong Reappointed to National Quantum Initiative Advisory Committee

Dec 13, 2022
UChicago CS News

Professor Fred Chong Named IEEE Fellow

Dec 09, 2022
UChicago CS News

Associate Professor Diana Franklin Named ACM Distinguished Member

Dec 07, 2022
In the News

UChicago CS Researchers Share in Special Prize on COVID-19 Research

Dec 01, 2022
UChicago CS News

The Computing Pipeline: A Foundation for Diversifying Computer Science

Nov 28, 2022
arrow-down-largearrow-left-largearrow-right-large-greyarrow-right-large-yellowarrow-right-largearrow-right-smallbutton-arrowclosedocumentfacebookfacet-arrow-down-whitefacet-arrow-downPage 1CheckedCheckedicon-apple-t5backgroundLayer 1icon-google-t5icon-office365-t5icon-outlook-t5backgroundLayer 1icon-outlookcom-t5backgroundLayer 1icon-yahoo-t5backgroundLayer 1internal-yellowinternalintranetlinkedinlinkoutpauseplaypresentationsearch-bluesearchshareslider-arrow-nextslider-arrow-prevtwittervideoyoutube