New Assistant Professor Rana Hanocka Combines AI, 3D, and Computer Graphics

Artificial intelligence has gotten better and better at understanding the visual world, classifying two-dimensional images and video without human intervention. But the application of neural networks, deep learning and other AI approaches to create new graphics and work with three-dimensional data is still in its earliest stages. The production of 3D graphics remains accessible only to experts, and the application of AI to analyze 3D data and produce new images is still limited in its scope and possibilities.

Rana Hanocka, a new assistant professor who joined UChicago CS in the summer of 2021, aims to democratize this technology, building AI capabilities for 3D data that transform aspiration into reality. Her research creates new frameworks and applications that allow users to more easily and expansively work with 3D geometric data: transforming two-dimensional images into 3D structures, changing features with descriptive text, and generating realistic synthetic data for the training of computer vision models.

Her research group 3DL — an acronym for “3D Deep Learning,” pronounced “threedle” — produces new tools that can help artists, designers, architects, and hobbyists create new 3D objects with limited technical expertise. For example, a user could take a 3D model of a table, chair or any other object and manipulate its appearance and texture through text inputs: “crochet,” “wicker,” “brick,” or more fantastical choices such as “cactus,” or “astronaut.”

“We’re interested in how to make things that work automatically, while being more intuitive and easier to understand and use,” Hanocka said. “We design tools for manipulating 3d objects, creating theatrics, and animating them. And we want to make our research available, enabling people to use these tools so that people can build off of and extend them.”

Hanocka joined the University of Chicago after completing her Ph.D. at Tel Aviv University in Israel, where she studied with Daniel Cohen-Or and Raja Giryes. She gravitated to studying computer graphics as a means of combining her interests in mathematics and visual art, seeing the potential of new AI approaches in this area.

“I think we have seen an AI explosion in structured data such as text and images, but we haven’t really seen that same success yet in unstructured 3D data and shape analysis,” Hanocka said. “So my hope is to be able to achieve that same success in this domain where we don’t yet see the same kind of amazing results.”

One obstacle is the relative shortage of 3D data compared to the deep well of 2D images that can be harvested from the Internet. Training neural networks requires lots of data, and several different research groups have developed methods to convert 2D imagery into 3D models. Hanocka’s research has focused on transforming point clouds into polygonal meshes – an efficient representation of 3D shapes.

By then designing neural networks that work with these mesh objects, Hanocka enables automated analyses and manipulations of 3D objects that previously required difficult and time-consuming software. These edits range from simple changes in characteristics such as color and lighting to 3D-specific alterations of texture and geometry. The networks also enable users to generate new data from limited examples, such as a particular motion sequence, opening up new graphics capabilities for virtual reality, animation, and other media. These new datasets could also help alleviate the shortage of 3D training data for computer vision models necessary for autonomous vehicles and other technologies.

In addition to her research, Hanocka brings new courses in computer graphics to UChicago CS. In Fall 2021, she taught a topics course on “Machine Learning: 3D Geometry Processing and Computer Vision with Deep Neural Networks.” This spring, she will revive and reinvent Introduction to Computer Graphics, a subject of high demand among UChicago students.

And it’s the opportunity to work with those students that attracted Hanocka to the University of Chicago, in addition to the rapidly-growing faculty, the university’s ties to national laboratories Argonne and Fermilab, and the experience of living in Chicago.

“The undergraduate students at the University are really top of the line,” Hanocka said. “I think that they can bring a different perspective and it’s a big strength that can be leveraged in research. When you have such talent, it’s a real asset.”