When the Unknown is a Shape
Broadening the computational toolbox, Justin Solomon investigates the links from a geometric idea to its mathematical and engineering counterparts.
Justin Solomon, X-Consortium Career Development Assistant Professor in the Department of Electrical Engineering and Computer Science (EECS) and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL), leads the Geometric Data Processing Group, which seeks to understand how the mathematics of differential geometry, probability, optimization, and metric geometry come into play when designing practical and efficient algorithms for computational challenges.
Solomon, who was named in Forbes 30 Under 30, did some of his earliest research at Pixar Research, where he worked on computational problems such as cloth simulation. “An artist looking for the technical tools to design a scene or a special effect shouldn’t have to animate every wrinkle and motion of a piece of cloth manually,” he says. “At the same time, they should have enough control to get the look that they want.” Swishing of the kitchen smocks worn by the rats in Ratatouille, for example, requires very low-level physics and number crunching coupled with high-level artistic control. Some of these creative processes, Solomon says, have led to developments in the worlds of physical simulation, partial differential equations, geometry processing, and computer-aided engineering. Other developments, such as those in fluid simulation, wind simulation and cloth originating in the computer graphics world, are feeding back into pipelines for designing cars and making household objects. “There are so many sources of challenging mathematical and engineering problems end-users are facing that there’s no reason for us in academia to invent new ones,” says Solomon.
Combining expertise in geometry, machine learning, and underlying mathematics allows Solomon and his group to tackle more challenging problems like shape and object recognition. “Geometry problems appear everywhere,” he says, “The challenge is to create the automatic tools used to transfer information painstakingly labelled on one shape, like the human brain, to another.
We try to help from the inception of a geometric idea—whether it’s understanding and digitizing an artist’s drawing of a scene or helping a shipping company route packages to its customers—to boiling it down to its mathematical essence, to designing software and algorithms, to bringing proof-of-concept demos back into the pipeline in a way that our collaborators don’t have to understand all the math. We’re involved in every single stage.
“While computers already can understand data points and discrete pieces of information, our group focuses on developing tools to solve problems whose unknown is a shape or piece of geometry,” he says. Although the context is different in each case, the underlying mathematical themes—ideas of shape, structure, distance, similarity and dissimilarity—are the same. He says, “The long-term goal is to view geometry as a fundamental piece of the computational toolbox.”
In the world of supply chain, a key challenge is optimal transportation. Solomon offers an example. “Say I’m running a big company and I have distributions of supply and demand, like gas stations and refineries, that have to be routed over a road network. I know how much comes out of every source and how much every target needs. The company has to match supply and demand to minimize the distance that everything must travel to minimize costs,” he says. Whether routing packages over a road network with millions of cars and roads or routing information over a network of computers, the structure and challenges in modern optimal transportation involve scalability, efficiency, and working in the context of an ever-changing landscape of supply and demand.
In worlds like medical imaging, the challenge is transferring information from one carefully studied object—like a manually-labeled human brain—to another recently scanned human brain. “Say I label and understand the folds in one subject’s brain and now I’m given a new subject who I know nothing about, except the 3D shape of their brain. Can I transfer information from one object—the brain I’ve just studied very carefully—onto one that I just scanned?”
In computer graphics and animation, the challenge involves taking a 3D scan of an object and turning it into to a water-tight, well-behaved shape: something somebody can animate, edit, or 3D print. “If I tug on the hand of a 3D model, we want to know how the entire object moves or responds from this one tug,” he says.
Computer-aided design (CAD) and drafting are overloaded with data, objects, and models that engineers have carefully designed: mechanical objects, furniture, gears. To search that kind of data right now requires typing in a keyword and relying on the engineer to carefully describe what it is he or she made. “We have few ways of intelligently navigating an object and saying, ‘Oh, that object is a refrigerator and now I can load up these fifty other refrigerators to help you with your design process.’ This is a really exciting search problem because it can help people use the computer-aided design pipeline in a more intelligent way,” he says.
One of the exciting and interesting problems we think about in our group is whether we can we leverage the fact that we have tons of 3D data, like the new cell phones that can sense depth and the world in front of us.
Where word-processing has revolutionized the way we communicate, we rarely think in 3D on a computer—at least not on the consumer end. “One of the exciting and interesting problems we think about in our group is whether we can we leverage the fact that we have tons of 3D data, like the new cell phones that can sense depth and the world in front of us,” says Solomon. Using that to inspire and change the design of 3D shapes requires understanding previously designed shapes, and then leveraging that information to assist in the creative and engineering processes. These, Solomon emphasizes, are completely open challenges.
The same themes and ideas invented for understanding geometry and shape also apply to understanding networks—not just in supply chain but also in understanding social networks. What is the distance between different people and different groups of people? Navigating the amount of information collected on websites like Facebook and Instagram is already a large systems challenge. “People like me with a more theoretical background are excited to give ideas and mathematical constructions for navigating and understanding this data,” says Solomon. But it’s not enough to just come up with a model. “We have to come up with algorithms and systems and even hardware that are specifically designed to deal with this modality,” he says.
What Solomon learned at Pixar and what he’s learned from working with some of his industrial collaborators—Adobe and Ford, for example—are the really serious challenges faced by industry. “It’s not just a question of hiring more coders,” he says. “These problems are mathematically difficult. It’s our role at MIT to identify and isolate the underlying fundamental challenges beneath giant systems.” He concedes that while they may not be writing the code that’s going into the final piece of software, they can help to prototype and eventually incorporate ideas into the pipeline. “We try to help from the inception of a geometric idea—whether it’s understanding and digitizing an artist’s drawing of a scene or helping a shipping company route packages to its customers—to boiling it down to its mathematical essence, to designing software and algorithms, to bringing proof-of-concept demos back into the pipeline in a way that our collaborators don’t have to understand all the math,” says Solomon. “We’re involved in every single stage.”