Computer-aided design (CAD) is the popular method for designing most of today's physical products. Engineers use CAD to convert 2D sketches into 3D models, which they will then test and refine before sending a final version to a production line. However, the software is notoriously complicated to learn, with hundreds of commands to pick from. To really master the software takes a whole lot of time and practice.
MIT engineers wish to simplify the CAD learning curve with an AI model that uses CAD software just like a human. Starting from a 2D sketch of an object, the model quickly creates a 3D version by clicking buttons and file options, just like how an engineer would use the software.
The MIT team created a brand new dataset called VideoCAD, which comprises greater than 41,000 examples of how 3D models are created in CAD software. By learning from these videos, which illustrate learn how to construct various shapes and objects step-by-step, the brand new AI system can now operate CAD software in the same solution to a human user.
With VideoCAD, the team is aiming for an AI-supported “CAD co-pilot”. They imagine that such a tool couldn’t only create 3D versions of a design, but in addition work with a human user to suggest next steps or robotically execute construction sequences that may otherwise be tedious and time-consuming to click through manually.
“AI offers the chance to extend the productivity of engineers and make CAD more accessible to more people,” says Ghadi Nehme, a doctoral candidate in MIT’s Department of Mechanical Engineering.
“This is essential since it lowers the barrier to entry into design and helps people without years of CAD training more easily create 3D models and unleash their creativity,” adds Faez Ahmed, associate professor of mechanical engineering at MIT.
Ahmed and Nehme, together with graduate student Brandon Man and postdoctoral fellow Ferdous Alam, will present their work on the Conference on Neural Information Processing Systems (NeurIPS) in December.
Click by click
The team's recent work expands on recent developments in AI-driven user interface (UI) agents – tools trained to make use of software programs to perform tasks akin to robotically gathering information online and organizing it in an Excel spreadsheet. Ahmed's group wondered whether such UI agents might be designed to make use of CAD, which incorporates many more features and functions and involves much more complicated tasks than the common UI agent can handle.
In their recent work, the team desired to design an AI-driven UI agent that may take the reins of the CAD program to create a 3D version of a 2D sketch, click by click. To do that, the team first examined an existing data set of objects designed by humans in CAD. Each object within the dataset comprises the sequence of high-level design commands akin to Sketch Line, Circle, and Extrude that were used to create the ultimate object.
However, the team realized that these high-level commands alone weren’t enough to show an AI agent to truly use CAD software. An actual agent also needs to grasp the small print behind every motion. For example: Which sketch area to pick out? When should it’s enlarged? And which a part of a sketch ought to be extruded? To address this gap, researchers developed a system to translate high-level commands into user interface interactions.
“For example, let’s say we created a sketch by drawing a line from point 1 to point 2,” says Nehme. “We translated these high-level actions into UI actions, which suggests we are saying: go from this pixel location, click, after which go to a second pixel location and click on while the Line operation is chosen.”
In the tip, the team created over 41,000 videos of human-designed CAD objects, each described in real time using the particular clicks, mouse movements, and other keyboard actions that the human originally performed. They then fed all of this data right into a model they developed to learn relationships between UI actions and CAD object generation.
Once trained on this data set, which they call VideoCAD, the brand new AI model could take a 2D sketch as input and directly control the CAD software by clicking, dragging, and choosing tools to construct the total 3D shape. Objects ranged in complexity from easy brackets to more complicated house designs. The team trains the model on more complex shapes and anticipates that each the model and data set could someday enable CAD co-pilots for designers in a wide selection of fields.
“VideoCAD is a priceless first step toward AI assistants that help onboard recent users and automate repetitive modeling work that follows familiar patterns,” says Mehdi Ataei, who was not involved within the study and is a senior research scientist at Autodesk Research, which develops recent design software tools. “This is an early foundation, and I would love to see successors that include multiple CAD systems, richer operations like assemblies and constraints, and more realistic, messy human workflows.”
