Sketch2Arti

Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects

Yi Yang1, Hao Pan2, Yijing Cui2, Alla Sheffer3, Changjian Li1

  1. 1 University of Edinburgh, United Kingdom
  2. 2 Tsinghua University, China
  3. 3 University of British Columbia, Canada

ACM Transactions on Graphics (Proc. SIGGRAPH 2026)

arXiv Video Code Data Hugging Face

To-do

  • We plan to release the Sketch2Arti code and dataset before SIGGRAPH 2026.

From quick motion sketches to articulated CAD objects.

Articulation modeling aims to infer movable parts and their motion parameters for a 3D object, enabling interactive animation, simulation, and shape editing. In this paper, we present Sketch2Arti, the first sketch-based articulation modeling system for CAD objects. Our key observation is that designers naturally communicate articulation intent through lightweight sketches (e.g., arrows and strokes) that indicate how parts should move, yet translating such sketches into articulated 3D models remains largely manual.

Sketch2Arti bridges this gap by enabling users to specify articulation through simple 2D sketches drawn from a chosen viewpoint. Given a CAD model and user sketches, our approach automatically discovers the corresponding movable parts and predicts their motion parameters, allowing iterative modeling of multiple articulations on complex objects with fine-grained control. Importantly, Sketch2Arti is trained in a category-agnostic manner without requiring object category information, leading to strong generalization to diverse objects beyond existing articulation datasets.

Moreover, for shell models lacking interior structures, Sketch2Arti supports controllable internal completion guided by user sketches, generating plausible internal components consistent with the existing geometry and predicted motion constraints. Comprehensive experiments and user evaluations demonstrate the effectiveness, controllability, and generalization of Sketch2Arti.

Sketch2Arti teaser with sketch-guided articulations.
Sketch-based articulation modeling. We present Sketch2Arti, the first sketch-based system for articulation modeling of CAD objects. Sketch2Arti is versatile. Top: through iterative sketch-based editing, Sketch2Arti progressively discovers multiple movable parts and recovers their motion parameters on a complex car model. Middle-left: Sketch2Arti offers high controllability, e.g., a car door can be opened in different user-specified manners (standard, backward-hinged, or butterfly) by simply changing the sketches. Middle-right: Sketch2Arti generalizes to diverse objects beyond public datasets, enabling articulation modeling for unseen categories. Bottom: for shell models lacking internal structures, users can sketch the missing components (e.g., the teddy bear) and Sketch2Arti faithfully generates plausible internal mechanisms that respect both the existing geometry and the predicted articulation parameters.

Overview

Where / How / What

Sketch2Arti addresses three coupled challenges in sketch-based articulation modeling: 1) locating the movable part, 2) estimating its motion, and 3) completing hidden geometry exposed by articulation.

Where

Predict a movable-part mask from the sketch region and select the corresponding 3D component through hierarchical segmentation.

How

Estimate joint type and motion parameters from local sketch, depth, and normal cues, then refine them geometrically.

What

Generate missing internal structure when articulation reveals an empty shell, while preserving existing geometry and motion constraints.

Sketch2Arti system overview pipeline.
Overview. (a) Given an input 3D shape and the user sketches, our method Sketch2Arti addresses the where and how challenges by (b) identifying movable parts (i.e., the two doors) and inferring their articulation parameters. (c) The predicted motion reveals missing internal structure (e.g., an empty drawer), which users can further specify via sketches. Sketch2Arti then tackles the what challenge by (d) generating the full drawer geometry while adhering to both the existing shape and the inferred articulation.

Pipeline 1

Sketch-based articulation analysis.

Pipeline 1 reconstructs the intended movable part and its articulation parameters from rough motion sketches. The input is a localized crop around the sketch, represented by sketch, depth, and surface-normal maps.

A UNet-based recognizer predicts image-space part evidence, joint type, pivot cues, and 3D motion direction. The part mask is matched to a 3D component with Partfield features, and the recovered motion is snapped to meaningful geometric anchors for physical plausibility.

Sketch2Arti articulation prediction pipeline.
Articulation prediction. Given a static 3D object, we apply category-agnostic articulation recognition on a localized region surrounding the sketch with the local context captured by the depth and normal maps. A trained U-Net module predicts the articulation parameters in 2D maps and 3D local camera coordinates, as well as motion type. The 2D part mask is then back-projected onto the object surface and used to filter through a hierarchy of segments produced by a foundation Partfield model, to select the best matching part at a level undetermined beforehand as the movable 3D component.

Interactive URDF Demo

Drag the articulated parts and inspect the predicted joints.

URDF Animation
Loading URDF...
Select or drag a colored part. Orbit with empty-space drag. Scroll to zoom.
Video Recording

Demo Video

Car articulation sketch record

Results Gallery

Generalization and controllability.

Sketch2Arti result gallery.
Results gallery. We show representative articulation modeling sessions using Sketch2Arti. For each example, user sketches are overlaid on the rendered shape under the chosen viewpoint, and the inferred movable parts are color-coded. The black arrow indicates the iterative modeling order across views/parts.

Interior Completion

Completing internal structures
for functional articulation.

Most existing assets within datasets or synthesized by generative models represent objects as surface-only models, containing only the geometry visible in a canonical static pose. Kinematic articulation exposes physical voids in regions that were previously occluded, rendering the object unsuitable for realistic physical interaction.

To resolve this incompleteness, we articulate the object to its open state, generate a completed 2D reference image for the exposed interior, and then lift that reference into 3D while preserving the existing shell and respecting the predicted articulation.

Interior shape completion pipeline.
Interior shape completion. Our approach leverages 2D and 3D generative models to complete the interior structures exposed by articulated parts. Given a 3D object with recognized articulation part and parameters, the top branch applies a 2D generative model (e.g., Nano Banana) to obtain a high-quality reference image, which is used to guide the 3D generative model (e.g., Trellis) to create the interior structure. Crucially for obtaining structure-preserving interiors, masks of loose and strict types are built to control the flow generative process of the 3D generative model and adjust the completed part interior, respectively. Finally, the completed part is refined for kinematic validity and turned into separate meshes that are readily usable as URDF models.
Video Recording
URDF Animation
Preparing URDF preview...

Citation

BibTeX

If you find our work useful for your research, please consider citing our paper. 

@article{yang2026sketch2arti,
  title = {Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects},
  author = {Yang, Yi and Pan, Hao and Cui, Yijing and Sheffer, Alla and Li, Changjian},
  journal = {ACM Transactions on Graphics},
  volume = {45},
  number = {4},
  year = {2026},
  doi = {10.1145/3811375}
}