CADGenBench Metrics

How CADGenBench scores one generated CAD part against the ground truth. These metrics are new, so this page explains each one.

CAD ScoreHow one part is scored

CADGenBench scores a generated part (STEP/BREP) against one ground-truth STEP. First a hard validity gate; if it passes, the CAD Score is a weighted mean of three independent metrics, each in [0, 1].

cad_score = 0                                                if not valid
          = 0.4*shape + 0.4*interface + 0.2*topology          otherwise

(This is the generation composition. Editing tasks renormalize the shape axis and reweight; see Editing tasks below.)

Component	Range	What it asks
CAD Validity (gate)	{0, 1}	Is the geometry valid?
Shape Similarity	[0, 1]	Does the bulk geometry match?
Topology Match	[0, 1]	Same pieces / holes / voids?
Interface Match	[0, 1]	Does it bolt up to the same fixture?

Why three axes

They are orthogonal by construction: each catches errors the others are blind to:

Shape catches wrong bulk geometry; blind to topology.
Topology catches wrong hole / piece / void counts; blind to feature position.
Interface catches a misplaced / mis-sized mating feature; blind to overall shape.

Outputs are rigidly aligned to the ground truth (rotation + translation only, never scale) before scoring.

GateCAD Validity

Runs before every other metric on the raw candidate. Any failure sets is_valid = False and forces cad_score = 0, so an invalid solid never beats a worse but valid one.

A candidate (output.step / output.stp) must pass all of:

Well-formed BREP: no per-face / edge / vertex errors (self-intersecting wires, edges off their surface, etc.).
Watertight: every shell is closed; no naked or free edges.
Meshable as a closed orientable manifold: tessellates to a manifold, closed (3F = 2E), orientation-consistent triangle mesh.

ShapeShape Similarity

Does the bulk geometry match? The mean of two complementary sub-metrics, each in [0, 1]:

shape_similarity = 0.5 * (surface_distance_F1 + volume_IoU)

Surface Distance F1

Checks the candidate's surface sits where the GT's does and faces the same way. Points are sampled across both surfaces with their outward normals; a point matches when the closest point on the other mesh's surface is within 0.5% of the GT bounding-box diagonal and the normals agree to within 20°. Precision and recall combine into F1.

Volume IoU

Shared volume of the two solids over their combined volume (intersection over union).

Both use a tolerance proportional to part size, so small features can move without shifting the score; those are covered by interface match.

TopoTopology Match

Does the candidate have the same number of pieces, through-holes, and internal voids? It compares the three Betti numbers of the solid:

b₀: connected solid components.
b₁: independent through-handles.
b₂: enclosed internal voids (cavities).

Each axis gets a fuzzy log-ratio against GT, sharpened by α = 2, and the three are multiplied:

s_i = ((min(cand,gt) + 1) / (max(cand,gt) + 1)) ^ 2
topology_match = s_0 * s_1 * s_2

The product means one wrong count collapses the score: topology is discrete, so two of three right is not a partial match. Example: GT (1,2,0) vs candidate (1,4,0) scores (3/5)² = 0.36. Blind features (blind pockets, fillets, chamfers) are topologically trivial and covered by the other axes.

InterfaceInterface Match

Would it bolt up to the same fixture? Each mating feature is a region of space the candidate must match in shape, size, and position:

Keep-out (KOR): must be empty (a bolt hole, a slot).
Keep-in (KIR): must be solid (a locating boss, a pin).

Mating groups

The features that must seat together against a single fixture form one mating group: here, two bolt holes and a slot that one jig drops into. A part can have several independent groups (say a bolt pattern on one face and a boss on another), and each group is scored on its own.

A jig with two pins and a slot key seating into a part's two bolt holes and slot — A mating group: a jig with two pins and a slot key seats into the part's two bolt holes and slot. The candidate has to fit the same fixture.

Scoring

Per group:

Per-feature fit: volumetric IoU against the region (with a thin shell of opposite material, so both oversize and undersize lose points).
Bounded pose search: ±1° and ±1% of part size per axis, so a feature isn't penalized for the residual of whole-part alignment.
Pass/fail ramp: IoU ≥ 0.95 → 1, ≤ 0.80 → 0, linear between; a sloppy fit scores 0.

A group scores as its worst feature (the minimum); the fixture scores as the mean over its groups, so nailing one interface and missing another still earns partial credit.

EditingEditing tasks: no-op renormalization

Editing fixtures ship an input.step plus an edit request; the GT is a small change to that input. Since all three axes measure global similarity, submitting the input unchanged (the no-op) already scores high, so the raw composition would reward doing nothing.

The fix renormalizes the shape axis against the no-op baseline b = shape_similarity(input, GT):

s_renorm  = max(0, (shape_similarity - b) / (1 - b))
cad_score = 0.6*s_renorm + 0.3*interface + 0.1*topology   (0 if not valid)

This maps the no-op to 0 and a perfect candidate to 1. Topology and interface stay raw (most edits leave them unchanged). A no-op therefore caps at 0.3 + 0.1 = 0.4, and any real shape improvement clears it.

For the full definitions and derivations, see the metrics reference in the code: docs/metrics.md.