CVDP Performance Analysis.

The CVDP Benchmark

The Comprehensive Verilog Design Problems (CVDP) benchmark from NVIDIA (arXiv:2506.14074) contains 649 RTL problems derived from real chip design workflows. Unlike VerilogEval (which tests short, textbook-style snippets), CVDP problems feature specifications averaging 18KB, multiple interacting submodules, and cocotb-based Python testbenches that exercise silicon-realistic edge cases.

CVDP organizes problems into 8 CID categories spanning code completion, spec-to-RTL generation, code modification, optimization, debugging, testbench generation, and assertion writing.

Non-Agentic Results

302 problems across 5 CID categories (easy + medium difficulty). ChipCraftX uses up to 5 iterations with EDA tool feedback: compile errors and simulation results inform each retry.

By Difficulty

Performance is remarkably consistent across difficulty levels, with medium problems only 3-5pp below easy in most categories. The one exception is cid016 (bug fixing), where medium-difficulty problems drop to 78.6%. These involve subtle timing bugs and multi-signal race conditions that are harder to diagnose from simulation output alone.

87.8% of problems pass on the first attempt (265 of 302). Only 37 problems required retries, and of those, 21 were resolved within 2-4 additional iterations.

Local Open-Model Results

Can the same pipeline run on a fully local, open-weight model instead of the Anthropic API? We evaluated Qwen3.6-35B-A3B-FP8 served by vLLM on two RTX 3090s: identical pipeline configuration, identical cocotb harnesses, zero API calls, zero marginal cost.

The local model reaches 86.4% at zero marginal cost: 91% of Claude's pass rate, needing ~46% more repair iterations. Two results stand out: Qwen3.6 essentially matches Claude on Spec-to-RTL (94.9% vs 96.2%), the core ChipCraftX use case, and beats Claude on Bug Fixing (94.3% vs 88.6%). The gap concentrates in RTL Modification (−18.2pp) and RTL Optimization (−20.0pp).

Head-to-Head: ChipCraftX vs ACE-RTL

ACE-RTL (NVIDIA, arXiv:2602.10218, February 2026) is the most directly comparable system: it combines a fine-tuned RTL-specialized LLM with an agentic feedback loop, evaluated on the same CVDP benchmark. Their paper reports results for 18 systems across four CID categories, the same four we share. All scores use APR (Agentic Pass Rate).

vs Agentic Systems

vs Frontier LLMs

vs RTL-Specialized Models

The Delta

ChipCraftX leads convincingly on code completion (+12.75pp) and code modification (+5.49pp), ties on spec-to-RTL, and trails by 2.83pp on bug fixing. The cid016 gap likely reflects ACE-RTL's restart mechanism: when a debugging trajectory stalls, their Coordinator discards it and starts fresh with distilled insights. Our pipeline does not yet implement trajectory restarts; when an iteration stalls, we continue refining the same code rather than starting over.

Agentic Results

92 problems across 4 CID categories (easy + medium + hard difficulty). Agentic problems provide multi-file context: existing RTL modules that the generated code must integrate with.

By Difficulty

The agentic subset is substantially harder: easy problems are universally solved (100% across all CIDs), but medium and hard problems expose the challenge of multi-file context. The 64.1% overall rate shows clear room for improvement, with module reuse/integration (40.9%) and hard code modification (33.3%) as primary bottlenecks.

Iteration Dynamics

Non-Agentic

Agentic

The pattern is striking: every problem that reaches 5 iterations fails. This is a hard ceiling, not a soft degradation. If the pipeline hasn't solved a problem in 4 attempts, the 5th attempt has zero empirical chance of succeeding.

Timing

Failures take 6.9x longer on average: the system exhausts its iteration budget before admitting defeat.

Key Takeaways

1. 94.7% non-agentic on CVDP demonstrates that iterative EDA feedback with a strong foundation model solves the vast majority of realistic RTL problems, not just textbook exercises.
2. Efficiency over brute force: ChipCraftX matches or beats ACE-RTL on 3/4 categories with 5 iterations vs their 150 attempts.
3. Agentic is the frontier: 64.1% on multi-file integration tasks shows where the real challenge lies. Code completion and spec-to-RTL are approaching saturation; module integration and debugging at scale are the next problems to solve.
4. Transparency builds trust: We report our cid016 gap honestly. ACE-RTL's restart mechanism is a good idea that we plan to adopt. The 16 non-agentic failures are understood and actively being addressed.