CVDP Performance Analysis

The CVDP Benchmark

The Comprehensive Verilog Design Problems (CVDP) benchmark from NVIDIA (arXiv:2506.14074) contains 649 RTL problems derived from real chip design workflows. Unlike VerilogEval — which tests short, textbook-style snippets — CVDP problems feature specifications averaging 18KB, multiple interacting submodules, and cocotb-based Python testbenches that exercise silicon-realistic edge cases.

CVDP organizes problems into 8 CID categories spanning code completion, spec-to-RTL generation, code modification, optimization, debugging, testbench generation, and assertion writing.

Non-Agentic Results

302 problems across 5 CID categories (easy + medium difficulty). ChipCraftX uses up to 5 iterations with EDA tool feedback — compile errors and simulation results inform each retry.

By Difficulty

Performance is remarkably consistent across difficulty levels, with medium problems only 3-5pp below easy in most categories. The one exception is cid016 (bug fixing), where medium-difficulty problems drop to 78.6% — these involve subtle timing bugs and multi-signal race conditions that are harder to diagnose from simulation output alone.

87.8% of problems pass on the first attempt (265 of 302). Only 37 problems required retries, and of those, 21 were resolved within 2-4 additional iterations.

Head-to-Head: ChipCraftX vs ACE-RTL

ACE-RTL (NVIDIA, arXiv:2602.10218, February 2026) is the most directly comparable system: it combines a fine-tuned RTL-specialized LLM with an agentic feedback loop, evaluated on the same CVDP benchmark. Their paper reports results for 18 systems across four CID categories — the same four we share. All scores use APR (Agentic Pass Rate).

vs Agentic Systems

vs Frontier LLMs

vs RTL-Specialized Models

The Delta

ChipCraftX leads convincingly on code completion (+12.75pp) and code modification (+5.49pp), ties on spec-to-RTL, and trails by 2.83pp on bug fixing. The cid016 gap likely reflects ACE-RTL's restart mechanism — when a debugging trajectory stalls, their Coordinator discards it and starts fresh with distilled insights. Our pipeline does not yet implement trajectory restarts; when an iteration stalls, we continue refining the same code rather than starting over.

Agentic Results

92 problems across 4 CID categories (easy + medium + hard difficulty). Agentic problems provide multi-file context — existing RTL modules that the generated code must integrate with.

By Difficulty

The agentic subset is substantially harder — easy problems are universally solved (100% across all CIDs), but medium and hard problems expose the challenge of multi-file context. The 64.1% overall rate shows clear room for improvement, with module reuse/integration (40.9%) and hard code modification (33.3%) as primary bottlenecks.

Iteration Dynamics

Non-Agentic

Agentic

The pattern is striking: every problem that reaches 5 iterations fails. This is a hard ceiling, not a soft degradation. If the pipeline hasn't solved a problem in 4 attempts, the 5th attempt has zero empirical chance of succeeding.

Timing

Failures take 6.9x longer on average — the system exhausts its iteration budget before admitting defeat.

Key Takeaways

1. 94.7% non-agentic on CVDP demonstrates that iterative EDA feedback with a strong foundation model solves the vast majority of realistic RTL problems — not just textbook exercises.
2. Efficiency over brute force: ChipCraftX matches or beats ACE-RTL on 3/4 categories with 5 iterations vs their 150 attempts.
3. Agentic is the frontier: 64.1% on multi-file integration tasks shows where the real challenge lies. Code completion and spec-to-RTL are approaching saturation; module integration and debugging at scale are the next problems to solve.
4. Transparency builds trust: We report our cid016 gap honestly. ACE-RTL's restart mechanism is a good idea that we plan to adopt. The 16 non-agentic failures are understood and actively being addressed.