news about the Open Verification Methodology (OVM)
and the advanced verification techniques it supports is that verification
engineers are now finding more bugs than ever. The bad news is that the
bottleneck is shifting to debugging. What are we going to do with all those
bugs? User and vendor representatives grappled with that question at a
Cadence-sponsored lunch panel at DVCon
Wednesday, Feb. 24.
Mike Stellfox, principal verification solutions architect at Cadence, noted
that OVM and techniques such as constrained-random test generation have
dramatically shortened the time it takes to build a testbench. "We're finding a
lot more bugs than we could with traditional directed-test approaches, but now
we're getting more bugs than we can debug in a timely manner," he said. In a
recent Cadence survey, he noted, advanced verification users said that debug
now takes about 50 percent of the overall verification effort.
Panelists included Steve Jorgenson, verification
specialist at Hewlett-Packard; Umer Yousafzai, solutions architect at Cadence;
Alicia Strang, principal verification engineer at Marvell; and Yuchin Hsu, vice
president of R&D for verification products at Springsoft.
emphasis on the user presenters, here are several perspectives that I thought
How OVM helps with debug
that OVM is very useful," said Strang. "If you just use that methodology, and
use the naming conventions, everybody is on the same page."
conventional directed-test environment, Yousafzai said, users spend a lot of
time debugging test stimulus. In OVM, which supports up-front planning, users
spend less time tracking down problems in the testbench and more time finding
bugs in the design. Jorgenson noted that OVM supports higher levels of
abstraction, and said that "I really think abstraction is what will save us in
the end from our debugging problems."
Testbenches need debugging, too
commented that verification teams can easily spend half their time just
debugging problems in the testbench. One antidote is to move up in abstraction
and reuse as much as possible from previous verification environments. Strang
had another suggestion. "In a modern software environment like a testbench, you
can do whatever you want. Keep it simple, and you can avoid a lot of bugs in
Constrained-random stimulus may
obscure bug causes
When a bug
is found in a directed-test environment, Jorgenson said, a designer can very
likely help identify the root cause. But with constrained-random techniques,
"you don't always know what the test did. One thing we've had to do is go in
there and get the test to tell us what it did."Assertions are important
that some kinds of bugs can be found only with assertions. These include bugs
that do not affect data. "If it's data in, garbage out, I know it's wrong, but
if it's data in, data out I don't see anything wrong," she said. Jorgenson said
his group has been using assertions for 8 or 9 years, long before SystemVerilog
assertions were standardized.
told of a customer who ran a single test 30,000 times on a block, and still had
to do a respin because all these tests couldn't find a "trivial" bug - a signal
that wasn't toggling correctly under some conditions. He noted that simple
assertions that can be added easily are often the most effective ones.
How to find software bugs
transaction monitors that generate waveforms that show what firmware is doing
in parallel with hardware signals. From this, she said, "I can tell whether
software issued a wrong command, or hardware is behaving incorrectly."
As I noted
in a blog
last year, based on part on an interview with Mike Stellfox, the bug diagnosis-and-repair
cycle is becoming the biggest bottleneck in the verification flow. If debug
really takes 50 percent of the verification effort - and verification takes 60
to 70 percent of the total design effort, as we've repeatedly heard - do the
math and you can see this is an area that deserves far more attention. The OVM
lunch panel was thus a very timely event.