While formalized methodologies now exist for many aspects of functional verification, verification planning is still largely an ad-hoc process. Yet verification planning is becoming increasingly important as chips get more complex. At the recent
DVCon 2013 conference, a panel of verification experts discussed why verification planning is important, what the challenges are, and what "best practices" can be applied to verification planning.
The panel, titled
Best Practices in Verification Planning, was part of a Cadence-sponsored luncheon Feb. 27. The panel was moderated by John Brennan, product director for verification at Cadence. Panelists were as follows, shown left to right (after Brennan at the podium) in the photo below: Jason Sprott - Chief Technology Officer, Verilab Mike Stellfox - Verification Fellow, Cadence Design Systems (see video clip at the end of this post for brief post-panel interview) Ambar Sarkar - Chief Verification Technologist, Paradigm Works Neyaz Khan - Distinguished Member of Technical Staff, Maxim Integrated Vigyan Singhal - President and CEO of OSKI Technology Meirav Nitzan - Lead Verification Methodologist, Xilinx Verification Planning panel at DVCon 2013 (photo by Joe Hupcey III)
Verification planning is still an "art form," Brennan noted at the beginning of the panel. "To get a good verification plan together takes a significant amount of work," he noted. "There are so many human factors that it is really hard to codify it and make it a true science."
Here's what panelists had to say about the three main points that came up in the panel - why planning is needed, what the challenges are, and what the best practices are.
Why is Verification Planning Needed? Khan - Maxim is a mixed-signal company, and we have tremendous challenges with mixed-signal designs. "A plan is what brings everything together for us. It's the blueprint for how we're going to build our verification environment." An executable plan is a must -- it's a "living document." Nitzan - Xilinx is transitioning from a directed test approach to a methodology based on verification planning and coverage-driven verification. An executable verification plan is feature-driven rather than a list of tests. "A check mark in a verification plan is real proof you have tested something. Once people understand that, it is much easier to promote correct and efficient planning." Stellfox - The biggest objection we hear is that people don't have time to capture a verification plan. But the reality is that you're going to waste a lot more time if you don't capture a plan where you can track your progress for various modules in the design. Sprott - "If someone asks how much time a verification plan will save, it's hard to quantify. But if you don't do it you're never going to have the opportunity to collect the status and record the history of how you verified." Sarkar -- Verification planning will save you from wasting time. What we want is to be more predictable. Singhal - Oski ran a 72-hour formal verification challenge at the Design Automation Conference last year [details in blog post here]. We spent 6-8 hours just doing the planning even though we only had 72 hours. "The harder it is, the more important the plan becomes. I've never seen people over-plan but I've seen people over-verify because they hadn't planned enough." What are the Challenges of Verification Planning? Sprott - "The things that make it difficult are the things we have to analyze as humans. It's not something that can be structurally analyzed." Sarkar - "My biggest challenge is getting marketing input into my verification plan." Aside from that, one problem with plans today is that they talk about features and coverage, but the plan is still a "wish list" unless you "tie it down with how on earth you are going to get there." Sprott (in response) - "Be careful. There can be too many details. I think you have to take broad strokes - you don't need details for each individual feature." Khan - There is a tendency to over-engineer plans. What's in the plan may be unachievable - for example, if somebody wants to cross every little item in a cover group with everything else.
Another challenge is that different stakeholders have different things they care about. How to look at a spec and extract features of interest for each key stakeholder is a huge challenge.
Nitzan - One challenge is getting management to understand the need for planning and the willingness to allocate enough time to build a plan properly. Stellfox - Here are the three biggest challenges: How to capture the plan at the right level at the right time - avoid putting in too much detail too early How to keep a plan relevant throughout a project where things are changing How to track your progress. You need to have milestones and track progress towards the plan. What are the Best Practices for Verification Planning? Sarkar - Remember a plan is not just a wish list, it's a plan. Get feedback from designers and architects at the right time. And have a way to "close the loop." Sprott --- "One thing we find very useful in verification planning is ruthless prioritization. The problem space is so massive you can never get it all done." Also, use statistics from previous projects - how many bugs, how long things took, and so forth. "Statistics are gold dust for planning." Singhal - Combine your formal-based plan with your simulation-based plan. Use coverage-based metrics for both simulation and formal verification. Khan - We have a mandatory "deep dive" before we start any project. We all get together in a single location - designers, verification people, software guys, anybody with any stake in the chip. We also bring in the product definers and the test engineers. "We find we save a lot of time towards the end and it really saves a lot of expense." Stellfox - When we work with customers, we start with a peer review process at the very beginning and bring in the stakeholders. "An amazing amount of verification work actually happens during the verification process. We're verifying things at the spec level before starting any executable work."
Tools that support verification planning have been lacking. Cadence, however, has "built a lot of technology" around creating verification plans, and that technology is provided in the
Incisive Enterprise Manager. Nitzan - An important part of planning is prioritization. Some features are more critical at given phases of the project than others. If you know what you need to do 5 days from deadline and what you don't need to do, it helps you be more efficient.
In the short video clip below, I asked Stellfox why verification planning is an important topic, what the main benefits of verification planning are, and what the main challenges are. If video fails to open,
Post-Panel Interview click here. VIDEO
Other DVCon 2013 Industry Insights posts:
DVCon 2013: Engineers Question EDA Standards Leaders at Accellera "Town Hall" Meeting DVCon 2013 Panel: 1 Million IC Design Starts - How Can We Get There?