Home > Community > Blogs > Custom IC Design > skill for the skilled simple testing macros
 
Login with a Cadence account.
Not a member yet?
Create a permanent login account to make interactions with Cadence more conveniennt.

Register | Membership benefits
Get email delivery of the Custom IC Design blog (individual posts).
 

Email

* Required Fields

Recipients email * (separate multiple addresses with commas)

Your name *

Your email *

Message *

Contact Us

* Required Fields
First Name *

Last Name *

Email *

Company / Institution *

Comments: *

SKILL for the Skilled: Simple Testing Macros

Comments(8)Filed under: SKILL, Team SKILL, programming, LISP, SKILL++, Jim Newton, SKILL for the Skilled, shuffle, macros In this post I want to look at an easy way to write simple self-testing code. This includes using the SKILL built-in assert macro and a few other macros which you can derive from it.

The assert macro

This new macro, assert, was added to SKILL in SKILL version 32. You can find out which version of SKILL you are using with the SKILL function getSkillVersion, which returns a string such as "SKILL32.00" or "SKILL33.00".

Using this macro is simple. In its simplest form, you wrap any single expression within (assert ...). At evaluation time this will trigger an error if the expression evaluates to nil.

CIW> (assert 1 + 1 == 2)
nil

CIW> (assert 2 + 2 == 5)
*Error* ASSERT FAILED: ((2 + 2) == 5)

<<< Stack Trace >>>
error("ASSERT FAILED: %L\n" '((2 + 2) == 5))
unless(((2 + 2) == 5) error("ASSERT FAILED: %L\n" '(& == 5)))

You can also specify the error message using printf-style arguments.

CIW> (defun testit (x "n")
       (assert x > 3 "expecting x > 3, not %L" x)
       x-3)

CIW> (testit 12)
9

CIW> (testit 2)
*Error* expecting x > 3, not 2
<<< Stack Trace >>>
error("expecting x > 3, not %L" x)
unless((x > 3) error("expecting x > 3, not %L" x))
testit(2)

What is a macro?

Macros are a feature in many lisps including emacs lisp, common lisp, and SKILL. Consequently, you can find tons of information on the Internet explaining lisp macros. A quick Google search for "lisp macro" returns pages of useful results.

In particular, a SKILL macro is a special type of function which computes and returns another piece of SKILL code in the form of an s-expression. This capability allows SKILL programs to write SKILL programs as a function of their raw, unevaluated, operands at the call-site. Although macros can certainly be abused, when used correctly SKILL macros can greatly enhance readability by abstracting away certain details, or by automating repetitive patterns.

You can find out more about SKILL macros by consulting the Cadence documentation. There is also an Advanced SKILL Programming class which Cadence Educational Services offers.

In-line tests

You can use assert inside your functions for run-time checks. You can also use assert at the top level of your SKILL files for load-time checks. This has an added advantage of helping the person who reads your code to understand how the function you are defining is used. In the following example anyone who reads your function definition can immediately see some examples of how to use the function.
;; Sort a list of strings case independently into alphabetical order.
(defun sort_case_independent (words "l")
  (sort words (lambda (w1 w2)
                (alphalessp (lowerCase w1)
                            (lowerCase w2)))))

(assert nil 
        ; works for empty list?
        == (sort_case_independent nil)) 

(assert '("a")
         ; works for singleton list?
         ==  (sort_case_independent '("a")))) 

(assert '("a" "B")
         == (sort_case_independent '("B" "a")))

(assert '("a" "B")
         == (sort_case_independent '("a" "B")))

(assert '("A" "b")
        == (sort_case_independent '("b" "A")))

(assert '("A" "b")
        == (sort_case_independent '("A" "b")))

(assert '("A" "b" "c" "D")
        == (sort_case_independent '("c" "D" "A" "b")))
Writing your SKILL files to include these top-level assertions has yet another advantage: if someone later modifies your function, sort_case_independent, the tests will run the next time anyone loads the file. This means if an error has been introduced in the function, some sanity testing happens at load time. Furthermore, if someone enhances the function in a way that breaks backward compatibility, the assertion will fail at load time.

Defining assert if it is missing

If you are using a version of Virtuoso, Allegro, etc, which does not contain a definition of assert, you can define it yourself.
(unless (isCallable 'assert)
  (defmacro assert (expression @rest printf_style_args) 
    (if printf_style_args 
        `(unless ,expression
           (error ,@printf_style_args))
        `(unless ,expression
           (error "ASSERTION FAILED: %L\n" ',expression)))))

The assert_test macro

Some unit testing frameworks supply assertion functions such as assert_less, assert_greater, assert_equal, assert_not_equal. It is possible in SKILL to define a single assertion macro called, assert_test, which provides all these capabilities in one. You don't really need assert_equal, assert_not_equal, asset_lessp etc...

This macro is useful for building test cases. This macro attempts to output a helpful message if the assertion fails. The message includes the parameters to the testing expression, and the values they evaluate to. For example:

CIW> A = 3
3

CIW> B = 2
2

CIW> (assert_test A+B == B+2)
 *Error* (A + B)
  --> 3
(B + 2)
  --> 4
FAILED ASSERTION: ((A + B) == (B + 2))
<<< Stack Trace >>>
...

CIW> (assert_test A+B > B+2)

*Error* (A + B)
  --> 3
(B + 2)
  --> 4
FAILED ASSERTION: ((A + B) > (B + 2))
<<< Stack Trace >>>
...

The intelligent thing about assert_test as can be seen from the above example, is that it constructs an error message which tells you the text of the assertion that failed: ((A + B) == (B + 2)). It also tells you the arguments to the testing function in both raw and evaluated form: (A + B) --> 3 and (B + 2) --> 4

The macro definition is not trivial, but the code is given here. You don't really need to understand how it works in order to use it.

;; ARGUMENTS:
;;   expr - an expression to evaluate, asserting that it does not return nil
;;   ?ident ident - specifies an optional identifier which will be printed with [%L] in
;;                     the output if the assertion fails.  This will help you identify the
;;                     exact assertion that failed when scanning a testing log file.
;;   printf_style_args - additional printed information which will be output if the
;;                     assertion fails.
(defmacro assert_test (expr @key ident @rest printf_style_args)
  (if (atom expr)
      `(assert ,expr)
      (let ((extra (if printf_style_args
                       `(strcat "\n" (sprintf nil ,@printf_style_args))
                       "")))
        (destructuringBind (operator @rest operands) expr
          (letseq ((vars (foreach mapcar _operand operands
                           (gensym)))
                   (bindings (foreach mapcar (var operand) vars operands
                               (list var operand)))
                   (assertion `(,operator ,@vars))
                   (errors (foreach mapcar (var operand) vars operands
                             `(sprintf nil "%L\n  --> %L" ',operand ,var))))
            `(let ,bindings
               (unless ,assertion
                 (error "%s%s%s"
                        (if ',ident
                            (sprintf nil "[%L] " ,ident)
                            "")
                        (buildString (list ,@errors
                                           (sprintf nil "FAILED ASSERTION: %L" ',expr))
                                     "\n")
                        ,extra))))))))

The assert_fails macro

With the assertion macros presented above you can pretty robustly make assertions about the return values of functions. A limitation, however, is you cannot easily make assertions about the corner cases where your function triggers an error.

The following macro, assert_fails, provides the ability to assert that an expression triggers an error. For example, the sort_case_independent function defined above will fail, triggering an error, if given a list containing a non-string.

CIW> (sort_case_independent '("a" "b" 42 "c" "d"))
*Error* lowerCase: argument #1 should be either a string or a symbol (type template = "S") at line 112 of file "*ciwInPort*" - 42
<<< Stack Trace >>>
lowerCase(w2)
alphalessp(lowerCase(w1) lowerCase(w2))
funobj@0x2cac49a8("a" 42)
sort(words lambda((w1 w2) alphalessp(lowerCase(w1) lowerCase(w2))))
sort_case_independent('("a" "b"))

You could fix this by enhancing the function to do something reasonable in such a situation. Or you could simply document the limitation, in which case you might want to extend the in-line test cases as well.

(assert_fails (sort_case_independent '("a" "b" 42 "c" "d")))

Here is an implementation of such a assert_fails macro.

(defmacro assert_fails (expression)
  `(assert (not (errset ,expression))
     "EXPECTING FAILURE: %L\n"
     ',expression))

Summary

In this article we looked at the assert macro, which is probably in the version of Virtuoso or Allegro you are using, and if not you can easily define it yourself. We also looked at assert_test and assert_fails which you can define yourself. You can use these three macros to easily improve the robustness of your SKILL programs.

See Also

Comments(8)

By TThomas on January 8, 2014
Could you also comment on performance hit with asserts? Is there a way to disable it if needed to improve performance.


By Team SKILL on January 9, 2014
Hi Thomas, thanks for the question.    As you imply by your question, there is often a tradeoff between safety and speed.  In fact some Lisp compilers (SKILL not included) have compiler settings where you can specify the relative importance of safety and speed.  You generally care more about safety while developing, but more about speed once the code is deployed.

The SKILL assert macro does not have a facility to disable the assertion at run-time, although it would be possible to write your own macro similar to assert which allows a run-time switch to disable the assertion check, or even a load-time switch to eliminate the check altogether.  I'll try to come up with a sample implementation and post it below.

But you also asked about the performance characteristics of the assert macro and the assert_test macros as described in the original SKILL for the Skilled article.  One advantage of using these top-level assertions in your file, is that you only suffer a performance penalty at load-time, not at run time.  I.e., if after defining a function in your source file, the next several lines are calls to assert (or assert_test) which effectively test the behavior of the function, then there is no run-time penalty of the assertion when calling the actual function thereafter.

The performance penalty of an assertion inside your function depends entirely on how complicated the actual asserted expression is.   For example asserting that a value is positive is very fast assert(A>0) or even assert(plusp(A)), but asserting something complicated about a list is of course slower:  assert( forall( x my_list and( x dtpr(x) plusp( car(x))))).

Please let me know whether this answers your question, or whether you'd like more explanation.

Kind regards

Jim


By Team SKILL on January 9, 2014
Hi Thomas, as promised in my previous response here is a macro similar to the SKILL
assert macro which has a subtle difference.  It checks the debugMode status by calling status(debugMode), and behaves differently depending on the result.
1) If at load time, the debugMode status is nil, the macro expands to nil.  Thus there is almost no run-time penalty, and the asserted expression is never checked.
2) If at load time, the debugMode status is t, the macros expands to contain an additional runtime check of the debugMode status.  When the code runs if the debugMode status is nil, the assertion expression is not evaluated.
3) If the debugMode status is true both at load time and also at run-time then the assertion expression gets evaluated and is asserted to be non-nil.
Thus using assert_debug only imposes penalties when in debug mode.
You can turn on debug mode directly from the CIWindow:
CIW> sstatus(debugMode t)     ; turn it on
CIW> sstatus(debugMode nil)    ; turn it off
CIW> status(debugMode)    ; returns t or nil indicating whether debug mode is on or off
Here is the macro definition.
(defmacro assert_debug (expression @rest printf_style_args)
 (cond
   ((null (status debugMode))
    nil)
   (printf_style_args
    `(when (and (status debugMode) (not ,expression))
       (error ,@printf_style_args)))
   (t
    `(when (and (status debugMode) (not ,expression))
       (error "ASSERT FAILED: %L\n" ',expression)))))
Here is an example usage:
(defun add_abs (a b)
 (plus (abs a) (abs b)))
(assert_debug 3 == (add_abs -1 -2))
(assert_debug 3 == (add_abs -1 2))

By tweeks on March 13, 2014
Thank you so much, Jim!  This is great stuff.  I've been wanting a SKILL unit testing library for a long time now, and was thinking I'd have to break down and write it all from scratch (maybe looking at Emacs "ert" and some Common Lisp stuff for inspiration), so I'm really glad I stumbled upon your article today for some real working SKILL examples!

By tweeks on March 13, 2014
Jim, I have a request for your next testing-related blog post: specification-based testing! Since SKILL procedures can have embedded type descriptions in the arglist(), we can leverage that to generate unit tests automatically!  (See en.wikipedia.org/.../QuickCheck for more)  

By Jim Newton on March 19, 2014
Hi Tweeks, thanks for your comments.  I'm not familiar with QuickCheck but I'd love to understand more about it.  The wiki page you referred me to is pretty sparse.

On the other hand I have developed a SKILL unit testing system for my own use.  I am more than happy to make it available to you.  You can either look at it and take ideas from it, or you may adapt it to your needs.    Let me know if you are interested and I'll make a copy available to you.

Kind regards,

Jim


By Riad on April 24, 2014
Hi Jim, This is great, I'll definitely consider these asserts for testing my SKILL codes. The one other thing that I find missing in SKILL is a documentation generation ala POD style, like in Perl for instance. Do you guys have anything hidden up sleeves ? Riad

By Team SKILL on April 25, 2014
Hi Riad, I'm glad you find the post helpful.
I do have a SKILL documentation system available as AE ware which I can make available to you.  It produces cdsfinder compatible documentation as well as HTML, but not POD style.  If you are interested I can send you a copy.
Jim

Leave a Comment


Name
E-mail (will not be published)
Comment
 I have read and agree to the Terms of use and Community Guidelines.
Community Guidelines
The Cadence Design Communities support Cadence users and technologists interacting to exchange ideas, news, technical information, and best practices to solve problems and get the most from Cadence technology. The community is open to everyone, and to provide the most value, we require participants to follow our Community Guidelines that facilitate a quality exchange of ideas and information. By accessing, contributing, using or downloading any materials from the site, you agree to be bound by the full Community Guidelines.