2018-11-18

ELISP: testing with ERT

Intro

In the past few weeks I’ve been working on a small project to get used to elisp programming, and I got to a point when I needed to make sure I wasn’t breaking anything in the process. There’re several solutions out there but the first I put my hands on was ERT.

What is ERT ?

ERT is a tool for automated testing in Emacs Lisp. Its main features are facilities for defining tests, running them and reporting the results, and for debugging test failures interactively.
— ERT main page

Writing a buggy function

In order to show how ERT works, I’m creating a buggy function. The function is supposed to sort a list of strings in ascending order by default. But in fact it returns by default the strings in descending order.

sort list tests
(defun sort-by-string-length (list &optional asc-desc)
  "Sort a LIST of strings by their length.
Apply direction by ASC-DESC value which could be 'asc' or 'desc'."
  (progn
    (fset 'direction
          (if (equal asc-desc "asc") '> '<))
    (sort list (lambda (a b)
                 (direction (length a) (length b))))))

Writing tests

In order to make sure the function does what it claims to do, we should write a test. An ERT test normally looks like the following:

basic form
(ert-deftest name-of-the-test ()
             (should ...)) ;; assertions

An ERT test uses the macro ert-deftest to declare a new test. Then it uses should to check assertions results. There’re more possibilities other than should: should-not or should-error. I’m talking about them a little bit later.

sort list tests
(ert-deftest test-sort-by-default () (1)
  (should
   (equal (sort-by-string-length '("a" "aaa" "aa")) '("a" "aa" "aaa"))))

(ert-deftest test-sort-by-desc () (2)
  (should (equal (sort-by-string-length '("a" "aaa" "aa") "desc") '("aaa" "aa" "a"))))
1 Tests default sort (ascending) by string length
2 Tests descending sort by string length

Executing tests

You can execute ERT test in different ways, but the one I prefer is batch execution via command line. Is the one I think I would use in a CI environmnet.

Of course, you will have to install Emacs in your CI environment in order to be able to execute these tests.
shell execution
emacs -batch -l ert -l /path/to/file_containing_tests.el -f ert-run-tests-batch-and-exit

And because I’ve made a mistake with the default sorting this is what I got:

test output
Running 2 tests (2018-11-18 16:08:14+0100)
   passed  1/2  test-sort-by-default
Test test-sort-by-desc backtrace:
...
Test test-sort-by-desc condition:
    (ert-test-failed
     ((should
       (equal
        (sort-by-string-length-ko ... "desc")
        '...))
      :form
      (equal
       ("a" "aa" "aaa") (1)
       ("aaa" "aa" "a")) (2)
      :value nil :explanation
      (list-elt 0
                (arrays-of-different-length 1 3 "a" "aaa" first-mismatch-at 1))))
   FAILED  2/2  test-sort-by-desc

Ran 2 tests, 1 results as expected, 1 unexpected (2018-11-18 16:08:14+0100)

1 unexpected results:
   FAILED  test-sort-by-desc
1 Expected
2 Actual result

Asserts with should

ERT provides different macros to help you make your tests easier to read and understand. The most common is should.

should
(ert-deftest test-sum-is-commutative ()
  (should (= (+ 1 2) (+ 2 1))))

Sometimes you may want to assert that the result is not what is expected, for example to assert that the division operation is not commutative. For that you can use should-not:

should-not
(ert-deftest test-division-not-commutative ()
  (should-not (= (/ 1 2) (/ 2 1))))

And I’m sure that at some point, you’ll need to assert that some function is throwing an error under some circumstances. In that case you can use should-error:

should-error
(ert-deftest test-error ()
  (should-error
   (signal 'singularity-error nil)
   :type 'singularity-error))

In this example I’m checking that the function throws an error and also that error is of type 'singularity-error.

:type is optional

Fixing function

Now it’s time to do the fix and make the test pass.

(defun sort-by-string-length (list &optional asc-desc)
  "Sort a LIST of strings by their length.
Apply direction by ASC-DESC value which could be 'asc' or 'desc'."
  (progn
    (fset 'direction
          (if (equal asc-desc "desc")
              '>
            '<))
    (sort list (lambda (a b)
                 (direction (length a) (length b))))))

Running tests again shows the following output:

Running 2 tests (2018-11-18 16:38:00+0100)
   passed  1/2  test-sort-by-default
   passed  2/2  test-sort-by-desc

Ran 2 tests, 2 results as expected (2018-11-18 16:38:00+0100)

Now everything works as expected

Resources