Property-Based Testing with Clojure and Datomic – Part 1

This is the first part of a multi-part series that explores clojure.spec, property-based testing, and how those tools can be used to test Datomic. This post serves mainly as an introduction to property-based testing and clojure.spec. Check out part 2 for the introduction of Datomic property-based testing.

Clojure.spec was recently released and the Clojure community is diving into it to see how specs can improve their programs. One of the more interesting design decisions made for clojure.spec is the connection between specs and property-based testing via another Clojure contrib library:
test.check.

clojure.spec and generators

Clojure specs are defined as predicate functions. So for instance if you wanted to create a spec called :db/id and the only constraint was that the value must be an integer than you can define the spec using the built in int? predicate function in clojure.core:

All examples in this post are evaluated with Clojure 1.9-alpha7. Due to the nature of Alpha releases, the examples are not guaranteed to work on newer versions of Clojure

(require '[clojure.spec :as s]
         '[clojure.spec.gen :as gen])

(s/def :db/id int?)

So with that spec defined, we can check that a given value is valid:

(s/valid? :db/id 23) ; => true

And get explanations for invalid values:

(s/explain :db/id 1)
;; val: "hey" fails spec: :db/id predicate: int?

The test.check integration works like this: any spec can be turned into a test.check generator with clojure.spec/gen, which can then be used with test.check to generate random values that conform to the spec.

(gen/sample (s/gen :db/id))
;; => (0 -1 -2 -1 -8 -1 -23 16 4 -1)

Let’s try a more complex spec:

(defn email-address? [st]
  (boolean (re-matches #".+\@.+\..+" st)))

(s/def :user/email (s/and string? email-address?))

(s/valid? :user/email "fake@email.com") ; => true
(s/valid? :user/email "not-an-email") ; => false

Now let’s generate an email!

(gen/generate (s/gen :user/email))
;; ExceptionInfo Couldn't satisfy such-that predicate after 100 tries.

What happened? Well clojure.spec isn’t magic, and it can’t generate data that matches any arbitrary predicate function that you
write. The way that the simple case works is that clojure.spec keeps a list of all the simple predicate functions included in clojure.core along with a generator programmed to create values that satisfy those predicates.

But we can create a generator that will generate “email addresses” for us.

(def non-empty-alphanumeric-string-gen
  (gen/such-that not-empty (gen/string-alphanumeric)))

(def email-generator
  (gen/fmap (fn [[a b c]] (str a "@" b "." c))
    (gen/tuple
      non-empty-alphanumeric-string-gen
      non-empty-alphanumeric-string-gen
      non-empty-alphanumeric-string-gen)))

(gen/sample email-generator)
;; => ("9@y.7"
       "W@k.M"
       "uv7@8G.9"
       "9@4q4.KA5Ep"
       "Hv3@8DpJ.7Da"
       "m@Ehu.D"
       "w@l.tRnC4U"
       "9pP6nJ3@9gL.I6S1"
       "9@i69eJN6X.9VcYUU3k"
       "U7lk@F2O9W3.RS9R")

And our custom generator can be attached to a spec with
clojure.spec/with-gen

(s/def :user/email
  (s/with-gen (s/and string? email-address?)
    (constantly email-generator)))
    ;; with-gen expects a zero-arg function that returns a generator,
    ;; so 'constantly creates that function here

Property-based testing

It’s cool to have Clojure generate a bunch of values for us, but generating values isn’t super useful on its own. Clojure.spec includes generation to run property-based tests which lend comfort that our code works correctly. Property-based tests require a different way of thinking about testing than unit tests. A typical unit test hard-codes an input value, passes that input to the function being tested, and then asserts the the function result is the same as a hard-coded output. As an example let’s imagine a function that takes a collection of collections and is supposed to return those collections sorted by length from longest to shortest.

We could write a unit test for this function that looks like this:

(deftest longest-first-test
  (testing "returns collections sorted by length"
    (is (= (longest-first [[1] [1 2 3] [1 2] [4 5]])
           [[1 2 3] [1 2] [4 5] [1]]))))

Our test passes so the function must work! Right? Well we really can’t be very sure that it does work. For all we know from this test, the implementation of longest-first could look like this:

(defn longest-first [colls]
  [[1 2 3] [1 2] [4 5] [1]])

As far-fetched as that implementation is, it still dashes some of the confidence we might have had in our unit test. For instance, what happens whenever you pass longest-first an empty collection? Also what if longest-first swapped the order of the two size 2 vectors? The functionality would have been correct, but the test we wrote would have failed unnecessarily.

By contrast, property tests are structured as: “For any valid input, [some condition] should hold”. So before you can test your code you have to state explicitly what type of inputs should be considered valid for your function. Test.check generators can help us then generate input samples. The next thing we need to do is find a way to assert a property about the output of our function without knowing what the input will be.

So first let’s generate some inputs for our function

(gen/sample
  (gen/vector
    (gen/vector
      (gen/simple-type-printable))))
;; => ([]
       []
       []
       []
       [[wB.md4/.kf] [] [-1.5625 I0O #uuid "093bcb69-167f-4ec2-be21-306a858adf34" -1]]
       [[\V -1/2 #uuid "585389d3-5360-4751-a2cc-4edf8e04c078" Q.*-v.R_a/oQ?M \P]
        [true \\ #uuid "80bcbf33-43d4-4f18-bf26-a1a2867ee369" -4/3]] ...)

The random element types will give us more confidence that our code will work with collections containing anything. Now we create a property test that makes sure the result is in an acceptable order.

(defspec prop-longest-first-correct-order 25
  (prop/for-all [colls (gen/vector
                         (gen/vector
                           (gen/simple-type-printable)))]
    (let [result (longest-first colls)]
      (->> result
        ;; split into neighbors
        (partition 2 1)
        (every? (fn [[fst scd]]
                  (>= (count fst) (count scd))))))))
;; => {:result true, :num-tests 25, :seed 1466549282870, :test-var "prop-longest-first-right-order"}

Our test ran 25 times with random collections and the property held true in each case. When we get the result from our function we check that all elements in that result are in descending order by count, just as we expect. Ordering is a nice property to test because we can easily check that the value is in the order we expect without duplicating the sorting logic code from longest-first.

That works great but actually this property test by itself is probably not sufficient. The pathological hard-coded implementation that I introduced above would still pass our shiny new property test. But we can pair it with a second property test that makes sure the result has the same number of elements as the input.

(defspec prop-longest-first-preserves-elements 25
  (prop/for-all [colls (gen/vector
                         (gen/vector
                           (gen/simple-type-printable)))]
    (= (count colls)
       (count (longest-first colls)))))
;; => {:result true, :num-tests 25, :seed 1466550267715, :test-var "prop-longest-first-preserves-elements"}

With these two property tests passing we can be much more certain that our function is implemented correctly without looking at the source. If you are super paranoid you could also check that every collection from the input was present in the output.

Clojure.Spec and property-based testing

It’s also worth noting that clojure.spec has an integration for property-based testing through the fdef form in the clojure.spec namespace. Our test.check test could be translated to part of the spec for that function.

(s/fdef longest-first
  :args (s/cat :colls (s/coll-of vector? []))
  :fn (fn [{:keys [ret]}]
        (->> ret
          ;; split into neighbors
          (partition 2 1)
          (every? (fn [[fst scd]]
                    (>= (count fst) (count scd)))))))

With our function spec in place, we can run the same kind of test using the clojure.spec.test namespace.

(require '[clojure.spec.test :as stest])

(stest/check-var #'longest-first :num-tests 25)
{:result true, :num-tests 25, :seed 1466605311988, :test-var "prop-longest-first-correct-order"}

which generates collections based on our :args spec and feed them to the function under :fn to determine if the property holds.

Hold on, why did the blog title mention Datomic?

We’ve seen how property-based testing can be can be used to robustly test a function. But one downside for property-based tests is that they are most well suited for pure functions. But often we need to test non pure code, for instance code that writes to a database, and it’s not immediately obvious how to write property-based tests for that code. Through some experience I’ve decided that the Datomic database is fairly uniquely suited for property-based testing, which I will explain in greater detail in part two.

Like it? Hate it? Let us know! Email feedback@altometrics.com.