Continuous Testing in Python, Clojure and Blub

John Jacobsen
Saturday, March 31, 2012

Home All Posts

Earlier post Programming Languages code

Later post Resources for Learning Clojure code clojure


Figure 1: A separate monitor is handy for showing tests results continuously while working. The paintbrushes are strictly optional.

What follows is a somewhat rambling introduction to continuous, test-driven development, focusing mainly on Python and influenced by Clojure tools and philosophy. At the end, a simple script is introduced to help facilitate continuous TDD in (almost) any language.

For the last four years I have increasingly followed a test-driven approach in my development. My approach continues to evolve and deepen even as some of the limits of TDD are becoming clearer to me.

Initially, I had a hard time getting my head around TDD. Writing tests AND production code seemed like twice as much work, and, even before I wrote tests, I typically ran the program under development, e.g. with print statements added, to test each change. But making changes to old code was always a fairly daunting proposition, since there was no way to validate all the assumptions I’d checked “by eye” just after I’d written the code.

TDD helps reduce risk by continuously verifying your assumptions about how the code should perform at any time. Using TDD for a fairly large project has saved my bacon any number of times.

The basic approach is that the test code and the production code evolve together more or less continuously, as one follows these rules:

  1. Don’t write any production code without a failing unit test;
  2. Write only enough production code needed to make the tests pass.

Once I started writing tests for all new production code, I found I could change that code and make it better without fear. That led to much better (and usually simpler) code. I realized I was spending much less time debugging; and when there were bugs, the tests helped find them much faster. As I have gained experience with this approach I have found that the reliability of, and my trust in, my code written with TDD is vastly superior than otherwise. The two-rule cycle also tends to foster simplicity, as one tends to eschew any modifications that don’t actually achieve the desired objectives (i.e. meet the goals of the software, as specified formally in the tests themselves). The process is also surprisingly agreeable!

It goes without saying that if you follow this approach you will be running your tests a lot. The natural next step is to automate this more. In the book Foundations of Agile Python Development, Jeff Younker explains how to make Eclipse run your unit tests every time you save a file in the project. The speed and convenience of this approach was enough to get me to switch from Emacs to Eclipse for awhile.

Most of my daily programming work is in Python, but I have been an avid hobbyist in Clojure for several months now. It wasn’t until I saw Bill Caputo’s preparatory talk for Clojure/West here in Chicago that I heard the term continuous testing and realized that this is what I was already doing; namely, the natural extension of TDD in which one runs tests continuously rather than “by hand.” Bill demoed the expectations module and the autoexpect plugin for Leiningen, which runs your tests after every save without incurring the overhead of starting a fresh JVM each time.

(One point Bill made in his talk was that if your tests are slow, i.e. if you introduce some new inefficiency, you really notice it. Ideally the tests should take a few seconds or less to complete.)

Back to Python-land. Not wanting to be always leashed to Eclipse, and inspired by the autoexpect plugin, I started looking for an alternative to using Eclipse’s auto-builders — something I could use with Emacs or any other editor. There are a lot of continuous build systems out there, but I wanted something simple which would just run on the command line on my laptop screen while I edited code on my larger external monitor. I found tdaemon on GitHub; this program walks a directory tree and runs tests whenever anything changes (as determined by keeping a dictionary/map of SHA values for all the files). This is most of what I want, but it restricts you to its own choices of test programs.

In a large project with many tests, some fast and some slow, I often need to specify a specific test program or arguments. For example, I have a wrapper for nosetests which will alternately run all my “fast” unit tests, check for PEP-8 compliance, run Django tests, etc. In some cases, such as debugging a system with multiple processes, I may need to do something complex at the shell prompt to set up and/or tear down enough infrastructure to perform an existing test in a new way.

One piece of Clojure philosophy (from Functional Programming, a.k.a. “FP”) that has been influencing my thinking of late is the notion of composability: the decoupling or disentanglement of the pieces of the systems one builds into small, general, composable pieces. This will make those pieces easier to reuse in new ways, and will also facilitate reasoning about their use and behaviors. (Unfortunately, the merits of the FP approach, which are many, have poisoned my enthusiasm for OO to the extent that I will typically use a function, or even a closure, before using an object, which would perhaps be more Pythonic in some cases).

So, in the current case under discussion (continuous testing), rather than making some kind of stateful object which knows about not only your current file system, but also what tests should be allowed, their underlying dependencies, etc., it would be better (or at least more “functional”) instead to simply provide a directory-watching function that checks a dictionary of file hashes, and compose that function with whatever test program suits your purposes at the moment.

The result of these thoughts is a small script called conttest which is a simplification of tdaemon that composes well with any test suite you can specify on the command line.

Some examples follow:

$ conttest nosetests  # Runs nosetests whenever the files on disk change

$ conttest nosetests test.path:harness  # Runs only tests in 'harness'
                                        # object in path/to/test — handy
                                        # for developing a single feature
                                        # or fixing a single bug.
# Check both PEP-8 style and unit tests:
$ conttest 'pep8 -r . ; nosetests'

It would work equally well with a different language (“blub language”) with a separate compilation step:

$ conttest 'make && ./run-tests'

Using this program, depending on my needs of the moment, I can continuously run a single unit test, all my “fast” unit tests, or, if I’m willing to deal with slower turnaround times, all my unit and integration tests.

The script is on GitHub for your continuous enjoyment of continuous testing. May you find it helpful.

(Ironically, this script does NOT work that well for JVM languages like Clojure since the startup time is lengthy (a couple of seconds on my MacBook Pro). Most of Clojure’s testing frameworks have autotest capabilities built in, which work great.)

Earlier post Programming Languages code

Later post Resources for Learning Clojure code clojure

Blog Posts (140)


Reflections on a Year of Daily Memory Drawings art

Repainting art

Daily Memory Drawings art

Questions to Ask art

Macro-writing Macros code clojure

Lazy Physics code clojure

Fun with Instaparse code clojure

Nucleotide Repetition Lengths code clojure

Updating the Genome Decoder code clojure

Getting Our Hands Dirty (with the Human Genome) code clojure

Validating the Genome Decoder code clojure

A Two Bit Decoder code clojure

Exploratory Genomics with Clojure code clojure

Rosalind Problems in Clojure code clojure

Introduction to Context Managers in Python code python

Processes vs. Threads for Integration Testing code python

Resources for Learning Clojure code clojure

Programming Languages code

Milvans and Container Malls southpole

Oxygen southpole

Ghost southpole

Turkey, Stuffing, Eclipse southpole

Wind Storm and Moon Dust southpole

Shower Instructions southpole

Fresh Air and Bananas southpole

Traveller and the Human Chain southpole

Reveille southpole

Drifts southpole

Bon Voyage southpole

A Nicer Guy? southpole

The Quiet Earth southpole

Ten southpole

ISO50 southpole art

In Defense of Hobbies misc code art

Closure southpole

Takeoff southpole

Mummification southpole

Eleventh Hour southpole

Diamond southpole

Baby, It’s Cold Outside southpole

Fruition southpole

Settling In southpole

Revolution Number Nine southpole

Red Eye to McMurdo southpole

Faults in Ice and Rock southpole

Bardo southpole

Chasing the Sun southpole

Downhole southpole

Coming Out of Hibernation southpole

Managing the Most Remote Data Center in the World code southpole

Photoshop on a Dime art

Man on Wire misc

Posing Rigs art

Metric art

Cuba southpole

Wickets southpole

Safe southpole

Broken Glasses southpole

End of the Second Act southpole

Pigs and Fish southpole

Last Arrivals southpole

Lily White southpole

In a Dry and Waterless Place southpole

Immortality southpole

Routine southpole

Tourists southpole

Passing Notes southpole

Translation southpole

The Usual Delays southpole

RNZAF southpole

CHC southpole

Wyeth on Another Planet art

Detox southpole

Packing southpole

Nails southpole

Gearing Up southpole

Gouache, and a new system for conquering the world art

Fall 2008 HPAC Studies art

YABP (Yet Another Blog Platform) southpole

A Bath southpole

Green Marathon southpole

Sprung southpole

Outta Here southpole

Lame Duck DAQer southpole

Eclipse Town southpole

One More Week southpole

IceCube Laboratory Video Tour; Midrats Finale southpole

SPIFF, Party, Shift Change southpole

Good things come in threes, or 18s southpole

Sleepless in the Station southpole

Post Deploy southpole

Midrats southpole

IceCube and The Beatles southpole

Video: Flight to South Pole southpole

The Pure Land southpole

Almost There southpole

There are no mice in the Hotel California Bunkroom southpole

Short Timer southpole

Sleepy in MacTown southpole

Superposition of Luggage States southpole

Sir Ed southpole

Shortcut to Toast southpole

Pynchon, Redux southpole

Flights: Round 1 southpole

Packing for the Pole southpole

Goals for Trip southpole

Balaklavas southpole

Tree and Man (Test Post) southpole

Schedule southpole

How to mail stuff to John at the South Pole southpole

Summer and Winter southpole

Homeward Bound southpole

Redeployment southpole

Short-timer southpole

The Cleanest Air in the World southpole

One more day (?) southpole

One more week (?) southpole

Closing Softly southpole

More Photos southpole

Super Bowl Wednesday southpole

Night Owls southpole

First Week southpole

More Ice Pix southpole

Settling In southpole

NPX southpole

Pole Bound southpole

Bad Dirt southpole

The Last Laugh southpole

Nope southpole

First Delay southpole

Batteries and Sheep southpole

All for McNaught southpole

The Big (Really really big…) Picture southpole

t=0 southpole

Giacometti southpole

Descent southpole

Video Tour southpole

How to subscribe to blog updates southpole

What The Blog is For southpole