Introduction to Context Managers in Python

John Jacobsen
Saturday, April 20, 2013

Home Other Posts

Earlier post Processes vs. Threads for Integration Testing code python

Later post Rosalind Problems in Clojure code clojure

Context managers are a way of allocating and releasing some sort of resource exactly where you need it. The simplest example is file access:

with file("/tmp/foo", "w") as foo:
    print >> foo, “Hello!”

This is essentially equivalent to:

foo = file("/tmp/foo", "w")
try:
    print >> foo, “Hello!”
finally:
    foo.close()

Locks are another example. Given:

import threading
lock = threading.Lock()

then

with lock:
    my_list.append(item)

replaces the more verbose:

lock.acquire()
try:
    my_list.append(item)
finally:
    lock.release()

In each case a bit of boilerplate is eliminated, and the “context” of the file or the lock is acquired, used, and released cleanly.

Context managers are a common idiom in Lisp, where they are usually defined using macros (examples are with-open and with-out-str in Clojure and with-open-file and with-output-to-string in Common Lisp).

Python, not having macros, must include context managers as part of the language. Since 2.5, it does so, providing an easy mechanism for rolling your own. Though the default, “low level” way to make a context manager is to make a class which follows the context management protocol, by implementing __enter__ and __exit__ methods, the simplest way is using the contextmanager decorator from the contextlib library, and invoking yield in your context manager function in between the setup and teardown steps.

Here is a context manager which you could use to time the use of threads vs. processes in Python:

import contextlib
import time

@contextlib.contextmanager
def time_print(task_name):
    t = time.time()
    try:
        yield
    finally:
        print task_name, “took", time.time() - t, “seconds.”

with time_print(“processes”):
    [doproc() for _ in range(500)]

# processes took 15.236166954 seconds.

with time_print(“threads”):
    [dothread() for _ in range(500)]

# threads took 0.11357998848 seconds.

Composition

Context managers can be composed very nicely. While you can certainly do the following,

with a(x, y) as A:
    with b(z) as B:
        # Do stuff

in Python 2.7 or above, the following also works:

with a(x, y) as A, b(z) as B:
    # Do stuff

with Python 2.6, using contextlib.nested does almost the same thing:

with contextlib.nested(a(x, y), b(z)) as (A, B):
    # Do the same stuff

the difference being that with the 2.7+ syntax, you can use the value yielded from the first context manager as the argument to the second (e.g., with a(x, y) as A, b(A) as C:...).

If multiple contexts occur together repeatedly, you can also roll them together into a new context manager:

import contextlib

@contextlib.contextmanager
def c(x, y, z):
    with a(x, y) as A, b(z) as B:
        yield (A, B)

with c(x, y, z) as C:  # C == (A, B)
    # Do that same old stuff

What does all this have to do with testing? I have found that for complex integration tests where there is a lot of setup and teardown, context managers provide a helpful pattern for making compact, simple code, by putting the “context” (state) needed for any given test close to where it is actually needed (and not everywhere else). Careful isolation of state is an important aspect of functional programming.

More on the use of context managers in actual integration tests in the next post.

Earlier post Processes vs. Threads for Integration Testing code python

Later post Rosalind Problems in Clojure code clojure

Posts (153)

Select from below, or choose only posts for:clojure southpole misc python art code


Home


Reflections on a Year of Daily Memory Drawings art

Repainting art

Marathon southpole

Daily Memory Drawings art

Questions to Ask art

Macro-writing Macros code clojure

Time Limits art

Lazy Physics code clojure

Fun with Instaparse code clojure

Nucleotide Repetition Lengths code clojure

Updating the Genome Decoder code clojure

Getting Our Hands Dirty (with the Human Genome) code clojure

Validating the Genome Decoder code clojure

A Two Bit Decoder code clojure

Exploratory Genomics with Clojure code clojure

Rosalind Problems in Clojure code clojure

Processes vs. Threads for Integration Testing code python

Resources for Learning Clojure code clojure

Continuous Testing in Python, Clojure and Blub code clojure python

Programming Languages code

Milvans and Container Malls southpole

Oxygen southpole

Ghost southpole

Turkey, Stuffing, Eclipse southpole

Wind Storm and Moon Dust southpole

Shower Instructions southpole

Fresh Air and Bananas southpole

Traveller and the Human Chain southpole

Reveille southpole

Drifts southpole

Bon Voyage southpole

A Nicer Guy? southpole

The Quiet Earth southpole

Ten southpole

Plein Air art

ISO50 southpole art

In Defense of Hobbies misc code art

Closure southpole

Takeoff southpole

Mummification southpole

Eleventh Hour southpole

Diamond southpole

Baby, It’s Cold Outside southpole

Fruition southpole

Friendly Radiation southpole

A Place That Wants You Dead southpole

Deep Fried Macaroni and Cheese Balls southpole

Retrograde southpole

Three southpole

Transitions southpole

The Future southpole

Sunday southpole

Radio Waves southpole

Settling In southpole

Revolution Number Nine southpole

Red Eye to McMurdo southpole

That’s the Way southpole

Faults in Ice and Rock southpole

Bardo southpole

Chasing the Sun southpole

Downhole southpole

Coming Out of Hibernation southpole

Managing the Most Remote Data Center in the World code southpole

Photoshop on a Dime art

Man on Wire misc

Posing Rigs art

Metric art

Cuba southpole

Wickets southpole

Safe southpole

Broken Glasses southpole

End of the Second Act southpole

Pigs and Fish southpole

Last Arrivals southpole

Lily White southpole

In a Dry and Waterless Place southpole

Immortality southpole

Routine southpole

Tourists southpole

Passing Notes southpole

Translation southpole

RNZAF southpole

The Usual Delays southpole

CHC southpole

Wyeth on Another Planet art

Detox southpole

Packing southpole

Nails southpole

Gearing Up southpole

Gouache, and a new system for conquering the world art

Fall 2008 HPAC Studies art

YABP (Yet Another Blog Platform) southpole

A Bath southpole

Green Marathon southpole

Sprung southpole

Outta Here southpole

Lame Duck DAQer southpole

Eclipse Town southpole

One More Week southpole

IceCube Laboratory Video Tour; Midrats Finale southpole

SPIFF, Party, Shift Change southpole

Good things come in threes, or 18s southpole

Sleepless in the Station southpole

Post Deploy southpole

Midrats southpole

IceCube and The Beatles southpole

Video: Flight to South Pole southpole

The Pure Land southpole

Almost There southpole

There are no mice in the Hotel California Bunkroom southpole

Short Timer southpole

Sleepy in MacTown southpole

Superposition of Luggage States southpole

Sir Ed southpole

Shortcut to Toast southpole

Pynchon, Redux southpole

Flights: Round 1 southpole

Packing for the Pole southpole

Goals for Trip southpole

Balaklavas southpole

Tree and Man (Test Post) southpole

Schedule southpole

How to mail stuff to John at the South Pole southpole

Summer and Winter southpole

Homeward Bound southpole

Redeployment southpole

Short-timer southpole

The Cleanest Air in the World southpole

One more day (?) southpole

One more week (?) southpole

Closing Softly southpole

More Photos southpole

Super Bowl Wednesday southpole

Night Owls southpole

First Week southpole

More Ice Pix southpole

Settling In southpole

NPX southpole

Pole Bound southpole

Bad Dirt southpole

The Last Laugh southpole

Nope southpole

First Delay southpole

Batteries and Sheep southpole

All for McNaught southpole

The Big (Really really big…) Picture southpole

t=0 southpole

Giacometti southpole

Descent southpole

Video Tour southpole

How to subscribe to blog updates southpole

What The Blog is For southpole

Home