John Jacobsen
Tuesday, January 18, 2011

Home All Posts

Earlier post Settling In southpole

Later post Baby, It’s Cold Outside southpole


Back in the Galley again, the smell of dozens of lunch burgers grilling and the very same licorice spice Stash tea I always drink here setting the olfactory tones for this post.

Almost exactly twenty years ago, when I was working at CERN in Geneva, Switzerland, I was preparing to return to Madison to take some art classes and I emailed some physics colleagues there to see if there was any student work to be had. Colleagues Bob Morse and Francis Halzen replied that there was a new project under way, which was to build a neutrino detector in the Ice in Antarctica. Bob had done an experiment at the Pole to detect cosmic rays, and Francis was a particle theorist who specialized increasingly in astro-particle physics, and neutrinos in particular. Computer simulations were needed to see how effective light sensors, placed deep in the ice, might be at detecting neutrinos.

What started as an art-school day-job turned into a doctorate in physics as the Antarctic Muon and Neutrino Detector Array (AMANDA) became a reality and the computer simulations I and other students wrote became important tools for the understanding of the data from the first instruments deployed by the growing collaboration in the polar ice cap. But in the very earliest years, it was understood even by using back-of-the-envelope calculations that a much larger detector would be needed… a Cubic kilometer of Ice, in fact … an Ice Cube. AMANDA paved the way by detecting high-energy neutrinos, by setting limits on astrophysical sources, and, perhaps most crucially, by providing technological prototypes for the hot water drilling which would be needed to deploy the sensors, and for the design of the sensors themselves which evolved from simple photomultiplier tubes at the ends of long coaxial cables to sophisticated computers in their own right which would provide the maximum signal quality by digitizing the light down in the ice and sending them to the surface with no loss of signal quality.

The difficulties faced by AMANDA were daunting. The drilling in particular was an immense engineering challenge, requiring a kind of specialized expertise found in only a handful of individuals (if that many) worldwide. One man in particular, the late Bruce Koci, brought a kind of preternatural understanding, intuition and tenacity to the task which allowed the 2-km holes for the AMANDA instrument to be drilled successfully, though not without delays, logistical and engineering nightmares, burst hoses, and endless worry Collaboration-wide. After the holes were drilled, deployments of the “strings” of sensors themselves were an ordeal in their own right, sometimes involving ‘stuck’ strings, free-falling cables, injuries and, in the best of times, 20-hour deployments out-of-doors in a single shift (one of my most vivid memories of these deployments was the enormous amounts of food one could eat during the breaks, and the utter contentment of exhaustion when the task was done).

Handling the data in AMANDA times was a challenge in and of itself. The data volume and mathematical complexity of the analysis strained the computing resources available at the time; crunching the numbers to find the first neutrino candidates required time on Cray supercomputers and a scrum of dedicated grad students and postdocs working at the limits of their programming abilities.

Since it was understood that AMANDA would eventually give way to a larger instrument, work began well before the final stages of AMANDA to design the IceCube detector and to procure funding. Several competing prototype designs for a next-generation sensor were created and tested in later AMANDA deployments, including a design created while I was at LBNL in Berkeley. The LBNL “Digital Optical Module” (DOM) took some of the signal processing computations down into the ice itself in order to get the maximum signal quality. Though more sophisticated (and expensive) than the usual AMANDA modules, the interface to the surface required only the usual copper coaxial cables, with far better quality of data. A test AMANDA “string” (#18) deployed in 2000 worked well enough (barely!) to show that the DOM technology was workable and could be used to detect particles in the ice. The early LBNL prototype would be extensively improved for IceCube.

IceCube itself received a first real jolt of funding in 2002 and work began to quickly implement the vision of a kilometer-cubed detector. I was put to work on the communications interface which would connect the deployed DOMs down in the ice to a network of off-the-shelf computers running on the surface, a design which would make IceCube an experiment with relatively little custom-built electronics from a high-energy physics standpoint (the DOMs, cables, and a computer interface card plugged into a commodity chassis are the only hardware we had to design ourselves… the rest of the experiment looks like a data center you’d find at any medium-sized Internet company).

The first three IceCube holes were planned for early 2005, but the first version of the new drilling equipment (an enormous engineering task in and of itself) proved exceptionally challenging, and after an injury in the drill camp and many other difficulties, the first IceCube string was deployed in early January. After freeze-in, the DOMs were powered on for the first time by National Science Foundation Representative Jerry Marty and Station Manager BK Grant, with all of us watching with bated breath to see if the DOMs communicated. They did.

The following seasons were all very challenging. Drilling improved substantially with experience, with 8, 12, 19, 19, 20 and 7 strings deployed in the following years, respectively. Software was written, and rewritten, to read out the data generated by 5500 DOMs and to progressively reduce the huge volume of noise to a size small enough to be sent over the satellites to the North. Tasks such as writing the embedded software for the DOMs themselves, to the system which sifts through the detected light to “trigger” on signals from real particles traveling through the Ice, to the filtering of these particles for neutrinos, to the overall control and monitoring of the system, to the simulation of the entire process to be able to make firm quantitative statements about what we are seeing — all this has taken a tremendous amount of effort, collaboration, acquisition of knowledge and skill, political struggles, failures and triumphs.

It has seemed that since we started this project, there has been barely enough time to catch one’s breath. Now we are at the end of the construction of this utterly strange device. I have put nearly a decade of my life into this project alone. If you count AMANDA, I have spent almost half my life in this field.

Last night we took the first runs with the completed IceCube detector — all 86 strings. Though hopefully the experiment will run for decades, in some sense last night’s runs were for me the fruition of years of effort, since I have been more a builder than a user of this instrument. I was going to write more about how it feels to hit this milestone (one which I was never at all certain we would reach), and less about the history… but how I feel seems somehow not that important right now. We built it. Many of us. It is good. May it run long and see deep into the fabric of things. May it teach all of us about many new and beautiful things. May we all benefit from working together and take the knowledge gained, from the trivial to the profound, out into the world to do good things.

Time to greet the dozen or so of new IceCube arrivals from the flight landing now — the last big batch of ‘Cubers for the season… maybe the last big batch ever.


Figure 2: IceCube on-Ice lead Greg Sullivan starts the first 86-string run using IceCube Live


Earlier post Settling In southpole

Later post Baby, It’s Cold Outside southpole

Blog Posts (140)


Reflections on a Year of Daily Memory Drawings art

Repainting art

Daily Memory Drawings art

Questions to Ask art

Macro-writing Macros code clojure

Lazy Physics code clojure

Fun with Instaparse code clojure

Nucleotide Repetition Lengths code clojure

Updating the Genome Decoder code clojure

Getting Our Hands Dirty (with the Human Genome) code clojure

Validating the Genome Decoder code clojure

A Two Bit Decoder code clojure

Exploratory Genomics with Clojure code clojure

Rosalind Problems in Clojure code clojure

Introduction to Context Managers in Python code python

Processes vs. Threads for Integration Testing code python

Resources for Learning Clojure code clojure

Continuous Testing in Python, Clojure and Blub code clojure python

Programming Languages code

Milvans and Container Malls southpole

Oxygen southpole

Ghost southpole

Turkey, Stuffing, Eclipse southpole

Wind Storm and Moon Dust southpole

Shower Instructions southpole

Fresh Air and Bananas southpole

Traveller and the Human Chain southpole

Reveille southpole

Drifts southpole

Bon Voyage southpole

A Nicer Guy? southpole

The Quiet Earth southpole

Ten southpole

ISO50 southpole art

In Defense of Hobbies misc code art

Closure southpole

Takeoff southpole

Mummification southpole

Eleventh Hour southpole

Diamond southpole

Baby, It’s Cold Outside southpole

Settling In southpole

Revolution Number Nine southpole

Red Eye to McMurdo southpole

Faults in Ice and Rock southpole

Bardo southpole

Chasing the Sun southpole

Downhole southpole

Coming Out of Hibernation southpole

Managing the Most Remote Data Center in the World code southpole

Photoshop on a Dime art

Man on Wire misc

Posing Rigs art

Metric art

Cuba southpole

Wickets southpole

Safe southpole

Broken Glasses southpole

End of the Second Act southpole

Pigs and Fish southpole

Last Arrivals southpole

Lily White southpole

In a Dry and Waterless Place southpole

Immortality southpole

Routine southpole

Tourists southpole

Passing Notes southpole

Translation southpole

The Usual Delays southpole

RNZAF southpole

CHC southpole

Wyeth on Another Planet art

Detox southpole

Packing southpole

Nails southpole

Gearing Up southpole

Gouache, and a new system for conquering the world art

Fall 2008 HPAC Studies art

YABP (Yet Another Blog Platform) southpole

A Bath southpole

Green Marathon southpole

Sprung southpole

Outta Here southpole

Lame Duck DAQer southpole

Eclipse Town southpole

One More Week southpole

IceCube Laboratory Video Tour; Midrats Finale southpole

SPIFF, Party, Shift Change southpole

Good things come in threes, or 18s southpole

Sleepless in the Station southpole

Post Deploy southpole

Midrats southpole

IceCube and The Beatles southpole

Video: Flight to South Pole southpole

The Pure Land southpole

Almost There southpole

There are no mice in the Hotel California Bunkroom southpole

Short Timer southpole

Sleepy in MacTown southpole

Superposition of Luggage States southpole

Sir Ed southpole

Shortcut to Toast southpole

Pynchon, Redux southpole

Flights: Round 1 southpole

Packing for the Pole southpole

Goals for Trip southpole

Balaklavas southpole

Tree and Man (Test Post) southpole

Schedule southpole

How to mail stuff to John at the South Pole southpole

Summer and Winter southpole

Homeward Bound southpole

Redeployment southpole

Short-timer southpole

The Cleanest Air in the World southpole

One more day (?) southpole

One more week (?) southpole

Closing Softly southpole

More Photos southpole

Super Bowl Wednesday southpole

Night Owls southpole

First Week southpole

More Ice Pix southpole

Settling In southpole

NPX southpole

Pole Bound southpole

Bad Dirt southpole

The Last Laugh southpole

Nope southpole

First Delay southpole

Batteries and Sheep southpole

All for McNaught southpole

The Big (Really really big…) Picture southpole

t=0 southpole

Giacometti southpole

Descent southpole

Video Tour southpole

How to subscribe to blog updates southpole

What The Blog is For southpole