A team of American scientists has recovered billions of dollars’ worth of “dark data” from the 1960s, pushing back the modern satellite record of sea ice extent by 17 years.
David Gallaher was eight years old in 1964, watching satellites twinkling high overhead. That year, the first American to orbit the planet left NASA, the Soviets put the first multi-person crew in orbit, and one tiny satellite, Nimbus 2, was taking grainy black-and-white images of the entire surface of the planet.
Seventeen years before the start of what we know as the “modern satellite record” of sea ice, Nimbus 2 was snapping images that would turn up on two huge pallets in Gallaher’s office in Colorado 50 years later.
“Holy crap,” Gallaher recalls thinking when he saw the daunting stack of canisters. “We took one box and looked at it, and said, ‘Is this even doable?’”
Gallaher had heard about the data from Nimbus and other weather satellites at a conference, and wondered what it might contain. He called up the National Climatic Data Centre (NCDC) in North Carolina where it was all stored, and asked if they might send him the satellite’s photos of Greenland. If he could get his hands on them, it would be a goldmine for a scientist who studies time-series data: they would be the earliest existing satellite photos of the island, giving him a special insight into the history of the largest body of ice in the northern hemisphere.
“You can’t build a time machine and go back and get this stuff,” he says. So the earlier the photos, the better. But it wasn’t going to be so easy; the NCDC office laughed off his request for sorted data. It had never been sorted before.
“If you want it, you’re going to have to scan all of it,” he remembers the office telling him.
‘All of it’ turned out to be 25 boxes full of tins containing several thousand 60-metre rolls, all unopened, all labeled with useless information on orbit numbers rather than locations. But the prize was too great, and he was running out of time: with the surviving NASA scientists who had taken the original images well into their 80s, he knew it wasn’t long before the knowledge he needed to decipher the data would be gone forever.
Gallaher started the process of sifting through roll after roll of film. The images he was scanning weren’t the originals: using the best technology of the time, the scientists had recorded the images on magnetic tape, played it back on a TV monitor, then snapped photographs of the TV monitor. What he had were those images, sporadically placed along rolls of film as long as the wingspan of a Boeing 787.
Gallaher sent the data off to a Montreal, Quebec-based company, JBI, which rescued the data for $10 a spool. By the end, Gallaher had over 200,000 images, amounting to several gigabytes of data. A truckload of film canisters fit on a thumb drive.
“That was an incredible amount of data,” he says. In the sixties, when the images were recorded, “that was more storage than there was available on the planet.”
It was worth the wait. What Gallaher and his NSIDC colleague Garrett Campbell had discovered was both the largest and the smallest Antarctic sea ice extent ever recorded, one year apart, as well as the earliest sea ice maximum ever just three years later; it was an inexplicable hole in the Arctic sea ice even while the overall extent agreed with modern trends; it was the earliest known picture of Europe from space; it was a picture of the Aral Sea with water still in it.
It was, as Gallaher puts it, like looking at “the Precambrian of satellite data.”
The team at the National Snow and Ice Data Center made the images available online in the searchable, standardized format that Gallaher originally wished it had been. The images hadn’t been intended for use in sea ice research, or in long-term study of trends, but it has been repurposed to serve a multitude of purposes.
The data dump is currently facilitating a flurry of activity as scientists around the world use the images to answer questions about deforestation, weather patterns, and any other line of inquiry that can benefit from an answer to the question, “what happened before that?”
And there is more of it out there – more canisters awaiting liberation in dusty back rooms of storage centres, and even data from entirely different sources besides satellites. Taken together, it’s called “dark data,” potentially valuable information locked away in unusable formats, unknown to most of the world, some of it never even seen by human eyes.
Spurred on by his discovery, Gallaher is in the process of starting an organization to recover more dark data, if he can raise enough money.
The original cost to American taxpayers to gather the images was in the billions (in today’s dollars). “For a few hundred thousand dollars I could get it all back,” he says.
“Before this stuff gets lost, let’s keep it.”