What went wrong with the Hubble Space Telescope (and what managers can learn from it)

There's nothing unusual about having a bad day at the office. But some people have worse days than others, and in his time Charles (Charlie) Pellerin has had a few notable ones. Not many people find themselves having to explain why an organisation has invested a decade and half and in the vicinity of $3 billion on a project that has failed.

That's the position Pellerin found himself in as NASA's director of astrophysics in the wake of the 1990 launch of the Hubble Space Telescope, which had what appeared to be an unfixable flaw in its optical system.

It's difficult to overstate what a disaster this was and the humiliation faced by NASA; not just as an organisation but also the individuals who worked for the agency. A good friend of Pellerin who worked on the telescope fell ill in the wake of the launch and died. Two of Pellerin's senior staffers had to be removed from their offices by guards and taken to alcohol rehab facilities. "These are PhDs sitting at their desk getting drunk; this is how bad the stress was," says Pellerin.

Most people faced with a disaster on the scale of Hubble might want to either bury themselves under a blanket in bed for a decade or two, or try (no doubt unsuccessfully) to forget it ever happened. Instead Pellerin set out to try to fix Hubble — and succeeded, in the process winning NASA's Distinguished Service Medal, the highest honour conferred by the agency. And with a stubbornness that some people may consider verges on the perverse, set out to discover exactly what went wrong. The problem with Hubble, Pellerin concluded, wasn't merely a technical failure. It was a leadership failure and a product of the culture surrounding the project.

Techworld Australia features:

Hadoop: How open source can whittle Big Data down to size

Dries' vision for Drupal 8

The road to a successful open source project: Learning lessons from Drupal

Open Source Ecology: Can open source save the planet?

Python vs. PHP: Choosing your next project's language

His study of why projects fail also led him to draw links between Hubble and an earlier NASA disaster: The disintegration of Space Shuttle Challenger on January 28, 1986, which killed seven astronauts.

"In 1986 I drove into my parking place at NASA headquarters and my staff members are waiting for me there and it's very unusual," Pellerin says. "They ran up to me and said, 'Charlie, Charlie, Challenger exploded." Pellerin's division had the biggest payload on Challenger, and in the lead-up to the launch there had been a lot of cross-training between his division and the astronauts. "I had this sinking feeling that maybe my payload had come loose, because it was a very low cost effort and it was big and heavy and maybe it had broken through the cargo bay doors and caused this accident."

Pellerin watched the Challenger failure review closely. "I had good friends who I thought were good solid engineering managers who worked on the boosters so I'm trying to figure out what happened," he says.

"I saw this guy, Richard Feynman, who was a review board member, take a piece of rubber O-ring and put it in his icy water on television, and showed that it stiffened up. So immediately I said, 'Oh, that's the technical problem, they didn't do the O-ring well.'"

"That was nuts," Pellerin says. "These guys understood the O-ring, but I put that story in my head because technical people look for technical answers. I never read the conclusion of [the review board] report that said it was a social shortfall."

Four years after Challenger Pellerin was getting ready to launch Hubble and grappling with the difficulties of readying a telescope that wasn't intended for in-atmosphere operation. The advantage of a telescope in space is that light from stars won't be moved around by atmospheric incoherencies. No-one had attempted to design a telescope that would offer the accuracy promised by Hubble.

"So the question is, what are you going to tell people if someone asks you if it's going to work? What would you say? 'Of course.' It's the only answer right?" Pellerin says. "You spend 15 years and $3 billion or whatever, so of course it's going to work — there's no other answer.

"The head of NASA congressional appropriations asked me, 'Is it going to work?' And I said, 'Of course!' So she went and identified herself very closely with the project. So then we find it doesn't…"

The circumstances under which Pellerin discovered Hubble's flawed mirror were "awkward". There was a 'first light' event for the opening of Hubble's aperture door — "we call it the toilet seat," Pellerin says. The aperture door was opened, and a little spot of light appears — the first light from Hubble. "Everybody whoops and cheers," Pellerin says. But he noticed the spot of light was fuzzy. He was reassured by a colleague that it was nothing serious; Hubble's secondary mirror was attached to a stepping motor that would allow minor alterations to cope with dimensional changes brought about by the outgassing of water vapour in space.

Page Break

Pellerin went and spent a week in Japan during which he was out of contact with NASA. He was unprepared for what awaited him. "I land in St Louis airport and I made my first mistake: I called Washington," he says. His secretary asked him if he had spoken to his boss lately, and when he replied in the negative she put him straight through. "I'm thinking, 'What good news could this be?'" Pellerin recalls.

"He says, 'What do you know about spherical aberration?' I said all I know is that when amateurs build mirrors and do it sloppily they get what's called a downturned edge, and on different radii of the mirror it focuses the light on different parts of the optical axis and it's physically impossible to ever focus a telescope; they're useless."

Pellerin's boss responded: "Well you launched Hubble Space Telescope with a spherically aberrated mirror."

"This was two PhDs and this was the maturity of the conversation," Pellerin says: "I said, 'Did not.' He said, 'Did so.' 'Did not.' 'Did so.' 'Did not.' 'Did so.'" Finally his boss told him to go and pick up a national newspaper and read out loud the front page headline above the fold.

"So I find the St. Louis Post Dispatch and I bring it back and I pick up the phone and I read it. It says 'National disaster: Hubble launched with flawed mirror'. So he said to me, 'Now what do you say?' And I said — 'You guys are too much; how'd you get a fake newspaper printed in this airport lounge?'"

Pellerin returned to Washington DC for a crisis meeting with NASA's top administrators. What followed were congressional hearings into the disaster. "It's a misnomer," Pellerin says. "There's no-body hearing at congressional hearings; they should be called congressional browbeatings or congressional yelling sessions."

Pellerin was appointed NASA liaison to the failure review board that was formed. The mirror was fabricated in 1977 and he didn't take his position until 1982, so "the thinking was I could have had nothing to do with whatever problem the mirror had. It turns out that was wrong."

The technical error related to the null corrector used to test Hubble's aspheric mirror. Using a null corrector, imperfections in a mirror can be found and fixed by the person fabricating it. The mirror was balanced on what NASA described as a 'bed of nails': A series of steel pins with little springs that maintained the mirror in the shape that it would have in zero gravity. NASA built a reflective null corrector, because even optical glass has inhomogeneities that can cause refraction; this isn't a problem with a mirror. "Looked foolproof," Pellerin says. The null corrector was used first used to build a 60-inch mirror. NASA tested the mirror extensively. "It's just a throwaway mirror, we did everything to it. Perfect mirror."

After that, to build the telescope's 96-inch mirror "all you had to do was respace and put a new field lens on [the null corrector], which is really simple. But because we're building the world's most perfect mirror we didn't use a more normal process like micrometers or something." Instead, they obtained precise metering bars from the US National Bureau of Standards (now called the National Institute of Standards and Technology). The metering bars were used to respace the reflective null corrector’s two mirrors  to produce the larger flight mirror.

NASA's Hubble Space Telescope Optical Systems Failure Report (known as the "Allen report", after Lew Allen, director NASA's Jet Propulsion Laboratory who headed the review in Hubble) states: "The ends of the metering rods were rounded and polished because the very precise positioning of the optics in the RNC [reflective null corrector] used an interferometer, rather than a mechanical measurement. This procedure involved auto-reflecting a focused beam of light off the end of a rod and observing an interference pattern from the beam that came back on itself. Centering the light beam on the rod end was essential for the measurement.

"To prevent the metering rod from being misaligned laterally with respect to the interferometer axis [so that the light hit the precise centre of the rod's curved end — RP], P-E decided to attach 'field caps' to one end of the rod…  The field caps were fitted over the rod ends and had a small aperture in the center to ensure centering of the rod on the beam."

"The instructions were take the cap and spray paint it," Pellerin says.

"The guy was working under great stress because we were angry about costly delays, and were threatening to put Kodak's mirror in the telescope, which would have been humiliating for our contractor [Perkin-Elmer Corporation was responsible for the mirror]. So he's working, he's really hurrying, he can't find the spray paint. So instead he puts black tape on [the cap] and he takes his X-Acto knife and cuts a hole. He doesn't notice he made a shiny little burr. So he puts the thing on, the light hits the burr and goes  back up.

"His instructions were to move the thing in every direction to see if the intensity of the light went down. So he moves it. Light goes onto the black tape, onto the black tape, onto the black tape, down the hole [in the field cap]. He assumes he got it centred."

However, the light had hit the burr instead of the top of the rod, which led to a gross misplacing of the two mirrors.

Page Break

Diagram from the Allen report.

It was a misalignment of about 1.3mm. "Holy Christ," says Pellerin. "In an optical system this is like missing by a thousand miles."

The review board had considered three possibilities when it came to the null corrector, according to the Allen report:

"(1) The field lens was inserted backward.

"(2) The index of refraction of the field lens was incorrect (i.e., the wrong glass was used).

"(3) The optical elements were incorrectly spaced (a circumstance that seemed highly unlikely because of the method used to set the lens spacings)."

After an analysis of the null corrector, which had been stored by the contractor after the mirror was finished, number three turned out to be the culprit,"

"So the next thing that happened was really kind of interesting," Pellerin says.

"Being a technically trained person and completely unaware of the power of social constructs, I thought 'Great it's a technical failure.' I mean it's not my fault, I wasn't even there. I'm off the hook — that's what I'm thinking. Because these things are usually unpleasant when you're responsible for them."

But the chair of the failure review board wanted to look into it further. What he found was that when the mirror was removed from the bed of nails and put in its three point mount, it was tested again. "We tested that mirror over and over and over with a different kind of device, the old style refractive null corrector," Pellerin says.

The results? "Half wave of error, half wave of error, half wave of error."

"So some people sat down and said, 'What's going on?" Pellerin recalls. "The mindset was that the mirror can not be other than perfect. So something else is happening. They concluded that the mirror was sagging under the force of gravity in the three point mount rather than being on the bed of nails by half a wave.

"Well it turned out that was wrong. But they rationalised, rationalised, rationalised. What kind of minds does a project like Hubble attract? The best. So [Allen] said, 'I want to understand why the smart guys working on it didn't go dig in and find out what's going on.'"

The project had suffered other challenges beyond fabricating and mounting the mirror; staff were being "hammered" all the time, Pellerin says. In addition there was constant angst about how far the project had gone over budget. "Hubble's initial budget was $434 million we closed it at $1.8 billion just for the flight segment; big overruns."

"So the way it works is you tend to blame the people doing the work," Pellerin says. "So we're hammering on them, hammering on them so they had no free time or no inclination to track down anything that wasn't a critical problem because we have other critical problems. Difficult technical things that we couldn't solve yet."

The review board also found that a hostile environment had been created for the contactor, which meant "they told us about any problem at their peril," Pellerin says.

1 2 Page 1
Page 1 of 2
7 secrets of successful remote IT teams