Ensuring Your Own Application Security

This past winter, a worm known as Slammer rattled the Internet violently enough to become what you might call a “CNN-level virus”?that is, it burrowed its way into the national consciousness.

Nearly everything about the SQL Slammer was old. It was an old hack that exploited a year-old vulnerability found in an old target, Microsoft software. There was a patch to block Slammer that was six months old, and that patch suffered from an old patch problem: It was so kludgy to install that the patch needed a patch. Above all, the reaction to Slammer?the call to use the event to build security awareness?was so old it called Bob Hope “kid.”

But this much was new: Everyone agreed that Slammer was your fault.

How to Save $60 Billion

The old game was to blame Microsoft. “Microsoft did not protect its customers,” read a letter to The New York Times after the Melissa virus hit in 1999. A year later, after the I Love You virus infected Microsoft Outlook, a Washington Post editorial stated, “This is a software development problem.” The Nimda worm (2001), according to Forrester Research, required 625 combinations of patches applied to Microsoft’s Internet Information Server. Nimda, along with its contemporary, the Code Red virus, eventually compelled Microsoft to implement and market Trustworthy Computing, an initiative aimed at helping Microsoft developers learn how to write secure code.

Slammer, though, hasn’t followed the old pattern. A developing consensual wisdom suggests that as woeful as Microsoft’s products may be, CIOs have been equally sloppy. A February poll of more than 200 IT professionals, by antivirus company Sophos, showed that 64 percent of respondents blamed their peers’ lax security practices for Slammer. Only 24 percent blamed Microsoft.

The poll also revealed that only 43 percent of the respondents said they subscribed to Microsoft’s vulnerability mailing list, which provides early alerts of viruses in the wild. Twelve percent said they relied on “mainstream news”?newspapers and TV?to learn about new viruses. Three percent said they “don’t really hear about them at all.” And 19 percent said they patched software when they “got around to it.”

“I’ve got to look around at my comrades and ask, Why aren’t you patching your systems?” says Bob Ferderer, vice president of IT internal operations and security at CUNA Mutual Group, the nation’s largest financial service provider for credit unions, with 5,000 employees and $9.3 billion in assets. “There’s a relationship between individuals not taking action and how these things spread out of control.”

What frustrates Ferderer and other security experts is the fact that this seemingly intractable problem is actually quite tractable. The tools and strategies to prevent another Slammer are just waiting to be used. In fact, the number of tools and strategies available to you?and available at a reasonable cost?makes it inexcusable for any CIO to fiddle while the software burns.

There is, after all, $60 billion on the table. A 2002 study by the National Institute of Standards and Technology (NIST) developed that number to describe buggy software’s cost to the national economy. Improved software testing alone, NIST suggests, could shave $22 billion off that.

Why can’t the software community motivate itself to grab all that cash? The answer lies in software culture.

Vendors, for the most part, value time-to-market over security. As long as they can get away with shipping buggy code, they will. (See “Sledgehammer,” Page 64.)

Developers live by deadlines, which compel them to work fast. At the same time, they’re being asked to provide ever more features.

And CIOs, as a group, have been passive, assuming there was little they could do to effect change.

They assumed wrong. In fact, a growing number of advocates believe CIOs should be leading the charge for secure software.

“CIOs must take action,” says Linda Northrop, director of the product-line systems program at the Software Engineering Institute (SEI) and coauthor of Software Product Lines: Practices and Patterns. “I think CIOs have done a deplorable job matching their software decisions to business goals, especially in its security and quality. What we need from the CIO ranks are leaders.”

Northrop could be talking about Al Schmidt, vice president of IT and CIO of $939 million Arch Chemicals. “What I’ve come to realize,” says Schmidt, “is that security is really about operational excellence. So why wouldn’t I jump on that? I mean, operational excellence?that’s what I’m supposed to be doing, right?”

Know Your Enemy

The vast majority of vulnerabilities in software arise from the following 10 basic development flaws.

1. Unvalidated parameters: Anyone can change a URL to access data. For example, switching “admin=no” to “admin=yes” within the URL gives the hacker admin privileges.

2. Broken access control: Software doesn’t check a user’s authorization properly, which means that credentials are cached and can be co-opted by anyone.

3. Broken account and session management: Hackers can take advantage of predictable validation schemes by, for example, manufacturing or altering cookies.

4. Cross-site scripting: User hits a poisoned webpage, which his browser accepts uncritically, allowing the hacker to access his machine.

5. Buffer overflows: Miscreant is allowed to input so much data that memory gets junked up and starts accepting malicious commands.

6. Command injection flaws: User’s Web application executes malicious commands because it doesn’t know any better.

7. Error-handling problems: Error messages give away information?for example, “ODBC Error” signals that a SQL injection (such as the SQL Slammer) is possible.

8. Insecure use of cryptography: Use of homegrown, weak cryptography is easily compromised by professional bad guys.

9. Remote administration flaws: Web apps allow remote control of machines on a known port with predictable default configurations, making hacking child’s play.

10. Web and application server misconfiguration: Unused features that leave ports open make servers vulnerable.

SEI studied 2,500 software products and 56,000 security incidents, and found the vast majority of problems are caused by those defects. Buffer overflows alone account for 40 percent to 60 percent of incidents, says CEO Steve Cross. What’s more, the flaws are, from a coding perspective, simple to fix.

“The thing that frustrates me is people think that since the problem is so prevalent, it must be complex,” says Cross. “It’s not. If the public understood that any freshman computer science student knows how to fix these problems, there would be an outcry. And there should be.”

The trick with code defects is to find them. A million-line software program, which is about the size of the average CAD/CAM package, comprises 20,000 pages of text. Finding errors?even in poorly written programs that will have two or so errors per page?is like finding needles in a binary haystack. And, once you do find the vulnerabilities, you have to make sure the fix doesn’t break anything else.

This used to provide a reasonable excuse for letting bad software practices continue. But a new class of application scanning software is emerging that makes the process of finding flaws almost trivial.

How to Take Responsibility for More Secure Software

Scan Everything…

Application scanning is based on the proposition that programming errors are reasonably predictable and limited, even if their consequences are not. (See “Application Scanning Vendors and Products,” Page 68, for eight top scanning tools on the market.)

Once you buy (one large insurance company recently paid $120,000) and then train your developers and IT staff to use the application (which should take anywhere from two days to two weeks), plan to use it in two ways.

First, make app scans a mandated part of all application audits. Whether it’s internally developed code or third-party software, create a checkpoint at which an application must contain fewer than a certain number of bugs, with all egregious errors eliminated.

“We’ve decided to push back,” says Ferderer’s partner, Tim Burke, IS manager at CUNA Mutual. “Any new software we’re developing or having developed must go through an error-free scan before it’s deployed.” Second, integrate the application scanners into the development process by training your developers and mandating that outside developers scan their work at predictable intervals. (This was made easier for .Net developers last February when Microsoft integrated the application scanner from Sanctum into Visual Studio .Net.)

Expect developers to freak out about that. “The first time we had one of our developers run the scan on some sample code he wrote, his eyes bugged out,” says Erick Weber, vice president of enterprise information security at IndyMac Bank, which holds $9.6 billion in assets and last year earned $600 million. “He thought he wrote pretty good code, but he had created significant security holes.

“But it made him want to learn how to integrate the scanning into his development. That’s what we want. The last thing we want is to have developers continue coding the same way, run scans on their work, and then throw the code back at them and say, Hey, fix this.”

…But Know Scanning’s Limitations

It would be folly to deploy app scanning and think the security problem is therefore licked. Know the limitations of these tools.

First, they don’t know what they don’t know. That is, they scan for known vulnerabilities, the most common ones. The tools will have to evolve as vulnerabilities evolve.

Second, the tools don’t know whether a flaw is dire or benign.

Third, some of these scanning apps just tell you where the holes in the roof are, some others put buckets under the holes, and still others even suggest how to patch the roof. But none of them actually fix anything.

“My concern is people buy them and think that alone fixes the problem,” says Bill Guttman, director of the Sustainable Computing Consortium?a collaborative designed to protect the nation’s computing infrastructure and improve the reliability of its IT systems. “It happened with antivirus [software]. Firewalls. People develop a false sense of security.”

The next generation of application scanners should address some of those issues. Plan for an ongoing investment in application scanning. A divisional IS officer at one of the nation’s largest banks (who asked not to be named) says his company will spend $30,000 to $40,000 per quarter scanning and auditing a major application.

IndyMac’s Weber has no trouble justifying the cost. As an ancillary benefit, he uses the tools to convince both developers and his executive peers of the value of security. He says security is like pollution; no one cares until they can see it. “I can rattle off really great reasons we need application security to users and management,” says Weber. “But they never really understand until they see, you know, eight buffer overflows, right there on their screen.”

Pull Rank on the Development Team

Application scanning is a useful tool, but it’s also fixing a problem 10,000 times rather than solving it once. If the 10 basic software flaws are so easy to avoid, why don’t developers do so?

The answer is that developers have two masters: features and deadlines. Until recently, security was not a feature. And dealing with it usually threatened a deadline. (There’s an adage among developers: Speed to market, number of features, level of quality. Pick two.)

“It’s why we published the 10 vulnerabilities,” says Mark Curphey, chairman of the Open Web Application Security Project (OWASP), an open-source project that develops Web application and Web services tools. “Developers understand the vulnerabilities technically, but they don’t change how they code because of them. But when CIOs get the list, right away they understand.”

Bring the list to a meeting with the person in charge of application development. Each of the 10 vulnerabilities can be assigned to either the development group or the IT-operations group, or in a few cases, both.

“As CIO, you need to be pretty intolerant of development that doesn’t address these basic issues,” says Schmidt, the Arch Chemicals CIO who, as a former developer, has come to appreciate the need for a sit-down between the CIO and the head of developers. “Absolutely get talking. And bring some of the numbers on how much you save by catching bugs early. That data is out there.” (See “Bug Economics,” Page 62.)

Give QA Some TLC

The development culture that rewards features delivered quickly also scorns quality assurance (QA). Here’s how several developers view the code testing piece of QA:

• “Thankless.”
• “Entry-level button pushing.”
• “You know your career is advancing when you don’t have to do testing anymore.”

It’s imperative for the CIO and the head of developers to reward developers who don’t feel that way.

At CUNA Mutual, Ferderer has recruited top developers and trained them to write secure code. He created a mentor program in which developers bring their code to their quality mentors and together they work on securing it. Ferderer says it’s early, but there’s anecdotal evidence that code quality is improving.

Make no mistake, forcing developers to get involved with QA puts you at the fulcrum of some wrenching cultural change in a guild whose practices are 30 years entrenched.

“There’s a fair bit of marketing involved because, no question, it’s a politically tricky situation,” says the aforementioned unnamed divisional IS officer of a large bank. “No matter how you approach development, they’ll resist. They’ll say you’re threatening their timetables. If you’re a jerk about it, you end up losing and things don’t get secured.”

“There is a delicacy to confronting development,” agrees IndyMac Bank’s Weber. “The application scanning tools add some objectivity. The CIO should also set development requirements and get management buy-in first. That adds more objectivity. If you clearly define the security parameters, development can’t make it a personal argument or a grudge match.”

It’s easy to see how it could devolve into politics and sniping when Weber explains what he’s really doing. “My job,” says Weber, “is to impose a security will on the developers.”

Use Freely Available Security Standards

Start with NSTISSP No. 11, the national security standard that mandates that any software used in a national security setting must pass certain government audits. Learn the criteria, and then demand that your developers and vendors meet them. (Go to www.cio.com/bugs for the NSTISSP No. 11 standard.)

The government has many other security standards. (Go to www.cio.com/bugs for a list.) None is a defining standard but virtually every one of them contains something useful. Special Publication 800-27, a NIST document, for example, contains 33 application security principles. (One of them: Implement least-user privilege, which means start with all access turned off and turn it on only as needed, not vice versa.)

It’s important to note that most of the standards are foundational. That is, they’re most useful for software at the design and requirements phase, and less useful for applications that have already been developed and deployed.

Put Security in Writing

Ferderer now requires that his vendors do application scanning on every software package Mutual deploys.

“The trend to put security right in contracts has become quite successful,” says OWASP’s Curphey. “It’s more common and more accepted than ever, in part because there are the tools which, to a degree, lend objectivity to the security of an application.”

A contract signed between General Electric and the software vendor General Magic last year excited security experts. (Go to www.cio.com/bugs to see the full contract.) Section 7.3 is called Code Integrity Warranty, and it holds the vendor financially accountable for bad software and requires the vendor to fix it.

Tick Off These To-Dos Too

After buying the software, reeducating your developers, poring over standards and hanging out with contract attorneys, you can (if you have the energy):

• Check out OWASP. Weber at IndyMac Bank lifts heavily from the OWASP guidelines for secure Web application development.
• Read Winning with Software, by Watts Humphrey, and have the developers read Writing Secure Code, by Michael Howard and David LeBlanc.
• Send your developers to school. College-level computer science classes in writing secure code are starting to appear. At SEI, fellow Humphrey (called “the Edwards Deming of software”) has developed an entire development methodology for secure coding called the Team Software Process (TSP). Microsoft recently put a small group of coders through TSP’s intense two-week training. The group was then charged with rewriting a 24,000-line application under the TSP process. The goal was to reduce the number of defects in the program from 350 to about 22. Microsoft says a post-production defect costs it $4,200. If the company meets its goal, the total cost of post-production defects in this small application (it’s an internal one) would shrink from $1,470,000 to $92,400.

Don’t Give Up, Don’t Ever Give Up

Asked if he was more optimistic now about security than he was four years ago, when Melissa first hit, Rich Pethia says flatly, “No.”

That’s an unsurprising response from the director of the CERT Coordination Center, charged with disseminating early notice of serious vulnerabilities. CERT has logged over 182,000 security incidents in 14 years, 82,084 in 2002. From 1995 to 2002, CERT mapped 9,162 vulnerabilities, nearly half of them last year.

So you’d expect Pethia, who has briefed President George W. Bush on such matters, to be gloomy. Still, Pethia thought for a minute and then revised his stance.

“Actually, I am a little more optimistic,” he says. “There’s far more awareness. The economy is terrible, so people can’t afford insecure applications. After 9/11, there’s a national sense of what these vulnerabilities mean. We’re starting to put numbers on the problem and use risk assessment tools. That means insurance will soon get in the game, which is always a big step for security.”

Combine these factors, Pethia says, and there’s a breath of hope. In fact, he’s convinced that one more ingredient will validate his optimism: proactive CIOs who demand better software.

So what exactly are you waiting for?