CSO: Not only are they possibly making themselves look bad if they find a problem, they're also creating more work for themselves, right?

JOHNSTON: Absolutely. In many cases when the fix is very simple, organizations are very reluctant to do it, because that is sometimes thought of as saying, "We've been screwing up all these years." So you don't want to go with people who have a history of doing a vulnerability assessment and then telling you everything is swell. There are always vulnerabilities, and they are always present in very large numbers. Any vulnerabilities assessment that finds zero vulnerabilities is completely useless.

CSO: When you actually do the assessment, are there warm-ups you can do to get yourself in the mind-set of a bad guy, or are there ways you should set up the room?

JOHNSTON: A lot of vulnerability assessment needs to be very similar to classic brainstorming. A lot of the tools that are applied to creative thinking in other fields can be applied directly to vulnerability assessments. This is kind of a radical position. A lot of people in the security business are not comfortable with this 1960s hippy, touchy-feely, "let's all get together" approach.

CSO: I'm imagining a bunch of beanbag chairs.

JOHNSTON: Yeah. A lot of people would much rather have a rigorous, quantitative approach, and I would claim that's largely a sham. I don't think it's a mistake to use analytical tools like a security survey, but we would like to combine those more closed-ended, straightforward tools with creative thinking. The fact is that creativity has been studied extensively over the last 50 years, and there's a lot of understanding of how you create an environment where people come up with good ideas. It's not quite the seat-of-the-pants, wacky kind of thing that it might look like from the outside.

CSO: Should the CSO even be there?

JOHNSTON: You don't want the boss in the room, because it constrains people. What you need are really nutty ideas, so we strongly encourage thinking about attacks that involve Elvis impersonators and flying monkeys and the use of space aliens. Early on, it's very important not to editorialize. Later on, we're going to prioritize them and think about the practicality of them. In many cases, we have people say, "Well, if I had the space aliens come down with a ray beam, they could do the following." Later on, it turns into a very viable attack, once we get rid of the space aliens and the laser beams.