Pirates, Cheats and IT Certs

Some ne'er-do-wells steal test questions and answers, and cheaters buy that information, share answers in chat rooms, pay other people to take tests for them and bring a range of technologies and techniques into test centers to gain an edge.

1 2 Page 2
Page 2 of 2

Internet Testing Systems LLC sells software and online proctoring services that IT certification programs and test centers can use via a private-label portal to deliver content over the Internet to test takers anywhere. "We stream encrypted test items one at a time and only decrypt them when rendered on the screen," says Cabell Greenwood, vice president of business development.

Kryterion offers IBT and online proctoring for IT certification programs. With online proctoring, "There's no opportunity for any level of collusion between the proctor and the test taker," says Dave Meissner, chief operating officer at Kryterion Inc.

CompTIA is working with Pearson VUE to deploy IBT, possibly later this year, and Bryan Kainrath, vice president for certification operations at CompTIA, is bullish on the technology's prospects. "We don't have to send the answer keys. We pull the items back, take it offline, do the scoring and send the results to the candidate. We can secure items for a lot longer."

But IBT isn't always a good fit. It requires significant bandwidth, and some testing centers, particularly in overseas locations where the most intellectual property theft occurs, don't have enough to reliably deliver tests in that way, Addicott says.

-- Robert L. Mitchell

That process can present an expensive challenge, however, because organized theft rings can compromise entire tests within three to five weeks of when they're first released, while most IT certification exams are refreshed every 12 to 15 months, Addicott says.

Kainraith admits that's a problem, but he thinks that questions take a bit longer to appear on brain-dump sites, and says CompTIA replaces tests at a rapid pace. "We're able to churn our items a lot faster than 12 to 15 months," he says, although he declined to say how fast.

While CompTIA has the scale and resources to turn over its test questions more quickly, smaller IT certification programs are more limited because the cost of building and maintaining tests ranges from hundreds of dollars per question to thousands of dollars per test item, according to Caveon.

Countermeasures: Tripping up the cheats

Catching cheaters has become its own science. "More candidates are sharing knowledge than we've seen in the past," says Kainrath. But both test centers and IT certification owners have ways of figuring out who's using stolen and shared test data, as well as who might be coming in to steal it.

In addition to using live proctors, Microsoft and others are moving toward online proctoring, which combines the use of a video camera with a live feed of the test taker's screen. While an online proctor is limited by what he can see on a video camera, it's easier to take immediate action against cheaters, Grieve says. Because they can look for suspicious activity at the question level, online proctors can identify cheating sooner and end the test before the candidate can see -- and possibly compromise -- the rest of the exam content.

How test centers stop cheaters

  • Use live and/or online proctors trained to spot suspicious activity
  • Ban all electronic devices from the test room
  • Perform forensic analysis of the test results to detect "anomalous" behavior that might indicate cheating
  • Use "Trojan horse" questions or other innovative test designs that tip off test program managers that the candidate studied stolen test content
  • Identity validation with photo ID, digital signatures, biometrics; photograph the subject and include it on the test report to thwart proxy test takers
  • Randomize order of multiple-choice test questions and answers
  • Use multiple exam versions containing completely different questions
  • Use scenario-based questions that require that the candidate perform an action by interacting with a simulation, rather than answering a multiple choice question
  • Use adaptive testing that varies each successive question based on the answer given to the previous one and stops the test as soon as proficiency is determined

-- Robert L. Mitchell

Test centers also have ways to tell if candidates have been memorizing stolen test questions and answers or sharing knowledge in chat rooms. "We leverage several different publication strategies and question types designed specifically to address cheating," Grieve says.

While Grieve declined to provide details, Addicott says some of the more basic anomalies include people who perform at "superhuman speeds" on the exam or who perform well on items that have been on the test a long time while scoring poorly on newer items -- an indicator that the individual may have memorized stolen test content.

Some IT certification exams also catch people who have memorized stolen test data by including "Trojan Horse" questions that deliberately include the wrong answer in the official answer keys. These questions don't count toward the candidate's overall score, but if the test taker answers a predetermined number of such questions with the incorrect answers listed in the answer key it's assumed that they used stolen information and the test is automatically invalidated, says Addicott.

Certification programs may also use different test designs in an attempt to thwart cheaters who have memorized test questions and answers. These include scrambling the order of questions on any given exam, randomizing the order of answers to multiple-choice questions, having a pool of questions from which to choose from for each test item and giving different candidates in the same test center entirely different versions of the test.

CompTIA and other certification organizations have also started to supplement or replace some of the standard multiple-choice test questions with adaptive and performance-based methodologies that are harder to compromise. With adaptive testing each successive question the user sees depends on whether or not he answered the previous one correctly. As soon as the test determines that the taker knows -- or doesn't know -- the content, the test ends. "It's a more refined manner of judging, but it also provides security," says Greenwood.

CompTIA is adding progressively more performance-based testing, which uses scenario-based questions that ask the user to perform specific actions in a simulated environment. Such questions are harder to memorize. "At that point it becomes easier just to study," says Kainrath.

And that, in a nutshell, is a key part of CompTIA's strategy. "We can't stop cheating, but we can make sure it takes a lot of time versus just studying."

Getting caught: A great way to kill a career

Wary of the damage that rampant cheating can have on an IT certification, like what some say happened in the 1990s (see sidebar, below), companies aren't just getting aggressive about catching cheats, they're clamping down by handing down more severe sanctions.

"We ban for life anyone who is caught cheating. They are not allowed to take any Microsoft exam ever again," says Grieve. And Microsoft, at its discretion, may also strip the candidate of any previously earned Microsoft IT certifications, she adds.

Devaluing a credential

As large numbers of people earned the Certified NetWare Engineer certification in the early 1990s, recalls Dave Meissner, chief operating officer at Kryterion, "there was concern about the quality of the professionals being certified. People could pass the CNE exam successfully purely by studying books," he says, which gave rise to the term "paper Certified NetWare Engineer." What's more, "there was a strong belief -- and perception is what matters -- that the test content was readily available, and the value of that CNE credential was diminished."

CompTIA is taking a harder line on cheating as well, "casting a wider net" by using data forensics in its investigations, says Kainrath. Today if you get caught cheating you won't get the certification and must wait a period of time, typically a year, before you can take the exam again. But CompTIA is considering changing that to a lifetime ban. "This year we'll roll out a harder policy," he says.

Poyiadgi says that he's seen cheaters lose their jobs in situations where employers sponsored the candidates. And if the person was selling test questions and answers, he or she may be prosecuted by law enforcement as well, he adds.

Kainrath marvels at the amount of time he says some people spend trying to cheat their way through IT certification exams. A certification like A+ serves only to validate the user's skills, he says, and if a cheater is hired or promoted based on false pretenses it hurts the cheater's career prospects as much as it does CompTIA's reputation. Ultimately, he says, "It's not doing them any good by faking it."

This article, Pirates, cheats and IT certs, was originally published at Computerworld.com.

Robert L. Mitchell is a national correspondent for Computerworld. Follow him on Twitter at twitter.com/rmitch, or email him at rmitchell@computerworld.com.

Read more about education/training in Computerworld's Education/Training Topic Center.

This story, "Pirates, Cheats and IT Certs" was originally published by Computerworld.

Copyright © 2014 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
7 secrets of successful remote IT teams