Consumer Reports, published by the non-profit Consumers Union, drew considerable criticism from antivirus vendors for creating 5,500 new viruses, all modifications of existing viruses, to test the heuristic capabilities of the software.
But in a test of anti-spyware software, CR went in another direction. All it did, according to the article in the September 2006 issue of the magazine, was to run Spycar on a system and see how well the anti-spyware software detected its changes.
Spycar is not spyware (define). It runs makes minor changes, like writing a harmless key to the Registry, and changes the user's default Web page in their browser. Even the Spycar Web site says point blank "Spycar alone cannot be used to determine how good or bad an anti-spyware product is."
"We were disappointed that they did not utilize a more comprehensive testing approach, of which Spycar can be one component for evaluating behavior-based defenses."
After going through the effort to make 5,500 viruses, CR's lack of effort for this test baffled at least one analyst who follows security software.
"In one test they go overboard in creating this whole zoo of viruses and then they do the exact opposite for this?" asked Peter Firstbrook, research manager for security products at Gartner.
Consumer Reports did not respond to inquiries from internetnews.com for comment.
Alex Eckelberry, president of Sunbelt Software, which develops CounterSpy, was particularly vocal in a blog posting about the testing methodology.
"I just cannot believe what they did," he said. "The thing is, Consumer Reports did something stupid but they didn't read the manual. The manual for Spycar says this is not to be used for testing on-demand scanning."
Webroot, developer of Spy Sweeper, kept things diplomatic, which was probably made easier by the fact its software came out on top in the test.
"Webroot is extremely honored to have its Spy Sweeper solution considered a leader by Consumer Reports and applaud Consumer Reports effort to play its part in increasing awareness of spyware and for bringing its heritage of consumer advocacy to this product and marketplace," the company said in a statement.
"We want the industry to be on some level playing field so acts like cheating aren't encouraged," said Eckelberry. "We just want a test metrics that's reasonable and fair."
And that may be the problem. CR has built its reputation on impartiality because it takes no advertising and purchases every product it tests rather than get free evaluation units from vendors.