Study Finds Fewer Flaws in Open-source Code
Software inspection firm Reasoning says MySQL source code contained fewer flaws than its commercial counterparts. But others question Reasoning's reasoning.
Code quality in a version of the MySQL open-source database was found to be six times superior to that of comparable proprietary code, according to a recent study of open-source software products by tech development firm Reasoning.
The results of the study come at a time when fierce debate rages as to whether open-source software, such as the Linux operating system, is safer, or more secure than proprietary products such as Microsoft Windows.
Although Windows has been the favorite target of criticism over its numerous security patches in recent years, commercial software proponents and even analysts note that open-source software can also be less secure than its commercial counterparts.
Fry said the study included products from some well-known commercial vendors, as well as those that are not so well known, including makers of embedded databases.
Asked what kind of conclusions Reasoning drew from the study, Frye told internetnews.com his company believed MySQL boasted fewer defects per line because proprietary software lacks the open peer review process of open-source projects such as MySQL.
Peer review, he said, enables many programmers to examine code, which often results in more flaw detections. Also, he said many users don't just report defects, as they would do with commercial software, but actually track them down to their root causes and fix them.
Fry also indicated stringent deadlines placed on proprietary software may force some products to the public realm before they are truly secure, or as bug-free as possible.
Mountain View, Calif.-based Reasoning, which provides code-review services to technology clients, said it compared MySQL v. 4.016 to several proprietary database products, finding that the "defect density" -- defined as the number of defects found per thousand lines of source code -- was greater in products that are shipped and sold by companies.
For example, Reasoning found 21 software defects in 236,000 lines of MySQL source code. The defect density of the MySQL code was 0.09 defects per thousand lines of source code. Reasoning, which scrutinized over 35 million lines of commercial code, found that the commercial average defect density of these projects came to 0.57 defects per thousand lines of source code.
IBM had no comment for this story. A Microsoft spokesperson expressed doubt that SQL Server was examined, and wondered how it was possible unless Reasoning had gotten its hands on its SQL Server code. Analyst Carl Olofson, who researches the database software market for IDC, said the criticism of Reasoning's methods is valid.
Fry said Reasoning's clients submitted source code as part of the software inspection services the company provides.
MySQL AB Co-founder and Vice President David Axmark, whose company's database benefited from the study because Reasoning helped his developers pick out defects in the software that had gone unattended, said the findings validate the open-source development method. MySQL will release a new version of its database software reflecting those changes this week.
"Reasoning's conclusion that the MySQL database software quality is significantly higher than proprietary code validates the Open Source development method, in which large communities of programmers 'battle test' the software," Axmark said in a public statement.
But IDC's Olofson said peer review testing comparisons between MySQL and commercial products may not be entirely balanced.
"It should be borne in mind that the leading RDBMS [relational database management system] products are probably huge in terms of source code compared with MySQL," Olofson told internetnews.com. "I suspect that most commercial proprietary RDBMS products that have been available for ten years or more (as is the case with all the leading ones) are pretty solid in terms of the core functionality that MySQL offers, and that their defect rates for just that functionality would be much lower than cited for the proprietary products overall."
Moreover, he said that because MySQL's development is guided by a strategy managed by the company MySQL AB, which owns the copyright on the code and trademark on the name, the company is more likely to address strategic needs of the marketplace than other open-source products.
While the open-source results for MySQL compared to commercial software were overwhelmingly positive, Fry said some open-source products fall short because the feature sets are not quite as advanced as the bells and whistles found in proprietary software. This is because commercial vendors are driven by competition to put the most advanced software they can out to the market. Open-source developers tend to be a bit more basic in the projects they undertake, he said.
"I would also suggest that MySQL is considerably simpler, not having to address the myriad features and backward compatibility issues that the proprietary products have had to do, which almost ensures that the code will have fewer defects," Olofson said. "As open source products become more complex and address more diverse requirements, their defect rate is likely to go up."