Mar 18 2015 · Software Engineering
On reliability, Robert Glass notes the following in Frequently Forgotten Fundamental Facts about Software Engineering:
Roughly 35 percent of software defects emerge from missing logic paths, and another 40 percent are from the execution of a unique combination of logic paths. They will not be caught by 100-percent coverage (100-percent coverage can, therefore, potentially detect only about 25 percent of the errors!).
John Cook dubs these “sins of omission”:
If these figures are correct, three out of four software bugs are sins of omission, errors due to things left undone. These are bugs due to contingencies the developers did not think to handle.
I can’t validate the statistics, but they do ring true to my experiences as well; simply forgetting to handle an edge case, or even a not-so-edge case, is something I’ve done more often than I’d like. I’ve introduced my fair share of “true bugs”, code paths that fails to do what’s intended, but with far less frequency.
The statistics also hint at the limitations of unit testing, as you can can’t test a logic path that doesn’t exist. While you can push for (and maybe even attain) 100% code coverage, it’s a fuzzy metric and by no means guarantees error-free functionality.