Some Financial Analysis in Software Engineering

Warning: Lots of navel-gazing in this post.

Example of Financial Analysis of Two Methods for Detecting Software Bugs (from Vienneau 1995)

Consider a specific industry, namely the production of software. (This industry would correspond to a column in an input-output matrix.) Many methods have been proposed for improving the productivity, the quality, and management control in software development, operations, and management. And, as software became a more important component in, for example, embedded systems, various measures, and various methods for choosing among possible measures, have been proposed for software processes and products. Some time ago, some researchers argued that analyses of such metrics should be oriented towards economic analysis of, for example, efforts to introduce new methods into the software lifecycle. I think of the following two articles as early exponents of this thesis:

• Sandra A. Slaughter, Donald E. Harter, and Mayuram S. Krishnan. Evaluating the Cost of Software Quality, Communications of the ACM. V. 41, No. 8 (Aug. 1998): pp. 67-73.
• Robert L. Vienneau. The Present Value of Software Maintenance, Journal of Parametrics (April 1995).

Slaughter et al. is more high level and is published in a more prestigious journal. My article is more a tutorial.

One can apply economics to analyze specific decisions during the software lifecycle. For instance, the following article looks at when testing should stop and software should be released:

• Robert L. Vienneau. The Cost of Testing Software, Proceedings of the Annual Reliability and Maintainability Symposium, Orlando, FL (29-31 Jan. 1991).

The optimal release date can be justified by a simple marginal rule: the final phase of testing should stop, and software should be released, when the cost of an increment of testing reaches or exceeds the marginal benefit to gained from that increment of testing. The marginal benefit can be conceptualized as the product of the expected number of bugs to be observed during that increment of time and the savings in finding those bugs during test, instead of operations. (Various software reliability models exist for estimating the parameters needed to implement this rule.)

I have not been aggressive in promoting my work in software engineering economics. Once in a while, I search the web to see who, if anybody, is referencing this stuff. This post concludes with a number of papers, theses, and selected technical reports citing one or the other of the above papers, or related work from me. I do not know why so many of these references are so recent; perhaps software processes must attain a certain level of maturity before these financial analyses become useful.

• Majed Alyahya, Rodina Ahmad, and Sai Peck Lee. Impact of CMMI-Based Process Maturity Levels on Effort, Productivity and Diseconomy of Scale, International Arab Journal of Information Technology. V. 9, No. 4 (Jul. 2012): pp. 352-360
• Hanna Farnstrand. Introducing Structured Testing, Lund University (16 Jun. 2008).
• Matt Ganis. Agile Methods: Fact or Fiction, Trenton Computer Festival. (Apr. 2010).
• Warren Harrison, David Raffo, John Settle, Nancy Eickelmann. Technology Review: Adapting Financial Measures: Making a Business Case for Software Process Improvement, Software Quality Journal. V. 8, No. 3 (1999): pp. 211-231.
• Chin-Yu Huang, Jung-Hua Lo, Sy-Yen Kuo, and Michael R. Lyu. Optimal Allocation of Testing-Resource Considering Cost, Reliability, and Testing Effort, Proceedings of the 10th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC'04) (2004): pp. 103-112.
• Chin-Yu Huang and Jung-Hua Lo. Optimal Resource Allocation for Cost and Reliability of Modular Software Systems in the Testing Phase, Journal of Systems and Software. V. 79, Iss. 5 (May 2006): pp. 653-664.
• Zhengrui Jiang, Sumit Sarkar, and Varghese S. Jacob. Postrelease Testing and Software Release Policy for Enterprise-Level Systems, Information Systems Research. V. 23, No. 4 (Dec. 2012).
• J. Dennis Lawrence. Software Reliability and Safety in Nuclear Reactor Protection Systems, Lawrence Livermore National Laboratory, UCRL-ID-114839 (11 Jun. 1993).
• Tuomas Paasonen. Methods for Improving the Maintainability of Application Software. Master Thesis, Aalto University (13 April 2012).
• David Raffo and Tim Menzies. Evaluating the Impact of a New Technology Using Simulation: The Case for Mining Software Repositories. [I am unsure of the status of this paper].
• Tobias Scherner and Lothar Fritsch. Early Evaluation of Security Functionality in Software Projects - some Experience on using the Common Criteria in a Quality Management Process, Johann Wolfgang Goethe - Universitat, Franfurt am Main (2007).
• Leon van Delft. Automated GUI Testing in Industry. Master Thesis, Delft University of Technology (28 Dec. 2011).

Update (28 February 2013): Added Huang and Lo (2006) to remind me to check if my search with Google Scholar yielded an accurate reference.

Update (13 March 2013): Added Huang et al. (2004).