The FAA Process For Certifying Flightworthy Software

1.0 Introduction

This post is on current events. The plane crashes of the Boeing 737 Max 8 airplane is arguably about more than a software bug. I point out in this post that you can read about how the United States' Federal Aviation Administration (FAA) is supposed to certify software and electronic hardware in airplanes yourself. In some sense, this post is not about software bugs, but rather about software engineering certification processes that fit into a larger systems engineering perspective.

2.0 FAA Resources

The FAA provides lots of resources associated with DO-278C, Software Considerations in Airborne Systems and Equipment Certification, the primary document of interest in this context.

• You can find an overview of the FAA Airplane Certification Service here.
• Position papers from the Certification Authorities Software Team (CAST) are available here.
• I was particularly interested in some of these Research Reports.

3.0 Conclusion of Recent FAA Research on Software Reliability Models

For historical reasons, I am interested in software reliability models. These models address an important problem. One could mandate that software be developed by rigourous processes, by some defined model or another. And one could require that developers produce certification arguments along with delivered software. But how does the variability of rigor relate to quantitative measures of software reliability and availability, as needed for system reliability? A recent report looks at software reliability and, as I understand, concludes the technology is not mature enough yet:

"the current position is that methods for estimating probabilities of software errors have not yet provided a level of confidence acceptable for software assurance purposes. Since the publication of the report and DO-178C document, work on software reliability has improved the level of confidence. However, the multiplicity of available models and absence of quantified performance objectives contribute to the confidence issue remaining open and added the issue of guidelines on selecting an adequate reliability method." Final Report for Software Service History and Airborne Electronic Hardware Service Experience in Airborne Systems (DOT/FAA/TC-16/18, Section 7.5.4, p. 102)

I don't know that I disagree with this conclusion. One important issue is not mentioned in that summary statement. If enough catastrophic or hazardous failures exist for estimating model parameters, the software is unlikely to ever meet reliability goals. One might apply the models to less severe failures, though. I do not think the range of models is quite as diverse as suggested. For example, the Goel-Okumoto model is a continuous version of the Jelinski-Moranda model. But no sign exists of settling down on a single model. I think there are a number of quantified performance objectives for these models: the LaPlace test for a decreasing failure rate, measures of the stability of parameter estimates, and Bev Littlewood's U and S tests of goodness of fit. Of course, one would have to agree to adopt some convention for confidence intervals or some such.

4.0 Further Thoughts

It is my impression that the FAA takes certification very seriously. Cost pressures and the evolution of technology lead to developers continually wanting to introduce new technology. And FAA-sponsored research has looked at the desirability of such introductions. For example, does the possibility of model checking make formal verification more practical? How should code coverage metrics be applied in testing object-oriented software? How should software developed with Model-Based Design be certified? Do MBD tools need to be certified and accredited, perhaps as installed on specific platforms? Should both the inputs and generated software be assessed?

Some of these considerations and developments in software technology are not all that new. The FAA should be conservative in what they will allow. For example, I like Java, but do not think it should be used for mission-critical software, with hard real-time and performance requirements. It might be used on an airplane for delivering consumer services, like music and movies to passengers, but in isolation from flight software.

At any rate, certification and accreditation by the FAA requires insight into engineering processes and cooperation with certification teams within developer organizations and their subcontractors. Even in an ideal world, decisions about who does what must be made here. Software technology is an important driver here, orthogonal to the need to align incentives.

But it does not matter how thoroughly certification and accreditation are defined if the FAA does not have the resources to provide oversight. And certification authorities must be willing to refuse to allow a plane to fly, with both commercial and government entities accepting such a decision. This is not a responsibility that I would want to have.