MCF > Detailed Design and Analysis > Embedded Software/Firmware >

Testing

Scope and Description

This topic page covers testing of embedded software (including firmware). This includes any software that is not what we categorize in this knowledge base as "flight software", which is responsible for command and data handling (C&DH) for the entire spacecraft. Software covered by this topic is typically bare-metal - running on a microcontroller or FPGA without an intervening operating system layer in the software architecture. This subsystem and payload software is usually much less complicated than the flight software and is responsible for lower-level functionality tied closely to analog inputs (e.g., power switch control and analog telemetry acquisition in the electrical power subsystem). Software flaws are one of the most common sources of on-orbit failure in small satellites. Because this software typically cannot be updated on-orbit, smallsat developers should ensure that it is thoroughly tested.

Resources under this topic area are primarily standards and handbooks that provide traditional software development processes and requirements that can inform smallsat software development practices. Note that this topic does not include software development environments, which usually include integrated testing frameworks and tools. These tools can be found on the Embedded Software/Firmware > Development topic page.

Best Practices and Lessons Learned

  • Test embedded software and firmware with flight-like hardware as soon as possible - on flight computer hardware and then end-to-end (e.g., on a complete flatsat) as soon as possible.
  • Institute an independent testing program if possible - a software developer should not test their own code.
  • Testing of this software should consider any radiation susceptibilities on the device being programmed and external radiation susceptibilities that could influence software performance. When using COTS parts throughout a system, testing of embedded software features that enable the device to support operation through or survival of single event effects requires close coordination between the software developer(s) and electrical engineer(s). These electrical engineers can provide important input to inform the development of test cases which address external radiation susceptibilities.

Resources

Complex electronics are "programmable and designable complex integrated circuits." Examples of these ... include, but are not limited to, FPGAs, CPLDs, ASICs, and SoCs. This handbook provides an overview of FPGA, CPLD, ASIC, and SoC devices, how they are designed and programmed, and assurance and verification activities for them. It also covers trends in design and assurance for these complex devices. This document is coming from a traditional, high-reliability perspective, but the intended audience and "overview" level of detail makes this a good introduction these design, assurance, and verification tasks. Section 3 provides a useful glossary of terms.

White Paper
NASA

A web-based software engineering handbook designed to provide guidance for the implementation of safe ... and reliable software on satellite projects. This handbook includes software engineering resources such as NASA standards, guides, lessons learned, and references for acquiring, developing, and maintaining NASA software systems.

Software Tool
NASA

NOS3 is an open-source software tool developed by the NASA Independent Verification and Validation (IV&V) ... Independent Test Capability (ITC) team. It is a collection of Linux executables and libraries used to test software for small satellites. This tool can be licensed under the NASA Open Source Agreement.

Standard
NASA

This NASA standard provides detailed information on the implementation of quality assurance, safety, ... and verification and validation (V&V) for software specifically made for and by NASA. While this is a detailed NASA standard which may be too stringent to implement on a smallsat project, smallsat teams can use the requirements to inform their policies for effective software assurance and safety. Table 2 provides a valuable causes and controls to consider when determining the root cause of various software-related faults (or designing the system to prevent them).

Standard
NASA

This NASA guidebook provides software safety procedures and guidance for the development and quality ... assurance of safety-critical software. This resource also provides software acquisition guidance for both commercial off-the-shelf (COTS) software and software developed under contract.

White Paper
Aerospace Corporation

This resource provides general and detailed requirements for software development on space systems. This ... document covers design, development, integration and testing for flight software and embedded software/firmware.

Get Involved