Category Verification & Validation

A Case Study on Automated Source-Code-Based Testing Methods

Date14 May 2013 - 17 May 2013
EventDASIA 2013
LocationPorto, Portugal
We present results of a case study on a test generation approach called Flow-optimized Automated Source-code-based unit Testing (FAST) which generates test stimuli from information available in the source code, in particular taken from the detailed software interfaces. This allows automation of a significant part of testing, ranging from the test stimuli generation to the generation of the test report. A huge number of stimuli can be generated exploring the behaviour of the software under test under nominal and non- nominal conditions. Symptoms like timeouts, unexpected termination, run-time exceptions, out-of-range conditions and missing coverage are applied for defect detection. The goal of this study was to evaluate the FAST process in context of a real spacecraft flight software application and to get a feedback on its scalability regarding larger applications, its sensitivity on detecting defects in the code, the achievable test coverage, its compliance with software standards and potential limitations. We also consider the impact of coding style on suitability for automated testing. The results confirm that the approach (1) provides acceptable code coverage results without requiring manual intervention for test preparation and execution, (2) raises the probability of activation of exotic fault conditions, (3) may provide hints on locations in the code where robustness needs to be verified, and (4) identifies defects not found before by static analysis and intensive testing


Lessens Learned on Quality (of) Standards

Date17 May 2011 - 20 May 2011
EventDASIA 2011

Standards are used to describe and ensure the quality of products, services and processes throughout almost all branches of industry, including the field of software engineering. Contractors and suppliers are obligated by their customers and certification authorities to follow a certain set of standards during development. For example, a customer can easier actively participate in and control the contractor’s process when enforcing a standard process.

However, as with any requirement, a standard may also impede the contractor or supplier in assuring actual quality of the product in the sense of fitness for the purpose intended by the customer.

This is the case when a standard defines specific quality assurance activities requiring a considerable amount of effort while other more efficient but equivalent or even superior approaches are blocked. Then improvement of the ratio between cost and quality exceeding miniscule advances is heavily impeded.

While in some parts being too specific in defining the mechanisms of the enforced process, standards are sometimes too weak in defining the principles or goals on control of product quality.

Therefore this paper addresses the following issues: (1) Which conclusions can be drawn on the quality and efficiency of a standard? (2) If and how is it possible to improve or evolve a standard? (3) How well does a standard guide a user towards high quality of the end product?

One conclusion is that the analyzed standards do interfere with technological innovation, though the standards leave a lot of freedom for concretization and are understood as technology-independent.

Another conclusion is that standards are not only a matter of quality but also a matter of competitiveness of the industry depending on resulting costs and time-to- market. When the costs induced by a standard are not adequate to the achievable quality, industry encounters a significant disadvantage.