Metrics for Test Reporting: Analysis and Reporting for Effective Test Management

Preface....5 Contents....12 1 Definition, History, and Benefits of Metrics....14 Abstract....14 1.1	Criteria for Metrics....15 1.2	Quality Criteria for Metrics....15 1.3	Success Criteria for Metrics....16 1.4	Validation of Measures....16 1.5	Problems in the Software Development Process....17 1.6	History of Test Metrics....17 1.7	Steps to Set Up Test Metrics....18 1.8	Lifecycle of Test Metrics....19 References....19 2 Classification of Metrics....20 Abstract....20 2.1	Key Figures....20 2.2	Indicators ....22 2.3	Classification of Test Metrics....22 2.4	Classification According to IEEE Standard 1061....26 References....28 3 Complexity Measures....29 Abstract....29 3.1	Evaluation of Project Complexity ....29 3.2	Calculation of Project Complexity....31 3.3	Halstead Metric....32 3.4	Number of Delivered Bugs (B)....36 3.5	McCabe Metric (Cyclomatic Complexity)....37 3.6	Complexity Measure According to Rechenberg....39 3.7	Maintainability Index....40 3.8	Hybrid Metrics....40 3.9	Object-Oriented Software Metrics....41 3.10	Number of Children (NOC)....42 3.11	RFC (Response for a Class)....42 3.12	CBO (Coupling Between Objects)....42 3.13	CBC (Coupling Between Classes)....43 3.14	LCOM (Lack of Cohesion in Methods)....43 3.15	WMC (Weighted Methods per Class)....43 3.16	Kiviat Diagrams....43 3.17	Metrics for Verifying Test Design and Evaluating the Test Process....44 References....45 4 Standards and Norms for Measurements....47 Abstract....47 4.1	PSM—Practical Software and Systems Measurement....47 4.2	ISO/IEC15939 Software Measurement Process....48 4.3	ISO14598....48 4.4	CMMI Process Area Measurement and Analysis....49 4.5	Measurement with ISO15504 and Automotive SPICE ®....49 4.6	Analysis Methods....51 4.7	Analysis Tools....52 4.8	Quality Improvement Paradigm (QIP Model)....56 References....57 5 Use of Different Representations....58 Abstract....58 5.1	Axis Diagrams....58 5.2	Block Diagrams....63 5.3	Design of Reporting....63 References....64 6 Application of Metrics....65 Abstract....65 6.1	Metrics and Software Industrialization....67 References....69 7 Software Metrics....70 Abstract....70 8 Metrics for Software Quality....72 Abstract....72 8.1	Boehm’s Property Tree....72 8.2	Function Measurement according to Gilb....74 8.3	McCall’s Quality Matrix....76 8.4	Automated Quality Measurement....78 References....79 9 Requirement-Based Test Metrics....80 Abstract....80 9.1	Requirements/specified requirements....81 10 Metrics for Effort Estimation....84 Abstract....84 10.1	Evaluation of Testability and Test Complexity....84 References....88 11 Metrics for Test Preparation and Follow-up....89 Abstract....89 12 Metrics for Test Planning....91 Abstract....91 References....94 13 Metrics for Different Test Levels....95 Abstract....95 13.1	Module Level Testing....95 13.2	Integration Level Testing....96 13.3	System Level Testing....96 14 Metrics for Measuring Test Coverage....99 Abstract....99 14.1	Concept Coverage....99 14.2	User Manual Coverage....99 14.3	Code Coverage....100 14.4	Design Coverage....100 14.5	Module Coverage....100 14.6	Architecture Coverage....100 14.7	Data Coverage....101 14.8	Function Coverage....101 14.9	Test Coverage Metrics....102 14.10	Test Case Coverage....102 14.11	Test Case Completeness....103 14.12	Lines of Code....103 14.13	Reliability as a Central Test Metric....105 14.14	Function Point Analysis (FPA)....105 14.15	Use Case Points....108 14.16	Test Case Points....109 14.17	Systematic and Deterministic Testing....110 14.18	Test Coverage Metrics....112 14.19	Stochastic Testing....113 References....113 15 Test Case-Based Metrics....115 Abstract....115 15.1	Chapin's Q Metric....116 15.2	Test Data Density....117 15.3	Test Data Volume....117 15.4	Test Case Efficiency....118 15.5	Test Case Reusability....118 15.6	Test Coverage....118 15.7	Test Conformity....118 15.8	Test Design Ratio....118 15.9	Test Coverage....119 References....120 16 Test Object-Based Metrics....121 Abstract....121 16.1	Code Coverage....121 16.2	Dialogue Coverage....122 16.3	Installation Variants....122 16.4	Coverage Measures....122 16.5	Instruction Coverage....123 16.6	Decision/Branch Coverage....123 16.7	Path Coverage....123 16.8	Condition Coverage....124 16.9	Tests in the Black-Box Method....125 16.10	Test Progress Metric....127 References....129 17 Metrics for Test Execution....130 Abstract....130 References....133 18 Metrics for Performance and Load Tests....134 Abstract....134 18.1	Requests per Second....134 18.2	Processor Utilization....135 18.3	Queued Requests....135 References....137 19 Metrics for Test Automation....138 Abstract....138 20 Cost-Based Test Metrics....140 Abstract....140 20.1	Test Effort....141 21 Test Productivity Metrics....143 Abstract....143 References....145 22 Goal Question Metric....146 Abstract....146 22.1	ISO9126....149 References....150 23 Defect-Based Test Metrics....152 Abstract....152 23.1	Error Classes....153 23.2	Defects Over Time....154 23.3	Metrics for Risks....154 23.4	Defect Reduction Forecasts....155 23.5	Examples of Defect Reduction Metrics....157 23.6	Error Seeding Method....160 23.7	Running Average of Errors....161 23.8	Defects by Project Phase....163 23.9	Test Quality Metric....164 23.10	Confidence Level....165 23.11	Escape Defects....166 References....168 24 Metrics for Test Documents....169 Abstract....169 24.1	Reviews....169 24.2	Status of the Documents....170 25 Metrics for Usability Tests....172 Abstract....172 25.1	A/B Testing and Multivariate Testing....175 References....177 26 TPI Next....178 Abstract....178 26.1	Criteria for Maturity Levels....181 26.2	Test Maturity Matrix....182 26.3	Maturity Levels per Criterion....184 26.4	Metrics in TPI Next....187 References....190 27 Metrics for Test Completion Criteria....191 Abstract....191 27.1	Defects per Unit of Time....192 References....194 28 Metrics and Test Reporting....195 Abstract....195 28.1	Perspectives on Test Reporting....195 28.2	Period of Consideration....196 28.3	Adjusting History of Metrics....197 28.4	Metric Defect....197 28.5	Problems with Test Progress Consideration....199 28.6	Other Viewing Periods....199 28.7	Recommended Reporting Rhythm....199 28.8	Summary of Metrics in a Metric....201 28.9	Other Metrics....203 28.10	Project Traffic Light....203 References....205 29 Comparison of Test Metrics....206 Abstract....206 29.1	Comparison of Different Metrics....207 29.2	Standardization of Test Metrics....207 29.3	Metrics No Longer Needed....209 29.4	Increasing Data Volume....209 References....211 30 Problem Solving for the Use of Test Metrics in Test Operations....212 Abstract....212 30.1	Number of Metrics Used....212 30.2	Selection of Metrics....213 30.3	Extension of Metrics....214 30.4	Creating Realistic Forecasts....214 30.5	Subsequent Change of Values....215 30.6	Evaluation Framework for Metrics....215 30.7	Meaningless Correlations....216 References....218 31 Impact of Reporting with the Help of Test Metrics....219 Abstract....219 31.1	Manipulative Effect of Statistics....226 31.2	Relative Values....226 31.3	Arithmetic Mean....226 31.4	Manipulated Graphics....226 Afterword....230 Weiterführende Literatur....235