Verification, Validation, and Testing of Engineered Systems

Preface xvii

Part I Introduction 1

1. Introduction 3

1.1 Opening 3

1.1.1 Background 4

1.1.2 Purpose 5

1.1.3 Intended audience 5

1.1.4 Book structure and contents 6

1.1.5 Scope of application 8

1.1.6 Terminology and notation 9

1.2 VVT Systems and Process 9

1.2.1 Introduction—VVT systems and process 9

1.2.2 Engineered systems 10

1.2.3 VVT concepts and defi nition 12

1.2.4 The fundamental VVT dilemma 19

1.2.5 Modeling systems and VVT lifecycle 20

1.2.6 Modeling VVT and risks as cost and time drivers 24

1.3 Canonical Systems VVT Paradigm 32

1.3.1 Introduction—Canonical systems VVT paradigm 32

1.3.2 Phases of the system lifecycle 34

1.3.3 Views of the system 37

1.3.4 VVT aspects of the system 39

1.4 Methodology Application 39

1.4.1 Introduction 39

1.4.2 VVT methodology overview 40

1.4.3 VVT tailoring 43

1.4.4 VVT documents 50

1.5 References 56

Part II VVT Activities and Methods 61

2. System VVT Activities: Development 63

2.1 Structure of Chapter 63

2.1.1 Systems development lifecycle phases and VVT activities 63

2.1.2 VVT activity aspects 64

2.1.3 VVT activity format 65

2.2 VVT Activities during Defi nition 65

2.2.1 Generate Requirements Verifi cation Matrix (RVM) 65

2.2.2 Generate VVT Management Plan (VVT-MP) 67

2.2.3 Assess the Request For Proposal (RFP) document 69

2.2.4 Assess System Requirements Specifi cation (SysRS) 71

2.2.5 Assess project Risk Management Plan (RMP) 72

2.2.6 Assess System Safety Program Plan (SSPP) 74

2.2.7 Participate in System Requirements Review (SysRR) 77

2.2.8 Participate in System Engineering Management Plan (SEMP) review 77

2.2.9 Conduct engineering peer review of the VVT-MP document 79

2.3 VVT Activities during Design 80

2.3.1 Optimize the VVT strategy 80

2.3.2 Assess System/Subsystem Design Description (SSDD) 83

2.3.3 Validate system design by means of virtual prototype 85

2.3.4 Validate system design tools 86

2.3.5 Assess system design for meeting future lifecycle needs 87

2.3.6 Participate in the System Design Review (SysDR) 90

2.4 VVT Activities during Implementation 91

2.4.1 Preparing the test cycle for subsystems and components 91

2.4.2 Assess suppliers’ subsystems test documents 96

2.4.3 Perform Acceptance Test Procedure—Subsystems/Enabling products 97

2.4.4 Assess system performance by way of simulation 100

2.4.5 Verify design versus implementation consistency 102

2.4.6 Participate in Acceptance Test Review—Subsystems/Enabling products 103

2.5 VVT Activities during Integration 104

2.5.1 Develop System Integration Laboratory (SIL) 104

2.5.2 Generate System Integration Test Plan (SysITP) 106

2.5.3 Generate System Integration Test Description (SysITD) 108

2.5.4 Validate supplied subsystems in a stand-alone configuration 111

2.5.5 Perform components, subsystem, enabling products integration tests 112

2.5.6 Generate System Integration Test Report (SysITR) 114

2.5.7 Assess effectiveness of the system Built In Test (BIT) 116

2.5.8 Conduct engineering peer review of the SysITR 120

2.6 VVT Activities during Qualification 120

2.6.1 Generate a qualification/acceptance System Test Plan (SysTP) 121

2.6.2 Create qualification/acceptance System Test Description (SysTD) 123

2.6.3 Perform virtual system testing by means of simulation 125

2.6.4 Perform qualification testing/Acceptance Test Procedure (ATP)—System 126

2.6.5 Generate qualification/acceptance System Test Report (SysTR) 129

2.6.6 Assess system testability, maintainability and availability 131

2.6.7 Perform environmental system testing 137

2.6.8 Perform system Certification and Accreditation (C&A) 140

2.6.9 Conduct Test Readiness Review (TRR) 144

2.6.10 Conduct engineering peer review of development enabling products 146

2.6.11 Conduct engineering peer review of program and project safety 148

2.7 References 149

3. Systems VVT Activities: Post-Development 153

3.1 Structure of Chapter 153

3.2 VVT Activities during Production 154

3.2.1 Participate in Functional Confi guration Audit (FCA) 154

3.2.2 Participate in Physical Confi guration Audit (PCA) 157

3.2.3 Plan system production VVT process 159

3.2.4 Generate a First Article Inspection (FAI) procedure 161

3.2.5 Validate the production-line test equipment 165

3.2.6 Verify quality of incoming components and subsystems 165

3.2.7 Perform First Article Inspection (FAI) 166

3.2.8 Validate pre-production process 167

3.2.9 Validate ongoing-production process 168

3.2.10 Perform manufacturing quality control 170

3.2.11 Verify the production operations strategy 172

3.2.12 Verify marketing and production forecasting 174

3.2.13 Verify aggregate production planning 176

3.2.14 Verify inventory control operation 177

3.2.15 Verify supply chain management 180

3.2.16 Verify production control systems 181

3.2.17 Verify production scheduling 183

3.2.18 Participate in Production Readiness Review (PRR) 184

3.3 VVT Activities during Use/Maintenance 186

3.3.1 Develop VVT plan for system maintenance 187

3.3.2 Verify the Integrated Logistics Support Plan (ILSP) 191

3.3.3 Perform ongoing system maintenance testing 200

3.3.4 Conduct engineering peer review on system maintenance process 204

3.4 VVT Activities during Disposal 208

3.4.1 Develop VVT plan for system disposal 209

3.4.2 Assess the system disposal plan 212

3.4.3 Assess system disposal strategies by means of simulation 214

3.4.4 Assess on-going system disposal process 215

3.4.5 Conduct engineering peer review to assess system disposal processes 219

3.5 References 221

4. System VVT Methods: Non-Testing 223

4.1 Introduction 223

4.2 Prepare VVT Products 223

4.2.1 Requirements Verification Matrix (RVM) 223

4.2.2 System Integration Laboratory (SIL) 226

4.2.3 Hierarchical VVT optimization 230

4.2.4 Defect management and tracking 234

4.2.5 Classification Tree Method 239

4.2.6 Design of Experiments (DOE) 243

4.3 Perform VVT Activities 256

4.3.1 VVT process planning 256

4.3.2 Compare images and documents 262

4.3.3 Requirements testability and quality 265

4.3.4 System test simulation 272

4.3.5 Failure mode effect analysis 280

4.3.6 Anticipatory Failure Determination 286

4.3.7 Model-based testing 293

4.3.8 Robust design analysis 302

4.4 Participate in Reviews 312

4.4.1 Expert team reviews 312

4.4.2 Formal technical reviews 326

4.4.3 Group evaluation and decision 331

4.5 References 346

5. Systems VVT Methods: Testing 351

5.1 Introduction 351

5.2 White Box Testing 356

5.2.1 Component and code coverage testing 356

5.2.2 Interface testing 360

5.3 Black Box—Basic Testing 365

5.3.1 Boundary value testing 365

5.3.2 Decision table testing 367

5.3.3 Finite State Machine testing 368

5.3.4 Human-system interface testing (HSI) 373

5.4 Black Box—High-Volume Testing 378

5.4.1 Automatic random testing 378

5.4.2 Performance testing 381

5.4.3 Recovery testing 385

5.4.4 Stress testing 386

5.5 Black Box—Special Testing 388

5.5.1 Usability testing 388

5.5.2 Security vulnerability testing 393

5.5.3 Reliability testing 402

5.5.4 Search-based testing 410

5.5.5 Mutation testing 418

5.6 Black Box—Environment Testing 422

5.6.1 Environmental Stress Screening (ESS) testing 422

5.6.2 EMI/EMC testing 424

5.6.3 Destructive testing 426

5.6.4 Reactive testing 431

5.6.5 Temporal testing 436

5.7 Black Box—Phase Testing 443

5.7.1 Sanity testing 444

5.7.2 Exploratory testing 445

5.7.3 Regression testing 447

5.7.4 Component and subsystem testing 452

5.7.5 Integration testing 455

5.7.6 Qualification testing 461

5.7.7 Acceptance testing 463

5.7.8 Certification and accreditation testing 466

5.7.9 First Article Inspection (FAI) 473

5.7.10 Production testing 477

5.7.11 Installation testing 481

5.7.12 Maintenance testing 484

5.7.13 Disposal testing 487

5.8 References 488

Part III Modeling and Optimizing VVT Process 495

6. Modeling Quality Cost, Time and Risk 497

6.1 Purpose and Basic Concepts 497

6.1.1 Historical models for cost of quality 498

6.1.2 Quantitative models for cost/time of quality 499

6.2 VVT Cost and Risk Modeling 500

6.2.1 Canonical VVT cost modeling 500

6.2.2 Modeling VVT strategy as a decision problem 502

6.2.3 Modeling appraisal risk cost 505

6.2.4 Modeling impact risk cost 511

6.2.5 Modeling total quality cost 516

6.2.6 VVT cost and risk example 517

6.3 VVT Time and Risk Modeling 521

6.3.1 System/VVT network 521

6.3.2 Modeling time of system/VVT lifecycle 524

6.3.3 Time and risk example 528

6.4 Fuzzy VVT Cost Modeling 530

6.4.1 Introduction 530

6.4.2 General fuzzy logic modeling 530

6.4.3 Fuzzy modeling of the VVT process 532

6.4.4 Fuzzy VVT cost and risk estimation example 541

6.4.5 Fuzzy logic versus probabilistic modeling 544

6.5 References 548

7. Obtaining Quality Data and Optimizing VVT Strategy 550

7.1 Systems’ Quality Costs in the Literature 550

7.2 Obtaining System Quality Data 554

7.2.1 Quality data acquisition 554

7.2.2 Quality data aggregation 555

7.3 IAI/Lahav Quality Data—An Illustration 557

7.3.1 IAI/Lahav pilot project 557

7.3.2 Obtaining raw system and quality data 559

7.3.3 Anchor system and quality data 560

7.3.4 Generating the VVT model database 561

7.4 The VVT-Tool 562

7.4.1 Background 562

7.4.2 Tool availability 563

7.5 VVT Cost, Time and Risk Optimization 564

7.5.1 Optimizing the VVT process 565

7.5.2 Loss function optimization—VVT cost 569

7.5.3 Weight optimization—VVT cost 576

7.5.4 Goal optimization—VVT cost 580

7.5.5 Genetic algorithm optimization—VVT time 584

7.5.6 Genetic multi-domain optimization—VVT cost and time 596

7.6 References 600

8. Methodology Validation and Examples 604

8.1 Methodology Validation Using a Pilot Project 604

8.1.1 VVT cost model validation 605

8.1.2 VVT time model validation 610

8.1.3 Fuzzy VVT cost model validation 617

8.2 Optimizing the VVT Strategy 618

8.2.1 Analytical optimization of cost 619

8.2.2 Cost distribution by phase 626

8.2.3 Weight optimization of cost 627

8.2.4 Goal optimization of cost 631

8.2.5 MPGA optimization for time 635

8.2.6 SSGA optimization of cost and time 637

8.3 Identifying and Avoiding Signifi cant Risks 639

8.3.1 Avoiding critical risks 640

8.3.2 Conjecture on future risk scenarios 642

8.4 Improving System Quality Process 644

Appendix A SysTest Project 646

A.1 About SysTest 646

A.2 SysTest Key Products 648

A.3 SysTest Pilot Projects 649

A.4 SysTest Team 653

A.5 EC Evaluation of SysTest Project 655

References 656

Appendix B Proposed Guide: System Verification, Validation and Testing Master Plan 657

B.1 Background 657

B.2 Creating the VVT-MP 658

B.3 Chapter 1: System Description 659

B.3.1 Project applicable documents 659

B.3.2 Mission description 659

B.3.3 System description 659

B.3.4 Critical technical parameters 660

B.4 Chapter 2: Integrated VVT Program Summary 660

B.4.1 Integrated VVT program schedule 660

B.4.2 VVT program management 661

B.5 Chapter 3: System VVT 662

B.5.1 VVT strategy 662

B.5.2 Planning VVT activities 665

B.5.3 VVT limitations 668

B.6 Chapter 4: VVT Resource Summary 669

B.6.1 Test articles 669

B.6.2 Test sites and instrumentation 669

B.6.3 Test support requisition 669

B.6.4 Expendables for testing 669

B.6.5 Operational force test support 670

B.6.6 Simulations, models and test beds 670

B.6.7 Manpower/personnel needs and training 670

B.6.8 Budget summary 670

Appendix C List of Acronyms 671

Index 679