IV&V a) Quantitatively Characterizing Software IV&V Activities Based on Defect History Shinsuke MATSUMOTO a), Hidetake UWANO, Akito MONDEN, Ken-ichi M

Size: px

Start display at page:

Download "IV&V a) Quantitatively Characterizing Software IV&V Activities Based on Defect History Shinsuke MATSUMOTO a), Hidetake UWANO, Akito MONDEN, Ken-ichi M"

たかよししげまつ
4 years ago
Views:

1 IV&V a) Quantitatively Characterizing Software IV&V Activities Based on Defect History Shinsuke MATSUMOTO a), Hidetake UWANO, Akito MONDEN, Ken-ichi MATSUMOTO, Masafumi KATAHIRA, Naohiko KOHTAKE, Yuko MIYAMOTO, Shohgo UJIHARA, and Shigeo YOSHIKAWA Independent Verification & Validation (IV&V) IV&V IV&V Verification Validation 6 IV&V Verification Validation (JAXA) IV&V IV&V IV&V 1. Software Independent Verification and Validation SW IV&V IV&V IV&V Graduate School of Information Science, Nara Institute of Science and Technology, Takayama, Ikoma-shi, Japan Japan Aerospace Exploration Agency, Sengen, Tsukuba-shi, Japan Japan Manned Space Systems Corporation, Kawaguchi, Tsuchiura-shi, Japan a) shinsuke-m@is.naist.jp [1], [2] NASA IV&V IV&V IV&V IV&V [2], [3] IV&V IV&V D Vol. J92 D No. 12 pp c

2 2009/12 Vol. J92 D No. 12 [4], [5] IEEE Std [6] Chillarege Orthogonal Defect Classification (ODC) [7] ODC ODC-CC [8] IV&V IV&V Lessons Learned IV&V Verification Validation 2 [9] IV&V IV&V Verification Validation 6 (JAXA) IV&V 2. IV&V JAXA IV&V Independent Verification and Validation 2. 1 IV&V IV&V Verification Validation [1], [2] IV&V Independent IEEE Std [10], [11] IV&V Verification Validation 1 Verification ISO Verification Validation ISO

3 IV&V 2 IV&V Fig. 2 Structure of IV&V activity. 1 Verification & Validation Fig. 1 Verification and Validation activities for software development process. JAXA Validation Verification Validation IV&V IV&V FTA FMEA IV&V IV&V 2 IV&V JAXA 1 IV&V 2. 2 IV&V IV&V [8] ODC [7] [12] Beizer [13] [9] ODC Function Interface Checking Assignment Timing/Serialization Build/Package/Merge Documentation Algorithm 1IV&V 2197

4 2009/12 Vol. J92 D No. 12 ODC Interface Documentation 30% [14] IV&V Verification Validation IV&V Verification Validation IV&V Verification Validation IV&V Verification Validation IV&V IV&V IV&V Verification Validation IV&V IV&V Verification Validation 6 Verification Validation IPO Input-Process- Output IPO 3 IPO 3 IPO Fig. 3 IPO model and our proposed functional types of defect. 2198

5 IV&V IPO / Verification/Validation Verification/Validation Verification Validation Verification Verification Validation Validation Fig. 4 A visualization of proposed defect classification. 5 Fig. 5 An example of visualization. Verification/Validation JAXA IV&V IV&V

6 2009/12 Vol. J92 D No. 12 Table 1 1 A list of metrics used in the analysis. ID 1, 2, 3 Lessons Learned Verification/Validation Verification Validation 3 (1) (2) 4 (a) (b) (c) IV&V JAXA IV&V IV&V Lessons Learned Lessons Learned [15] DOORS [16] FTA Fault Tree Analysis [17] FMEA Failure Mode and Effects Analysis [18] 2200

7 IV&V JAXA IV&V IV&V IV&V IV&V JAXA 2 Table 2 The number of detected defects in each perspective. # % Lessons Learned % % % % % % 6 Fig. 6 Classification of defects by detection perspective. IV&V 7 IV&V JAXA IV&V IV&V 6 7 Lessons Learned Validation /14 12/14 7 Lessons Learned Validation IV&V Lessons Learned Verification 4/9 2201

8 2009/12 Vol. J92 D No Fig. 7 Assumed defect classification in each detection perspective. 8 Fig. 8 Classification of defects by detection technique. Verification 3/9 IV&V IV&V Lessons Learned 23/43 Lessons Learned IV&V IV&V Table 3 The number of detected defects in each method. # % % % % 38/ Validation [19] Verification Validation Validation Verification Validation Veri- 2202

9 IV&V 9 Fig. 9 Risk presence/absence of each defect. 4 Table 4 The number of detected defects of risk presence/absence. # % % % % fication Validation IV&V Verification Validation Verification Validation IV&V 57% 8/14 59% 13/22 86% 6/7 IV&V IV&V IV&V IV&V 2 JAXA % 7/35 51% 18/35 A.1 Verification Validation IV&V Verification Validation Verification Verification Validation IV&V IV&V Verification 2203

10 2009/12 Vol. J92 D No. 12 Validation IV&V IV&V IV&V IV&V IV&V IV&V JAXA IV&V 1 6. IV&V JAXA IV&V Verification Validation IV&V JAXA IV&V IV&V IV&V Verification Validation IV&V IV&V IT [1] J.D. Arthur, M.K. Groener, K.J. Hayhurst, and C.M. Holloway, Adding value to the software development process: A study in independent verification and validation, Technical Report, TR-98-15, Virginia Tech, [2] R.O. Lewis, Independent verification and validation: A life cycle engineering process for quality software, Wiley, [3] A.B. Bassam, B. Jonathan, and R.M. Philippe, Independent validation and verification of the tcas ii collision avoidance subsystem, Aerospace and Electronic Systems Magazine, vol.15, no.8, pp.3 21, [4] M.E. Fagan, Design and code inspection to reduce errors in program development, IBM Syst. J., vol.15, no.3, pp , [5] O. Laitenberger, Studying the effects of code inspection and structural testing on software quality, ISSRE 98: Proc. Ninth International Symposium on Software Reliability Engineering, pp , IEEE Computer Society, Washington, DC, USA, [6] IEEE/ANSI, IEEE standard classification for software anomalies [7] R. Chillarege, I.S. Bhandari, J.K. Chaar, M.J. Halliday, D.S. Moebus, B.K. Ray, and M.Y. Wong, Orthogonal defect classification A concept for in-process measurements IEEE Trans. Softw. Eng., vol.18, no.11, pp , [8] D. Kelly and T. Shepard, A case study in the use of defect classification in inspections, CASCON 01: Proc Conference of the Centre for Advanced 2204

IV&V Studies on Collaborative Research, p.7, IBM Press, 2001. [9] L.O. Damm and L.

4th International Symposium on Empirical Software Engineering, pp.152 161, 2005. [10] IEEE/ANSI, IEEE standard for software verification and validation plans 2004.

Grady, Practical software metrics for project management and process improvement, Prentice- Hall, Upper Saddle River, NJ, USA, 1992. [13] B.

1st ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp.99 106, 1993. [16] Telelogic, Doors tools, http://www.telelogic.com/doors [17] U.S. Nuclear Regulatory Commission NRC, Fault tree handbook, NUREG-0492, 1981.

Peled, Model checking, The MIT Press, 1999. A 1 Table A 1 The number of differences with our classification result.

11 IV&V Studies on Collaborative Research, p.7, IBM Press, [9] L.O. Damm and L. Lundberg, Identification of test process improvements by combining fault trigger classification and faults-slip-through measurement, ISESE 05: Proc. 4th International Symposium on Empirical Software Engineering, pp , [10] IEEE/ANSI, IEEE standard for software verification and validation plans [11] IV&V Independent Verification and Validation 155 vol.2007-se-155, no.33, pp.81 87, [12] R.B. Grady, Practical software metrics for project management and process improvement, Prentice- Hall, Upper Saddle River, NJ, USA, [13] B. Beizer, Software testing techniques, 2nd ed., Van Nostrand Reinhold New York, NY, USA, [14] vol.48, no.5, pp , May [15] R.R. Lutz, Targeting safety-related errors during software requirements analysis, Proc. 1st ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp , [16] Telelogic, Doors tools, [17] U.S. Nuclear Regulatory Commission NRC, Fault tree handbook, NUREG-0492, [18] D.J. Reifer, Software failure modes and effects analysis, IEEE Trans. Reliab., vol.28, no.2, pp , [19] M.C. Edmund, O. Grumberg, and D.A. Peled, Model checking, The MIT Press, A 1 Table A 1 The number of differences with our classification result. A B 5 (14%) 9 (26%) 7 (20%) 17 (49%) 19 (54%) 18 (51%) DC2 IEEE ACM IEEE ACM Auckland IEEE ACM / ACM IEEE Senior Member 1987 NASDA H-IIA Complex Systems Research Lab (Prof. Nancy G. Leveson) IV&V RTOS IEEE 2205

12 2009/12 Vol. J92 D No H-IIA IV&V PMP INCOSE IEEE IV&V JSTQB Foundation Level 2008 IV&V IV&V 2206

2

Copyright 2008 Nara Institute of Science and Technology / Osaka University 2 Copyright 2008 Nara Institute of Science and Technology / Osaka University CHAOS Report in US 1994 http://www.standishgroup.com/sample_research/