72204ReportBackPerformanceAnalysisTrack

1、17/22/04 Report Back:Performance Analysis TrackDr. Carol SmidtsWes Deadrick2Track Members Carol Smidts (UMD) Track Chair Integrating Software into PRA Ted Bennett and Paul Wennberg (Triakis) Empirical Assurance of Embedded Software Using Realistic Simulated Failure Modes Dolores Wallace (GSFC) Syste

2、m and Software Reliability Bojan Cukic (WVU) Compositional Approach to Formal Models Kalynnda Berens (GRC) Software Safety Assurance of Programmable Logic Injecting Faults for Software Error Evaluation of Flight Software Hany Ammar (WVU) Risk Assessment of Software Architectures3Agenda Characterizat

3、ion of the Field Problem Statement Benefits of Performance Analysis Future Directions Limitations Technology Readiness Levels4Characterization of Field Goal: Prediction and Assessment of Software Risk/Assurance Level (Mitigation optimization) System Characteristics of interest Risk (Off-nominal situ

4、ations) Reliability, availability, maintainability = Dependability Failures - general sense Performance Analysis Techniques - modeling and simulation, data analysis, failure analysis, design analysis focused on criticality5Problem Statement Why should NASA do performance analysis? - We care if thing

5、s fail! Successfully conducting SW and System Performance Analysis gives us the data necessary to make informed decisions in order to improve performance and overall quality Performance analysis permits: Ability to determine if/when system meets requirements Risk reduction and quantification Applica

6、tion of new knowledge to future systems A better understanding of the processes by which systems are developed and therefore enables NASA to exercise continual improvement6Benefits of Performance Analysis Reduced development and operating costs Manage and optimize current processes thereby resulting

7、 in more efficient and effective processes Defined and repeatable process reduced time to do same volume of work Reduces risk and increases safety and reliability Better software architecture designs More maintainable systems Enable NASA to handle more complex systems in the future Put the responsib

8、ility where it belongs from a organizational perspective - focuses accountability7Future Directions for Performance Analysis Automation of modeling and data collection increased efficiency and accuracy A more useful, better reliability model useful = user friendly (enable the masses not just the dom

9、ain experts), increased usability of the data (learn more from what we have) better = greater accuracy and predictability Define and follow repeatable methods/processes for data collection and analysis including: education and training use of simulation gold nugget = accurate and complete data8Futur

10、e Directions for Performance Analysis (Cont.) Develop a method for establishing accurate performance predictions earlier in life cycle Evolve to refine system level assessment factor in the human element Establish and define an approach to performing trade-off of attributes reliability, etc. Need fo

11、r early guidance on criticality of components Optimize a defect removal model Methods and metrics for calculating/defending return on investment of conducting performance analysis9Why not Standard traps - Obstacles Uncertainty about scalability User friendliness Lack of generality “Not invented here

12、” syndrome Costs and benefits Difficult to assess and quantify Long term project benefit tracking recommended10Technology Readiness Level Integrating Software into PRA Taxonomy (7) Test-Based Approach for Integrating SW in PRA (3) Empirical Assurance of Embedded Software Using Realistic Simulated Fa

13、ilure Modes (5) Maintaining system and SW test consistency (8) System Reliability (3) Software Reliability (9) Compositional Approach to Formal Models (2) Software Safety Assurance of Programmable Logic (2) Injecting Faults for Software Error Evaluation of Flight Software (9) Risk Assessment of Soft

14、ware Architectures (5)11Research Project Summaries12Integrating Software Into PRADr. Carol Smidts, Bin LiObjective: PRA is a methodology to assess the risk of large technological systems The objective of this research is to extend current classical PRA methodology to account for the impact of softwa

15、re onto mission risk13Integrating Software Into PRA (Cont)AchievementsDeveloped a software related failure mode taxonomy Validated the taxonomy on multiple projects (ISS, Space Shuttle, X38)Proposed a step-by-step approach to integration in the classical PRA framework with quantification of input an

16、d functional failures. 14ProblemDisconnect exists between System and software development loopsSYSTEMDesign/DebugAnalyze/Test/V&VModel,Simulate,Prototype,ES, etc.SWInterpretationRequirementsAnalyze/Test/VerifyDesign/DebugBuildIntegration TestingMost embedded SW faults found at integ. test traceable

17、to Rqmts. & interface misunderstandingTRIAKIS Corporation15Approach Develop & simulate entire system design using executable specifications (ES) Verify total system design with suite of tests Simulate controller hardware Replace controller ES with simulated HW running object (flight) software Test S

18、W using system verification testsWhen SW passes all system verification tests, it has correctly implemented all of the tested requirementsTRIAKIS Corporation16Mini-AERCamIV&V FacilityEmpirical Assurance of Embedded SWUsing Realistic Simulated Failure Modes Problem: FMEA Limitations Expensive & time-

19、consuming List of possible failure modes extensive Focuses on prioritized subset of failure modes Approach: Test SW w/simd Failures Create pure virtual simulation of Mini-AERCam HW & flight environment running on PC Induce realistic component/subsystem failures Observe flight SW response to induced

20、failures Can we improve coverage by testing SW resp. to simd failures?nCompare results with project-sponsored FMEA, FTA, etc.:#Issues uncovered?#Failure modes evaluated?Effort involved?TRIAKIS Corporation17Software and System ReliabilityDolores Wallace, Bill Farr, Swapna Gokhale Addresses the need t

21、o evaluate and assess the reliability and availability of large complex software intensive systems by predicting (with associated confidence intervals): The number of software/system faults, Mean time to failure and restore/repair, Availability, Estimated release time from testing.182003 & 2004 Rese

22、arch Literature search completed New models were selected: 1) Enhanced Schneidewind (includes risk assessment and trade-off analysis) and 2) Hypergeometric Model Incorporated the new software models into the established public domain tool SMERFS3 Applied the new models on a Goddard software project

23、Made the latest version of SMERFS3 available to the general public Conducted similar research effort for System Reliability and Availability Will enhance SMERFS3 and validate the system models on a Goddard data set19A Compositional approach to Validation of Formal Models Problem Significant number o

24、f faults in real systems can be traced back to specifications. Current methodologies of specification assurance have problems: Theorem Proving: Complex Model Checking:State explosion problems Testing:Incomplete. Approach Combine them! Use test coverage to build abstractions. Abstractions reduce the

25、size of the state space for model checking. Develop visual interfaces to improve the usability of the method.Dejan Desovski, Bojan Cukic20Software Fault Injection ProcessKalynnda Berens, Dr. John Crigler, Richard Plastow Standardized approach to test systems with COTS and hardware interfaces Provide

26、s a roadmap of where to look to determine what to testIdentify Interfaces and Critical SectionsError/Fault ResearchEstimate Effort RequiredObtain Source Code and DocumentationStartSufficient time and funds?Importance AnalysisSelect SubsetTest Case Generation Fault Injection TestingDocument Results,

27、Metrics, Lessons LearnedFeedback to FCF ProjectEndYes21Programmable Logic at NASA Kalynnda Berens, Jacqueline Somos Issues Lack of good assurance of PLCs and PLDs Increasing complexity = increasing problems Usage and Assurance Survey - SA involved in less than 1/3 of the projects; limited knowledge

28、Recommendations Trained SA for PLCs PLDs determine what is complex; use process assurance (SA or QA) Training Created Basic PLC and PLD training aimed at SA Process assurance for hardware QA22Year 2 of Research What is industry and other government agencies doing for assurance and verification? An i

29、ntensive literature search of white papers, manuals, standards, and other documents that illustrated what various organizations were doing. Focused interviews with industry practitioners. Interviews were conducted with assurance personnel (both hardware and software) and engineering practitioners in

30、 various industries, including biomedical, aerospace, and control systems. Meeting with FAA representatives. Discussions with FAA representatives lead to a more thorough understanding of their approach and the pitfalls they have encountered along the way. Position paper, with recommendations for NAS

31、A Code Q23Current Effort Implement some of the recommendations Develop coursework to educate software and hardware assurance engineers Three courses PLCs for Software Assurance personnel PLDs for Software Assurance personnel Process Assurance for Hardware QA Guidebook Other recommendations For Code Q to implement if des