Software Fault Tolerance Techniques and Implementation phần 4 pdf

3.3.2 Dependable System Development Model Given the complexity of computer-based critical software, the diversity offaults to be handled by these systems, and the consequences and severi

3.3.1.7 Summary of Design Considerations

Little specific assistance is available to make the required decisions aboutdesign, particularly since many of them are at least partially application-dependent However, cost and overhead information (such as that describedearlier in this chapter), performance analysis (such as that described in [63]),design methodologies (e.g., those described in the next sections), and proto-type design assistance tools (e.g., the Software Fault Tolerance Design Assis-tant (SFTDA) [87]) provide valuable guidance and input to the necessarydecisions

3.3.2 Dependable System Development Model

Given the complexity of computer-based critical software, the diversity offaults to be handled by these systems, and the consequences and severity

of their failure, a systematic and structured design framework that integratesdependability concerns and requirements at the early stages of (and through-out) the development process is needed [88–90] Software design faults arerecognized as the current obstacle to successful dependable systems develop-ment [91] Conventional development methods do not incorporate theprocesses and key activities required for effective development of dependablesystems To fill this need, Kaaniche, Blanquart, Laprie, and colleagues [91,92] developed the dependability-explicit development model The modelprovides guidelines emphasizing the key issues to be addressed during themain stages of dependable systems development [91, 92] In this section, weprovide an overview of the development model’s key activities for the faulttolerance process and refer the reader to the sources [91, 92] for additionaldetails and activities in other processes

The dependability-explicit development model provides lists of keyactivities related to system development phases The requirements phasebegins with a detailed description of the system’s intended functions anddefinition of the system’s dependability objectives The following list[91, 92] summarizes the key activities in the fault tolerance process forthis phase

• Description of system behavior in the presence of failures:

• Identification of relevant dependability attributes and necessarytrade-offs;

• Failure modes and acceptable degraded operation modes;

• Maximum tolerable duration of service interruption for eachdegraded operation mode;

• Number of consecutive and simultaneous failures to be toleratedfor each degraded operation mode

The main objective of the design phase is to define an architecture thatwill allow the system requirements to be met The following list [91, 92]summarizes the key fault tolerance activities and issues for this phase

• Description of system behavior in presence of faults:

• Fault assumptions (faults considered, faults discarded);

• System partitioning:

• Fault tolerance structuring: fault-containment regions, containment regions;

error-• Fault tolerance application layers;

• Fault tolerance strategies:

• Redundancy, functional diversity, defensive programming, tection techniques and others;

pro-• Error-handling mechanisms:

• Error detection, error diagnosis, error recovery;

• Fault-handling mechanisms:

• Fault diagnosis, fault passivation, reconfiguration;

• Identification of single points of failure

The realization phase consists of implementing the system componentsbased on the design specification Below is a summary of the key fault toler-ance process activities for the implementation or realization phase [91, 92]

• Collect the number of faults discovered during this stage:

• Use as indicator of component dependability;

• Use to identify system components requiring reengineering.The integration phase consists of assembling the system componentsand integrating the system into its environment to make sure that the final

product meets its requirements Following is a summary of the key fault erance activities for the integration phase [91, 92].

tol-• Verification of integration of fault and error processing mechanisms:

• Use analysis and experimentation to ensure validated tolerant subsystems satisfy dependability requirements when inte-grated;

fault-• Use fault injection (multiple and near-coincident faults);

• Evaluate fault tolerance mechanisms’ efficiency;

• Estimate fault tolerance mechanism coverage—fault injectionexperiments

The dependability-explicit development model is provided to ensurethat dependability related issues are considered at each stage of the develop-ment process The model is generic enough to be applied to a wide range ofsystems and application domains and can be customized as needed Since thekey activities and guidelines of the model focus on the nature of activities to

be performed and the objectives to be met, they can be applied regardless ofwhich development methods are used

3.3.3 Design Paradigm for N-Version Programming

Although the NVP technique is presented in Section 4.2, we describe in thissection a design paradigm for NVP because it contains guidelines and rulesthat can be useful in the design of many software fault tolerance tech-niques It is generally agreed that a high degree of variant independenceand a low probability of failure correlation are vital to successful operation ofN-version software (NVS) This requires attaining the lowest possible prob-ability that the effects of similar errors in the variants will coincide atthe DM The design paradigm for NVP was developed and refined by Lyuand Avizienis [93–96] to achieve these goals Hence, the objectives of thedesign paradigm, as stated in [96] are:

• To reduce the possibility of oversights, mistakes, and inconsistencies

in the process of software development and testing;

• To eliminate most perceivable causes of related design faults in theindependently generated versions of a program, and to identifycauses of those that slip through the design process;

• To minimize the probability that two or more versions will producesimilar erroneous results that coincide in time for a decision (con-sensus) action of the N-version executive (NVX).

The design paradigm for NVP is illustrated in Figure 3.7 [96] Asshown, it consists of two groups of activities On the left side of the figure arethe standard software development activities To the right are the activitiesspecifying the concurrent implementation of NVS Table 3.4 summarizesthe NVS design paradigm’s activities and guidelines incorporated into thesoftware development life cycle (The table was developed by combininginformation found in [95] (the table structure and initial entries) and [96](updated information on the refined paradigm) For more detail on theparadigm and a discussion of the associated issues, the reader is referred

to [95, 96]

3.4 Summary

This chapter presented software fault tolerance problems and issues, gramming techniques, and design and development considerations andmodels The advantages of software fault tolerance are accompanied by dis-advantages, issues to consider, and costs Those common to most tech-niques were covered here We covered perhaps the greatest bane of designdiversity—similar errors If these are not avoided then software faulttolerance techniques based on design diversity will not be effective Otherissues and potential problems to be considered were covered, including theCCP with FPA applications, the domino effect in backward recovery,and overhead (not just cost, but time, operation overhead, redundancy, andmemory)

pro-Then, to help in development, we described several programmingmethods that are used by several software fault tolerance techniques Theseinclude assertions (that can be used by fault tolerant or non-fault-tolerantsoftware), checkpointing (typically used in techniques employing backwardrecovery), and atomic actions (also used in non-fault-tolerant software, butpresented here in reference to concurrent systems)

Backing out the scope, we then present methods to assist in the designand development of critical, fault-tolerant software systems Design consid-erations, a development model for dependable systems, and a design para-digm specific to NVP are presented

Yes No

Determine method of NVS supervision

Conduct NVS development protocol

Figure 3.7 Design paradigm for N-version programming (Source: [96], © 1995 John

Wiley and Sons Reproduced with permission.)

This chapter has focused on issues that are fairly common across ware fault tolerance techniques In the following Chapters 4, 5, and 6, weexamine individual techniques, including technique-specific issues.

soft-96 Software Fault Tolerance Techniques and Implementation

Table 3.4 N-Version Programming Design Paradigm Activities and Guidelines Software Life

Cycle Phase Enforcement ofFault Tolerance Design Guidelines and Rules System

requirement Determine methodof NVS supervision 1 Choose NVS execution method and allocate required1 resources

2 Develop support mechanisms and tools

3 Select hardware architecture Software

requirement Select softwaredesign diversity

dimensions

1 Assess random diversity versus required diversity

2 Evaluate required design diversity

3 Specify diversity under application constraints Software

specification Install errordetection and

recovery algorithms

1 Specify the matching features needed by NVX

2 Avoid diversity-limiting factors

3 Diversify the specification Design and

coding Conduct NVSdevelopment

protocol

1 Impose a set of mandatory rules of isolation

2 Define a rigorous communication and documentation

1 Define NVS acceptance criteria

2 Assess evidence of diversity

3 Make NVS dependability predictions Operational Choose and

implement an NVS maintenance policy

1 Assure and monitor NVX functionality

2 Follow the NVP paradigm for NVS modification

Team-Fly®

[6] Littlewood, B., and D R Miller, “Conceptual Modeling of Coincident Failures in Multiversion Software,” IEEE Transactions on Software Engineering, Vol 15, No 12,

[9] Vouk, M A., et al., “An Empirical Evaluation of Consensus Voting and Consensus Recovery Block Reliability in the Presence of Failure Correlation,” Journal of Computer and Software Engineering, Vol 1, No 4, 1993, pp 367–388.

[10] Anderson, T., and P A Lee, Fault Tolerance: Principles and Practice, Englewood Cliffs, NJ: Prentice-Hall, 1981.

[11] Pullum, L L., “Fault Tolerant Software Decision-Making Under the Occurrence of Multiple Correct Results,” Doctoral dissertation, Southeastern Institute of Technol- ogy, 1992.

[12] Brilliant, S., J C Knight, and N G Leveson, “The Consistent Comparison Problem

in N-Version Software,” ACM SIGSOFT Software Engineering Notes, Vol 12, No 1,

1987, pp 29–34.

[13] Brilliant, S., J C Knight, and N G Leveson, “The Consistent Comparison Problem

in N-Version Software,” IEEE Transactions on Software Engineering, Vol 15, No 11,

1989, pp 1481–1485.

[14] Knight, J C., and P E Ammann, “Issues Influencing the Use of N-Version ming,” Information Processing ’89, 1989, pp 217–222.

Program-[15] Ammann, P E., and J C Knight, “Data Diversity: An Approach to Software Fault Tolerance,” IEEE Transactions on Computers, Vol 37, No 4, 1989, pp 418–425 [16] Knuth, D E., The Art of Computer Programming, Reading, MA: Addison-Wesley, 1969.

[17] Randell, B., “System Structure for Software Fault Tolerance,” IEEE Transactions on Software Engineering, Vol SE-1, No 2, 1975, pp 220–232.

[18] Kim, K H., “An Approach to Programmer-Transparent Coordination of Recovering Parallel Processes and Its Efficient Implementation Rules,” Proceedings IEEE Computer Society’s International Conference on Parallel Processing, 1978, pp 58–68.

[19] Nelson, V P., and B D Carroll, “Software Fault Tolerance,” in V P Nelson and

B D Carroll (eds.), IEEE Tutorial on Fault Tolerant Computing, Washington, D.C.: IEEE Computer Society Press, 1987, pp 247–256.

[20] Randell, B., and J Xu, “The Evolution of the Recovery Block Concept,” in M R Lyu (ed.), Software Fault Tolerance, New York: John Wiley and Sons, 1995, pp 1–21 [21] Anderson, T., and J Knight, “A Framework for Software Fault Tolerance in Real- Time Systems,” IEEE Trans on Software Engineering, Vol SE-9, No 5, 1983,

pp 355–364.

[22] Kelly, J P J., T I McVittie, and W I Yamamoto, “Implementing Design Diversity

to Achieve Fault Tolerance,” IEEE Software, July 1991, pp 61–71.

[23] Levi, S.-T., and A K Agrawala, Fault-Tolerant System Design, New York: Hill, 1994.

McGraw-[24] Kim, K H., “Approaches to Mechanization of the Conversation Scheme Based on Monitors,” IEEE Trans on Software Engineering, Vol SE-8, No 5, 1993,

pp 189–197.

[25] Kim, K H., “Distributed Execution of Recovery Blocks: Approach to Uniform ment of Hardware and Software Faults,” Proceedings IEEE 4th International Conference

Treat-on Distributed Computing Systems, 1984, pp 526–532.

[26] Kim, K H., “Programmer-Transparent Coordination of Recovering Concurrent Processes: Philosophy & Rules,” IEEE Transactions on Software Engineering, Vol 14,

[29] Xu, J., A Bondavalli, and F Di Giandomenico, “Dynamic Adjustment of ability and Efficiency in Fault-Tolerant Software,” in B Randell, et al (eds.), Predict- ably Dependable Computing Systems, New York: Springer-Verlag, 1995, pp 155–172 [30] Halton, L., “N-Version Design Versus One Good Design,” IEEE Software, Nov./Dec.

Experi-[38] Bhargava, B., and C Hua, “Cost Analysis of Recovery Block Scheme and Its mentation Issues,” International Journal of Computer and Information Sciences, Vol 10,

Imple-No 6, 1981, pp 359–382.

[39] McAllister, D F., “Some Observations on Costs and Reliability in Software Tolerant Techniques,” Proceedings TIMS-ORSA Conference, Boston, MA, 1985 [40] Saglietti, F., and W Ehrenberger, “Software Diversity—Some Considerations about Benefits and Its Limitations,” Proceedings IFAC SAFECOMP ‘86, Sarlet, France, 1986,

Jour-B PWR, Churchill College, Cambridge, 1992.

[45] Ward, N J., “Rigorous Retrospective Static Analysis of the Sizewell B Primary tection System Software,” Proceedings IFAC SAFECOMP ’93, Poznan-Kiekrz, Poland, 1993.

Pro-[46] Kanoun, K., “Cost of Software Design Diversity: An Empirical Evaluation,” LAAS Report No 9163, Toulouse, France: LAAS, 1999.

[47] Ramamoorthy, C V., et al., “Software Engineering: Problems and Perspectives,” puter, Vol 17, No 10, 1984, pp 191–209.

Com-[48] Boehm, B W., Software Engineering Economics, Englewood Cliffs, NJ: Prentice-Hall, 1981.

[49] Parnas, D L., A J Van Schouwen, and A Po Kwan, “Evaluation of Safety-Critical Software,” Communications of the ACM, June 1990, pp 636–648.

[50] Mili, A., An Introduction to Program Fault Tolerance: A Structured Programming Approach, New York: Prentice-Hall, 1990.

[51] Bjork, L A., “Generalized Audit Trail Requirements and Concepts for Data Base Applications,” IBM Systems Journal, Vol 14, No 3, 1975, pp 229–245.

[52] Horning, J., et al., “A Program Structure for Error Detection and Recovery,” in ture Notes in Computer Science, Vol 16, New York: Springer-Verlag, 1974,

Lec-pp 171–187.

[53] Lee, P A., N Ghani, and K Heron, “A Recovery Cache for the PDP-11,” IEEE Transactions on Computers, June 1980, pp 546–549.

[54] Saglietti, F., “Location of Checkpoints by Considering Information Reduction,” in

M Kersken and F Saglietti (eds.), Software Fault Tolerance: Achievement and ment Strategies, New York: Springer-Verlag, 1992, pp 225–236.

Assess-[55] Saglietti, F., “Location of Checkpoints in Fault-Tolerant Software,” IEEE, 1990,

pp 270–277.

[56] Nicola, V F., and J M Spanje, “Comparative Analysis of Different Models of pointing and Recovery,” IEEE Transactions on Software Engineering, Vol 16, No 8,

Check-1990, pp 807–821.

[57] Nicola, V F., “Checkpointing and the Modeling of Program Execution Time,” in

M R Lyu (ed.), Software Fault Tolerance, New York: John Wiley & Sons, 1995,

pp 167–188.

[58] Kulkarni, V G., V F Nicola, and K S Trivedi, “Effects of Checkpointing and Queuing on Program Performance,” Communications on Statistics—Stochastic Models, Vol 6, No 4, 1990, pp 615–648.

[59] Wood, W G., “A Decentralized Recovery Protocol,” Proceedings of FTCS-11, land, OR, 1981, pp 159–164.

Port-[60] Juang, T T Y., and S Venkatesan, “Efficient Algorithm for Crash Recovery in tributed Systems,” 10th Conference on Foundations of Software Technology and Theoreti- cal Computer Science (LNCS), 1990, pp 349–361.

Dis-[61] Juang, T T Y., and S Venkatesan, “Crash Recovery with Little Overhead,” 11th International Conference on Distributed Computing Systems, 1991, pp 454–461 [62] Maier, J., “Fault-Tolerant Parallel Programming with Atomic Actions,” in D Pradhan and D Avresky (eds.), Fault-Tolerant Parallel and Distributed Systems, Los Alamitos, CA: IEEE Computer Society Press, 1995, pp 210–219.

[63] Tai, A T., J F Meyer, and A Avizienis, Software Performability: From Concepts to Applications, Norwell, MA: Kluwer Academic Publishers, 1996.

[64] Chandy, K M., and L Lamport, “Distributed Snapshots: Determining Global States

of Distributed Systems,” ACM Transactions on Computer Systems, Vol 3, No 1, 1985,

[67] Tong, Z., R Y Kain, and W T Tsai, “Rollback Recovery in Distributed Systems Using Loosely Synchronized Clocks,” IEEE Transactions on Parallel and Distributed Systems, Vol 3, No 2, 1992, pp 246–251.

[68] Venkatesh, K., T Radhakrishnan, and H F Li, “Optimal Checkpointing and Local Recording for Domino-Free Rollback Recovery,” Proceedings of FTCS-11, Portland,

in Multi-Threaded Object-Oriented Environment,” IEEE Transactions on Reliability, Vol 48, No 4, 1999, pp 325–339.

[74] Garcia, J C R., et al., Optimized Object State Checkpointing Using Compile-Time Reflection, LAAS Report 98173, 1998.

[75] Rafnel, B A., “A Transaction Approach to Error Handling,” Hewlett-Packard Journal, June 1993, pp 71–77.

[76] Lee, P A., and T Anderson, Fault Tolerance: Principles and Practice, 2nd ed., New York: Springer-Verlag, 1990.

[77] Lomet, D B., “Process Structuring, Synchronization and Recovery Using Atomic Actions,” ACM SIGPLAN Notices, Vol 12, No 3, 1977, pp 128–137.

[78] Liskov, B., and R Scheifler, “Guardians and Actions: Linguistic Support for Robust Distributed Programs,” ACM TOPLAS, Vol 5, No 3, 1983, pp 381–404.

[79] Reed, D P., “Implementing Atomic Actions on Decentralized Data,” ACM TOCS, Vol 1, No 1, 1975, pp 3–23.

[80] Campbell, R H., and B Randell, “Error Recovery in Asynchronous Systems,” IEEE Transactions on Software Engineering, Vol SE-12, No 8, 1986, pp 811–826 [81] Moss, J E B., An Introduction to Nested Transactions, Technical Report 86-41, Amherst, MA: University of Amherst Massachusetts, 1986.

[82] Burns, A., and A J Wellings, Real-Time Systems and Their Programming Languages, Reading, MA: Addison-Wesley, 1990.

[83] Wellings, A J., and A Burns, “Implementing Atomic Actions in Ada 95,” IEEE Transactions on Software Engineering, Vol 23, No 2, 1997, pp 107–123.

[84] Detlefs, D., M P Herlihy, and J M Wing, “Inheritance of Synchronization and Recovery Properties in Avalon/C++,” IEEE Computer, Vol 21, No 2, 1988,

pp 57–69.

[85] Shrivastava, S K., G D Dixon, and G D Parrington, “An Overview of the Arjuna Distributed Programming System,” IEEE Software, Vol 8, No 1, 1991, pp 66–73 [86] Laprie, J -C., et al., “Definition and Analysis of Hardware-and-Software Fault- Tolerant Architectures,” in B Randell et al (eds.), Predictably Dependable Computing Systems, New York: Springer, 1995, pp 103–122.

[87] Pullum, L L., Software Fault Tolerance Design Assistant (SWFTDA) User’s Manual, Quality Research Associates Technical Report, QRA-SWFTDA-SUM, 1997 [88] Avizienis, A., “Building Dependable Systems: How to Keep Up with Complexity,” Proceedings of FTCS-25—Special Issue, Pasadena, CA, 1995, pp 4–14.

[89] Laprie, J -C., “Dependability of Computer Systems: from Concepts to Limits,” 1998 IFIP International Workshop on Dependable Computing and Its Applications (DCIA98), Johannesburg, South Africa, 1998, pp 108–126.

[90] Laprie, J -C., “Software-Based Critical Systems,” Proceedings of the 15th International Conference on Computer Safety, Reliability and Security (SAFECOMP’96), Vienna, Aus- tria, 1996, pp 157–170.

[91] Kaaniche, M., J -P Blanquart, and J -C Laprie, A Dependability-Explicit ment Model, LAAS Report 98341, 1998.

Develop-[92] Kaaniche, M., J -C Laprie and J -P Blanquart, “A Dependability-Explicit Model for the Development of Computing Systems,” 19th International Conference on Com- puter Safety, Reliability and Security (SAFECOMP-2000), Rotterdam, NL, 2000,

or modules) By using design diversity, the designer assumes that coincidentcomponent failure is rare and that, if it does occur, the results are differentenough to enable error detection and to distinguish a correct result or “best”answer The philosophy behind design diversity is to increase the probabilitythat the variants fail on disjoint subsets of the input space when they do fail,

so as to decrease the probability of occurrence of coincident failure

It is interesting to note some early observations and how they relate todesign diversity Dionysius Lardner stated in 1834 [1] that

The most certain and effectual check upon errors which arise in theprocess of computation, is to cause the same computations to be made

by separate and independent computers; and this check is rendered stillmore decisive if they make their computations by different methods

When Lardner uses the term “computer” above, he is referring to the son performing the computation, not to the calculating engine or machine.Charles Babbage, in 1837 [2] stated that

per-105

When the formula to be computed is very complicated, it may bealgebraically arranged for computation in two or more totally distinctways, and two or more sets of cards may be made If the same constantsare now employed with each set, and if under these circumstances theresults agree, we may then be quite secure of the accuracy of them all.

The design diverse techniques discussed in this chapter use these principles todetect and tolerate software faults This chapter covers the original and basicdesign diverse software fault tolerance techniques—recovery blocks (RcB)and N-version programming (NVP) The results of research and applica-tion of these techniques have highlighted several issues regarding these tech-niques Modifications and new techniques have been proposed to overcomethe weaknesses in the original RcB and NVP techniques, while attempting

to maintain the strengths of these foundational techniques The additionaltechniques described in this chapter are distributed recovery blocks (DRB),

N self-checking programming (NSCP), consensus recovery blocks (CRB),and acceptance voting (AV) All the techniques have advantages and disad-vantages that will be discussed in this chapter, along with the techniques’operations

A significant amount of material is presented in this chapter for eachtechnique, broadly divided into operation, example, and issues The first fig-ure shown for each technique describes its operation For those wanting abrief introduction to the technique, this operational figure is a good place tostart Operational details are provided via scenarios and an example Issuesrelated to each technique are summarized in a table in the third subsectionfor each technique A similar format for discussion of each technique is fol-lowed in this chapter and in Chapters 5 and 6

4.1 Recovery BlocksThe basic RcB scheme is one of the two original design diverse software faulttolerance techniques It was introduced in 1974 by Horning, et al [3], withearly implementations developed by Randell [4] in 1975 and Hecht [5] in

1981 In addition to being a design diverse technique, the RcB is furthercategorized as a dynamic technique In dynamic software fault tolerancetechniques, the selection of a variant result to forward as the adjudicated out-put is made during program execution based on the result of the acceptancetest (AT) This will be clarified when we examine the operation of the tech-nique below The hardware fault-tolerant architecture related to the RcBscheme is stand-by sparing or passive dynamic redundancy

106 Software Fault Tolerance Techniques and Implementation

Team-Fly®

RcB uses an AT (see Section 7.2) and backward recovery (see Section1.4.1) to accomplish fault tolerance We know that most program functionscan be performed in more than one way, using different algorithms anddesigns These differently implemented function variants have varyingdegrees of efficiency in terms of memory management and utilization, execu-tion time, reliability, and other criteria RcB incorporates these variants suchthat the most efficient module is located first in the series, and is termed theprimary alternate or primary try block The less efficient variant(s) are placedserially after the primary try block and are referred to as (secondary) alter-nates or alternate try blocks Thus, the resulting rank of the variants reflectsthe graceful degradation in the performance of the variants.

The operation of RcB is described in 4.1.1, with an example provided

in 4.1.2 The advantages and disadvantages of RcB are presented in 4.1.3

4.1.1 Recovery Block Operation

The basic RcB scheme consists of an executive, an acceptance test, andprimary and alternate try blocks (variants) Many implementations of RcB,especially for real-time applications, include a watchdog timer (WDT) [6].The executive orchestrates the operation of the RcB technique, which has thegeneral syntax:

else failure exception

The RcB syntax above states that the technique will first attempt to ensurethe AT (e.g., pass a test on the acceptability of a result of an alternate) byusing the primary alternate (or try block) If the primary algorithm’s resultdoes not pass the AT, then n-1 alternates will be attempted until an alter-nate’s results pass the AT If no alternates are successful, an error occurs.Figure 4.1 illustrates the structure and operation of the basic RcB tech-nique with a WDT We examine several scenarios to describe RcB operation:

• Failure-free operation;

• Exception or time-out in primary alternate execution;