CIS 314

• Chapter 1 - Software Management
• Chapter 2 - Process Modelling
• Chapter 3 - Software Measurement
• Chapter 4 - Project Planning and Estimation
• Chapter 5 - Cost and Schedule Estimation
• Chapter 6 - Comparison and Critique of Cost Models
• Chapter 7 - Function Points
• Chapter 8 - Quality Measurement
• Chapter 9 - Fagan Inspection
• Chapter 10 - Dependability Measurement
• Chapter 11
• Chapter 12 - Safety-Critical Software
• Chapter 13 - Data Collection
• Chapter 14 - Process Maturity Framework
• Chapter 15 - Maintenance Cost Estimation

Chapter 1 - Software Management

References

• Pressman, chapter 1
• Pfleeger, chapter 1
• Sommerville, chapter 1
• Fenton, chapter 3

Why do we need Software Engineering?

Problems

1. Increase in hardware speed has outpaced software development speed.
2. Cannot keep up with the demand from programs
3. Poor design, outdated documentation, unmanageable coding affects the difficulty in maintenance.

Crisis

• What crisis?
• The perspectives from the sofwtare industry
• How is software being developed?
• Light at the end of the tunnel

Software makes News

• Business Week: ``Software -- The New Driving force" (1984)
• Fortune: ``A Growing Gap in Software" (1986)
• Newsweek: ``Can we trust our software?" (1990)

A US $125M Hiccup

System: CONFIRM

• supposed to be a leading-edge comprehensive travel industry reservation program.
• combining airline, car rental & hotel information.
• developer: AMR Information Services, a subsidiary of American Airlines Corporation.
• partners and users
  • Hilton Hotels Corporation
  • Marriot Corporation
  • Budget Rent-A-Car Corporation

Background of CONFIRM

• 1987 AMR discovered the potentially huge market of ``centralized hotel reservation".
• Only 20% of hotel reservation were made through a centralized service.
• More than 80through a centralized system.
• AMR has developed the highly successful airline reservation system SABRE
• 1986, AMR formed AMRIS to expand SABRE into other business.

Partnership Formed

• Oct 1987, Marriott, Hilton, Budget join the consortium.
• A new company Intricon was formed.
• May 1989 AMR announced commencement of the design of CONFIRM.
• Sept 1988 Marriot, Hilton, and Budget signed the partnership agreement.
• AMRIS was designated the managing partner and development prime of CONFIRM.
• Partner undertook to pat US $55M for development.

Rosy Promises

• Design will be completed in 7 months.
• Development will be done in 45 months leading to deadline in June 1992.
• Expenditure will not exceed US $55.7M.
• Operation cost per reservation will be below US $1.05.

Ambitious Goals

• March 1987 AMRIS representatives made a presentation to Marriot to build CONFIRM.
• They claimed CONFIRM will be a state-of-the-art reservation system meeting all business needs of hotel and car-rental partners in the joint venture.
• AMRIS would be in charge of the design and development.
• The partners besides undertook to use the system will also join AMR to market the system to other business.
• AMRIS would operate the data processing center.

Development Plan

• AMRIS announced design completed in Spet 1989 (took less than 6 months).
• Project price revised to $72.6M (from $55.7M).
• Per reservation operation cost revised to $1.3 (from $1.05).
• Sept 1989, partners accepted the development plan and agreed to move the deadline to July 1992 (from June 1992).
• Oct 1989 AMRIS assured the partners the project was on time and schedule.

Shaky Development

• Two months l;ater, Jan 1990, AMRIS missed a milestone, failed to complete some screen design.
• Also missed another milestobe of business model.
• Feb 1990 AMRIS admitted it was behind schedule for 13 weeks and claimed it will be caught up.
• May 1990 AMRIS again claimed it was on time and schedule.
• Employee at project office were instructed to revise project data to follow the original calendar even though milestones were missed.

Chaotic Begin

• Aug 1990 AMRIS declared first phase completed.
• But refused to deliver ``deliverables" to Marriott.
• Oct 1990 AMRIS admitted it was one year behind schedule.
• Feb 1990 AMRIS presented a ``re-plan", only Hilton would be able to use the system by JJune 1992 (deadline).
• Marriot would not receive all the functionalities.
• Re-plan new price tag: US $92M )vs. 5.5M).
• AMRIS president resigned.
• 20 employees resigned.

Morale Drop to the Lowest Level

• AMRIS employee believed their managers kept stating unrealistic schedules and lied about projest status.
• Schedule could not be met even with 7x9 hours work week.
• Summer 1991 half people working in CONFIRM were looking for new position.
• A consultant was hired by AMRIS to evaluate the project.
• A V.P. was very upset by the findings from the consultant.
• He burried the report and dismissed the consultant.

Final Chance

• Marriot's evaluation shown AMR could not complete the project.
• AMR continue to claim it will be on time and continue to bill Marriott US $1M per month.
• April 1992 AMRIS admitted it is behind schedule, but only for 2 to 6 months.

Death Sentence

• April 92, AMRIS deliver the system to Hilton for Beta-test; Major failure occurred during the test.
• AMRIS's chief call for emergency actions and predicts 15-18 months.
• AMRIS promised to repay 100% all partner's investment.
• 8 top executives were fired.
• July 1992, after 3-and-1/2 years and US $125M investment, Intrico was disband and CONFIRM was scraped.

Epilogue

• it was discovered later, the database in CONFIRM can not even recover from a system crash.
• AMR faced law suit of breaching contract.
• Sept 92 AMR filed countersuit against the partners.
• Jan 1994 AMR reached out-of-court settlement with all of its partners.
• Tumors said AMR was facing damages suit of more than US $500M.
• Eventually AMR agreed to pay about US $160M.

No Phone?

What Went Wrong?

Life After

• Federal givernment step in.
• Down-time report regularly.
• Media scrutiny.
• Sprint and MCR put out big promotion campaign.
• All vendors rush to check out their STP's.
• Top down complete review of STP program.
• Review of development process.
• Stop all current related development programs.

ISDN glitch in Hamburg

• Sept 1993
• 12,000 customers lost services.
• Recovered after more than 12 hours.
• 25% of 50,000 ISDN lines cannot use phone, fax and data transfer.

What's happened?

• 2:00am a new version was loaded.
• System crashed.
• Cannot reload the old software.
• 1:15pm restart the old system version.
• ISDN's reputation is tainted.

Complexity

• Too complicate
• Too large
• Too many functionalities

The London Ambulance Service

• Largest in the world
• Covers 600 square miles, 6.8M people/residents.
• 5,000 patients, about 2 to 2.5 thousand calls per day.
• 1300-1600 emergencies/day

Current Ambulance Dispatch System

• Receiving calls for assistance.
• Identifying the positions of the nearest ambulance and crews.
• Dispatching the vehicle which can most rapidly attend to an outstanding call.
• Monitoring its status.

To computerize it -- LAS Compuer-Aided Dispatch (CAD) System.

An Ideal Case to Study

• A failure
• A large safety-critical system
• Investigated by an independent teams of experts.

Collapse of CAD

System done by Systems Options Ltd. with an attractive-priced bid.

• Fully live at 7am Oct 26, 92.
• Old paper-based system scrapped.
• Duplicated file server as a backup.
• Keep voice communication minimum.
• Use Automatic Vehicle Locating System to identify the nearest available ambulance.
• Automatic communication between centre.
• Mobile Data Terminals on the vehicles.
• At 10am, number of calls increases
  • AVLS cannot keep track accurately of the position and status of each vehicle.
• Dispatching vehicles
  • not closest the scene.
  • Make multiple assignments to cover a single call
    • Large number of exception messages
    • System slowed down
    • Where was that exeception message? The screen cannot hold all exception messages as there is not scrolling function.
• Ambulance crews
  • Frustrated
  • Under pressure
  • Failure to communicate their status using MDTs
    • AVLS no longer KNEW
      • Which vehicles were ready for dispatch
      • Shortage of resource ==>
        longer to search an available ambulance!!
• One man
  • Placed a call in the afternoon when his mother suffered a stroke.
  • 6 hours later, he drove his mother to a hospital.
  • 2am he had a phone call that asked if he wanted an ambulance.

Life After

Semi-manual operation before Oct 26

• Calls were taken on the CAD system.
• Incident details printed.
• Optimum vehicle was identified through contact with the nearest station to the incident.
• A message sent via CAD to the station printer or to the MDT.
  System seemed okay afterwards.

Nov 24, 2am

• System began to slow down.
• Rebooting of the workstation did not clear the problem.
• backup server could not use (only for fully auto-system).
• Calls could not be printed.

Total manual operation afterwards.

What went Wrong?

Software went life

• SW incomplete
• Not been tuned
• Not been fully tested
• Not been tested under a realistic workload
• Backup file server had not been tested

Staff in CAC room

• Inadequate preparation/training
• Negligence of human factors
• No planning of the change-over

SW Design

• Assume perfect working conditions.

General Imperfections

• Failure to identify all duplicate calls.
• Lack of prioritisation of exception message.
• Exception message and awaiting queues scrolling off the top of the allocators' and exception rectifiers' screens.
• SW resource allocation errors.
• Lack of general robustness of the system.
• Slow response times for certain screen based activities.

Software Development Tech

• Most people still think it is a black art.
• No standard technology.
• Demanding learning curve.
• No standard process.

Industry Perspectives

• Long development time.
• Very high cost.
• High error rate in delivery.
• Hard to monitor development.

High failure rate in delivery

Management's Myths

• Standards and procedures are all that is required to build software.
• Tools and computers will solve most of the problems.
• Add more programmers will speed up the development. (Fred Brooks, The Mythical Man-Month).

Customer's Myths

• A general statement of objective is sufficient to start the development.
• Software is flexible so that requirement change can always be accommodated.
• Time needed to develop a asystem can always be shortened.

Practitioner's Myths

• System can always be designed along with implementation (coding).
• The job is to write a program that works.
• Quality can only be assessed after the program has been completed.

Software Poses Challenges

• How do we ensure the quality of the SW that we produce?
• How do we meet growing demand and still maintain budget control?
• How do we upgrade an aging ``software plant"?
• How do we avoid disastrous time delays?
• How do we successfully institute new software technology?

What is Software Engineering?

A discipline for design, development and maintenance of software products.

Software Products

• Program code of substantial size/complexity.
• Has multiple users.
• Has multiple developers and maintainers.
• Users, developers and maintainers are distinct persons (groups).
• Is a technological artifact which has much in common with the objects that are designed and developed by traditional engineers.

Aspects of Software Products

• Intangible, only its effect can be observed.
• Does not degrade with time as hardware.
• No physical property
  • No fundamental guidelines to constraint its development.

Software Products Size Categories

Sample systems

• Personal software
• Small data processing applications
• Compiler, POS systems
• Database management system, airline reservation system
• Telephone switching system

Hardware/Software Differences

Design	Once, harder to change	All, easy to change
Component failures	down	many still be running
Modification	hard	easy
Maintenance	men and material	information
System failures	design, mechanical, ...	design

Classic Problems of Software Designs

• Projects get completed late.
• During development, no one is able to give a proper estimate of expected completion date.
• Projects end up costing substantially more than was initially estimated.
• The completed product falls short of expectations (capabilities, performance, user friendliness, robustness, reliability).
• Maintenance is a nigghtmare because of poor design concepts, poor documentation, poor code design, ....

Quality Software

High qulaity if

• Does not what the users want it to do
• Uses computer resources correctly and efficiently
• Easy for the user to learn and use
• Developers can design, code, test, and maintain the system with relative ease

Distribution of Computing Costs

Year	Size $M	HW%	SW%
1980	4100	35	65
1985	13920	20	80
1990	37990	15	85

• Development, 40%
  • Requirement development, 14%
  • Code and debug, 6%
  • Test, 20%
• Maintenance, 60%
  • Enhance software functionality
  • Adapt the product to new processing requirements.
  • Fixing bugs.

A Real Example

Complexity of the DMS switch

Lines of code	24 million
Modules	18,819
Subsystems	5378
Areas	134
Archids	118
Memory	50 Mbytes

Code Architecture

Data Size

DMS Data

Lines of code	2.3 million
Modules	1905
Subsystems	1780
Areas	29605
Archids	100 M(2x code)

Data Architecture

The GAMMA Paradigm

• Goal: set measurable targets for what to be achieved.
• Attribute: entities to be measured.
• Measure: meaningful measures to represent each of the attributes.
• Means of Measurement: the method used to obtain the measurements.
• Action: perfom actions depending on values measured.

Other paradigms

• GQM (Goal, Question, Metric), Basili and Rombach 1988.
• DBO (Design by Objectives), Tom Gilb 1987.

Chapter 2 - Process Modelling

References

• Pressman, chapter 1.5
• Pfleeger, chapter 1.4
• Sommerville, chapter 1.2, 1.3

A Product Life-Cycle

1. Concept and feasibility study
2. Requirement definition
3. High level design
4. Low level design
5. Construuct and test a prototype
6. Manufacture
7. Maintenance
8. Obsolescence and decommissioning

The Ideal Software Life-Cycle

1. Concept and feasibility
2. Requirements capture
3. Requirements specification
4. High level design
5. Low level design
6. Implementation
7. Testing
8. Trial
9. Shipment
10. Operation and maintenance
11. Obsolescence

Software development process as any description of software development that contains the stages above.

The Real Software Life-Cycle

1. Concept and feasibility
2. Coding
3. More coding
4. Test and trial
5. User documentation
6. Design
7. Functional specification
8. Requirements defintion
9. Shipment
10. Operation and maintenance
11. Obsolescence

Verification and Validation

• Verification
  • Are we doing it right?
• Validation
  • Are we doing the right thing?

They are used in

• Requirements capture
• Requirements specification
• Input products, intermediate products, end products

Software Life-Cycle Models

• The Waterfall
• The ``V"
• rapid prototyping
• Incremental delivery
• The ``Spiral"

Waterfall Model

Key aspects of Waterfall

• Each phase is completed by a validation and verification activity; to eliminate as many problems as possible.
• Each phase has well defined input and output (work products) and procedures (or templates).

Key Activities

Phase 1: System Requirement Specification

• System user and system analyst are sufficiently informed about the intention of the system.
• System requirement is identified to support future design.

Phase 2: Software Requirement Specification

• Define nature of the software to be developed.
• Could be part of the system requirement if the system is not too large.
• System and software requirement will be reviewed to ensure their completeness and consistency.

Phase 3: High Level Design

• System level architecture
• System decomposition
• Important data structures
• Data definition/Database design

Phase 4: Detail Design

• Program modules definition
• Intermodules interfaces
• Algorithms
• Pseudo codes
• Tracability between high level and detail designs

Phase 5: Coding

• Detail design is translated into source code

Phase 6: Testing

• Verified against design
• Verified against requirement

Phase 7: Maintenance

Focus of Waterfall

• A phase should not be started unless the previous phase has been stablized.
• A flexible life-cycle model should also support considerable iteration and feedback.
• Waterfall model depends heavily on the effectiveness of requirement identification.

Modified Waterfall

An error may force one to backtrack to an earlier stage of development (Royce, 1970).

The V Model

• Same phases as in the Waterfall
• Bent in the middle
  • Each development phase is matched by a corresponding verification or validation phase.
• Top-down design
  • Bottom-up implementation
• Time flows from left to right horizontally.
• Disadvantage
  • The whole product is designed before any of it is implemented.

The Prototype Model

The Evolutionary-Prototype Model

• Proposed by McCracken and Jackeson
• Use prototyping as the means to communicate with the user to obtain requirement specification
• Apply to case in which user is unsure of the system requirement

Incremental Delivery

• Early versions do not discard as in ``prototyping" but are actual products.
• Can be viewed as a series of ``waterfall" or ``V" models running in parallel, but with a time lag between the start of each.
• E.g. OS developments, ICLVMS, IBM OS36.
• Scheduling new releases is important
  • Do not introduce too many new functions
  • Fizing old bugs via patches

Spiral Model

• First proposed by Boehm 1986.
• Introduce a risk-driven approach into development cycle; Waterfall is a specification-driven approach.
• Spiral model requires prototyping and calls for risk asssessment during each cycle.
• It incorporates waterfall model constructs with an iteration style supported by risk analysis.

What are Requirements?

Software requirement specification (SRS) is a document containing a complete description of WHAT the software will do without describing HOW it will do it.

Requirements Specification

Why are requirement specification important?

• The later in the development life cucle that a software error is detected, the more expensive it will to repair.

Many erros detected after coding and testing are attributed to requirements.

• Boehm 1975
• 54% of all errors were detected after coding and testing
• Among this 54%, 45% were attributed to requirement

Errors made in requirement specifications are typically incorrect facts, omissions, inconsistencies, and ambiguities.

Incorrect facts	49%
Omission	31%
Inconsistency	13%
Ambiguity	5%
Misplaced requirment	2%

Requirement errors can be detected.
Myth: it is useless to analyse non-executable forms of software

Conclusion

• Many requirement errors are being made.
• Many of these errors are not being detected early.
• Many of these errors can be detected early.
• Not detecting these errors may contribute to sky-rocketing software costs.

Chapter 3 - Software Measurement

References

• Pressman, chapter 2.2-2.5
• Fenton, chapter 1-3

Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe them according to clearly defined rules.

Purposes of Measurements

1. Assessment and certification of the product
2. Maintenance cost estimation
3. Improvement of the development process
4. Control of the project
5. Estimation of future projects

Principles of Measurements

Entities and attributes

• Measurements describe entities by assigning numbers to their attributes
• Mapping of entities into some number system
• Representation condition
  • Numerical relationship
• In order to makde meaningful measurements
  • Select entities
  • Decide which attributes to measure
  • Define each attribute clearly and quantitatively
  • Choose appropriate scales

Scales of Measurement

1. Nominal: arbitrarily labelled categories, e.g. logic error, uninitialized variables
2. Ordinal: an ordering is introduced, e.g. critical, major, minor
3. Interval: defined on real numbers, but gradation and origin are arbitrary, e.g. days.
4. Ratio: fixed origin, e.g. CPU seconds.
5. Absolute: counting, e.g. number of errors.

Hard Data

Hard data are facts. You can measure such data with little subjectivity

• Number of people assigned to a project
• Effort (person-months) expended on project tasks.
• Duration (calendar units) for each project task.
• Volume of documentation, source code, test cases.
• Number of defects found and reported.

The resolution of hard data varies widely. Measuring hard data by software engineering task is not difficult, but rarely done. Hard data can also be referred as raw data.

• Immediate observations
• Incident report, entry in running time log, source lines in a given module.

• Direct measurement
  • Refined data extracted from raw data
  • No. of failures, running time, KLOC written, hours worked.
• Indirect measurement
  • Derived from other measures
  • Failure rate (number of failures/running time)
  • Productivity (KLOC/hours worked)

Soft Data

Soft data are subjective: different people might have different opinios when such data are evaluated.

• Business constraints on the project
• Skill of the project team
• Experience of the project manager
• Stability of requirements as a function of time
• Difficulty of the problem to be solved
• Adequacy of the software engineering environment (methods and tools)
• User satisfaction with end-product
• Value of the software to the business

Normalized data are used to evaluate the process and the product (but never individual people).

• Size-oriented normalization: the line of code approach (KLOC)
• Function-oriented normalization: the function point approach (FP)

Size-Oriented Metrics

This is a direct measures of software and the process by which it is developed.

Productivity = KLOC/person-month

Duality = defects/KLOC

Cost = $/LOC

Documentation = pages of doc/KLOC

Problems of using size-oriented metrics?

Function-Oriented Metrics

• This is an indirect measures
• It focuses on program ``functionality"
• A popular empirical approach is to measure the ``function point" (FP)
• Advantages
  • It is independent of programming language
  • It uses readily countable characteristics of the ``information domain" of the problem
  • It does not ``penalize" inventive implementations that require few LOC than others.

Chapter 4 - Project Planning and Estimation

References

• Pressman, chapter 2.1, 4
• Pfleeger, chapter 2
• Sommerville, chapter 26

Purposes of Planning

• Cost
  • Finite available resources
  • All aspects of costs must be taken into account, e.g. plant, raw material, effort, overhead
  • Job done within budget
• Schedule
  • Limited time available
  • Meet contractual delivery dates or grasp market opportunities
  • ``Loss of face" or severe financial penalties
• Quality
  • Nothing is ever perfect
  • Different aspects of qualities, e.g. reliability, functionality
  • Product fits for purpose

A Project Plan Should Have

• Relationship
  • The relationship between the development of the software and the development of the total system within which it will be executed.
• Organization
  • The organization of the development team, the roles that must be filled, and the amount of effort required for each.
• Life-cycle model
  • The life-cycle model to be used and the expenditure of resources within each phase, and an estimate of the total cost of the project.
• Milestones
  • Times at which defined intermediate products should have been delivered, and an estimate of the total time to completion.
• Quality goals
  • They are assessed at each milestone.
• Identified risks
  • Things which could go wrong to prevent successful completion, and a defined course of action to be taken if the risk materialises.

Planning Activities and Tasks

1. Set goals and objectives
   • What to accomplish?
   • When must be accomplished?
   • What resources are needed?
   • Determine management constraints on resource/schedule.
   • Success criteria, quality?, profit? future contract?
2. Develop project strategy
   • Project team can have its own strategic plan.
   • It should be correlated to the corporate's strategy plan.
   • Example: develop new expertize, new market penetration.
3. Develop policies for the project
   • Policies are predetermined guidelines for making management decisions.
   • E.g. loading build policy, customer support policy.
   • Relation between policy and procedures and guidelines.
4. Determine possible courses of action
   • There is always more than one way yo conduct a project
   • Each course of action must be examied to determine advantages, disadvantages, risks, and benefits.
5. Make planning decisions
   • Project planning team is responsible for choosing the most appropriate course of action.
   • Make trade off among cost, schedule, design and risks.
6. Set procedures and rules for the project
   • Procedures specify customary methods and provide guidelines for project activities
   • Rules establishes specific and definite actions to be taken or not taken with respect to a given situation.
   • E.g. design procedures, planning procedures, budget allocation rules.
7. Develop a software project plan. It should specify
   • Tasks: to be performed by different staff, break project activities into small, well-specified work units.
   • Cost and resources to accomplish the tasks.
   • Schedule: duration of each task and its dependency.
8. Prepare a budget
   • Budgeting is the process of allocating estaimted costs to project tasks.
9. Conduct a risk assessment and prepare contingency plans
   • Risk analysis
     • Identification of risk factors
     • Impact analysis
   • Contingency plans
     • Specify the actions to be taken should when a risk becomes a real problem.
     • When does a risk becomes a real problem? When a risk-indicator metric crosses a predetermined threshold. E.g. schedule overuns by more than 10%.
     • Planning team should define these metrics during planning stage.
10. Docuement the software project plan

Organizing a Software Engineering Project

Major issues

• Difficult to determine the best organizational structure.
• Organization structure may leave responsibilities for some project activities and tasks undefined or unclear.
• The matrix structure presents special challenges to both the organization and the individuals assigned to the organization.

Organizing activities

1. Identify and group required tasks
   • Map product requirement into work tasks
   • Size tasks and group them into units
   • Assign task names and map them into organizational units
   • Supporting tasks must be identified as well
2. Select an organizational structure
   • Conventional organization structures
     • A line organization has the responsibility and authority to perform the work that represents the primary mission.
     • A staff organization is a group of functional experts that perform special activities that help the line organization to do its work.
   • Project structures
     • Functional project organization - coordination has no responsibility in hiring or training, etc.
     • Project organization
       • Project manager has the sole responsiblility of the project
       • Responsible for hiring, discharge, training and evaluation
     • Matrix project organization
       • Project manager has the responsiblility and authority of the project
       • Functional manages provide the resources needed
       • However, project manager usually does not having hiring/discharging/evluation responsibility

Clean Room

Harlan Mills - developing a large system for the New York Times

• Programmers, only write code
• Testers, find faults in the code
• The programmers had to get their code past the Test Team, and the Test Team to stop them
• Code get out the door will be thoroughly tested
  • Successful because of Mill's personal enthusiasm
  • Independent verification and validation
    • Two different contractors: development, testing
    • Overhead cost can be 60%!

Chief Programmer Teams

Brooks 1975, Hog-butchering vs. surgery

• A cheif programmer
  • Junior pprogrammers, assistant, secretary, clerk, technical writer.
• The chief programmer is ultimatelt responsible for all design decisions.

Possible Risks

• Loss of key personnel
  • Age, health
  • Maternity/paternity
  • Moving to higher paid job
  • Job satisfaction
• Requirements may change in mid-project
  • High chance
  • Adequate attention during requirements capture
  • A contractual basis for re-negotiation
• Non-availability of essential hardware or software
  • Identidy dependencies
  • It is recognized in the contract that the project will not be penalized under such situations.

Project Roles

• Project manager
• Software manager
  • Designer
  • Programmer
  • Tester
  • Secretary
  • Technical clerk

Chapter 5 - Cost and Schedule Estimation

References

• Pressman, chapter 3.5 - 3.7
• Pfleeger, chapter 2.3
• Sommerville, chapter 29
• Fenton, chapter 9

Scheduling

The purpose is to determine that the deadline is feasible.

• Rule of Thumb
  • schedule for a complete development is
    proportional to
    
• Two common fetal mistakes
  • Gutless estimation: people who makes the estimates does not inform the managment how long they really believe the project will take. It can be due to fear, insufficient knowledge, etc.
  • 90% syndrome: informal estimates as ``it is 90% complete!"
    • Myth of programmers: management will not cancel a project in which they have already made a large investment.

Project Scheduling

Scheduling

• Anaylze the project and specify the paradigm(s) to be used.
• Break the project into a reasonable number of tasks (applied to a reasonable set of functions).
• Estimate the effort and time required to perform each task.
• Create a ``task network" that depicts the time dependencies of tasks.
• Develop schedule to conform to defined completion date.
• Use automated tools to assist in this process.

Creating a Task Network

Subtasks

Design is a major software engineering task. To adequately schedule a design, the following subtasks might be specified

• Review of software specification
• Refinement of DFDs for each major software function
• Architectural mapping for each software function
• Review of software architecture
• Data design
  • Refine data dictionary
  • Identify file structure and contents
  • Normalize files, if required.
• Review of data design
• Interface design
• Assign modules or clusters of modules for detail design
• Procedural design of module n
  • Develop module prologue
  • Algorithm design for module
• Review procedural design for module n
• Develop integration strategy for software architecture

Project Scheduling Technique

PERT charts

• PERT - program evluation and review techniques (Lockheed Corporation)
• A network of nodes
• Node: project activities and their associated durations
• Link: procedence relations between pairs of activities

Construction of PERT chart

• Starts at the finish node
• Go backward until the start node

Critical Path Analysis

Given a PERT chart

1. What's the minimum time it will take to complete the project?
2. What activitiies are critical so that the project can be completed in the minimum time?

Critical path: the path between the start and finish node which is the lingest path through the network in terms of total duration of activities.

Critical Path Procedure

1. Label the start node (0,0).
2. For all unlabeled nodes N whose predecessors are all labeled nodes, compute the earliest possible start time as the latest finishing time of all its predecessor nodes. I.e.
   S_N = max (F_i) where i belongs to P(N)
   which is the set of predecessor nodes of N.
3. Compute the corresponding finish time
   F_N = S_N + D_N
   where D_N is the duration of activity N. Label the node N as (S_N,F_N).
4. Repeat step 2 and 3 until no unlabeled nodes remain.

If N₁N₂ ... N_k is the cirtical path, then F_i = S_i+1 for i=1,...,k-1. The earliest finishing time of node i is equal to the earliest start time of node i+1.

Late-Start and Slack Time

After the critical path has been determined, then

1. What's the latest time that each activity can start without delaying the overall project finsih time?
2. How much slack is there in the completion time for each activity?

Latest-Start Procedure

1. Underlabel the Finish node F with its start and finish times as determined from the critical-path procedure.
   (S_F', F_F') = (S_F, F_F)
2. For all non-underlabeled nodes N whose successors are all underlabeled nodes, compute the latest possible finish time as the earliest starting time of all its successor nodes.
   F_N' = min [S_i'] for i belongs to S(N) which
   is the set of predecessor nodes of N.
3. Compute the corresponding latest-start time
   S_N' = F_N' - D_N where D_N is the duration of activity N. Underlabel the node N as (S_N', F_N').
4. Compute the slack time for the activity as
   L_N = S_N' - S_N.
5. Repeat steps 2-4 until no non-underlabeled nodes remain.

Planning and Control Implications

• Use the slack time associated with an activity to delay its start time oor to lengthen its duration.
• If the duration of any critical path activity is lengthened by X time units, then the project's completion tie is lengthened by X time units.
• If only a single critical path exists, we can shorten the project's overall duration by shortening the duration of a critical path activity.

Aspects of Cost

• Vendor
  • Cost of development
  • Marketing, maintenance
• Ownership/user
  • Purchase, procurement
  • Disposal
  • Training
  • Sofwtare reliability

Cost Estimation

• Effort
  • Cost of employing highly skilled people
    • No. of hours
    • Different pay rates
  • Cost relating to LOC, not easy as system is not yet built.
• Plant
  • Computer equipment, software utilities, CASE tools.
• Raw materials
  • Paper, office stationery, diskettes, tapes.
• Overheads
  • Heating, lighting, rent of office space, poweer for computer equipment,...
• External costs
  • Subcontracts, legal expenses, ...

Classification - Size-based Estimation

• Parametric Cost Models (C)
  • Stengths: objective, repeatable, permit sensitivity analysis, objectively calibrated to experience
  • Weakness: inputs are often subjective, exceptional circumstances not consider, calibrated to past not future
• Expert Judgement (A)
  • Stengths: can access representativeness of past experience, can estimate the effect of new environmental factors and production techniques, can cope with exceptional circumstances
  • Weakness: no better than the quality of the experts, subject to biases.
• Analogy (B)
  • Stengths: based on representative experience
  • Weakness: past experience may happen not to be representative

Expert Judgement

• Another name for inspired guess work!
• It needs
  • An experience estimator
  • A database of past projects
  • Cf. the ``chicken sexing".
• Delphic technique
  • More than 1 expert is involved
  • Each individual makes his/her own estimates
  • Discuss to have concensus or combine using an algebraic formula
• E.g. PERT
  • Three estimates: smallest s, most likely m, largest l.
  • Expect value = E = (s+4m + l)/6
  • Variance = V = (l-s)/6.

Computer-Aided Design System

• User interface and control facilities (UICF)
• Two-dimensional geometric analysis (2DGA)
• Three-dimensional geometric analysis (3DGA)
• Database management (DBM)
• Computer graphics display facilities (CGDF)
• Peripheral control (PC)
• Design analysis modules (DAM)

Analogy (B)

• Corporate memory (boehm 1981)
  • for each project we keep:
    size, cost, development time, ...
• To have a new estimate
  • Match the similarities with the old projects
  • Put in the essential paramters

Paramteric Cost Models (C)

• Size + Schedule ==> Total effort
  • LOC + elapsed development time ==> person-months.
• Salries + overhead ==> cost
• Many models
  • COCOMO
  • Putnam

COCOMO - COnstructive COst MOdel

• BASIC
  • A static single-valued model that computes software development effort (cost) as a function of program size expressed in estimated LOC.
  • It is quick, easy, first approximation.
• INTERMEDIATE
  • Computes software devellopment effort as afunction of program size and a set of ``cost drivers" that include subjective assessments of products, hardware, personnel, and project attributes.
  • Effort multipliers, result of 15 cost drivers.
• ADVANCED
  • Incorporates all characteristics of the intermediate version with an assessment of the cost drivers' impact on each step of the software engineering process
  • Includes still more cost drivers

Basic COCOMO Model

• COCOMO is an algorithmic software cost estimation model
• Man-month estimation
  • MM = 2.4 (KDSI)^(1.05) where MM is man-months and KDSI is thousands of delivered source instructions.
• Development schedule estimation
  • TDEV = 2.5 (MM)^0.38 where TDEV is in months.
• Software using this equation belongs to organic mode.
• Delivered: exclude nondelivered support software, e.g. test drivers
• Source instructions
  • Program instructions
  • Excludes comments
  • Utility, tools, etc.
• Development period: design start to integration and test end
• Cost covers all development related activities, e.g. code, document; but not include support activities, e.g. training, installation, ...
• Cost covers all direct charges, include project managers, librarian; but not human resource, secretaries, etc.
• COCOMO man-month consists of 152 hours of working time
  • 1 MM is 152 M hours
  • 1 MM is 19 Mdays
  • 1 MM is 1/12 Myear
• COCOMO assumes good management and hence very small slack time
• COCOMO assumes requirements specification is not substantially changed after the plans and requirements phase.
• To convert COCOMO man-month to $ estimates, Beohm suuggested to it by phases. I.e. $6000/MM for planning and requirment, $5000/MM for detailed design etc.

Example

Du Bridge Chemical is planning to develop a new system to keep track of raw materials. It will be developed by an in-house team with several years of similar experience. This is a good example of an organic-mode software project.

• Initial estimation = 32 KDSI
• Effort = MM = 2.4 (32)^1.05 = 91 man-months
• Productivity = 32,000/91 = 352 DSI/MM.
• Schedule TDEV = 2.5 (91)^0.38 = 14 months
• Average staffing = 91 man-months/14 months = 6.5 FSP full-time equavalent software personnel.

Overall Project Profile

Standard size and effort for projects in organic mode.

Product Size	Effort	Productivity	Schedule(months)	Avg. staffing
Small 2KDSI	5.0MM	400 DSI/MM	4.6	1.1 FSP
Intermediate 8KDSI	21.3MM	376 DSI/MM	8.0	2.7 FSP
Medium 32KDSI	91.0MM	352 DSI/MM	14.0	6.5 FSP
Large 128KDSI	392.0MM	327 DSI/MM	24.0	16.0 FSP

Phase Distribution

Effort

Phase	Small	Intermediate	Medium	Large
Plans and requriements	6%	6%	6%	6%
Product design	16%	16%	16%	16%
Programming - detailed design	26%	25	24%	23%
Programming - code and unit test	42%	40%	38%	36%
Integration and test	16%	19%	22%	25%

Schedule

Phase	Small	Intermediate	Medium	Large
Plans and requriements	10%	11%	12%	13%
Product design	19%	19%	19%	19%
Programming	63%	59%	55%	51%
Integration and test	18%	22%	26%	30%

• The percentage distribution of schedule for all phases does not sum up to 100%. Why?
• Difference in the distribution between small and large projects?

• Code size estimation = 32 KDSI
  • Effort = 2.4 (32)^1.05= 91 MM
  • Schedule = 2.5 (91)^0.38= 14 months
• For the programming phase
  • Effort = (62%) (91 MM) = 56 MM
  • Schedule = (55%) (14 months) = 7.7 months
  • Avg. staffing = 56 MM / 7.7 months = 7.3 FSP

Nominal Project Profiles

• How different fields are computed?
• Note: productivity on large project is decreasing -- diseconomy of scale.
• Reason contribute to the diseconomy?
• Which phases increase more rapidly?

Interpolation

How to find out the percentage of effort or schedule for different phase given the size of a software product.

• Size of a product = 12.8 KDSI
  • Effort = 2.4 (12.8)^1.05= 35 MM
  • Schedule = 2.5 (35)^0.38= 9.7 months
• From the tables, for programming,
  • Effort 32 K ==> 62%
  • Effort 8 K ==> 65%
• By linear interpolation
  • (Effort - 65)/ (12.8 - 8) = (62-65)/(32-8)
  • Effort (128K) = 64%
  • Effort = (0.644)(35 MM) = 22.5 MM
• Similarly, schedule for programming is 58.2%
  • Schedule = (0.582)(9.7) = 5.6 months
  • Avg staffing = 22.5/5.6 = 4.0 FSP

Maintenance Effort Estimation

Use basic COCOMO estimate for annual sofwtare maintenance

Annual charge traffic (ACT): the fraction of the software product's source instructions which undergo change during a (typical) year, either through addition or modification.

Suppose the 32 KDSI software system for Du bridge had 4000 DSI added and 2400 DSI modified during its first year.

• ACT = (4000+2400)/32,000 = 0.2 0
• Basic annual maintenance effort (MM)_AM
  • (MM)_AM = 1.0 (ACT) (MM)_D where D is for development.
• FSP level for annual maintenance (FSP)_M
  • (FSP)_M = (MM)_AM/12
• Since MM_D = 91 MM
  • MM_AM = (0.2) 91 = 18 MM
  • (FSP)_M = 18/12 = 1.5 FSP

Three COCOMO Modes of Software Development

• Organic mode
  • Small software team in a stable environemnt
  • Extensice experience in similar projects
  • Most team memebers can contribute to the project in an early stage (low overhead)
  • Relatively relax about requirement and specification; extensive rework (design problem) can generally be resolved by modification of specifications.
  • Minimal need for innovative architecture
  • Low premium on early completion
  • Relatively small size
• Embedded mode
  • Product is tightly coupled with hardware, software, regulation, operational requirement, etc.
  • Expected to conform to original specification
  • Very few options of negotiating charge in specification and requirement
  • A lot of uncertainty and risk
  • Generally use a small team to do initial specication and design
  • Bring in a lot more manpower to do programming and testing at a later stage
  • Involve some new/advance technology
• Semidetached mode
  • An intermediate level of project characteristics
  • A mixture of the organic and embedded mode characteristics.

Basic Effort and Schedule Equations for Different Modes

• Organic
  • MM = 2.4 (KDSI)^1.05
  • TDEV = 2.5 (MM)^0.38
• Semidetached
  • MM = 3.0 (KDSI)^1.12
  • TDEV = 2.5 (MM)^0.35
• Embedded
  • MM = 3.6 (KDSI)^1.20
  • TDEV = 2.5 (MM)^0.32

COCOMO Database

Boehm introduced the COCOMO equations by analyzing a sample of 63 software project data points. He claimed that the database have some representative points from all major sectors.

• Effort equations
  • Data points are plotted on a log scale
  • Boehm claimed basic COCOMO estimates are within a factor of 1.3 of the actuals only 29% of the time, and within a factor of 2 only 60% of the time.
  • The estimating equations are not very accurate. However, it is very useful for rough early estimation.
• Schedule equations
  • Schedule estaimaing equations are more accurate than the effort equations.
  • The estimates are within 20% of the actual 58% of the time.

Phase Distribution of EFFORT and SCHEDULE

• Embedded mode can consume more effort in integration and test phase
• Embedded moe consumes proportionally less effort in the code and unit test phase
• Embedded project consumes considerably more schedule in both the plans and requirement phase and the product design phase.

Size - 32 KDSI

Phase Distribution for all Modes

Basic Project Profile: Medium Size Project (32K)

• Note the large variation of FSP for the embedded mode and a much smaller variationn for the organic mode
• Note the difference in schedule time for plan requirement and design for different modes
• COCOMO is an easy and practical tool to develop project labour plan

Intermediate COCOMO

• Basic COCOMO uses only a single predictor variable (KDSI)
• Only correct wihtin a factor of 1.3 of the actuals only 29% of the time, and within a factor of 2 only 60% of the time.
• Intermediate COCOMO incorporates 15 predictor variables. Its estimates are within 20% of actuals 68% of the time.
• Boehm has identified 15 factors out of a large number of candidate factors
• These 15 factors are grouped into 4 categories
  • Software product attributes
    • RELY: required software reliability
    • DATA: database size
    • CPLX: product complexity
  • Computer attributes
    • TIME: executionn time constraints
    • STOR: main storage constraint
    • VIRT: virtual machine volatility
    • TURN: computer turnaround time
  • Personal attributes
    • ACAP: analyst capability
    • AEXP: applications experience
    • PCAP: programmer capability
    • VEXP: virtual machine experience
    • LEXP: programming language experience
  • Project attribute
    • MODP: modern programming practices
    • TOOL: use of software tools
    • SCED: required development schedule

Nominal Scaling Equations

Intermediate COCOMO NOminal Effort Estimating Equations

• Organic
  • Nominal effort equation: (MM)_NOM = 3.2 (KDSI)^1.05
• Semidetached
  • Nominal effort equation: (MM)_NOM = 3.0 (KDSI)^1.12
• Embedded
  • Nominal effort equation: (MM)_NOM = 2.8 (KDSI)^1.20

• The scale equation for nominal effort is of the same form in Basic COCOMO
• Scale factors (exponents) are the same as in Basic COCOMO
• Coefficients are different from that of Basic COCOMO
• The nominal effort calculated from the equations will be adjusted by applying effort multipliers.

Second Example

Norminal effort = 3.0 (32)^1.12 = 146 MM

• A 32-KDSI prototype of a voice input system. Hence, very low reliability
  RELY effort multiplier = 0.75
  • Adjusted effort estimate = (146)(0.75) = 110 MM
• A 32-KDSI climate forecasting model. Hence, low reliability
  RELY effort multiplier = 0.88
  • Adjusted effort estimate = (146)(0.88) = 128 MM
• A 32-KDSI inventory control system. Effect of a fault is often nontriavial, but generally recoverable without exterme penalty. Hence, nominal required reliability
  RELY effort multiplier = 1.0
  • Adjusted effort estimate = (146)(1.0) = 146 MM
• A 32-KDSI electric power distribution control system. The effect of a software fault can be a major finanicial lost. Hence, high required reliability
  RELY effort multiplier = 1.15
  • Adjusted effort estimate = (146)(1.15) = 168 MM
• A 32-KDSI nuclear reactor control system. The effect of a software fault can be loss of human lives. Hence, very high required reliability
  RELY effort multiplier = 1.4
  • Adjusted effort estimate = (146)(1.4) = 204 MM

Phase and Activity Distribution of Effort and Schedul
for Intermediate COCOMO

• Development schedule TDEV is calculated using the same formula in Basic COCOMO
• The percentage distribution effort and schedule by phase is the same as in Basic COCOMO
• The percentage distribution of effort by activity and phase again is the same as in Basic COCOMO

Sensitivity Analysis - Example

To develop a 10 KDSI embedded-mode communication product on a microprocessor. need to determine the effect of various features on the overall development effort and cost.

Communication software generally is very complicate with high complexity. Planning to use high capability analyst and programmers.

• Balancing effect between high complexity and high capability of analysts and programmers
  • Man months: 2.8 (10)^1.20(1.17) = 51 MM
• Assume avg cost of $6000 per month
  • Devlop cost: (51 MM) ($6000/MM) = $306,000

Can the project cost be reduced by using less expensive personnel which would cost $5000 per month only?

• Replace effort multippliers
  • ACAP: 0.86 --> 1.0
  • PCAP: 0.86 --> 1.0
  • Effort adjustment factor: 1.17 --> 1.58
  • Man months: 2.8 (10)^1.20(1.58) = 70 MM
  • Devlop cost: (70 MM) ($5000/MM) = $350,000
• What strategy should be used?

$10,000 could buy 96K memory to replace the 64K. Is it cost effective?

• Main storgae constraint = (45K/96K) = 47%
  • STOR: high --> nominal
  • Multiplier: 1.06 --> 1.0
  • Effort adjustment factor: 1.17 --> 1.10
  • Man months: (44 MM) (1.10) = 48 MM
  • Devlop cost: (48) (6000) = $288,000

The Intermediate Effort Equation

The equation is determined through a 4-step process.

• Review of data and literarure to identify the most significant candidate cost driver attributes.
• A two-round Delphi group consensus activities with ten experienced software managers
• Calibration of the multipliers using a 36-project subset of the COCOMO database
• Final adjustment to correct some discrepancies in the phase distribution of effort, using a 56-project subset of the COCOMO database

Large variation between the Basic COCOMO estimates and the actuals are mostly eliminated by the use of the cost driver factors in the Intermediate COCOMO.

The Intermediate COCOMO estimates are within 20% of the actuals 68% of the time.

Putnam Estimation Model (1978)

It was developed to fit the ``Rayleigh curve" to data extracted from a large project.

It is a dynamic multivariable model that assumes a specific distribution of effort over the life a software development project.

K = L³/ (C_k³ t_d⁴)
where

• K: effort expended (person years)
• L: lines of code
• C_k: state of art constant
  • 2000, poor software development environment
  • 8000, poor software development environment
  • 11000, poor software development environment
• t_d: development times in years

The mathematics is fairly complex and is not needed for this subject.

Three different types of curves

• Requirement and specification
• Design and code
• Total project

Chapter 6 - Comparison and Critique of Cost Models

References

• Pressman, chapter 3
• Pfleeger, chapter 2
• Sommerville, chapter 26-27
• Fenton, chapter 9

Brooks 1982
Cost does not indeed vary as the product of the number of men and the number of months. Progress does not.

Hence the man-month as a unit for measuring the size of a job is a dangerous and decptive myth.

McNicholos'Law 1986
The utility of a model is inversely proportional to the number of its parameters.

Many models were derived from an examination of the empiral statistical relationships between size, effort, schedule, using multivariate model to come up with

estimate = constant * variate^exponent

The two fundamental agreements betwen people so far are

• Effort is proportional to LOC
• Schedule is proportional to (effort)^1/3

The choice of fundamental measures and parameters are often criticised

• Source code: ?object code
• Rayleigh curves: ?Poisson process
• paramters in COCOMO: ?Subjectivity
• C_k in Putnam: ?Good estimates for all

No good way now to solve the problem. The advice is

• Record data
• Use several models
• Trust none of them!

Chapter 7 - Function Points

References

• Pressman, chapter 2, 3.6-3.7
• Pfleeger, chapter 2.3
• Sommerville, chapter 25.3, 27.1
• Fenton, chapter 9.4

A model is most useful for prediction at the beginning of product development, well before the exact number of lines of code can be known.

Albrecht's Method

A framework that relies on the functionality of software as derived from the requirement. I.e. The model is based on an evluation of several measures of the domain for which a software project is defined.

Domain Item	Count	Simple (wt)	Avg	Complex	Fi
Distinct input data items	-	3	4	6	-
Output screens or reports	-	4	5	7	-
Files	-	3	4	6	-
Interfaces to other systems	-	7	10	15	-
Types of online queries	-	5	7	10	-

Function Points

1. Analyze information domain of the application and develop counts
   • Establish count for input domain and system interfaces
2. Weight each count by assessing complexity
   • Assign level of complexity or weight to each count
3. Assess influence of global factors that affect the application
   • Grade significane of external factors, F_i, such as reuse, concurrency, OS, ...
4. Compute function points
   • Function points = Sum (count x weight) x C where
     • complexity multiplier: C = (0.65 + 0.01 x N)
     • degree of influence: N = Sum_i F_i

Forteen Factors

1. Reliable backup and recovery
2. Data communication
3. Distributed processing function
4. Performace critical
5. Run in heavily utilized environment
6. Online data enrty
7. Multiple screens or operations
8. Master file online update
9. Complex input, output, files or inquires
10. Complex internal processing
11. Designing re-usable code
12. Design including conversion and installation
13. Design for multiple installation in different organization
14. Design to facilitate change and ease of use

Weights

Value	Stage
0	no influence
1	incidental
2	moderate
3	average
4	significant
5	essential

Defining an Algorithm

For the purposes of feature point computation, an algorithm

• is a set of logical and compuatational steps that solves a specifi, bounded problem or sub-problem
• has input (or a start value) and produces output
• often corresponds to a cohesive computational or control module in a well designed program

Mark II function Points (Symons 1988)

• A variant of function points based on logical transaction objects
  • Any object, real or abstract, about which the system provides information
• The intention was to relate system size to the effort required to analyze, design, and develop the system
• Unadjust function point (UFP)
  • UFP = N_I W_I + N_E W_E + N_O W_O
  • N_I number of input data element types
  • W_I weight of N_I
  • N_E number of entity-type references
  • W_E weight of N_E
  • N_O number of output data element types
  • W_O weight of N_O
• By calibrating the UFP equation with data generated from actual projects
  • UFP = 0.44N_I + 1.67N_E + 0.38 N_O
• 6 more questions added for a ``technical complexity adjustment"
  • Is there a need to interface with other applications?
  • Is there a need for special security features?
  • Is there a need for direct access for third parties?
  • Are these extraordinary documentation requirements?
  • Is there a requirement for special user training facilities such as training subsystem?
  • Is there a need to define, select, and install special hardware or software unique to this application?
• After calibrating
  • 14 factors from Albrecht's formula plus 6 new factors
    • Coefficients of some factors should be changed from 0.01 to 0.005 ==> FPs are not independent of the technology used in development and implementation

Advantages pf FP Models

• An estimate can be made early in the development process as it is based on the requirements definition provided by the customers
• The estimate is based on input and output, rather than on modular breakdown, so it is applicable to all development environment

Disadvantages pf FP Models

• Much of the evaluation is based on subjective data
• May not be appropriate for radically different types of applications
• The nature of business software changes, the empiral constants must be evluated and possibly revised.

Function Point - Example

Spelling chcker specification: the chcker accepts as input a document file and an optional personal directory file. The checker lists all words not contained in either the dictionary or personal dictionary files. The user can query the number of words processed and the number of spelling errors found at any stage during the processing.

Size-Functionality: UFC (unadjusted function count)

• External inputs: A
  • A = number of external inputs = 2, doc. name and personal dict.
• External outputs: B
  • B = 3, misspelt words, no. of words processed, no. of errors so far
• External inquires: C
  • C = 2, words processed, errors so far
• External files: D
  • D = 2, doc. file, personal dict.
• Internal files: E
  • E = 1, dict.

UFS = Sum Item_i * Weight_i

Item	Simple (wt)	Avg	Complex
A	3	4	6
B	4	5	7
C	3	4	6
D	7	10	15
E	5	7	10

\ \
Technical complexity factor
TCF = 0.65 + 0.01 Sum F_i FP = UFC x TCF

Assume average weights,
UFC = 4A + 5B + 4C + 10D + 7E = 58

Assume dict. file and misspelt word report are considered complex,
UFC = 4A + (5x2 + 7x1) + 4C + 10D + 10 E = 63

Say

• F3, F5, F9, F11, F12, F13 = 0
• F1, F2, F6, F7, F8, F14 = 3
• F2, F10 = 5

FP = 63 * 0.678 = 59

If each FP is about 2 person-days ==> this project needs 118 days.

Chapter 8 - Quality Measurement

References

• Pressman, chapter 17
• Pfleeger, chapter 5.4
• Sommerville, chapter 31.4
• Fenton, chapter 10

Quality == conformance to Requiremnts (Crossby)

Axiom of software development
``Good internal structure" ==> Good external quality

Quality Measurement

• Internal attributes of a product, process, or resource are those which can be measured purely in terms of the product, process, or resource itself (see chapter 8 in Tutorial Guide)
• External attributes of a product, process, or resource are those which can only be measured with respect to how the product, process, or resource relates to its environment

Internal attributes

• Used for qulaity control and assurance
• The building blocks for measuring complexity
• Can be measured early during SW projects
• Are curcial for evlauting the efficiency of software methods
• Will also assure
  • The external attributes expected by SW users, e.g. reliability, usability
  • The external process attributes expected by managers, e.g. productivity, cost-effectiveness

Measurement Metrics

1. Size
   • Crucial inputs to cost and effort prediction systems
   • Specification documents, e.g. length, functionality, complexity of the underlying problem
2. Modularity and information flow
   • High/medium level design
   • Measures various aspects of modularity, e.g. coupling, cohesion, tree impurity, reuse and information flow
3. Control-flow structure
   • Detail design
   • Identify those programs or modules which are likely to be difficult to test and maintain
4. Data structure
   • Detail design
   • Identify those programs or modules which are likely to be difficult to test and maintain

Size

1. There is some consenus view on measuring length of programs but not of specifications or design
2. There is some work on measuring functionality of specifications
3. There is very little work on measuring problem complexity other than some under the subject of computational complexity

Size - Length

A line of code is any line of program text that is not a comment or blank line, regardless of the number of statements or fragments of statements on the line. This sepcifically includes all lines containing program headers, declarations and executable and non-executable statements.

LOC = NCLOC (non-comment) + CLOC (comment)

Density of comment = CLOC/LOC

Size of Specification and Design

• Not easy to have a good measure
• Frequently used in the industry, no. of pages
• Counts of different atomic attributes
  • Text - pages, lines, words, ...
  • Diagrams - uniform syntax; atomic objects for different types of diagrams and symbols

DeMarco 1982

View	Diagram	Atomic object
Functional	Data-flow diagram	Bubbles
	Data dictionary	Data elements
Data	Entity relation diagram	objects, relations
State	State transition diagram	states, transitions

Modularity and Information Flow

A module is a contiguous sequence of program statements, bounded by boundary elements, having an aggregate identifier. (Yourdon 1979)

• Intra-modular attributes
  • Attributes of individual modules
• Inter-modular attributes
  • Attributes of a system viewed as a collection of modules with dependencies

• Module A calls B,C
• Module B calls D
• Module C calls D,E

Module Call Graph: dependencies between data

Initialize

If (x<y)

then A = B

else A = C

D = A

Related Metrics

1. Views of modularity
   • M₁ = modules/procedures
   • M₂ = modules/variables
2. Morphology
   • Size: no. of nodes, number of arcs
   • Depth: length of the longest path from root to a leaf node
   • Width: max. number of nodes at any one level
   • Arc-to-node ratio: connectivity density measure
3. Tree impurity

   

   • The extent to which a graph deviates from being a tree (Fenton chapter 10.4.4)
4. Reuse
   • Public reuse: the proportion of a product which was constructed externally
     PR = |P₂| / (|P₁| + |P₂|) where |P₁| is for new and |P₂| is external
   • Private reuse: the extent to which modules within a product are reused within the same product
     r(G) = e - n + 1 where e is no. of edges and n is number of nodes
5. Coupling
   • A measure of the degree of interdependence between modules
   • No standard numerical characterization of this attribute
     • Suggested to use an ordinal scale of measurement
6. Cohesion
   • An attribute of individual modules, describing their relative functional strength, i.e. the extent to which the individual module components are needed to perform the same task
7. Information Flow
   • Fan-in of a module M is the number of local flows that terminate at M, plus the number of data structures from which information is retrieved by M
   • Fan-out of a module M is the number of local flows that emanate from M, plus the number of data structures that are updated by M

Coupling

For a pair of modules, x and y, we have

Bad	Content, R5(x,y)	x refers to the inside of y
	Common, R4(x,y)	x,y refer to some global data
	Control, R3(x,y)	x passes a parameter to y in order to control its behavior
	Stamp, R2(x,y)	x,y accept the same record type as a paramter
	Data, R1(x,y)	x,y communicate by parameters
Good	Data, R0(x,y)	x,y no communication

A Coupling-Model Graph

• Coupling between modules x and y
  • c(x,y) = i + n/(n+1)
  • i: number of worst coupling type between x, y
  • n: number of interconnections between x,y
• Global coupling of a system S
  • C(S) = median of {C(D_i,D_j): 1<= i < j <= n} for modules {D₁, ..., D_n}.

Cohesion

Good	6, function	performs a single function
	5, sequential	performs sequential functions
	4, communicational	> 1 function, but these are all on the same body of data
	3, procedural	> 1 function, and these are only related to a general procedure
	2, temporal	> 1 function, and these are only by the fact that they must occur within the same time span
	1, logical	> 1 function, and these are only related logically
Bad	0, coincidental	> 1 function, and these are unrelated

Cohesion ratio = no. of modules having functional cohesion/ total no. of modules

Information Flow

Henry-Kafura, 1981
CM = Length(M) x [fan-in(M) x fan-out(M)]²

where length is the size of the module M.

Shepperd 1990

CM = [fan-in(M) x fan-out(M)]²

Word Processor

Module	FI	FO	(FI x FO)2	Length	CM
WC	2	2	16	30	480
FD	2	3	36	11	396
CW	4	4	256	40	10240
DR	2	0	0	23	0
GDN	0	2	0	14	0
RD	3	1	9	28	252
FWS	1	2	4	46	184
PW	2	1	4	29	156

Control-Flow Structure

Direct graphs (flow graphs)

• Nodes correspond to program statements
• Edges correspond to flow of control between the corresponding statements

Flow graphs

Hierarchical Measures

• Depth of nesting measure
• Number of nodes measure
• Number of edges measure
• Largest prime measure
• Number of occurrences of named primes measure
• Is D-structured measure

Recursive measure: number of simple paths through F for each prime F (Fenton chapter 10.3)

Data Structures

• Global measures
  • Number of variables
  • Halstead, u₂
    • u₂ = no. of variables + no. of unique constants + no. of labels
  • Fail to identify complexity if it is hidden in the data structure
• Boehm 1981
  • D/P = Database size in bytes or char. /
    Program size in DSI where DSI is the delivered source instruction
  • Used by the DATA field in the COCOMO model

DATA	Multiplier
Low (D/P < 10)	0.94
Nominal (10 <= D/P < 100)	1.00
High (100 <= D/P < 1000)	1.08
Very high (D/P >= 1000)	1.16

Complexity

A term to captuure the totality of all internal attributes.

• McCabe's Cyclomatic Complexity Measures (1976)
  • v(F) = e - n + 2
  • It measures the number of linearly independent walks through a program (flowgraph)
• Halstead's Volume Metric (1977)
  • Program length = n₁ log₂ n₁ + n₂ log₂ n₂
  • Program volume = N log₂ (n₁ + n₂)
  • where n₁ is the number of distinct operators that appear in a program
    n₂ is the number of distinct operands
    N₁ is the total number of operator occurrences
    N₂ is the total number of operands occureences
• Kitchenham 1984
  • No better correlation with external measures than a simple count of LOC.
• Fenton
  • Largest prime subgraph within the control flowgraph of a program
    • Does not lend itself to measurement on a ratio scale

Chapter 9 - Fagan Inspection

References

• Pressman, chapter 17.2 - 17.3
• Pfleeger, chapter 7.2
• Sommerville, chapter 24.1

Fagan in 1970's

He noticed that no inspection of designs was routinely practised during software development ==>

He developed a set of procedures for inspection of software designs, source code, test plans and test cases.

He argued that software development should

• Clearly define the programming process as a series of operations, each with its own exit criteria
• Measure the completeness of the product at any point of its development by inspections or test
• Use the measurements to control the process

Types of Inspection

1. I₀, Initial design
   • The initial design specification is inspected against the statement of requirements in order to remove defects and ambiguities, and ensure that it provides a sound basis for detailed design
2. I₁, Detail design
   • The logical specification are inspected against the initial design specification to remove defects and ambiguities and provides a sound basis for coding
3. I₂, Coding
   • From source code to testing
4. IT₁, Test plan preparation
   • It is inspected against the functional specification to ensure that it will accurately and comprehensively verify the program functions
5. IT₂, Test case preparation
   • The test case listings are inspected against the test plan to ensure that they match the plan, that proper instructions are provided for carrying out the tests, and that the test harness or stub code is corrected.

Testing

• Testing of software is a means of measuring or assessing the software to determine its quality
• Testing is dynamic assessment of the software
  • Sample input
  • Actual outcome compare with expected outcome

Testing and Debugging

• Testing is different from debugging
• Debugging is removal of defects in the software, a correction process
• Testing is an assessment process
• Testing consumes 40-50% of the development effort

Purpose of Testing

• Testing paradox
• To gain confidence: a successful test is one where the software does what is should
• To reveal error: a successful test is one that finds an error
• In practice: a mixture of defect-revealing and correct-operation

The Testing Team

• Planning
  • Moderator
  • Schedule activities, distribute materials
• Overview
  • Authors and other team members
  • Education
• Preparation
  • All members
  • Familiarization
• Inspection
  • All members
  • Find defects
• Re-work
  • Authors
  • Correct defects
• Follow-up
  • Moderator and authors
  • Assure rework is correct, improve development process, improve inspection efficiency

Necessary Condition for Testing

• A controlled/observed environment, because tests must be exactly reproducible
• Sample input - test uses only small sample input (limitation)
• Predicted result - the results of a test ideally should be predictable
• Actual output must be able to compare with the expected output

Exit Criteria

To judge whether a given development phase is complete. It is usually defined locally by individual organization. Some examples here

• I₀: external specifications are completed
• I₁: design specifications must be structured
• I₂: module prologue up-oto-date and complete

Test Design and Test Execution

When are tests designed

• As late as possible - just before it is needed for test execution - worst case
• As early as possible - right after the requirement specification is ready - ideal case
• Strike a good balance

Level of Testing in Software Development

• Unit test
  • Function of the basic unit of software is tested in isolation
  • performed by programmers who produce the code
  • Tests are derived from the detail logic of the unit
  • Test is to find errors in the individual units, in either data or logic
  • Beware of OO
• Component integrating testing
  • Test functionality of components formed by the combinations of units
  • Two kinds of erros
    • Interface between units
    • Functions supported by multiple units in the components
• System testing
  • After integration testing is completed
  • Entire system is tested as a whole
  • Requirements specification is used to derive test cases
• Acceptance testing
  • Mark the transition from ownership by the developers to ownership by the users
  • Different from development testings
    • Executed by the accepting organization
    • A demonstration not an error revealing process
    • Include testing of the user organization's working practices, ensure the system is compatible with the business practices

Check Lists

Sets of questions which the testers will ask, and which will prompt them to look for certain types of defect

1. I₀, Is the design consistent with the program requirements? External specifications
2. I₁, Are all constants defined? Logic
3. I₂, Is each field initialized properly before its first use? storage, usage

Defect Recording and Classification

• Severity
  • Major
  • Minor
• Category
  • Missing
  • Wrong
  • Extra
• Type
  • The type code as in checklists

Management of Testing

What test metrics are being used?

• Error rates: in design, in test execution, in operations or field use
• Error severity and distribution in different development stages
• Time/cost in test design
• Time/cost in test running
• Time/cost in bug fixing
• Time/cost in inspections or reviews

More Test Metrics

• Defect present/remaining
  • An estimate of the no. of defects in a given module prior to test/still remaining after testing
• Defect density
  • Counts of defects in a module/size of the module
• Defect removal efficiency
  • Percentage of defects present that is removed at a given test step

Management of Testing - continue

What testing policies are used?

• Objectives of testing
• Economic constraints
• Who decide how many, what and severity standard
• Documentation of tests
• Quality and acceptance criteria for test documents
• What tools will be available

Cost of testing

• Test script preparation
• Test data set up
• Debugging tests
• Test execution
• Debugging software
• re-test
• Regression

Inspection rates in NCSS/hour

Inspection Step	T0	I1	I2
Overview	500	500	-
Preparation	200	100	125
Inspection/testing	250	130	150
Rework	-	50	60

Effective Testing

• Right balance between the cost of testing and the cost of not testing
• Concentrate on the most difficult and complex part of the system
• Sofwtare which is difficult to use or least liked is often the most error-prone
• The most likely place to find erros is where you have found the most errors already (Demarco'82)

Management of Testing - continue

How much testing is enough?

• Stop when you have no more time
• Stop when no error has been discovered in the current test cases
• When enough errors have been found
• Coverage measurement -% of the structural elements of the software exercised (Graham)

Personnel assessment - Must not be DONE

Who makes the best tester?

• Make it an important/prestigious job
• Make the tester to represent the true interest of the user
• Independent test team
• A professional but friendly team

Test Documentation

• Test plan
  • Features to be tested and not to be tested
  • Test approach
  • Pass/fail criteria
  • Test deliverables
  • Test environment
  • Schedule
  • Risk
• Test case specification
  • Test case id
  • Input specification
  • Expected outcome including non-functional qualities, e.g. response time
  • Environments needs
  • Procedural requirement: setup, precondition
  • Intercase dependencies
• Test procedure specification
  • Purpose, reference to test case
  • Special requirement, if any
  • Procedure steps
    • Logging
    • Set up
    • Start
    • Actions
    • Measurement taken
    • Shut down/restart/stop
• Test log
  • Items or cases tested
  • Activities/events/time
  • Execution description
  • Success/failure of the test
  • Unexpected events
  • Incident report id
• Incident report
  • Summary
  • Incident description
  • Impact on test plan
  • Severity assessment
• Test summary report

Test Management Pitfalls

• Absence of policy/strategy/plan
• Inadequate and inefficient test data
• Uneven testing
  • Important test omitted
  • Too much testing a simpler area
• Lack of support
  • Tool support
  • Developer support
  • Technical support

Test Quality

How to ensure testing quality?

• Inspection
  • Pre-test document: plan, cases, ...
  • Test results: log, incident, ...
  • Test result analysis
• On-time support
  • Enough tool support
  • Standby design support
  • Use team approach

Static Testing

• Software is examined but not executed to discover programming errors
• Error types
  • Dead code
  • Infinite loops
  • Uninitialized variables
  • Unused data values
  • Standard violations
• Use compiler like techniques to build tool to examine code

Black Box (Functional) Test

• Given all functionality defined in the requirements
• Testing to validate each function is fully operational
• Practitioner found black box approach more useful
• What are the techniques available to design functional tests

Equivalence Partitioning

• Input values which are treated the same way by the software can be regarded as equivalent.
  • e.g. if x>5 then y := y+ 1
    else y: = y - 1
  • Equivalent classes: { x>5 } and { X | X <l= 5 }

Boundary Value Analysis

• Values lie at the edge of an equivalence partition are boundary values
• Boundary values are very sensitive; very often, software may treat it as a value in a wrong equivalent class

Other techniques

• Finite State Machine Testing
  • Seek to exercise sufficient states and transitions
• Random Testing
  • Generate randomly according to user profile
• Error guessing
  • Supported by experiment

White Box (Structural) Test

• Assume full knowledge of designs and implementation
• To validate all components work as specified in the design (documentation)
• Academic tends to emphasize more on white box test

Code Structure Coverage

• Run a set of test cases
• Analysis the % of structures that the test cases have exercised
• Module coverage
• Procedure coverage
• Code statement coverage

It is found (Open Univ.'84) that a good set of functional tests often achieve only 60%-80% code coverage

Branch or Decision Coverage

• IF-THEN-ELSE has 2 branches
• Case statement has multiple branches
• Decision coverage is the % of branches exercised by a test case
• Typical ``good" functional test cases only achieve 40% branch coverage

Testing in Maintenance

• Tests performed for
  • Planned changes
  • Emergency fixes
• Objectives
  • Establish confidence on the changed system
  • Use a cost-effective approach to test maintenance changes
• Breath testing: to get a basic level of confidence on the working of the whole system
• Depth testing: probes the weak area in the system which has a high risk of generating errors due to the change
• Regression testing: a continuous effort to ensure that the software has not degraded from its previous quality

Audit of Testing Activities

What should be looked at if you want to audit the testing activities of the vendor/contractor???

Testing Audit

• Test plan, test case
• Test log,, incident report
• Test progress, result analysis
• Test result database
• Regression test

Critique of Testings

The final number of defects is only known at the end of the product life-cycle, when we assume that any remaining defects can be regarded as causes of failure.

We want to retrospectively estimate the ``defect removal efficiency" of testings at each step of development ==>

To predict the quality of the current product by analogy with past products and psat tests.

Remus and Zilles 1979

• Assume 2 defect removal steps
  • Inspection
  • Testing
• Number of defects after step i =
  d_i+1 = (1 - p_i) d_i + b_ip_id_ior
  d_i+1 = (1 - p_i (1- b_i)) d_i
  • d_i is the number of defects existing in the product on entry to step i
  • p_i is the fraction of defects detected at step i
  • b_i is the fraction of detected defects that are incorrectly repaired in step i
• Assuming
  • Detection fraction p_i is constant at all steps i
  • Bad fix fraction b_i is constants at all steps i
• Defect retention fraction = (1 - p_i(1 - b_i)) = A is a constant

• Let n = total defects inserted at all steps
• k₁ = defects detected in step 1 (inspection)
• k₂ = defects detected in step 2 (testing)
  • k₂ = n - k₁= An
  • d₃ = n - (k₁+ k₂₎= A²n
  • Therefore, A = k₁/k₂ , and n = k₁²/(k₁ - k₂)
  • d₃ = A² n = k₂²/ (k₁ - k₂) = number of defects shipped

A note from the authors: n is not measurable during the development process but it can be measured for products that have expended their useful life already. These values of n can be used to estimate other quantities ..... and to manage the defect removal process.

Problems with Defect Density Estimates

• Every software product is a ``prototype" ==>
  No objective assessment of the predictive value of such techniques seems to be available
• Inspection and testing effort may not be constant ==>
  We are certainly not justified in assuming similar efficiency of defect detection
• The estimation takes no account of product use ==>
  Meaning of ``product life"?

Testing Standard

• ANSI/IEEE Std 829-1983, Software Testing Documentation
• ANSI/IEEE Std 1008-1987, Software Unit Testing
• ANSI/IEEE Std 1012-1988, Software Verification and Validation Plans

Chapter 10 - Dependability Measurement

References

• Pressman, chapter 17
• Pfleeger, chapter 7.6
• Sommerville, chapter 19-20
• Fenton chapter 8, 11

Failure of Complex Systems

Physical: A hardware component develops a fault. The cause if physical and the fault appears at a certain time during operation. Repair consists of replacement of the faulty component to restore the system to its previous good state.

Design: A fault in the design of the system is activated under certain trigger conditions. The fault may have been present for some time, but latent. Repair consists of a corrective change to remove the fault.

Dependability

The extend to which the user can justifiably depend on the service delivered by a system (Laprie 1992)

Dependability is not a single attribute, but is consisted of ``RAMURSES"

1. Reliability
2. Availability item Maintainability
3. Usability
4. Recoverability
5. Safety
6. Extendability
7. Security

Dependability - Indirect Measures

The probability that the system will deliver a required service, under given conditions of use, for a given time interval, without relevant incident.

• Maintainability
• Availability
• Recoverability
• Extendability

Dependability - Direct Measures

Two types

• Event time data: list of inter-event times
• Event count data: counts of events, total operational time, ...

Software Quality Assurance

What is SQA?

• Is SQA an organization?
• Is SQA a philosophy?
• Is SQA a set of activities?

Software Quality Assurance

• Stage one of SQA - 1960-1979
  • Starting with late 1960's, several IBM locations occasionally used the term software quality assurance. The context is primary final product testing
  • 1960-1970, requests for proposal for complex military equipment would have ``software quality assurance": conduct test program on embedded software
• Influence from DOD - 1974
  • DOD Std MIL-S-52779 (AD) ``Software Quality Assurance Program Requirements"
  • Require contractors to address the following aspects
    • Methods to evaluate design and design documentation
    • Documentation of standards
    • Library control procedures
    • Testing, plans, test results, etc. ...
  • 75 SQA Program Requirement
    • Force contractor to identify technique to deliver quality product
      
    • No address on quality improvements
    • SQA is basically an extension of testing and review, and book keeping of all the results
    • In short, SQA is a police or arbiter, not a direct agent to promote quality
• Stage two of SQA - 1979-1988
  • Commissioned by the DOD, a workshop was held to hopefully define a minimum acceptable standard for SQA
  • DOD-STD-2168 was released in 1988
  • After that many major defense contractors have started to establish a SQA dept, usually under quality assurance directorate
• Stage three of SQA - 1979-
  • Influence from TQM
  • Bell Lab: in the ESS program, SQA us ``a set of rules and procedures used to guide the development, administration, maintenance, and improvement of the ESS program".
• Three distinct meanings of SQA - 80's
  • A comprehensive approach to improve software, the process of programming, and the control of software projects
  • Diligent testing
  • Verification and validation
• Impact of the new approach: SQA does not assure the quality of software; it ensures the planning and execution of a quality program
  • Emphasis on analyzing the attributes of software structures
  • Emphasis on accumulating and analyzing fault data
  • Emphasis on accumulating and analyzing process data
• Bellcore SQA criteria (85)
  • Software life cycle plan
  • Management commitment and organization
  • Development support environment
  • Documentation
  • Verification and validation procedures
  • Configuration management
  • Problem reporting and corrective action
  • Data collection, analysis, and use
  • Customer engineering and operations
  • Gear more towards broader sense of SQA

Current Practices

Emphasis on defects removing. Passing methods:

• Requirement and design reviews
• Inspection of pseudo code
• Code coverage analysis
• Code inspection

Emphasis on defects removing. Active methods:

• Functional testing
• Structural testing

SQA role:

• Independent test team
• Independent reviewers
• Oversee review process
• In-house audit team

SQA - besides defect removal

SQA also has effect on:

• Delivery of reliable and maintainable product
• Control of project to reduce risk of late delivery
• General improvement in the quality of future software products
• These goals are driven by customer satisfaction

SQA - also a TQM issue

Also a technology-people-management combination.

TQM: A quality system is the agreed on, companywide and plantwide operation work structure, documented in effective, integrated technical and managerial procedures, for guiding the coordinated actions of the people, the machines, and the information of the company and plant in the best and most practical ways to assure customer quality satisfaction and economical costs of quality.

SQA Quality Program

• Training programming staff in new techniques, methods, and tool use
• Evaluation of effectiveness of current development methods and tools
• Project, quality program, and test program planning as policy or standards dictate
• Use of reviews, analysis tools, and tests to find defects at the earliest possible time
• Library control, change control, distribution, and storage per project plan and relevant policies or standards
• Recording of all defects data and subsequent analysis of defect, fault detection and failure modes
• Use of defect data to improve processes
• Generation and analysis of various data for early indication of adverse product or project control trends
• Gathering, analyzing, and evaluation user feedback
• Survey of potential software vendors and surveillance of their performance
• Objective evaluation of the fidelity with which plans and applicable standards are followed
• Empowerment of staff to prevent defective code, artifacts of development, and user documentation from being entered into the system or delivered

SQA Techniques

• Pareto Analysis
  • Vilfredo Pareto (1906) published a treatise on skewed distribution on wealth among various people
• Albert Endres of IBM found that a remarkably small number of modules in a large system produced most of the errors
  • E.g. 0.7% of the modules accounts for 15% of the faults
  • 10% accounts for 24%
  • Look for trouble making modules or process
• Trend Data
  • Use trend data to evaluate quality of software product
  • Set control curves with the help of past data
  • E.g. a code inspection that results in a density of detected errors greater than a given upper limit suggests disturbingly faulty code; one with a density smaller than lower limit may be an ineffective testing

    

• Audits
  • To confirm compliance with approved procedures or standard
  • Audit is different from review
  • It is directed to seeing that things are being done in the correct way, while reviews focus on the substance of the material or product under review
• Failure analysis
  • Look for common failure modes
  • Correlation of failure with operation conditions
  • Root cause analysis
  • Support quality improvement
• Direct measurement of software
  • Measurement of goodness
    Number of defects with severity
    Time required to fix a bug (maintainability)
    Time to take a user to do an operation (usability)
  • Structure measurement
    Complexity of the software: cyclomatic measure
    Module coupling and cohesion
• Error seeding
  • Plant A number of seeds in the software
  • B is the number of bugs found
  • C is the number of seeds recovered in testing
    Then N = A x B / C is the number of bugs expected in the software
    
• Regression
  • New code for bug fixing is the most likely place for introduction of new bugs
  • Best case: re-run all test cases in the new load
  • Practical case: select a subset that have good coverage to be re-tested

SQA Organization

• SQA should be an independent organization

SQA Costs

• Prevention cost: action taken to prevent defects in products
• Appraisal cost: tests, evaluations used to detect defects
• Failure cost: correction of defects
• Cost in general ranges from 2% to 8% of staffing cost

Quality Factors

It provides

• A goal-oriented methodology for measuring software quality
• Another dimension that software requirements can be addressed
• Identify attributes important in a software product that will have impact later in the life cycle
• A basis for process improvement

Definitions of Quality Factors

• Coreectness: extent to which a program satisfies its specifications and fulfills the user's mission objectives
• Reliability: extent to which a program can be expected to perform its intended function with required precision
• Efficiency: the amount of computing resources required by a program to perform a function
• Integrity: extent to which access to software or data by unauthorized persons can be controlled
• Usability: effort required to learn, operate, prepare input, and interpret output of a program
• Maintainability: effort required to locate and fix an error in an operational program
• Testability: effort required to test a program to ensure it performs its intended function
• Flexibility: effort required to modify an operational program
• Portability: effort required to transfer a program from one hardware configuration and/or software system to another
• Reusability: extent to which a program can be used in other applications
• Interoperability: effort required to couple one system with another

SQA Plan

Each project should have a software quality assurance plan (SQAP). The IEEE Std for SQAP (84)

• Purpose
• Reference documents
• Management
• Documentation
• Standards, practices and conventions
• Reviews and audits
• Software configuration management
• Problem reporting and corrective action
• Tools, techniques and methodologies
• Code control
• Media control
• Supplier control
• Records collection, maintenance, and retention

SQA Considerations

• SQA organization are rarely staffed with sufficiently experienced or knowledgeable people
• The SQA management team is often not capable of negotiating with development
• Senior management often backs development over SQA on a large percent
• Many SQA organizations operate without suitable documented and approved development standards and procedures
• Software development groups rarely produce verifiable quality plans
• SQA should avoid to fix development problem, they are not fire fighters

SQA Staffing

Good people is hard to find without paying enough $$$ !!!

Chapter 11

References

• Pressman, chapter 17
• Pfleeger, chapter 7.6
• Sommerville, chapter 19-20
• Fenton chapter 8, 11

Software Reliability

Software Reliability Engineering (Musa): apply a statistical approach to measuring, managing, and containing the risk of software failure, which never goes to zero.

Ref. Musa, Iannino, Okumoto, Software Reliability, Measurement, Prediction, Application

The Fundamental Modelling Problem

A prediction system which will allow us to predict the future (T_i, T_i+1, ...) from the past (t₀, t₁, ..., t_i-1) comprises:

1. The probabilistic model which specifies the distribution of any subset of the T_j's conditional on an unknown parameter p.
2. A statistical inference procedure for p involving use of available data (realization of T_j's)
3. A prediction procedure combining (1) and (2) to allow us to make probability statements about future T_j's.

Basic Concepts

Fault density: faults removed (after system test) per thousand lines of delivered source code.

Wrong and misleading: faults removed has no correlation with reliability. Removal of a large number of faults can indicate either high quality testing or low quality coding

Customer-oriented approach

• They are not concern about the number of faults in the system
• They are concern about how often failure occurs and the cost associated with it
• What is error, bug, fault and failure??

• Failure: a department of operation from user expectation
• Fault: a defect in the program, when executed under particular condition, causes failure (also known as bug)
• Error: a mistake that a programmer made which creates a fault
• Time measurements in reliability
  • Execution time, calendar time, clock (elapse) time
• Four general ways to characterize failure occurrence
  • Time of failure
  • Time interval between failures
  • Cumulative failures experienced up to a given time
  • failures experienced in a time interval
  • These are all random variables
• Failure intensity: the number of failures per unit time
  • E.g. 0.001 failure/hour or 1 failure/100 hour
• Reliability: the probability of failure-free operation of a software for a specified time in a specified environment
  • E.g. a system may have a reliability of 0.92 for 8 hours when used by the ``average users". I.e. there will be 92 failures if the system is run 100 eight-hour periods

Failure Behaviour

Affected by two principal factors:

1. The number of faults in the software
2. The operational profile of execution

Mean Time to Failure

Mean time to failure (MTTF) - average of the next failure interval

• MTTF is used mainly in hardware failure
• Failure intensity is more preferable in software reliability study
• MTTF and FI, nonrigorously, are inverses of each other

Classification of Models

Models can be classified by their mathematical structure, assumptions, parameter estimation method, ...

1. Time domain
2. Stochastic/deterministic
3. Failure modeling
   • Models directly
   • The manifestations of faults
4. Repair modeling
   • The repair process is modeled, and imperfect and delayed repair have been taken into account
5. Additional features
6. Inclusion of considering structure of program
7. Parameters
8. Data types
9. Functional form of the derived measures

Reliability Modelling

• A software reliability model specifies the general form of the dependence of the failure process on the factors concerned in the model
• To develop a model to predict failure intensity (reliability) on execution time
• Parameters of the model can be established by
  • Estimation - statistical inference applied on failure data collected
  • Prediction - determine from the properties of the software product or the development process
• Once the model has been chosen and parameter estimated, many failure characteristics can be determined
  • Average number of failures experienced at any point in time
  • Average number of failures in a time interval
  • Failure intensity at any point in time
  • Probability distribution of failure intervals
• What constitutes a good software reliability model
  • Good predictions of future failure behavior
  • Produce useful quantities
  • Simple to use
  • Widely application
  • Based on sound assumption

Geol-Okumoto Model

• The failure process is assumed as a non-homogeneous poisson process
• Number of faults in the program, N
  • A random variable
  • Have a poisson distribution
• n = mean (N)
• m(u) = expected number of faults found by time u =
  n [ 1 - e^(-Zu) ] where
  • m'(u) = Z (n - m(u)) and
  • m(0) = 0
• Z - a constant of proportionality between the failure rate and (n - m(u)), the expected number of faults left
• Assumption: all faults make the same contribution to the failure rate

Duane's Learning Curve

It was originally to model the trend of the increasing familiarity of a user with a machine ==> After a fault once found is removed, it will not be repeated. No assumption about the repair mechanism

• Cumulative failure rate
  • q(u) = c(u)/u over operational time u
• The points (u, q(u)) tended to lie on a straight line with negative slope when plotted on log-log graph paper
  • E(Q(u)) = E(C(u))/u = m(U)/u = a u^b-1 for b<1
  • m(u) = a u^bb<1 reliability growth
  • b=11 constant reliability
  • b>1 decaying reliability

Jelinski-Moranda

It models failure as manifestation of a fault, and assumes perfect immediate repair

• pdf(t/c) = z(n-c) e^{( -Z(n-c)t)}
  • n: number of faults in the program
  • Z: the rate at which each individual fault is activated

Problems with Early Models

All faults cause failure at the same rate

Wrong assumption (Adams 1984)
Two lessons to learn

• Large software contains many small faults
• The manifestation rates of individual faults can differ by many orders of magnitude

The Bayesian Approach

A measure of our subjective degree of belief about an event

• Manifestation rate of fault i = random variable
• Failure rates can decrease between failures
  • If we use a program for a long period and find no faults, then our subjective belief that the program will not fail increases in strength

Littlewood-Verrall Model

It represents the inter-failure times directly as random variables

• Assume: after each failure a repair is attempted, and that this changes the failure rate
• It can allow for imperfect repair, since there is only a certain probability that the failure rate will decrease
• 3 parameters
  • , b₁, b₂ : estimated by ML from continuous data
• 2 sources of randomness
  • input selection, debugging

Littlewood Stochastic Reliability Growth Model

It takes account of the difference in fault manifestation rates by modeling the individual rates Z_i as independent identically distributed random variables with a gamma, gamd(z,h,s), distribution

• Failure rate = R
  = Sum (n-c) of the rv's has gamd(r,h,(n-c)s) distribution
• r(u|c) = (n-c) s / (h + u + t)
• pdf(t| R_i = r_i) = r_I e^{(-r_{i t}})
• Assumes that times to failure T_i after ith fix are independent exponentially distributed random variable's whose rates are themselves random variable's with a gamma distribution
  • pdf(r_i) = gamd(r_i, h(i), s)
• Rate of occurrence of failure
  • r(c) = s / (t + h(c))
  • h(c): scale factor of the gamma distribution

Most Common Models

• Basic execution time model
• Logarithmic poisson execution time model (Musa, Okumoto 1984)

• Tools exist to perform the statistical inference for different models
• To produce various prediction results

• SRE Toolkit (Unix or DOS, from AT&T)
• SMERFS - Naval weapon lab
• SQMS - SQT Corp

The failure intensity for the basic model as a function of failures is

• l(u) = l₀ (1 - u/v₀)
  • u: mean failures experienced
  • l₀: initial failure intensity
  • l: failure intensity
  • v₀: total failures
• E.g. Assume that a program will experience 100 failures in infinite time. It has now experienced 50. The initial failure intensity was 10 failures/CPU hr. The current failure intensity is
  • l(u) = l₀(1 - u/v₀) =
    10 ( 1 - 50/100) = 5 failures/CPU hr

Logarithmic Possion Execution Model

The decrement of failure intensity per failure is not constant, it becomes smaller as more failures are being experienced.

The failure intensity for the logarithmic possion model is

• l(u) = l₀exp( -t u)
  • t: failure intensity decay parameter
• E.g. Assume that the initial failure intensity was 10 failures/CPU hr. The failure intensity decay parameter is 0.02/failure. We will assume that 50 failures have been experienced. The current failure intensity is
  • l(u) = l₀exp( -tu) =
    10 exp ( - 0.02 x 50) = 3.68 failures/CPU hr

Parameter Determination

• Initial failure intensity
• Total failures for the basic model
• The decay parameter for logarithmic model

Incorporate Reliability in Life Cycle

Chapter 12 - Safety-Critical Software

References

• Pressman, chapter 9
• Sommerville, chapter 21
• Software Safety in Embedded Computer Systems, CACM Vol. 34, No. 2, Feb 91

Criticisms of Models

1. Lack of independence of failures
   • Faults can mask one another
   • A single fault will manifest itself in a ``burst" failure
   • Several faults in a single area will manifest themselves together
2. Usage effects
3. Imperfect observation and diagnosis of faults
4. Imperfect repair
   • Errors in previous repairs
5. Delay in reporting, diagnosis and repair
6. Update
   • Read software is enhanced and re-issued frequently
7. Discrete data

Software Safety

A software quality assurance activity that focuses on the identification and assessment of potential hazards that may impact software negatively and cause an entire system to fail

E.g. A computer-based cruise control for an automobile

• Causes uncontrolled acceleration that cannot stop
• Does not respond to depression of brake pedal
• Does not engage when switch is activated
• Slowly loses or gains speed

Software Ultra-Reliability

• Fault avoidance
  • Proof of correctness
  • Static analysis
  • Structured programming
• Fault removal
• Fault tolerance: ``N-version" technique

Hazard Analysis

All hazards are identified: features of the system with a potential for leading to an accident

• Risks are assessed with regard to their probability of materializing, and the likely severity of the consequences if they do
• Software hazard analysis: an examination of the risks posed by the possibility of software failure

Hazard Assessment

1. Hazard can be viewed as falling along a continuum in terms of severity
   • A threshold
   • Several cutoff points
2. Assessing their likelihood
   • Probabilities of individual events and the overall probability for the hazard is calculated
     difficult because sufficient design information is usually not available
   • Qualitative assessments
     they may be sufficient to provide the information needed for resource allocation during development

Hazard Control

• Eliminate the hazards
• Minimize the hazards
  • Include control devices: automatic control, lockouts, lock-ins, interlocks

Fault-tree Analysis

• A given top-level event
  • An accident that we wish to avoid
• All events that could lead up to it are identified
  • The causes of them are in turn identified
  • Repeat until a sufficient level of detail has been achieved
• A graph/tree of the sequential and concurrent combinations of events that can lead to a hazardous event
• Z - a constant of proportionality between the failure rate and (n - m(u)), the expected number of faults left
• Failure modes and effect analysis
  • Works from bottom-up
  • Each way in which each component can fail is examined, together with its effect on the whole system

Patient Monitoring System Fault Tree

Process Certification

Faced with the impossibility of achieving and assuring ultra-dependability

• Process certification
  • Follows a set of guidelines which are essentially a code of good development practice, and produces evidence to the certifying authority that this has been done

Chapter 13 - Data Collection

References

• Pressman, chapter 2.2 - 2.6
• Fenton, chapter 6, 8

Guiding Principle of Data Collection

Data should be collected with a clear purpose and a clear idea as to the precise way in which they will be analyzed so as to yield the desired information (Moroney 1951)

Six Crieria for Data Collection

1. The data must contain information permitting identification of the types of faults and changes made
2. The data must include the cost of making changes and correcting faults
3. Data to be collected must be defined as a result of clear specification of the goals of the study
4. Data should include studies of projects from production environments, involving teams of programmers
5. Data analysis should be historical, but data must be collected and validated concurrently with development
6. Data classification schemes to be used must be carefully specified for the sake of repeatability of the study in the same and different environments

Some Observations

• Most data collection depends at some point on the willingness of people to provide it
• Under pressure, data collection is always the first thing to be ditched, and therefore
• Schemes which ask for too much are doomed

Data Requirement for Reliability Modeling

Mellor 1986

• Time on all sample systems up to each failure
• Is failure the first manifestation of a fault or a repeat?
• What's the type of the fault and severity of failure?
• in which product, version and module is the fault located?
• Data must be ``clean"
  • Apply to a single version of the product
  • Run on a defined sample of systems for a given time
  • Not to subject to interference from outside events

Common Problems

• Inadequate definition
  • Definition of time
  • Definition of failure
  • Definition of modular structure
  • Classification of severity of failure
  • Type of fault
• Nature of the products
  • Multiple installations
  • Multiple products
  • Multiple versions of product in field together
  • Multiple repair levels
  • Module breakdown
• Support activities
  • Imperfect diagnosis
  • Imperfect repair
  • Delay in diagnosis
  • Suppression of data
• Data collection
  • Omission of running time
  • Omission of failure
  • Omission of time of failure
  • Omission of other essential item
  • Wrong classification
• Database integrity
  • Cannot correlate running time with failure
  • Faults and failures not distinguished and cross-referred
  • Missing cross-references
  • Confusing classifications
  • Incompleteness
  • Lack of standard identifier formats

Particular problems

1. Lack of running time records
   • Often due to the fact that often reporting systems are set up solely to chase progress on fault clearance rather than measure reliability
2. Mission of occurrence time from failure report
3. Suppression of failure reports
4. Use of inappropriate classification
   • Failures are not necessarily diagnosed in the order in which they occurred
   • Faults which were manifest on the sample as repeats but not as first manifestations

Need for Automatic Recording

To overcome the problems, automate collection systems are developed

• Static
  • Kitchenham 1984. A program history record system
• Dynamic

Chapter 14 - Process Maturity Framework

Reference

• Pfleeger, chapter 11.4
• Sommerville, chapter 18
• Fenton, chapter 13

Process Improvement Movement and CMM

Process Improvement program (Denning)

1. Understand the current status of the process
2. Develop a vision of the desired process
3. Establish a list of required process-improvement actions in priority order
4. Produce a plan to accomplish these actions
5. Commit the resources and execute the plan
6. Start over at step 1

Some Principles

• Major changes to the software process must start at the top
  • Process problems are management's responsibility
• Ultimately, everyone must be involved
  • With a more mature process, individual actions are more structured, efficient, and reinforcing
• Change is continuous. Human intensive processes are never static
  • Reactive changes generally make things worse
  • Every defect is an improvement opportunity
  • Crisis prevention is more important that crisis recovery
• Software process changes won't stick by themselves - in the absence of conscious effort, human processes behave like entropy
• It takes time, skill, and money to improve the software process
  • Dedicating resources to improvement is self evident

Capability Maturity Model

• A framework to improve software process
• A basis for process evaluation and assessment
• CMM consists of five maturity levels
• CMM was developed by SEI (Software Engineering Institute)
• Sponsored by US Defense Department
• Used by DoD as a reference to evaluate tenders from software contractors

History

• Initiated by the DoD in 1986
• Mov. 1986 SEI began developing a process maturity framework
• Sept 1987, SEI released the first version of the framework
• 1990 SEI evolved the framework into CMM 1.0
• 1992 complete review of CMM 1.0
• 1993 CMM 1.1 was released based on the 92 review

Inmature Software Organization

• No clearly defined processes to follow
• If processes are specified, not rigorously followed
• A reactionary software organization, focus on solving immediate crisis (fire fighting)
• Schedule and budgets routinely exceeded
• If hard deadlines are imposed, product functionality and quality are often comprised
• No objective way to judge quality of product or process
• Quality activities like testing and reviews often are scarified to meet schedules

Mature Software Organization

• Roles and responsibilities are clear within a project and across organization
• Managers monitor quality of products and processes
• Objective and quantitative basis exist to measure quality
• Schedules and budgets based on historical performance
• Expected results for cost, schedule, functionality, and quality are usually achieved
• Process very adaptive to new technology

Five Mature Levels

Level 1: Initial

Behavior characterization:

• No stable environment for developing and maintaining software
• Difficult to meet commitment with an orderly engineering process
• During crisis, projects typically abandon planned procedures and revert to coding and testing
• Success depends entirely on having an exceptional manager and a seasoned development team
• Frequently over budget and behind schedule
• Success depends on individual but not the organization
• No guarantee that a successful product can be repeated in the next project

Level 1: Key Challenges

• Project management
• Project planning
• Configuration management
• Software quality assurance

Level 2: Repeatable

Behavior characterization:

• Policy and procedures for managing software project established
• Planning and management of new project based on experience with similar project
• Basic management controls have installed
• Project managers track costs, schedules, and functionality and identify problems in meeting commitments when they arise
• Software requirements and work products are baselined
• Project standards are defined, and the organization ensures they are faithfully followed
• Process capability can be summarized as disciplined because project planning and tracking are stables and early success can be repeated
• Strength in doing similar work, but faces risk when presented with new challenges
• Lacks orderly framework for improvement

Level 2: Key Process Areas

Key: to establish basic project management

• Requirement management
• Software project planning
• Software project tracking and oversight
• Software subcontract management
• Software configuration management
• Software quality assurance

Level 3: Defined

Behavior characterization:

• Typical process for development and maintenance across the organization is documented (standard software process of the organization)
• Exploits effective software-engineering practices when standardizing its processes
• A Software Engineering Process Group (SEPG) is responsible for the organization's process activities
• An organization-wide training program ensures that the staff and managers have the knowledges and skills
• Project team tailor an organization's standard software process to develop their own defined process
• A well-defined process includes readiness criteria, inputs, standards and procedures for performing the work, verification mechanisms, outputs, completion criteria
• The capability can be summarized as standard and consistent because both software engineering and management activities are stable and repeatable

Level 3: Key Process Areas

Need to establish an infrastructure to institutionalizes effective software engineering and management process:

• Organization process focus
• organization process definition
• Training program
• Integrated software management
• Software product engineering
• Intergroup coordination
• Peer reviews

Level 4: Managed

Behavior characterization:

• Well-defined quantitative and quality goals for products and process set by the organization
• Productivity and quality are measured for important process activities across all projects
• Products are of predictably high quality
• Organization-wide process database is established with resource to analyze its data and maintain it
• Data in the database is used to evaluate key process activities
• Projects control their products and processes by narrowing the variation in their performance to fall within acceptable quantitative boundaries
• Capability can be summarized as being quantifiable and predictable
• Organization can identify and address special cause when some exceptional circumstance occurs

Level 4: Key Process Areas

To establish a quantitative understanding of both the software process:

• Quantitative process management- control a project's process performance quantitatively
• Software quality management - quantitative understanding of the quality of a project's products to achieve specific quality goals

Level 5: Optimizing

Behavior characterization:

• Focus on continuous improvement
• Organization has the means to identify weaknesses and strengthen the process proactively, with the goal of preventing defeats
• Data on process effectiveness is used to perform cost-benefit analysis of new technologies and propose changes to process
• Best SE practices are identified and transferred throughout the organization
• Data in the database is used to evaluate key process activities
• Rigorous defect-cause analysis and defect prevent
• Organized efforts to remove waste by changing the common cause of inefficiency
• Capability can be summarized as continuous improving by striving to improve the range of their process capability

Level 5: Key Process Areas

To implement continuous and measurable process improvement:

• Defect prevention: identify cause and propose prevention
• Technology change management: identify beneficial new technology and transfer them in an orderly manner
• Process change management: continuously improve the processes orderly

Software Indicators

CMM has a set of software indicators which is consistent with the key practices (key processing areas) in CMM

• Progress
• Effort
• Cost
• Quality
• Stability
• Computer resource utilization
• Training

SEI Process Assessment

Chapter 15 - Maintenance Cost Estimation

References

• Pfleeger, chapter 20
• Pfleeger, chapter 10
• Sommerville, chapter 28

Any reasonably large software product is bound to need maintenance, and the bear of maintenance cost is lumbering up on all vendors

Two Main Aspects to Modelling Maintenance Cost

1. The flow of failure reports from the whole field
2. The distribution of cost over failure reports

Maintenance Cost

• McCracken 1980
• Other intangible costs
  • Customer dissatisfaction when seemingly legitimate requests for repair or modification cannot be addressed in a timely manner
  • Reduction in overall software quality as a result of changes that introduce latent errors in the maintained software
  • Upheaval caused during development efforts when staff must be pulled to work on a maintenance task
• Final cost: a dramatic decrease in productivity

An Example - Belady 1972

Total effort expended on maintenance

• M = P + K e ^(c-d)
• P is the productivity effort, e.g. 40:1, $40 per line of code
• K is a constant
• c is a measure of complexity that can be attributed to a lack of good design and documentation
• d is a measure of the degree of familiarity with the software

A Second Example - COCOMO

• E.maint = ACT x 2.4 x KLOC^1.05
• ACT = annual change traffic
  = KLOC for a system undergoing maintenance / CI
• CI is the number of source code instructions that are modified or added during 1 year of maintenance

General Problems

• Difficult or impossible to trace the evolution of the software through many versions or releases
• Difficult or impossible to trace the process through which software was created
• Hard to understand some other's program
• Previous developers are not here
• Documentation does not exists or is useful
• Most software is not designed for change

Maintainability Metrics

Gilb 1979

• Problem recognition time
• Administrative delay time
• Maintenance tools collection time
• Problem analysis time
• Change specification time
• Active correction time
• Local testing time
• Global testing time
• Maintenance review time
• Total recovery time