close

Вход

Забыли?

вход по аккаунту

код для вставкиСкачать
Writing a Test Strategy
Tor Stålhane
Why a testing strategy
We need a testing strategy to help
• System and software testers plus Test-andEvaluation staff to determine the overall test
strategy when developing or modifying a
software intensive system
• Project stakeholders – customers and senior
management – to approve the test strategy
• Testers and system and software analysis to
determine
– Test objectives
– Qualification requirements
– Verification and validation criteria
Testing strategy concepts
We will discuss the following concepts:
• Purpose of a test strategy
• Testing focus
• Contents of a test strategy
• Software integrity levels
• Test objectives and priorities
Purpose of a test strategy – 1
The test strategy is important in order to
• Obtain consensus on test goals and objectives
from stakeholders – e.g. management,
developers, testers, customers and users
• Manage expectations right from the start
• Be sure that we are heading in the right
direction
• Identify the type of tests to be conducted at
all test levels
Purpose of a test strategy – 2
When we write a test strategy is important to
remember that:
• Whatever we do, some kind of test strategy will
emerge. Thus, we might as well specify the one
we think is the best one
• A documented strategy is the most effective way
to get an early agreement on goals and objectives
• We need to address:
– Human factors – usability
– Interoperability, except for stand-alone systems
Testing focus
Our focus will depend on which stakeholder we
are considering at the moment:
• Users – acceptance test and operational tests
• Analysts – systems test, qualification tests
• Designer – integration tests
• Programmer – unit tests
The main point is that we need to define the
stakeholder first – then the tests to be run.
Contents of a test strategy – 1
The following is a list of what can be specified in
a test strategy. Not all of it is needed in all
cases – only use what is necessary.
• Project plan, risks and activities
• Relevant regulations – depending ofn
application area
• Required processes and standards
• Supporting guide lines
Contents of a test strategy – 2
• Stakeholders – e.g. users, testers, maintainers
– and their objectives
• Necessary resources – people, computers
• Test levels and phases
• Test environment – e.g. lab equipment
• Completion criteria for each phase
• Required documentation and review method
for each document
Software integrity level
There are several ways to define software
integrity levels. When we choose an integrity
level this will strongly influence the way we do
testing.
We will look at three definitions of integrity
levels:
• IEEE 1012 – general software
• ISO 26262 – automotive software
• IEC 61508 – general safety critical software
Warning
The next slides are taken from the two standards
ISO 26262 (automotive) and IEC 61508 (industrial
automation).
They show the accepted techniques for some testing
activities – integration testing, unit testing and
module testing.
Note that these lists
• Show what is recommended depending on SIL or
ASIL level.
• Do not include any techniques related to TDD or
agile techniques.
IEEE 1012 – general software
• 4, High – some functions affect critical system
performance.
• 3, Major – some functions affects important
system performance
• 2, Moderate – some functions effect system
performance but workarounds can be
implemented to compensate.
• 1, Low – some functions have noticeable
effect on system performance but creates only
inconveniences
V&V Activities
V&V activity
SW Integrity Level
Development
requirements
level
4
Design
level
3 2 1 4 3 2 1
Implementation
level
4
3
2
1
Acceptance test execution
Test level
4 3 2
X X X
Acceptance test plan
X
X X
Interface analysis
X
X X
X X X
X
X X
Management and review
support
X
X
X X
X
X
Management review of
V&V
X
X X
X X X X X
X X
X X
X
1
ISO 26262 – automotive software
The ASIL level – A, B, C or D – is the outcome of
the combination of three factors:
• S – Severity. How dangerous is a event
• E – Probability. How likely is the event
• C – Controllability. How easy the event to
control if it occurs
Finding the ASIL level
Severity
S1
S2
S3
Probability
C1
C2
C3
E1
QM
QM
QM
E2
QM
QM
QM
E3
QM
QM
A
E4
QM
A
B
E1
QM
QM
QM
E2
QM
QM
A
E3
QM
A
B
E4
A
B
C
E1
QM
QM
A
E2
QM
A
B
E3
A
B
C
E4
B
C
D
Methods for software integration
testing
Methods and Measures
ASIL
According
to req.
A
B
C
D
1
Requirements based test
9.4.4
++
++
++
++
2
External interface test
9.4.4
+
++
++
++
3
Fault injection test
9.4.4
+
+
++
++
4
Error guessing test
9.4.4
+
+
++
++
Methods for software unit testing
Methods and Measures
According
to req.
ASIL
A
B
C
D
1
Functional tests
8.4.2
See table 8.2
2
Structural coverage
8.4.2
See table 8.3
3
Resource usage measurement
8.4.2
+
+
+
4
Back-to-back test between simulation 8.4.2
model and code, if applicable
+
+
++ ++
++
IEC 61508 – safety critical software
The level of criticality uses two parameters:
• Ft = tolerable risk frequency
• Fnp = protection system demand rate
PFDavg = Ft / Fnp .
E.g. we can accept one dangerous situation per year, thus
Ft =10-4. Problems will cause the protection system to be
invoked 10 times per hour.
Thus Fpn = 10 => PFDavg = 10-6.
This value decides the SIL level – see table on the next
slide
IEC 61508 – safety critical software
Safety
integrity
level
High demand or continuous mode of
operation
(Probability of a dangerous failure per
hour)
4
 10 -9 to < 10 -8
3
 10 -8 to < 10 -7
2
1
 10 -7 to < 10 -6
 10 -6 to < 10 -5
Detailed design
Technique/Measure
SIL1
SIL2
SIL3
SIL4
HR
HR
HR
HR
Table B.7
R
HR
HR
HR
1c Formal methods including for example,
CCS, CSP, HOL, LOTOS, OBJ, temporal
logic, VDM and Z
2 Computer-aided design tools
C.2.4
---
R
R
HR
B.3.5
R
R
HR
HR
3
Defensive programming
C.2.5
---
R
HR
HR
4
Modular approach
Table B.9
HR
HR
HR
HR
5
Design and coding standards
Table B.1
R
HR
HR
HR
6
Structured programming
C.2.7
HR
HR
HR
HR
7
Use of trusted/verified software modules
and components (if available)
C.2.10
C.4.5
R
HR
HR
HR
1a Structured methods including for example,
JSD, MASCOT, SADT and Yourdon
1b Semi-formal methods
Ref
C.2.1
Appropriate techniques/measures shall be selected according to the safety integrity level.
Alternate or equivalent techniques/measures are indicated by a letter following the number. Only
one of the alternate or equivalent techniques/measures has to be satisfied.
Module testing and integration
Technique/Measure
1
Probabilistic testing
Ref
C.5.1
SIL1
---
SIL2
R
SIL3
R
SIL4
HR
2
Dynamic analysis and testing
B.6.5
Table B.2
R
HR
HR
HR
3
4
Data recording and analysis
Functional and black box testing
C.5.2
B.5.1
B.5.2
Table B.3
HR
HR
HR
HR
HR
HR
HR
HR
5
Performance testing
C.5.20
Table B.6
R
R
HR
HR
6
Interface testing
C.5.3
R
R
HR
HR
a) Software module and integration testing are verification activities (see table A.9).
b) A numbered technique/measure shall be selected according to the safety integrity level.
c) Appropriate techniques/measures shall be selected according to the safety integrity level.
Test objectives and priorities
Only in rather special cases can we test all input
– binary input / output and few parameters.
Thus, we need to know
• The overall objective of testing
• The objective of every test case
• The test case design techniques needed to
achieve our goals in a systematic way.
The test objectives are our requirements
specification for testing.
Test data selection
One of the important decisions in selecting a
test strategy is how to select test data. We will
look at five popular methods
• Random testing
• Domain partition testing
• Risk based testing
• User profile testing
• Bach’s heuristic risk-based testing
Random testing
The idea of random testing is simple:
1. Define all input parameters – e.g. integer, real,
string
2. Use a random test / number generator to produce
inputs to the SUT
The main problem with this method is the lack of an
oracle to check the results against. Thus, manual
checking is necessary.
The method is mostly used for crash testing (robustness
testing) – will the system survive this input?
Domain partition testing – 1
Definitions:
• A domain is a set of input values for which the
program performs the same computation for
every number of the set. We want to define
the domains so that the program performs
different computations on adjacent domains
• A program is said to have a domain error if the
program incorrectly performs input
classification – selects the wrong domain
Domain testing – simple example
aX
2
 bX  c  0
If a >< 0, this equation has the following solution
b
X  
2a

b
2
4a
2

c
a
Otherwise we have that
X  
c
b
Testing domains – 1
= 00
aA ><
0
aA == 0
b
D3
2
4a
c
A= 0
2
 0
a
D1
D2
b
2
4a
c
A= 0< 0
2
a
Testing domains – 2
We have four input domains:
• D0: a = b = 0, which is a line in the threedimensional space
• D1: a = 0 and b >< 0, which a plane in the
three-dimensional space
• D3: b*b >= 4*ac and a >< 0.
• D2: b*b < 4*ac and a >< 0.
Testing domains – 3
Domains for a fixed c
b
-value
D1
D
D3
2* sqrt(ac )
D2
D0
a
D2
D3
D1
Risk based testing
The idea of risk based testing is to
1. Identify the risk or cost of not delivering a
certain functionality.
2. Use this info to prioritize tests. We will cover
this in more details later under “Test
prioritation”
User profile testing
The main idea with this type of testing is to
generate tests that mirror the user’s way of
using the system.
Consider a situation where we know that the
users in 80% of all case
• Fetch a table from the database
• Update one or more info items
• Save the table back to the database
Then 80% of all tests should test these three
actions.
Bach’s risk-based testing
Bach’s heuristics is based on his experience as a
tester. Based on this experience he has
identified
• A generic risk list – things that are important
to test
• A risk catalogue – things that often go wrong
We will give a short summary of the first of
Bach’s lists.
Bach’s generic risk list – 1
Look out for anything that is:
• Complex – large, intricate or convoluted
• New – no past history in this product
• Changed – anything that has been tampered with or
“improved”
• Upstream dependency – a failure here will cascade
through the system
• Downstream dependency – sensitive to failure in the
rest of the system
• Critical – a failure here will cause serious damage
Bach’s generic risk list – 2
• Precise – must meet the requirements exactly
• Popular – will be used a lot
• Strategic – of special importance to the users or
customers
• Third-party – developed outside the project
• Distributed – spread out over time or space but still
required to work together
• Buggy – known to have a lot of problems
• Recent failure – has a recent history of failures.
Test and system level – 1
Test and system level – 2
From the diagram on the previous slide we see
that we can test on the
• Electronics level – e.g. DoorActuator sends the
right signal
• State / signal level – e.g. door is closed iff
DoorStateClosed
• Logical level – e.g. the door remain closed as
long as the speed is non-zero
• Safety level – e.g. the door remain closed as
long as the train is moving
Acknowledgement
The first part of this presentation is mainly taken
from Gregory T. Daich’s presentation “Defining
a Software Testing Strategy”, 30 April 2002.
1/--страниц
Пожаловаться на содержимое документа