close

Вход

Забыли?

вход по аккаунту

код для вставкиСкачать
Will your study have
enough power?
CYRUS SAMII
NEW YORK UNIVERSITY
ieGovern Impact Evaluation Workshop
Istanbul, Turkey
January 27-30, 2015
: #ieGovern
Definition
THE POWER OF YOUR STUDY MEASURES
THE CONFIDENCE YOU CAN HAVE THAT IT
WILL LEAD YOU TO DRAW THE RIGHT
CONCLUSION.
What we will not do
today
What we will not do
today
Definition
THE POWER OF YOUR STUDY MEASURES
THE CONFIDENCE YOU CAN HAVE THAT IT
WILL ALLOW YOU TO DRAW THE RIGHT
CONCLUSION.
BUT WHY MIGHT YOU DRAW THE
WRONG CONCLUSION?
Definition
THE POWER OF YOUR STUDY MEASURES
THE CONFIDENCE YOU CAN HAVE THAT IT
WILL ALLOW YOU TO DRAW THE RIGHT
CONCLUSION.
BUT WHY MIGHT YOU DRAW THE
WRONG CONCLUSION?
THE WORLD IS MESSY.
Definition
POWER = EFFECT SIZE + NUMBERS - MESSY
Messy
“WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?”
Messy
“WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?”
EFFECTS ARE MESSY.
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
DEPENDS ON THE KIND OF PERSON:
“EAGER BEAVER”
“NORMAL BUREAUCRAT”
“L AZY SOB”
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
Type of person
With incentive
Without incentive
Effect of incentive
Eager beaver
10 cases/day
10 cases/day
0
Normal bureaucrat
10 cases/day
2 cases/day
8
Lazy SOB
2 cases/day
2 cases/day
0
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
Type of person
With incentive
Without
incentive
Effect of
incentive
Number of
these types?
Eager beaver
10 cases/day
10 cases/day
0
8
Normal
bureaucrat
10 cases/day
2 cases/day
8
8
Lazy SOB
2 cases/day
2 cases/day
0
8
IF INCENTIVE SCHEME APPLIED TO WHOLE DEPARTMENT:
EFFECT
= (8 X 0) + (8 X 8) + (8 X 0)
= 64 MORE CASES/DAY
Numbers and messy
SUPPOSE OUR STUDY WORKS WITH 2 OFFICES.
1 GETS THE INCENTIVE SCHEME AND 1 DOESN’ T.
THEN WE MEASURE THE PRODUCTIVITY EFFECT.
WHAT COULD HAPPEN?
32
16
16 X 6 = 96
40
16
24 X 6 = 144
8 X 6 = 48
16
8
40
-24 X 6 = -144
16
Numbers and messy
IF WE ONLY WORK WITH 2 OFFICES, WE HAVE A
STRONG POSSIBLY OF DRAWING WILDLY MISTAKEN
CONCLUSIONS.
Numbers and messy
WHAT IF WE WORK WITH ALL SIX OFFICES?
32
16
40
16
16
8
2 X 48 = 96
24
32
40
40
8
16
2 X 16 = 32
24
32
40
16
8
16
2 X 40 = 80
32
32
40
16
8
16
2 X 16 = 32
Numbers and messy
COMPARE:
−150
−100
−50
0
25
50
75 100
150
50
75 100
150
With 2 offices
−150
−100
−50
0
25
With 6 offices
Numbers and messy
WITH EVEN MORE:
−150
−125
−100
−75
−50
−25
0
25
50
75
100
125
150
50
75
100
125
150
50
75
100
125
150
With 2 offices
−150
−125
−100
−75
−50
−25
0
25
With 6 offices
−150
−125
−100
−75
−50
−25
0
25
With 36 offices
POWER = EFFECT SIZE + NUMBERS - MESSY
POWER = EFFECT SIZE + NUMBERS - MESSY
Numbers help to compensate for messy.
POWER = EFFECT SIZE + NUMBERS - MESSY
Numbers help to compensate for messy.
Sometimes increasing numbers is straighforward.
Sometimes you need to be creative:
• For example, information campaigns experiments for
universal policies.
POWER = EFFECT SIZE + NUMBERS - MESSY
Other things you can do to reduce the problem of messy:
• Stratification techniques
• Analytical techniques (regression adjustment)
POWER = EFFECT
SIZE + NUMBERS - MESSY
POWER = EFFECT
SIZE + NUMBERS - MESSY
If the effect is REALLY large or small, then even with
small numbers and lots of messy you can draw
reasonable conclusions.
But if the effect is small, you need lots of numbers
and ways to reduce messy to draw good
conclusions.
POWER = EFFECT
SIZE + NUMBERS - MESSY
Of course, you do not know the effect size.
So, what assumption should you make about it?
Define a minimal desirable effect that would lead
you to adopt the intervention you are studying.
You want to have power to detect this minimal
desirable effect.
MDE = 50
−150
−100
−50
−25
0
25
MDE = 20
50
75
100
125
150 −150
−100
−50
With 2 offices
−150
−100
−50
−25
0
25
−100
−50
−25
0
25
With 36 offices
0
25
50
75
100
125
150
50
75
100
125
150
50
75
100
125
150
With 2 offices
50
75
100
125
150 −150
−100
−50
With 6 offices
−150
−25
−25
0
25
With 6 offices
50
75
100
125
150 −150
−100
−50
−25
0
25
With 36 offices
Formally…
Test critical value
Power critical value
MDE = (ta /2 + t1-k )s bˆ (n)
Minimal detectable effect
Effect estimator standard
deviation (“messy”) as a
function of sample size
(numbers)
Conclusion
THE WORLD IS MESSY. EFFECTS ARE MESSY.
RELIABLE EVIDENCE REQUIRES THAT WE HAVE ADEQUATE
NUMBERS. SOMETIMES WE NEED TO BE CREATIVE WITH
THIS.
WE WANT ADEQUATE POWER SO WE DON’ T WASTE $$$
ON UNRELIABLE OR INCONCLUSIVE STUDIES.
RESEARCH DESIGN COURSE MATERIALS:
HTTP://CYRUSSAMII.COM/?PAGE_ID=1603
1/--страниц
Пожаловаться на содержимое документа