Internal Consistency

Page Contents

Pilot Testing

The Pilot Test: In search of internal consistency

Difficulty Index

The purpose for pilot testing the Rosebud and Swordfish Scales and Tests are to determine the difficulty index (p-values) of the items. The items, acquired from various sources, have all received some minimal amount of validation in other studies around this four-year age group.

P-values for each of the 102 items will be determined based on the percentage of students who accurately compute the given mathematical problem. The goal of the pilot test is to determine which items can be successfully computed by representative groups around the .50 (50%) level.

The pilot testing will also reveal the discrimination index (D-value) of each item in the measures. These may range from -1.0 to +1.0 and will indicate the existing levels of skills and knowledge of the participants, and distinguish between them. Language used below is from Goodwin handout.

This will be computed as follows: 1. N = the number of participants

2. Order the tests from high to low

3. Multiply N by 27 and round off to integer (= n)

4. Count off n from the top ("high" group)

5. Count off n from bottom ("low" group)

6. Calculate the proportion of students in the high group (PH) who answered the item correctly.

PH = (number of correct answers)/n

6. Calculate the proportion of students in the low group (PL who answered the item correctly.

PL= (number of correct answers)/n

7. The discrimination index (D) is obtained by

D = PH - PL

________________________________________________________

Another way to get at it: 1. The biserial correlation, rb, and the point-biserial correlation, rpb, are two other common ways to calculate the discrimination index. Both give a correlation between the item and the rest of the test (sometimes called the "item-to-total correlation")

2. High discrimination index is desireable (usually means .30 or higher)

3. If the discrimination index is low (near 0) the difficulty index is near 0 or 1.0, or the item is measuring something different from what the rest of the items are measuring, or the item is poorly written

4. If the index is negative, item is tricky or sample may be too small and influenced by single deviant response