How are INPECT^® items developed?

INSPECT^® items are written specifically to address California Content Standards, not back-aligned to the standards.

A professional statewide team of item writers consisting of both current and former teachers as well as administrators write and review each item. Throughout the process, each item writer maintains a focus on content accuracy, grade level standard alignment, and difficulty level (i.e., low, medium, and high).

The item writers provide the most likely wrong answers so that the most common cognitive disconnect may be revealed and targeted for intervention. The item writers also provide a clear rationale for each wrong answer so these cognitive disconnects can easily be identified.

Once an item has been completed and submitted for evaluation, it goes through an intensive review process. The INSPECT^® Coordinator evaluates each item and assesses the content, standard alignment, and difficulty level of each item. Two additional Content Area Specialists then further scrutinize each item. Only items that make it through all three levels of review receive final acceptance and are processed into INSPECT^® for use in testing.

How can the INSPECT^® items be administered/delivered?

INSPECT^® items can be delivered in two ways.

1. On-line Delivery and Scoring.

This is the most efficient, immediate, and economical means of delivery. Implementation of the on-line delivery option allows for the fullest and most complete use of INSPECT^® and the WHAT^® reports.

2. Paper-Pencil Delivery, Local Scanning Mode (School Site or District).

Booklets and answer sheets printed and scanned on-site. Answer sheets scanned into the server (at school site or district office) so that standards based reports of results can be displayed via OARS and EADMS. This can be performed using most manufacturers’ scanners.

Reporting is done standard-by-standard at the individual, classroom, site, and district levels.

How are reliability and validity of the items established?

Several steps are taken in order to ensure the reliability and validity of each INSPECT^® item. This level of quality control begins with the initial writing of items and continues well after each item has been administered. Our treatment of these issues include quantitative and statistical approaches. When problems with an item are intercepted, steps are taken to remedy the problem.

Our first efforts at establishing validity are centered around content validity. This is sometimes referred to as content definition (Messick, 1989) and is critical for score interpretation and item response validation (Haladyna, 1999). The best way to establish content validity is through the use of expert judges. For each INSPECT^® item, three content experts judge whether or not a particular item is aligned to the standard that the item writer has chosen.

In addition to having content experts review each item, we also conduct focus groups with grade level teams of teachers. Information from these teams are compiled and used to modify problem items or other areas of concern.

Item discrimination is also assessed to ensure that each item has the ability to sensitively measure individual differences in knowledge and ability among test-takers. Each item should be able to discriminate between high performing students and lower performing students. That is, a higher percent of students with more knowledge in a given subject area should answer those questions with greater accuracy than students with lower ability or knowledge in that area. Item discrimination is estimated using a point-biserial. This gives an estimate of the relationship between a given item and overall test performance. The point-biserial has a direct relationship to alpha (the reliability coefficient that is used to assess the reliability of a given test).