Inter-rater Reliability Overview

There are three dimensions of Inter-rater Reliability testing. The three tests are used collectively to provide evidence of generalizability to support the validity of your competency assessment plan.  The most familiar of these three Reliability tests is inter-rater Reliability, which involves the consistency of assessor scoring practices.  The second test deals with the Inter-rater Reliability of Scoring Criteria in rubrics and the third test involves the authenticity of Learning Tasks.

Inter-Rater Reliability:

Inter-rater Reliability is the most common form of Inter-rater Reliability testing. Inter-rater Reliability testing examines the degree to which two or more assessors agree on scores for a single work sample.

The Inter-rater Reliability of assessor scoring is expressed as the percentage of scoring criteria that received the same score. Eighty-percent or greater is considered reasonable. If the percentage agreement is lower than eighty percent, there should be a collaborative effort to arrive at a consensus as to the meaning of the scoring criteria on which the assessors disagreed and the interpretation of the work sample. When there are more than two assessors involved, inter-rater Reliability is expressed as the mean of the difference scores for each pair of scores. A mean of < 20% is considered reasonable.

Step 1: Access Inter-rater Reliability

Step 1: Access Reliability

First ensure that you have a sufficient number of assessments that have been completed using the regular online scoring method. You must have one completed assessment for each reallocation you intend to send.

1. Click on the Main Menu icon.

2. Select Reporting.

3. Select Inter-rater Reliability.

Step 2: Allocate Re-Assessments

Step 2: Allocate Re-Assessments

On the Inter-rater Reliability screen:

4. Click the Allocate Re-Assessments button.

5. Using the Sources column, select the department that contains the Assessment Instrument you wish to use.

6. Click on the instrument that you would like to select. Use the Add Selected button or drag-and-drop it into the third column.

7. Select your Assessor Choice Method using the drop down menu.

8. Using the Assessors Available column select the assessor(s) to whom you wish to assign the reassessment and use the >> button to move the assessor(s) into the Assessors that will Reassess column.

9. If you wish to use a different assessment instrument for the reassessment click the Choose Instrument button. If you wish to use the same assessment instrument, skip this step.

If you have clicked the Choose Instrument button:

10. Using the Sources column select the department to which the assessment instrument is assigned.

11. In the Assessment Instruments column select the assessment instrument you wish to use for the reassessment and add it to the Chosen Instruments column by clicking the Add Selected button or dragging and dropping.

12. Click Done.

13. Select the number of Reassessments Per Portfolio.

14. Select whether or not you would like to Reallocate Past Reassessments.

15. Select whether or not you would like to Anonymize Students.

Step 3: Save and Prompt Allocation

Step 3: Save and Prompt Allocation

16. Once you have finished making your selections for the reassessment, scroll to the top of the screen and click Save.