Progress-Over-Time Reports with Chi Square
Progress-Over-Time reports are used to answer the question, “Are our students learning?” To investigate progress over time, we can look at an Outcome Set with two different date ranges to determine if the scoring distribution has changed over time. To do this, we want to add a second time interval.
In a Progress-Over-Time report, two Point-in-Time reports are compared to examine progress in learning over time. We expect the mean score for the first dataset (pre-scores) be less than the mean score for the second dataset (post-scores).
If you have sufficient granularity in your performance levels, you can use the Chi Square test to see whether the difference in the two scoring distributions is significant or could have just happened randomly. Scoring Criteria with less than six performance levels are not granular enough to yield reliable correlations. If you have less than six performance levels, consider revamping your default levels to increase the number to at least six. Use the Chi Square test to make comparisons between two or more scoring distributions.
Step 1: Select Report Scope Options
1. Use the List By drop-down menu to select any option.
2. Use the Statistics drop-down menu to select Performance Level (Counts). If you select any other option, you will not be able to run the Chi Square test comparing time intervals.
Step 2: Select Time Intervals
3. Use the From and To date fields to select the dates for your report.
Step 3: Add Comparison Intervals
4. Click on the Add Interval button to reveal a second set of date range boxes. Use the boxes to select comparison From and To dates.
Step 4: Select Filter Options
5. Click on the Choose Outcome/Instrument button. This button will change depending on the Report Scope options that were selected in Step 1.
1. Use the Sources column to select the department in which the Outcome or Outcome Set exists.
2. Use the Outcome Sets/Rubrics column to select the appropriate Outcome or Outcome Set. Use the Add Selected button or Drag-and-Drop the Outcome to the Chosen Outcomes/Instruments column.
3. Click Done.
Step 5: Select Calculation & Output Options
6. Select the Calculation Options that you would like to use:
- In Case of Multiple Submissions: If more than one submission was made and scored, this option allows you to select to include all submissions, average student submission scores, use the latest student submission score, or use the earliest student submission score.
- Filter by Date: This option allows you to select to filter by the date the submissions were made or the date the submissions were assessed.
- Statistics Mode: This option allows you to select to display population or sample statistics.
- Calculate Instrument Means Using: This option allows you to select how the means will be calculated.
- Reliability Assessment Scores: If you have performed reliability tests using Assessment Instrument linked to this outcome set you can select to include or exclude the reliability assessment scores.
- Held Scores: This option allows you to select whether you would like to include, exclude, or who only held scores.
7. Select the Output Options that you would like to use:
Select the output options you wish to include in your report by clicking on the associated checkboxes.
8. If you would like to use a Pegging Scheme, use the drop-down menu to select it.
Step 6: Generate Report
9. Click on the Generate Report button.
Step 7: Run-Chi Square Simulator
Once you have generated the table report,
10. Click on the Chi Square Simulator button.
The simulator allows you to select the distribution score columns that you want to compare. This is important because you cannot use cell counts of less than 5.
If all cell counts are greater than 5 and you wish to compare all columns, then you can skip this step and click the Chi-Square Test button right away.
11. Click and drag your cursor over the cells in the row that you wish to compare.
Select only the performance level columns and for one time interval at a time. Repeat for additional time intervals.
Step 8: Review the Chi-Square Probability Chart
In the Chi Square Probability Chart, look at the appropriate Degrees of Freedom (DF) row and the .05 alpha level column in the Probability chart. This represents an acceptable level of probability for the type of data we are using and a small to medium sample size.
Compare the critical probability value for 6 DF at the .05 alpha level with the calculated Chi Square value. If the calculated value is greater than the value in the chart, then the difference between the two scoring distributions is significant. This can be interpreted as meaning that the students did not learn what they know simply by chance. It does NOT necessarily mean that they learned anything because of your instructional program. But, if your student sample is broad and heterogeneous enough, significant learning progress can indicate that you are reaching learning goals. This information supports system validity, and you have come a long way from believing that ANY number generated by digital assessment software is necessarily true.
A table will appear with the selections that you have made.
12. Click on the Calculate Chi-Square button below the table.