Criteria as observable differences
Your rubric says ‘thorough analysis’ and grades itself differently every time
You and your TA scored the same submission two tiers apart, and neither of you was wrong under the rubric as written.
What it is
Three tiers per component, written as differences someone could see rather than as adjectives. “Thorough analysis, clear interpretation” looks like criteria and functions as nothing.
The test is whether a student could predict their own grade within one tier. If they could not, you have written a vocabulary for defending grades rather than criteria for earning them.
Run it
- For each component, write three lines: full credit, partial, none.
- Write them as observable differences, not adjectives.
- Use the near-miss you actually get as the partial-credit line. You already know what it looks like; you have graded it forty times.
- Test it against the prediction question above.
Say this
An adjective rewritten as a difference
Before: “Limitations are discussed thoroughly.”
After: “Full credit names a specific limitation and says which way it biases the estimate. Partial credit names a specific limitation without a direction. No credit offers a generic statement that the study has limitations.”
Students improve immediately, because the thing being asked for was never a mystery, only unstated.
In my courses
Adapted from the EPI 553 final report rubric. Planned for EPI 601, where rubrics and exemplars for the study critiques and the final paper are to be written before the semester rather than during it.
That timing is the point. Front-loading them is also what makes the stress practice work, because most of what students experience as workload pressure in a methods course is ambiguity about what is being asked.
The dominant failure is that the criteria become adjectives again on the second assignment, when you are writing quickly.
Evidence
Transparency in criteria can be scored reliably enough to guide revision (Palmer et al., 2018). That measures the artifact, not student outcomes.
The rest of the case is the same evidence that supports transparent assignments, which is to say one replication on perceptions and nothing on learning.
The one thing you can check without a study is grader agreement. Score five submissions independently with your TA before norming, once under the old criteria and once under tiered criteria. A narrower spread is what the practice actually predicts, and unlike confidence it is observable.