We recommend these general guidelines for your rescoring implementation.

When should rescoring be used?

The rescoring process should be used when an authoring oversight or misconfiguration in a Question has caused a cohort of students to receive the wrong autoscores for their responses. It allows the scores and Question validation UI to be fixed so they will display accurately to end users.

Changing the scoring rules of a published Item can make it hard to compare student results from before the change with those that come after the change. Rescoring can be used to retroactively apply the same scoring rules to historical sessions, so that the results remain comparable.

Note that some Learnosity reports validate that the Question content and scoring rules are identical for all students included in the report. Selectively applying rescoring to a subset of students can cause errors, inconsistencies or misleading data in those reports.

If you wish to override the autoscores for students on an ad-hoc basis, consider using Data API's UPDATE /sessions/responses/scores endpoint instead.

Consider whether rescoring is required for your use case:

• Use rescoring only where the autoscoring rules of an Item/Question need to be permanently updated, and need to be applied to existing historical sessions.
• If the change in autoscoring rules should only apply to future sessions, there is no need to update historical sessions via rescoring.
• It may be misleading to rewrite the existing scoring rules of historical sessions if those scores have already been published to end users. Consider the importance of auditability, and that rescoring may cause inconsistencies in historical records.
• If only a small number of sessions need a once-off score adjustment, it may be better to override their scores using the UPDATE /sessions/responses/scores endpoint .
  • The drawbacks of overrides is they can be erased if the student resubmits their test, and they don't modify the validation UI shown in review mode (which may show misleading feedback that is mismatched with the overridden score).
• Fixing typos or cosmetic issues in the Item/Question content should not warrant rescoring, since they should not affect autoscoring.
• A change to the wording or intent of a Question should not warrant rescoring, since a learner's response may be highly dependent on how the original Question was presented to them. If the wording or intent of the Question was unclear, it may be better to nullify all scores for that Question, or grant the full score to all sessions (see the UPDATE /sessions/responses/scores endpoint).
  
  

Make the fewest changes possible

When modifying an Item for the purposes of rescoring, make the fewest changes possible.

• Changing anything other than a Question's autoscoring parameters may cause rescoring to fail.
• Do not change the possible response options on a Question, as this will cause errors for those students who submitted a response option before it was changed.
• Do not change Dynamic Data associated with the Item.
• Do not change the Item layout, number of Questions or type of Questions on the Item.
• Avoid fixing typos or cosmetic issues in the Item/Question content.
  
  

Apply rescoring consistently to a whole cohort of students

Apply the rescore to the whole cohort of students who saw that Item. The cohort should include all students whose scores contribute to your largest combined report. For example:

• If the affected Item was intended to be attempted by students across multiple classes/sections, apply the rescore to all those classes/sections.
• If the affected Item was intended to be attempted by all students in an enrolment/year group, apply the rescore to everyone in that enrolment/year group.
• It may be possible to delineate cohorts by the "versions" of content they saw, for example it may be appropriate to rescore all students from the current enrolment year group, and leave the previous enrolment year group's scores untouched. Consider whether this will invalidate comparisons between the year groups.
• Note that certain Learnosity reports like the response-analysis-by-item and activity-analysis-by-question validate that all sessions in the report contain identical Questions. If you try to report on a combination of original and rescored sessions, you may encounter errors due to mismatched Questions.

Wait until students have submitted their work

Wait until all students in the cohort have submitted their sessions against the affected Item before attempting to rescore them. Then perform a single rescoring operation against all the affected sessions at once.

• Avoid inconsistencies where different sessions are in different states, with different scoring rules. This always leads to mismatched scores that are confusing and hard to troubleshoot.
• The rescoring process will validate that all the given sessions are in "Completed" status.
• Up to 100 sessions can be rescored in a single job. If more sessions must be rescored, batch the sessions into multiple sequential jobs.

Common rescoring scenarios

We provide recommendations for common rescoring scenarios below.

Scenario 1: Editable Item bank without Item pools

This scenario applies where you have full authoring control of the Item that needs to be rescored, and you do not use Item pools to maintain your content.

We recommend the following process:

1. Wait until all students in the cohort have submitted the learning activities containing the affected Item.
2. Update the scoring rules of the affected Item in your Item bank. Note the time of the update.
3. Use Data API's UPDATE /sessions/item endpoint to rescore sessions that included the Item, and were initialized before the Item was updated. Just pass the target_item in your Data API request; the rescoring will be performed using the current definition of the Item in the Item bank.

Consider:

• From the time you modify the definition of the Item, any sessions initialized with the Item thereafter will be scored according to the new definition.
• All historical sessions initialized prior to the change will be scored according to the old definition, unless/until they are included in a rescoring job.
• Rescoring all historical sessions against the updated scoring rules will ensure they remain comparable with the newer sessions.

Scenario 2: Editable Item bank with Item pools

This scenario applies where you have full authoring control of the Item that needs to be rescored, and you use Item pools to manage your content.

Depending on your use case for Item pools, decide whether to:

Scenario 2a. Mutate the Item stored within the existing Item pool as follows:

• Update the affected Item in your Item bank, then update the Item pool with that Item.
• Use Data API's UPDATE /sessions/item endpoint to rescore historical sessions against the updated Item definition. Pass the target_item.reference, target_item.organisation_id and target_item.item_pool_id matching your updated Item pool; the rescoring will be performed using the current definition of the Item in the Item pool.

OR

Scenario 2b. Create a new Item pool containing an updated version of the Item as follows:

• • Update the affected Item in your Item bank, then create a new Item pool with that Item.
  • Use Data API's UPDATE /sessions/item endpoint to rescore historical sessions against the updated Item definition in the new Item pool. Set the target_item parameters to match the old Item pool, and the copy_item parameters matching the new Item pool.

Consider:

• Plan how you want to cutover from the historical scoring rules to the new scoring rules.
  • Historical sessions which were initialized with the read-only Item may not be comparable with new sessions which include the writable Item. Attempting to include historical and new sessions in the same report may throw an error due to mismatched content.
  • Historical sessions can always be made comparable with other historical sessions by rescoring them against the same instance of the Item.
• It may be preferable to plan a cutover in the scoring rules that coincides with a separation of your student cohorts (for example, the end of a course or academic year). This way, the two cohorts will remain internally comparable, though the results between the two cohorts may not be.
• If you created a new Item pool, you may wish to update any Activity Template definitions to point to the new Item pool. Be aware that resuming a historical session using the new Activity Template will append the new Item to the session, and prevent learners from reviewing/modifying their original response.

Scenario 3: Read-only Items, including licensed content

This scenario applies where the Item that needs to be rescored resides in a read-only Item bank, which may be maintained inside or outside your organisation.

We recommend the following process:

1. Liaise with the Item bank owner to have them fix the scoring rules on the Item. They may choose to fix the scoring rules on a reissued version of the Item with a new Item reference, so as not to disrupt legacy usage under the old Item reference.
2. Once the Item's scoring rules have been fixed, use Data API's UPDATE /sessions/item endpoint to rescore sessions that included the Item, and were initialized before the Item was updated.
   • If the original Item has been updated directly, just pass target_item; it will be rescored using the latest version of the Item from the licensed Item bank.
   • If the Item was reissued with a new Item reference, specify the reissued Item via the copy_item parameter.

Consider:

• Plan how you want to cut-over from the legacy Item to the reissued Item.
  • Historical sessions which were initialized with the legacy Item may not be comparable with new sessions which include the reissued Item. Attempting to include historical and new sessions in the same report may throw an error due to mismatched content.
  • Historical sessions can always be made comparable with other historical sessions by rescoring them against the same instance of the Item.

• It may be preferable to plan a cut-over in the scoring rules that coincides with a separation of your student cohorts (for example, the end of a course or academic year). This way, the two cohorts will remain internally comparable, though the results between the two cohorts may not be.
• If the Item was reissued, you may wish to update any Activity Template definitions to point to the new Item pool. Be aware that resuming a historical session using the new Activity Template will append the reissued Item to the session, and prevent learners from reviewing/modifying their legacy response.

See also

• Rescoring: change or fix the autoscoring rules for an assessment