Re-scoring: change or fix the autoscoring rules for an assessment

Overview

Sometimes it is necessary to change or update a Question's autoscoring rules after students have already seen and responded to it. For example:

  • The Question's autoscoring rules may have been misconfigured, so that students who answered with a correct response were not awarded their full score.
  • Students may have provided unexpected but valid responses, which were not included for autoscoring so were not awarded the right score.

For these scenarios, it is possible to re-score students' responses against an updated or alternate version of the Question using Data API's UPDATE /sessions/item endpoint.

The endpoint creates a re-scoring job to operate on a single Item across up to 100 session_ids. The job updates each session with new Question definitions from the Item bank, and then updates the sessions' scores according to the new scoring rules of those Questions. If the Item contains multiple Questions, the endpoint will update and re-score all of them.


rescoring-1.png

Figure 1: re-scoring updates the autoscoring rules for students' responses,
then updates the student scores based on those rules.

 

When should re-scoring be used?

The re-scoring process should be used when an authoring oversight or misconfiguration in a Question has caused a cohort of students to receive the wrong autoscores for their responses. It allows the scores and Question validation UI to be fixed so they will display accurately to end users.

Changing the scoring rules of a published Item can make it hard to compare student results from before the change with those that come after the change. Re-scoring can be used to retroactively apply the same scoring rules to historical sessions, so that the results remain comparable.

Note that some Learnosity reports validate that the Question content and scoring rules are identical for all students included in the report. Selectively applying re-scoring to a subset of students can cause errors, inconsistencies or misleading data in those reports.

If you wish to override the autoscores for students on an ad-hoc basis, consider using Data API's UPDATE /sessions/responses/scores endpoint instead.

 

Re-scoring best practices

Re-scoring is an advanced feature that has important consequences for reporting and data analysis. Read our separate ariticle on best practices for re-scoring before beginning your implementation.

 

The UPDATE /sessions/item endpoint

A re-scoring job is initiated using Data API's UPDATE /sessions/item endpoint.
 
The endpoint accepts a request payload as follows:
{
    "session_ids": ["f91a7773-c5b0-440a-86a4-b8f82c8a8119", "2761f570-c4ec-413d-ac9d-26e429e7f8df"]
    "target_item": {
            "reference": "LRN-000045",
            "organisation_id": 32, // optional
            "item_pool_id": "ipv1" // optional
    },
    "copy_item": { // optional
            "reference": "LRN-000046",
            "organisation_id": 32, // optional
            "item_pool_id": "ipv2" // optional
      }
}

The session_ids array may contain up to 100 session_id strings.

target_item specifies which Item in the provided session_ids should be updated.

copy_item can point to an alternative version of the Item that has been updated with the intended scoring rules. If provided, the Question definitions of copy_item will be copied over to the session and used for scoring purposes. This does not rewrite the original Item references in the session, therefore:

  • The session's updated scores will continue to be reported against target_item.
  • The session's Item source metadata accessible via Data API will still refer to target_item.

If copy_item is omitted, it defaults to the current definition of target_item in the Item bank.

The endpoint validates that the provided Item(s) and session(s) are compatible for re-scoring. Specifically:

  • target_item and copy_item must contain the same number of Questions.
  • target_item and copy_item must contain identical Question types, in the same order.
  • Each provided session_id must be in Completed status.

If any validation errors occur at this stage, the endpoint will return a descriptive message and the sessions will not be modified.

If the validation is successful, the endpoint returns a job_reference. The status of the re-scoring job can be retrieved by passing that job_reference to Data API's GET /jobs endpoint.

  • The job may take up to 60 minutes to complete, depending on other load on Learnosity systems.
  • Reviewing any of the given sessions while the re-scoring job is in progress may cause inconsistent data to be displayed in the reviewed session.
  • Resuming any of the given sessions while the re-scoring job is in progress is not a supported use case and may cause errors in the session and/or the job.

Note that this endpoint currently has a rate limit of 1 request per 5 second window, per consumer key.

 

Re-scoring job result

Once the job completes, its result can be retrieved from Data API's GET /jobs endpoint.

If any of the given sessions did not contain the target_item, those session_ids will be listed in the job result. These are informational warnings only; the other sessions will still be updated.

The job result also lists any sessions for which a scoring error resulted when trying to re-score the student's response. The most likely cause of a scoring error is that target_item and copy_item are incompatible, for example:

  • A single-select MCQ is swapped to a multi-select MCQ.
  • The available response options in an MCQ are edited.
  • Rows or columns are added to/removed from a choicematrix Question.
  • The number of response areas in a cloze Question is changed.

In the case where scoring errors occurred, it is necessary to repair the sessions by lodging a new re-scoring job against a compatible Item/Question. It may be necessary to audit the variations from the original Item to understand how to rebuild a compatible Item.

A sample job result payload is shown below:


{
    "reference": "85198c2e-d6b4-4cbe-b6bd-9fc816580d46",
    "type": "UpdateSessionsItem",
    "status": "completed",  // Can also be "halted", signifying a general failure
    "results": {
        "error": null,      // May contain null, or a string error message that affected the whole job.
        "sessions": [       // Array of result objects per session
            {
                "session_id": "0442bb7e-5dc9-4e90-88c9-858dfe95dbb0",
                "success": true,  // true if this session was processed without errors, else false.
                // additional fields may be present describing specific failures.
            }
        ]
    },
    "dt_created": "2020-10-02 06:33:58",
    "dt_completed": "2020-10-02 06:33:58.317265"
}

 

Resuming/reviewing a session after re-scoring

If your system will allow a session to be resumed or reviewed again after re-scoring, consider the following:

  • The Item list of the resumed the session (specified via Items API's request.items or the activity_template_id) must include the original Item source identifiers of target_item. Using the copy_item identifiers instead will result in copy_item being appended to the session as an additional Item.
  • You will need to pass question_source: "scorable" in your session initialization to display the corrected score and validation feedback:
    • If a student, teacher or other user can view the session via Items API, Assess API or Questions API in resume or review mode, pass Items API's question_source: "scorable"
    • If a student, teacher or other user can load the session via Reports API's session-detail-by-item or the session-detail-by-question reports, pass Reports API's question_source: "scorable"

 

Understanding the question_source option

Specify the question_source in your initialization options to Items API, Assess API, Questions API and/or Reports API to control which Question state is rendered when resuming or reviewing a session:

  • Specify question_source: "scorable" to render all Questions using the version being used for backend scoring. This is the same as "original", unless re-scoring has been performed.
  • Specify question_source: "original" to render all Questions as they were originally presented to the student when they first initialized their session. This can be used to audit the Question as the student first experienced it.
  • Omit the question_source option for the default behaviour, which will render each Question using the current live version in the Item bank.

Screen_Shot_2020-10-27_at_7.54.01_pm.png

Figure 2: use the question_source option to display either the
original or the re-scored version of the Question.

 

Limitations

Note the following limitations of re-scoring:

 

Also note the following limitations when specifying copy_item:

  • The copy_item does not replace target_item on the session. The session will be updated so that its Question definitions for target_item are a copy of those on the copy_item.
  • A re-scoring operation does not rewrite any session metadata to reflect copy_item's source identifiers (i.e. Item/Question reference, id, organisation_id or item_pool_id). Session metadata will continue to list the target_item.
    • That metadata can be retrieved via the GET /sessions/responses endpoint; it will continue to reflect the Item and Question identifiers of the original target_item.
  • Each session's scores will continue to be reported against target_item's reference, organisation_id and item_pool_id, even if copy_item is specified. This applies to all reports that display an item_reference, including Item Analysis, Response Analysis and *-by-item report types (eg. lastscore-by-activity-by-item).
  • Each session's scores will continue to be reported against target_item's tags, even if copy_item is specified. To change the tag breakdown, you will need to update the tags on the original target_item and possibly resubmit the session (depending on where you need the tagging changes to appear).
  • If multiple re-score operations are applied to the same Item and session(s), target_item must always be specified using the original reference, organisation_id and item_pool_id of the Item that was served to the student.
  • If your system will allow a session to be resumed or reviewed again after re-scoring it against a copy_item:
    • The Items of the session (either via Items API's request.items or the activity_template_id) must include the original Item source identifiers of target_item. Using the copy_item identifiers instead will result in copy_item being appended to the session as an additional Item.
    • You will need to pass question_source: "scorable" in your session initialization to display the corrected validation feedback. Consider these scenarios:
      • If a student, teacher or other user can view the session via Items API, Assess API or Questions API in resume or review mode, pass the question_source: "scorable" option.
      • If a student, teacher or other user can load the session via Reports API's session-detail-by-item or the session-detail-by-question reports, pass Reports API's question_source: "scorable" option.
Was this article helpful?

Did you arrive here by accident? If so, learn more about Learnosity.