Overview
Sometimes it is necessary to change or update a Question's autoscoring rules after students have already seen and responded to it. For example:
- The Question's autoscoring rules may have been misconfigured, so that students who answered with a correct response were not awarded their full score.
- Students may have provided unexpected but valid responses, which were not included for autoscoring so were not awarded the right score.
For these scenarios, it is possible to re-score students' responses against an updated or alternate version of the Question using Data API's UPDATE /sessions/item
endpoint.
The endpoint creates a re-scoring job to operate on a single Item across up to 100 session_ids. The job updates each session with new Question definitions from the Item bank, and then updates the sessions' scores according to the new scoring rules of those Questions. If the Item contains multiple Questions, the endpoint will update and re-score all of them.
Figure 1: re-scoring updates the autoscoring rules for students' responses,
then updates the student scores based on those rules.
When should re-scoring be used?
The re-scoring process should be used when an authoring oversight or misconfiguration in a Question has caused a cohort of students to receive the wrong autoscores for their responses. It allows the scores and Question validation UI to be fixed so they will display accurately to end users.
Changing the scoring rules of a published Item can make it hard to compare student results from before the change with those that come after the change. Re-scoring can be used to retroactively apply the same scoring rules to historical sessions, so that the results remain comparable.
Note that some Learnosity reports validate that the Question content and scoring rules are identical for all students included in the report. Selectively applying re-scoring to a subset of students can cause errors, inconsistencies or misleading data in those reports.
If you wish to override the autoscores for students on an ad-hoc basis, consider using Data API's UPDATE /sessions/responses/scores
endpoint instead.
Re-scoring best practices
Re-scoring is an advanced feature that has important consequences for reporting and data analysis. Read our separate ariticle on best practices for re-scoring before beginning your implementation.
The UPDATE /sessions/item
endpoint
/sessions/item
endpoint.{
"session_ids": ["f91a7773-c5b0-440a-86a4-b8f82c8a8119", "2761f570-c4ec-413d-ac9d-26e429e7f8df"]
"target_item": {
"reference": "LRN-000045",
"organisation_id": 32, // optional
"item_pool_id": "ipv1" // optional
},
"copy_item": { // optional
"reference": "LRN-000046",
"organisation_id": 32, // optional
"item_pool_id": "ipv2" // optional
}
}
The session_ids
array may contain up to 100 session_id
strings.
target_item
specifies which Item in the provided session_ids
should be updated.
copy_item
can point to an alternative version of the Item that has been updated with the intended scoring rules. If provided, the Question definitions of copy_item
will be copied over to the session and used for scoring purposes. This does not rewrite the original Item references in the session, therefore:
- The session's updated scores will continue to be reported against
target_item
. - The session's Item source metadata accessible via Data API will still refer to
target_item
.
If copy_item
is omitted, it defaults to the current definition of target_item
in the Item bank.
The endpoint validates that the provided Item(s) and session(s) are compatible for re-scoring. Specifically:
target_item
andcopy_item
must contain the same number of Questions.target_item
andcopy_item
must contain identical Question types, in the same order.- Each provided
session_id
must be inCompleted
status.
If any validation errors occur at this stage, the endpoint will return a descriptive message and the sessions will not be modified.
If the validation is successful, the endpoint returns a job_reference
. The status of the re-scoring job can be retrieved by passing that job_reference
to Data API's GET /jobs endpoint.
- The job may take up to 60 minutes to complete, depending on other load on Learnosity systems.
- Reviewing any of the given sessions while the re-scoring job is in progress may cause inconsistent data to be displayed in the reviewed session.
- Resuming any of the given sessions while the re-scoring job is in progress is not a supported use case and may cause errors in the session and/or the job.
Note that this endpoint currently has a rate limit of 1 request per 5 second window, per consumer key.
Re-scoring job result
Once the job completes, its result can be retrieved from Data API's GET /jobs endpoint.
If any of the given sessions did not contain the target_item
, those session_ids
will be listed in the job result. These are informational warnings only; the other sessions will still be updated.
The job result also lists any sessions for which a scoring error resulted when trying to re-score the student's response. The most likely cause of a scoring error is that target_item
and copy_item
are incompatible, for example:
- A single-select MCQ is swapped to a multi-select MCQ.
- The available response options in an MCQ are edited.
- Rows or columns are added to/removed from a choicematrix Question.
- The number of response areas in a cloze Question is changed.
In the case where scoring errors occurred, it is necessary to repair the sessions by lodging a new re-scoring job against a compatible Item/Question. It may be necessary to audit the variations from the original Item to understand how to rebuild a compatible Item.
A sample job result payload is shown below:
{
"reference": "85198c2e-d6b4-4cbe-b6bd-9fc816580d46",
"type": "UpdateSessionsItem",
"status": "completed", // Can also be "halted", signifying a general failure
"results": {
"error": null, // May contain null, or a string error message that affected the whole job.
"sessions": [ // Array of result objects per session
{
"session_id": "0442bb7e-5dc9-4e90-88c9-858dfe95dbb0",
"success": true, // true if this session was processed without errors, else false.
// additional fields may be present describing specific failures.
}
]
},
"dt_created": "2020-10-02 06:33:58",
"dt_completed": "2020-10-02 06:33:58.317265"
}
Resuming/reviewing a session after re-scoring
If your system will allow a session to be resumed or reviewed again after re-scoring, consider the following:
- The Item list of the resumed the session (specified via Items API's
request.items
or the activity_template_id) must include the original Item source identifiers of target_item. Using the copy_item identifiers instead will result in copy_item being appended to the session as an additional Item. - You will need to pass
question_source: "scorable"
in your session initialization to display the corrected score and validation feedback:- If a student, teacher or other user can view the session via Items API, Assess API or Questions API in resume or review mode, pass Items API's
question_source: "scorable"
- If a student, teacher or other user can load the session via Reports API's session-detail-by-item or the session-detail-by-question reports, pass Reports API's
question_source: "scorable"
- If a student, teacher or other user can view the session via Items API, Assess API or Questions API in resume or review mode, pass Items API's
Understanding the question_source option
Specify the question_source
in your initialization options to Items API, Assess API, Questions API and/or Reports API to control which Question state is rendered when resuming or reviewing a session:
- Specify
question_source: "scorable"
to render all Questions using the version being used for backend scoring. This is the same as"original"
, unless re-scoring has been performed. - Specify
question_source: "original"
to render all Questions as they were originally presented to the student when they first initialized their session. This can be used to audit the Question as the student first experienced it. - Omit the
question_source
option for the default behaviour, which will render each Question using the current live version in the Item bank.
Figure 2: use the question_source
option to display either the
original or the re-scored version of the Question.
Limitations
Note the following limitations of re-scoring:
- Re-scoring does not update pre-existing
activity-summary-by-group
orsessions-summary-by-group
datasets. Those datasets can be regenerated via Data API to reflect the new scores. However the updated scores will automatically flow into other score-based reports.
Also note the following limitations when specifying copy_item
:
- The
copy_item
does not replacetarget_item
on the session. The session will be updated so that its Question definitions fortarget_item
are a copy of those on thecopy_item
. -
A re-scoring operation does not rewrite any session metadata to reflect
copy_item
's source identifiers (i.e. Item/Questionreference
,id
,organisation_id
oritem_pool_id
). Session metadata will continue to list thetarget_item
.- That metadata can be retrieved via the GET
/sessions/responses
endpoint; it will continue to reflect the Item and Question identifiers of the originaltarget_item
.
- That metadata can be retrieved via the GET
- Each session's scores will continue to be reported against
target_item
'sreference
,organisation_id
anditem_pool_id
, even ifcopy_item
is specified. This applies to all reports that display anitem_reference
, including Item Analysis, Response Analysis and *-by-item report types (eg.lastscore-by-activity-by-item
). - Each session's scores will continue to be reported against
target_item
's tags, even ifcopy_item
is specified. To change the tag breakdown, you will need to update the tags on the originaltarget_item
and possibly resubmit the session (depending on where you need the tagging changes to appear). - If multiple re-score operations are applied to the same Item and session(s),
target_item
must always be specified using the originalreference
,organisation_id
anditem_pool_id
of the Item that was served to the student. -
If your system will allow a session to be resumed or reviewed again after re-scoring it against a
copy_item
:- The Items of the session (either via Items API's
request.items
or theactivity_template_id
) must include the original Item source identifiers oftarget_item
. Using thecopy_item
identifiers instead will result incopy_item
being appended to the session as an additional Item. -
You will need to pass
question_source: "scorable"
in your session initialization to display the corrected validation feedback. Consider these scenarios:- If a student, teacher or other user can view the session via Items API, Assess API or Questions API in resume or review mode, pass the
question_source: "scorable"
option. - If a student, teacher or other user can load the session via Reports API's session-detail-by-item or the session-detail-by-question reports, pass Reports API's
question_source: "scorable"
option.
- If a student, teacher or other user can view the session via Items API, Assess API or Questions API in resume or review mode, pass the
- The Items of the session (either via Items API's