Overview
A key component of Feedback Aide is the Rubric. The rubric is the scoring criteria definition, that a human or AI can use to evaluate and provide a score and relevant feedback to the learner.
The rubric format is designed to allow existing rubrics that are currently in use to be easily migrated to the Feedback Aide Rubric format.
Feedback Aide Rubric Types
Feedback Aide currently supports two different types of rubrics, Essay and Short Response. Essay rubrics are for longer form responses and are fall into two formats, Analytic and Holistic rubrics, each with discrete and multi-discrete scoring. Short Response rubrics are for shorter questions, where there is a right answer students are trying to demonstrate. There are currently four different formats of Short Response rubric: General, Keypoints, Categorize and X from Y.
Essay - Analytic
The following is a discrete analytic rubric which has four traits and four levels. In a discrete rubric, there is a fixed score for each level.
The rubric above is represented by the following JSON data:
{
"title": "Narrative Writing Rubric",
"type": "analytic",
"criteria": [
{
"title": "Ideas",
"type": "discrete",
"levels": [
{
"score": 3,
"description": "Tells a story with ideas that are clearly focused on the topic and are thoroughly developed with specific, relevant details."
},
{
"score": 2,
"description": "Tells a story with ideas that are somewhat focused on the topic and are developed with a mix of specific and/or general details."
},
{
"score": 1,
"description": "Tells a story with ideas that are minimally focused on the topic and developed with limited and/or general details."
},
{
"score": 0,
"description": "Ideas are not focused on the task and/or are undeveloped."
}
]
},
{
"title": "Organization",
"type": "discrete",
"levels": [
{
"score": 3,
"description": "Organization and connections between ideas and/or events are clear and logically sequenced."
},
{
"score": 2,
"description": "Organization and connections between ideas and/or events are logically sequenced."
},
{
"score": 1,
"description": "Organization and connections between ideas and/or events are weak."
},
{
"score": 0,
"description": "No organization evident."
}
]
},
{
"title": "Style",
"type": "discrete",
"levels": [
{
"score": 3,
"description": "Command of language, including effective and compelling word choice and varied sentence structure, clearly supports the writer's purpose and audience."
},
{
"score": 2,
"description": "Adequate command of language, including effective word choice and clear sentences, supports the writer's purpose and audience."
},
{
"score": 1,
"description": "Limited use of language, including lack of variety in word choice and sentences, may hinder support for the writer's purpose and audience."
},
{
"score": 0,
"description": "Ineffective use of language for the writer's purpose and audience."
}
]
},
{
"title": "Conventions",
"type": "discrete",
"levels": [
{
"score": 3,
"description": "Consistent, appropriate use of conventions of Standard English for grammar, usage, spelling, capitalization, and punctuation for the grade level."
},
{
"score": 2,
"description": "Adequate use of conventions of Standard English for grammar, usage, spelling, capitalization, and punctuation for the grade level."
},
{
"score": 1,
"description": "Limited use of conventions of Standard English for grammar, usage, spelling, capitalization, and punctuation for the grade level."
},
{
"score": 0,
"description": "Ineffective use of conventions of Standard English for grammar, usage, spelling, capitalization, and punctuation."
}
]
}
]
}The below example is a multi-discrete rubric (or "range rubric") with five criteria and five levels. Instead of assigning a fixed score for each level, the grader can choose from a range of possible points within that level, giving the grader more precision in scoring.
Essay - Holistic
The following is a holistic discrete rubric. Holistic rubrics provide a single score, capturing the general quality and impact of the work as a whole. Holistic rubrics streamline grading and provide students with a big-picture view. We also support multi-discrete (range) holistic rubrics.
A holistic rubric can be selected by specifying the global type as 'holistic.'
"type": "holistic"
Essay Rubric JSON Schema
The Feedback Aide essay rubric format is defined in the following JSON schema. This can be used to validate schemas for use with Feedback Aide.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://feedbackaide.learnosity.com/docs/rubric.schema.json",
"title": "Learnosity Rubric v0.1",
"description": "Rubric definition for essay scoring",
"type": "object",
"required": ["title", "type", "criteria"],
"additionalProperties": false,
"properties": {
"title": {
"description": "The title of the rubric",
"type": "string",
"minLength": 0,
"maxLength": 60
},
"type": {
"description": "Type of rubric - holistic or analytic",
"type": "string",
"enum": ["holistic", "analytic"]
},
"criteria":{
"description": "Array of Criteria for scoring",
"type": "array",
"minItems": 1,
"maxItems": 10,
"items":{
"type": "object",
"required": ["title", "type", "levels"],
"additionalProperties": false,
"properties": {
"title": {
"description": "The title of the criteria",
"type": "string",
"minLength": 0,
"maxLength": 60
},
"description": {
"description": "Detailed description of the criteria",
"type": "string",
"minLength": 0,
"maxLength": 500
},
"type": {
"description": "Type of criteria - discrete or discrete_multi",
"type": "string",
"enum": ["discrete", "discrete_multi"]
},
"levels": {
"description": "Array of levels for the criteria",
"type": "array",
"minItems": 1,
"maxItems": 6,
"items": {
"type": "object",
"required": ["description"],
"additionalProperties": false,
"properties": {
"title": {
"description": "The title of the rubric",
"type": "string",
"minLength": 0,
"maxLength": 60
},
"description": {
"description": "The full description of the rubric",
"type": "string",
"minLength": 0,
"maxLength": 500
},
"score": {
"description": "The score of the level - only for discrete type criteria",
"type": "integer",
"minimum": 0,
"maximum": 100
},
"scores": {
"description": "Array of possible scores at this level - only for discrete_multi type criteria",
"type": "array",
"minItems": 1,
"maxItems": 6,
"items": {
"type": "integer",
"minimum": 0,
"maximum": 100
}
}
}
}
}
},
"$comment": "Conditional validation based on type of criteria - TODO investigate if can be done with to support more than 2 types of criteria",
"if": {
"properties": {
"type": {
"const": "discrete"
}
}
},
"then": {
"properties": {
"levels": {
"type": "array",
"items": {
"type": "object",
"required": ["score"]
}
}
}
},
"else": {
"properties": {
"levels": {
"type": "array",
"items": {
"type": "object",
"required": ["scores"]
}
}
}
}
}
}
}
}Short Response - General
This rubric type is a good fit for questions that require a short answer or explanation. The student's response is evaluated based on whether it is correct or incorrect and the score for this question type is binary (1 or 0), with 1 indicating that the student's response is correct and 0 indicating that it is incorrect.
The rubric includes the correct answer along with two lists, one for alternate correct answers and one for incorrect answers. The answers in the two lists should be close to the correct answer but missing details or vague. Just like calibration responses do with teachers, these provide indicators to the AI of how harsh or generously to mark certain responses.
The rubric above is represented by the following JSON data:
{
"type": "sr-general",
"criteria": [
{
"correct": "If the string is shorter, the pitch will be higher. If the string is longer, the pitch will be lower.",
"alternate": [
"Short strings make higher sounds, long strings make lower sounds.",
"The shorter string will produce a higher pitch.",
"Short strings make a higher pitch, but I'm not sure about long strings.",
"Short strings make a high-pitched sound when plucked."
],
"incorrect": [
"The sound of a plucked string depends on its length.",
"Long strings vibrate slower, making a different sound.",
"The length of the string affects the sound produced.",
"The sound is different because of the string length, but I'm not sure how."
]
}
]
}
Short Response - Keypoints
This rubric type is used to evaluate the correctness of a students response based on how many 'key points' they correctly identified/calculated.
The student's response is scored based on which key points they identified and whether they have provided enough detail or explanation to score for each key point. Key points can be assigned a 'score' which the student will receive if they score that point. The score for this question type is calculated by summing up the value of each achieved key point.
The rubric for this question type includes a list of the key points that the student should have included in their response, along with a score for each point.
The rubric above is represented by the following JSON data:
{
"type": "sr-keypoints",
"criteria": [
{
"point": "Contribution per unit = $6.40",
"score": 1
},
{
"point": "Break Even point is 34,375 units",
"score": 2
}
]
}Short Response - Categorize
This rubric type is used to evaluate the student's ability to categorize or classify information. The student is given a list of items and asked to categorize them into different groups or categories.
The score for this question type is binary (1 or 0) for each item, with 1 indicating that the student has correctly categorized the item, and 0 indicating that they have not. The total score is calculated by summing the number of correct answers.
The rubric for this question type should include a list of criteria, which each has a category name and a list of items that are in that category.
The rubric above is represented by the following JSON data:
{
"type": "sr-categorize",
"criteria": [
{
"category": "Fixed Costs",
"items": [
{
"name": "Davids’s salary"
},
{
"name": "Van depreciation"
},
{
"name": "Van expenses (tax, insurance, and maintenance)"
}
]
},
{
"category": "Variable Costs",
"items": [
{
"name": "Petrol"
},
{
"name": "Electricity and water"
},
{
"name": "Grooming Products"
},
{
"name": "Casual labour"
},
{
"name": "Grooming equipment depreciation"
}
]
}
]
}Short Response - X from Y
This rubric type is used when a student must provide 1 or more (X) correct answers from a longer list of (Y) correct answers. This is useful in cases such as if the student is asked to provide one (or more) examples of something where there are many correct answers.
The score for this question type is binary (1 or 0) for each point, with 1 indicating that the student has included that point and provided enough detail, and 0 indicating that they have not. The total is counted by summing the number of points achieved up to the number needed.
The rubric for this question type should include a list of the valid points that the student may have included in their response. Each valid point should identify the correct answer as well as a small number of examples of an incorrect response which don't provide enough detail etc. The rubric should also include the number of points needed for full marks.
The rubric above is represented by the following JSON data:
{
"type": "sr-xfromy",
"number_needed": 3,
"criteria": [
{
"correct": "Britain’s debt from the French and Indian War led it to try to consolidate control over its colonies and raise revenue through direct taxation (e.g., Stamp Act, Townshend Acts, Tea Act, and Intolerable Acts), generating tensions between Great Britain and its North American colonies.",
"altenate": [],
"incorrect": []
},
{
"correct": "Colonists resented the end of “salutary neglect,” the curtailment of self-government, and inability to set taxation policy (“no taxation without representation”).",
"altenate": [],
"incorrect": []
},
{
"correct": "Colonial organizations (e.g., Sons of Liberty) and publications (e.g., Common Sense) created structure for revolutionary activity.",
"altenate": [],
"incorrect": []
},
{
"correct": "Colonial confrontations (e.g., Boston Massacre and Boston Tea Party) exacerbated tensions.",
"altenate": [],
"incorrect": []
},
{
"correct": "Military participation in the French and Indian War not only provided military experience but also established Americans’ sense of themselves as an independent people.",
"altenate": [],
"incorrect": []
},
{
"correct": "Westward population movement provoked British restrictions (Proclamation of 1763) as well as discontent with those restrictions.",
"altenate": [],
"incorrect": []
},
{
"correct": "The Enlightenment inspired rethinking of concepts, such as rights of individuals, the rights of British subjects, and republican self-government.",
"altenate": [],
"incorrect": []
},
{
"correct": "Intercolonial connections strengthened earlier in the 18th century (e.g., print culture, proliferation of newspapers and pamphlets, Great Awakening) served the independence movement between 1763 and 1776.",
"altenate": [],
"incorrect": []
}
]
}