Learnosity's Item analysis reporting capabilities allow you to perform detailed analysis on your Item bank content. This can range from large scale analysis of your whole Item bank, and all of your sessions, to more focused analysis of a single Activity, or specific selection of sessions. Our solution aims to provide insight in the following areas:

• Performance of individual Questions - difficulty (p-value) and discrimination index.
• Identification of misleading or misconfigured Questions.
• Usage of and engagement with Questions across an Item bank.

Learnosity's Item analysis is implemented in Data API using datasets. There are two different dataset types used for Item analysis:

• ibk-analysis-by-question

  Item bank scale analysis across all Questions and sessions.

• activity-analysis-by-question

  Analysis of a specific set of identical sessions. All sessions being analyzed must contain exactly the same Questions (in any order). This allows the overall session scores to be compared, which is required for certain measures e.g. discrimination index.

Generating Item analysis datasets

For performance and scalability, Item analysis reports are generated asynchronously via Data API datasets. See the datasets implementation guide for details.

Examples

These are sample requests made to Data API's SET reports/datasets endpoint.

• Calculate p_value for all Questions across all sessions, sorted by p_value.

   

  {
      "dataset_type": "ibk-analysis-by-question",
      "organisation_id": 1,
      "file_count": 0,
      "options": {
          "default_sort_field": "p_value",
          "default_sort_order": "asc",
          "fields": [
              "organisation_id",
              "item_reference",
              "question_number",
              "question_reference",
              "count_sessions",
              "count_attempted",
              "p_value",
              "stddev_p_value"
          ],
          "summary_fields": [
              "count_questions"
          ]
      }
  }

   

• Calculate discrimination_index for Questions in sessions with a specific Activity ID, and within a date range.

   

  {
      "dataset_type": "activity-analysis-by-question",
      "filters": {
          "activity_id": ["ELA_comprehension"],
          "maxtime": "2017-07-31T23:59:59Z",
          "mintime": "2017-07-01T00:00:00Z"
      },
      "file_count": 0,
      "options": {
          "default_sort_field": "item_position",
          "default_sort_order": "asc",
          "fields": [
              "organisation_id",
              "item_position",
              "item_reference",
              "question_number",
              "question_reference",
              "count_sessions",
              "count_attempted",
              "p_value",
              "stddev_p_value",
              "discrimination_index"
          ],
          "summary_fields": [
              "count_questions",
              "count_sessions",
              "count_sessions_discarded",
              "count_sessions_analyzed"
          ]
      }
  }

   

• Calculate discrimination_index for a specific list of sessions. The session IDs are uploaded in a separate data file, as indicated by file_count. See step 2 of the implementation guide for further details.

   

  {
      "dataset_type": "activity-analysis-by-question",
      "file_count": 1,
      "options": {
          "default_sort_field": "item_position",
          "default_sort_order": "asc",
          "fields": [
              "organisation_id",
              "item_position",
              "item_reference",
              "question_number",
              "question_reference",
              "count_attempted",
              "p_value",
              "stddev_p_value",
              "discrimination_index"
          ],
          "summary_fields": [
              "count_questions",
              "count_sessions",
              "count_sessions_discarded",
              "count_sessions_analyzed"
          ]
      }
  }