Solution Description
This solution goes beyond our typical reporting features to illustrate the use of more advanced analytics. These are used to create actionable reports and produce large-scale cohort analyses for Business Intelligence systems. The system incorporates data exports to 3rd party data warehouses for aggregate processing.
This solution is ideal for:
- Clients with advanced reporting needs
- Data-centric clients
- Clients who have a large number of users
Who:
- Want to extract real meaning from their assessment results
- Have custom reporting needs requiring access to raw data
Example company
OmniCorp produces a range of educational products for the K-12 market, with a particular focus on mathematics. In addition to printed textbooks, they provide companion web-based workbooks with interactive assessments. Typically, an assessment appears at the end of each chapter in the workbook. Each of these assessments, or activities, contains 20 questions. OmniCorp currently has 40 such assessments in its online system.
The existing content creation and delivery system allows OmniCorp to create multiple-choice questions, fill-in-the-blank questions, and some basic drag-and-drop questions. Some questions are presented alongside videos or images which the student refers to when giving their response. Reporting is in the form of a simple percentage grade per student.
Having grown its organization for 10 years, OmniCorp is now eager to bid for larger contracts at state and district levels. The specification for these contracts outlines a requirement that OmniCorp does not currently fulfill. District administrators need bidders to deliver advanced analytics functionality for their Business Intelligence systems. These include cohort analysis and detailed datasets for data warehouse processing.
There are three distinct areas where OmniCorp needs to improve:
- They need a mechanism to generate analytics on subsets of the subject matter covered in their existing assessments
- They need to deliver actionable reports and large-scale analytics across their entire customer base
- They need to make their aggregated data available for 3rd party systems to consume
Knowing how a student performed overall in an assessment is no longer sufficient. The teachers need to know what parts of a particular subject each individual student needs help with. On completion of a general math exam, the teacher should be able to tell at a glance if the student needs help with trigonometry, algebra, calculus, etc. They also need a view of how the class, as a whole, is doing in each area. At the district level, policymakers will need to examine hundreds of thousands of students across various schools to help them examine the effects of existing policies and identify areas for improvement.
Solution
OmniCorp can meet all of its requirements using the Learnosity APIs and hosted services. We will walk through an example where OmniCorp has used:
- The Author Site (using tagging for tag-based reporting)
- The Learnosity-hosted authoring environment will allow OmniCorp to author questions in a user-friendly instinctual manner
- This tool also provides a powerful and flexible tagging mechanism. This is an open taxonomy that allows OmniCorp to create any number of tags and tag hierarchies and then apply those to their content
- OmniCorp has a series of 40 existing assessments, these can be re-authored or imported into their Learnosity-hosted Item Bank
- Learnosity Math
- As OmniCorp is a math-focussed company they will also benefit from state-of-the-art math authoring and delivery capabilities through the Learnosity Math product
- The Items API
- Used to present the assessment to the student
- The Items API pulls items from OmniCorp's Learnosity-hosted Item bank
- Along with CSS styling and other customizations, will render a branded version of the assessment inside the clients existing web-based workbook
- Questions can be rendered one by one in an inline fashion, intermixed with the learning material. Alternatively, the questions can be rendered in a full-featured assessment 'player' which is typically placed at the end of each chapter in OmniCorp’s workbooks
- The Items API saves the student's work and submits it to the Learnosity servers for automatic scoring
- The Reports API
- Automatically builds a series of standard report types based on the results of OmniCorp’s auto-scored assessments
- Tag-based reports include information on which area of the subject matter the students require assistance with, on a class or individual basis
- The reports will be embedded in OmniCorp's back-end admin system, and can optionally be shown to students immediately on completion of an assessment
- The Data API
- A server-to-server API that gives the client access to all of their raw data
- Extract data to local or third-party servers for bespoke data warehousing operations as required by district administrators
- Data Analysis Engine
- A cloud-based data processing service used in conjunction with the Data API
- Can be used to create flexible and complex cohorts of students and compare them in browsable reports using up to 20 statistical measures (mean, median, percentile scores, standard deviation, etc)
Walkthrough
The Authoring Experience
Authoring of content (Items) can be done using the Learnosity Author Site. This requires no development work whatsoever. Users simply log in and start creating Items.
OmniCorp authors can create Items using 65 different question types. They can then tag those items, as mentioned above, tagging items during the authoring process allows us to generate complex actionable reporting and data analysis for all future uses of these items. All of this item data is stored in OmniCorp’s Learnosity-hosted Item bank.
Authors can create advanced math questions using Learnosity’s tools. Our Learnosity Math engine provides fast, flexible, and intelligent scoring and has been used to deliver well in excess of half a billion questions to date. Math authors can customize keypads to specific student needs, they can choose from over 200 symbols, or remove keys that students won’t need. All mathematics topics are covered including algebra, trigonometry, geometry, statistics, calculus, and interactive graphing.
The authoring tools can be used to create a series of 'activities'. An activity is a sequence of Items that are grouped together for the purpose of presenting them to the students as an assessment. OmniCorp will use these tools to recreate or import their existing 40 activities. All of this is handled by Learnosity, the client only needs to store references to those activities in their database.
The Assessment Experience
The assessment experience is embedded into the end of each chapter within OmniCorp's web-based workbooks. Users will then log in and take the tests. The assessment is handled by the Learnosity Items API.
The assessment experience powered by Learnosity provides optional configurable features such as timers, item counters, save/resume buttons, pause/resume buttons, calculators, rulers & protractors, video & audio players, item flagging & response masking, as well as a full-featured range of accessibility features. All math questions, with the exception of the math essay, are automatically scored when the student submits their responses.
Learnosity does not enforce a particular approach to user management. We only require that you send us a user Id each time an assessment is rendered. This open approach allows OmniCorp to build flexible rostering options such as routing a logged-in student directly to the latest assigned assessment or presenting a menu of available assessments. The Learnosity Items API displays the appropriate activity for them. It will take the Id of the appropriate activity and the Id of the logged-in user as two of its parameters. No personally identifying information (PII) is used.
When the student completes their end-of-chapter activity, their student responses are saved by Learnosity. The data are stored in the Learnosity Student Response Database.
The user Id, session Id, and activity Id can later be used to generate a range of reports for individuals or groups.
The Analytics Experience
Learnosity will use three mechanisms to deliver the analytics required by OmniCorp in this scenario.
- Tag-based reporting
- Cohort analysis
- Data warehouse exports
Tag-based reporting
The first, and perhaps the simplest, will be to use the Learnosity Reports API to deliver a suitable selection of our pre-made reports, including the tag-based reports to show weaknesses and strengths within a particular subject. These can be rendered in the back-end admin system where the teacher or administrator can view them. They can optionally be displayed to the student immediately upon completion of the assessment. This part of the client system we will refer to as the OmniCorp Analytics Environment which will use the Learnosity Reports API to render the reports.
To facilitate these reports OmniCorp stores the user Id, activity id, and tag hierarchy data in their database, these are then provided as parameters when calling the Reports API and reports such as the lastscore-by-tag-by-user can be displayed, giving a visual and numerical breakdown of each student’s performance.
Cohort Analysis
Learnosity offers a cloud-based analysis engine to facilitate cohort analysis, also known as aggregate reporting. This enables OmniCorp to compare and explore cohorts of hundreds of thousands of students, producing actionable reports for Business Intelligence systems. These are precomputed, Learnosity runs the numbers for you after you send a request. These reports are visualized as a browsable hierarchical structure of the cohorts defined by the client, yet no PII (Personal Identifiable Information) is required to be held on Learnosity's side.
An example report might examine the median and percentile algebra scores for students with a 95%+ school attendance record, and compare these to cohorts of students with 90%, 80%, and those with less than 70% attendance records. Learnosity does not store students' names or attendance records but we provide a mechanism by which this data can be received as part of an analytics request and subsequently used in your analytics.
OmniCorp sets up an analysis request and then provides anonymized student data, that data includes the user Id which allows Learnosity to relate the anonymized data to the student response data in our database. We then apply the statistical measures requested and schedule the analysis. When complete the Analysis Engine will pass the results back to the client in the form of a browsable report, which is rendered using the Reports API inside of the OmniCorp analytics environment. If raw data is preferred that can also be delivered.
Data warehouse exports
District administrators require that student data be exported to centralized data warehousing applications. OmniCorp would build a back-end data processing system for this purpose. This system would use the Data API to pull data from the Learnosity Student Response Database and push that data to the 3rd party data warehouse application. The system could optionally pull data back if needed.
The three processes described above allow OmniCorp to meet all of the analytics requirements.
Summary
OmniCorp can meet all authoring, assessment, and analytics requirements and take part in district and state-level RFPs using Learnosity’s products.
In this final diagram, we see the entire system:
OmniCorp's step-by-step tasks
Here are the steps OmniCorp needs to follow to integrate this plan:
- Import or author the items needed for their 40 assessments. Assemble these into 40 'activities'. The Learnosity Authoring team will provide training for this. Store the activity Ids in their database.
- Add an HTML container at the end of each chapter in their web-based workbooks where their logged-in users can take an assessment. Initialize the Items API using the required Ids, which include user Id and activity Id from the OmniCorp database
- Add a section in the administrator's area where the teacher can view reports from the Reports API
- Build a backend processing system to perform these steps:
- Use the Data API to retrieve Student Responses and export this data to the 3rd party data warehouse
- Set up a request for a Learnosity cohort analysis
- Create hierarchical cohorts of OmniCorp’s data on their students and pass this to the Analysis Engine
- Notify the user when the cohort analysis has been completed and display the resulting report
Return to the index of technical solutions.