Implementing and Creating Datasets for Download or Reporting

This series describes the technical implementation steps to generate and retrieve a dataset. The dataset generation process is a series of steps using Data API endpoints. Once generated, the dataset can be downloaded, or used with Reports API to render a completed report, depending on the dataset type.

 

A dataset is a set of data files (sometimes many thousands of files) containing aggregated results and analysis in JSON and CSV formats. The exact contents and format of a dataset will vary depending on the dataset type. The dataset is generated asynchronously using Data API. Once completed, the dataset can be retrieved via Data API or rendered in-browser using Reports API, if the dataset type supports it.

 

Report generation is performed asynchronously as follows:

1. Initialize a new dataset using Data AP (see Step 1, below): Create and configure a new dataset. You must specify a dataset_id when you do this, in UUID format. The endpoint will return one or more URLs to which input files need to be uploaded.

If input files are not required to be uploaded, the dataset_type and a job_reference are returned, and you can proceed directly to Step 3 below.

2. Upload input files:

a. Upload input files (see Step 2a, below). Upload the input files required for the dataset. The data is uploaded to a specific URL returned from Step 1.

b. Commence dataset generation job using Data API (see Step 2b, below). Notify Data API that input files have been uploaded and kick off the dataset generation job.

3. Poll for job completion using Data API (see Step 3, below). The status of the dataset generation job can be obtained by polling the /jobs endpoint.

4. Retrieve results via Data API or Reports API (if applicable, see Step 4, below). On completion, raw report data can be retrieved by your server application using Data API. If the dataset type has a corresponding report type in Reports API, the report can render the new dataset, and raw data can also be accessed from the Reports API public methods.

 

Note that you can also refresh an existing dataset (see heading under Step 1, below) with new group and/or new score data.

 

Implementation Step 1: Initialize dataset

Initialize a new report dataset using the Data API SET /reports/datasets endpoint. The request data specifies the configuration for the dataset including which fields to calculate.

The specific request and response format depends on the type of dataset being generated.
Note: you must specify a dataset_id when you do this, in UUID format.
See the attributes relevant for each type of dataset:

The endpoint returns a dataset_id for the new dataset.

If you are required to upload input files as part of your request, as indicated by the file_count param, the response also provides a list of input_files URLs, which are used in Step 2.

If you are not providing input files, the response will contain a job_reference, and you can proceed to Step 3.

 

Refresh an existing dataset

Note that you can also refresh an existing dataset with new group data, and then regenerate it based on the latest score data. To do so, simply include the dataset_id as a parameter in your initialization request.

 

{
    "dataset_id": "ce6c3842-5366-486d-a68f-ab9e6160e9de",
    "dataset_type": "activity-summary-by-group",
    "file_count": 1,
    "filters": {
        "activity_id": [
            "20170106b_ELA_comprehension"
        ]
    },
    "options": {
        "default_sort_field": "mean_percent",
        "default_sort_order": "desc",
        "fields": [
            "group_count",
            "population",
            "lowest_score",
            "highest_score",
            "mean_seen_activities",
            "mean_attempted_percent",
            "mean_score",
            "median_score",
            "p25_score",
            "p75_score",
            "p90_score",
            "stddev_score",
            "lowest_percent",
            "highest_percent",
            "mean_percent",
            "median_percent",
            "p25_percent",
            "p75_percent",
            "p90_percent",
            "stddev_percent"
        ],
        "default_users_sort_field": "score",
        "default_users_sort_order": "desc",
        "user_fields": [
            "score",
            "attempted_max_score",
            "unscored_max_score",
            "max_score",
            "seen_activities"
        ]
    },
    "groups": [
        {
            "key": "district",
            "label": "District"
        },
        {
            "key": "school",
            "label": "School"
        },
        {
            "key": "class",
            "label": "Class"
        }
    ]
}

 

{
  "meta": {
    "status": true,
    "timestamp": 1483585276
  },
  "data": {
    "dataset_id": "ce6c3842-5366-486d-a68f-ab9e6160e9de",
    "input_files": [
      "https://learnosity-reportdatasets-va.s3.amazonaws.com/reports/0034/activity-summary-by-group/ce6c3842-5366-486d-a68f-ab9e6160e9de/infiles/0.ndjson?AWSAccessKeyId=AKIAJB5XQL2VQTD4KG6Q&Expires=1483588877&Signature=QUXjR0beW83bB7l1OpiTMOKRsKc%3D"
    ]
  }
}

 

Implementation Step 2(a): Upload input files

This step involves uploading the list of users and sessions that should be included in this report, and their group information, if applicable. The data is uploaded as one or more NDJSON files.

  • Upload the NDJSON files with an HTTP PUT request to the input_files URL(s) returned in Step 1.
  • Set the Content-Type: application/x-www-form-urlencoded header on the PUT request.
  • Set the Expect: 100-continue header if your HTTP client supports it, to avoid transmitting unneccessary NDJSON data in certain error cases. See the Expect-continue handshake optimisation introduced in HTTP 1.1.

The signed input_files URLs expire 60 minutes after they're issued. If one or more signed URLs have expired, start again from Step 1 to initialize a new dataset_id.

The expected format of your NDJSON file can be found in the documentation for the dataset type you are creating:

 

The data can be uploaded using a simple Curl call, or the HTTP client of your choice. For example, the following command will upload the users.ndjson file from a Curl-enabled terminal:

$ curl --request PUT --header "Expect: 100-continue" --header "Content-Type: application/x-www-form-urlencoded" --data-binary "@users.ndjson" "https://learnosity-reportdatasets-va.s3.amazonaws.com/reports/0034/activity-summary-by-group/a951cc14-0316-4c12-bb13-1f386338a094/infiles/0.ndjson?AWSAccessKeyId=AKIAJB5XQL2VQTD4KG6Q&Expires=1484208440&Signature=lEANv4k3aIK1CelRhXsYHgXNTDQ%3D"

 

To debug issues with constructing a valid PUT request, it may help to inspect the raw request content and compare it to the following sample request produced by Curl:

PUT https://learnosity-reportdatasets-va.s3.amazonaws.com/reports/0034/sessions-summary-by-group/a951cc14-0316-4c12-bb13-1f386338a094/infiles/0.ndjson?AWSAccessKeyId=AKIAJB5XQL2VQTD4KG6Q&Expires=1484204382&Signature=3zqRDJvtif%2B5wYmdCsWXHZLCkrM%3D HTTP/1.1
Host: learnosity-reportdatasets-va.s3.amazonaws.com
User-Agent: curl/7.43.0
Accept: */*
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded
Content-Length: 10376

{"user_id":"bart_simpson",           "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"milhouse_vanhouten",     "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"nelson_muntz",           "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"martin_prince",          "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"wendell_borton",         "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"lewis_clark",            "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"sherri_mackleberry",     "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"terri_mackleberry",      "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"adrian_belew",           "group_path":["Simpson","Springfield High","7_krabappel"]}
{"user_id":"lisa_simpson",           "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"ralph_wiggum",           "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"janey_powell",           "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"allison_taylor",         "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"alex_whitney",           "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"francine_rhenquist",     "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"hubert_wong",            "group_path":["Simpson","Springfield High","7_hoover"]}
{"user_id":"maggie_simpson",         "group_path":["Simpson","Cypress Creek College","7_doyle"]}
{"user_id":"dot_warner",             "group_path":["Simpson","Cypress Creek College","7_doyle"]}
{"user_id":"gordy_scribe",           "group_path":["Simpson","Cypress Creek College","7_doyle"]}
{"user_id":"warren_watson",          "group_path":["Simpson","Cypress Creek College","7_doyle"]}
{"user_id":"melvin_earnest",         "group_path":["Simpson","Cypress Creek College","7_doyle"]}
{"user_id":"gary_gray",              "group_path":["Simpson","Cypress Creek College","7_doyle"]}
{"user_id":"draco_malfoy",           "group_path":["Simpson","Hogwarts School","7_slytherin"]}
{"user_id":"vincent_crabbe",         "group_path":["Simpson","Hogwarts

 

Implementation Step 2(b): Commence dataset job

Once the input files are uploaded, commence the dataset generation job using the Data API SET /jobs/reports/datasets endpoint. This call validates the provided report configuration, input files and commences the dataset aggregation processing. The request object contains the dataset_type and dataset_id from step 1.

 

Sample request

{
    "dataset_type": "activity-summary-by-group",
    "dataset_id": "e63de7cf-8a11-4b87-9a98-aba1af4e5340"
}

 

Sample response

{
    "meta": {
        "status": true,
        "timestamp": 1474336936
    },
    "data": {
        "job_reference": "402ead85-5a27-4d40-b68b-7b1e77924ed4",
        "dataset_type": "activity-summary-by-group",
        "dataset_id": "e63de7cf-8a11-4b87-9a98-aba1af4e5340"
    }
}

 

 

Implementation Step 3: Poll for job completion

Use the Data API GET /jobs endpoint to poll the dataset generation job's status. Pass the job_reference returned earlier. A status of "completed" indicates that the dataset has been compiled successfully and can now be retrieved using Reports API and Data API. A status of "halted" indicates that the job has not completed successfully. Consult the error property of the job for further information.

For sample request and response objects, visit the GET /jobs documentation.

 

Implementation Step 4: Retrieve data

Raw data from compiled datasets is accessible via the Data API GET /reports/datasets endpoint. Some dataset types have a corresponding Reports API report type, allowing the dataset to be retrieved and explored interactively using the in-browser GUI in Reports API.

 

The raw dataset files can be retrieved via Data API using the GET /reports/datasets endpoint. The endpoint returns pre-signed URLs. Send an HTTP GET request to the pre-signed URL to retrieve a JSON/CSV file. See the documentation for the dataset type you're creating for more details:

 

See the Reports API documentation for the supported report types corresponding to these datasets:

 

Related articles


Did you arrive here by accident? If so, learn more about Learnosity by clicking here.