Feedback Aide includes a content moderation agent, which is helpful for identifying inappropriate content in essays and short responses written by learners. Additionally, the agent's guardrails option masks PII in learner responses and applies independent safety guards to Feedback Aide feedback.

Types of content which will be flagged

AI-driven moderation helps ensure that sensitive, inappropriate, or crisis content in the learner response is flagged and managed efficiently. This reduces manual workload, improves accuracy, and enhances the overall user experience.

The content flagged includes:

• Expressions or promotion of hate towards any target.
• Threats of violence or harm towards any target.
• Depictions of violent or sexual acts.
• Depictions, promotion, or encouragement of acts of self harm.
• Disclosures that the learner is engaging or intends to engage in acts of self-harm, such as suicide, cutting, or eating disorders.

Types of PII which will be masked

The following categories of PII will all be masked if the moderation agent's guardrails option is enabled:

• Name
• Phone number
• Email address
• Address
• Username
• Password

Enabling content moderation

AI content moderation is not enabled by default. To activate it, developers must explicitly request it by passing the moderation evaluation option when creating the feedback session with the feedbackApp.feedbackSession() method such as below:   

const feedbackSession = await feedbackApp.feedbackSession(
    // security
    {
        ...
    },

    // feedback session options
    {
        state: 'grade',
        session_uuid: '36eebda5-b6fd-4e74-ad06-8e69dfb89e3e',
        stimulus: 'Write an essay about obesity and its impact on society',
        response: 'Obesity is ...',
        rubric: {
            ...
        },
        options: 
            agents: {
                moderation: {
                    inappropriate_content: {
                        enabled: true
                    },
                    critical_safety_content: {
                        enabled: true
                    },
                    guardrails: {
                        enabled: true
                    }
                }
            }
        }
    }
);

Each moderation agent option can be independently enabled. This ensures that moderation is only applied when needed, giving developers control over when to leverage AI moderation for their specific use case. 

Critical safety content looks for high risk content such as self harm, inappropriate content covers more general moderation like harassment, hate etc. and guardrails applies PII masking and an independent safety check to returned feedback.

This is the full list of possible moderation return types which are sorted into two main categories.

Critical safety content

• sexual_minors
• self_harm
• self_harm_intent
• self_harm_instructions

Inappropriate content

• hate
• hate_threatening
• harassment
• harassment_threatening
• violence
• violence_graphic
• sexual

Moderation workflow examples

Content flagging

When the essay is first graded by Feedback Aide, the user interface will display a warning message to the grader indicating it has detected content that may be of concern. The grader needs to acknowledge the message in order to continue.

When the grader has finished marking the essay and the feedback is ready for learner review, the grader will click ‘Submit to student.’ They will then be shown a second dialog window, asking for their acknowledgement before proceeding.

PII masking

If a learner includes PII in their response and guardrails is enabled, the PII will be masked such that Feedback Aide's scoring engine never sees the unmasked text. PII is still saved as part of the original response for display back to the learner.

Rogue feedback

Extensive testing of all Feedback Aide scoring engines and agents means there have been no reports in customer environments of unsafe feedback being returned to end users. However, for safety-conscious users it is possible to enable an independent safety check on all feedback before a client ever sees it. If the independent check detects any unsafe content, the feedback is not returned to the user.

Note: This independent safety check slightly increases the total amount of time it takes to fully score a response.