Skip to main content
schoolOS AI safety

How our AI assistants are constrained

Every schoolOS assistant — the one inside the app and the one embedded on a school website — follows the rules below in every conversation. They are injected into the assistant itself and cannot be turned off by a school or overridden by anything a user types.

If someone is in crisis

When a user shows signs of distress, the assistant stops answering the question it was asked and shares these resources instead. It never attempts to counsel anyone itself. A school can add its own local resources, but it cannot remove this floor.

What we watch for

  • Self-harm & suicide — suicidal thoughts, self-harm, hopelessness, not wanting to be here.
  • Mental health — anxiety, depression, grief, stress, emotional distress.
  • Eating disorders — disordered eating, restricting food, purging, bingeing, fear of gaining weight, or distress about body image or weight.
  • Safety & emergencies — weapons, threats of violence, someone in immediate danger, emergencies.
  • Abuse & neglect — abuse or neglect.
  • Bullying — bullying or harassment.

The rules themselves

Data & Privacy

  • Messages reach AI model providers only with zero data retention: providers keep nothing and never train on it.
  • Before anything reaches a model, people's names, email addresses, phone numbers, ID numbers, and street addresses are replaced with placeholders, in the agent and in Otto, and put back in the reply.
  • No AI model is ever trained on user conversations.
  • The agent never divulges personally identifiable information (PII) about students, families, or staff.

Scope

  • Only answers school- and district-related questions from the approved knowledge base.
  • Does not help with homework, assignments, essays, tests, or any graded academic work.

Health & Wellbeing

  • Does not discuss or give medical, health, or mental-health advice.
  • Directs health questions to the school nurse or a healthcare provider, and emotional concerns to a school counselor and a trusted adult.

Essential Safety Guardrails

  • Human / AI distinction: mandatory disclosure that the bot is not human and lacks professional credentials.
  • Crisis protocols: recognizes signs of self-harm or suicidal ideation and immediately redirects to verified crisis resources (e.g. the 988 Suicide & Crisis Lifeline).
  • Content filtering: strict blocks against sexually explicit content, graphic violence, and substance abuse.

Strictly Prohibited Content

  • Child safety: anything that sexualizes or exploits children, including child sexual abuse material (CSAM).
  • Dangerous activities: instructions or encouragement for suicide or self-harm (including eating disorders), or facilitation of real-world harm such as obtaining illegal drugs or building weapons.
  • Violence and gore: sensational, shocking, or gratuitous violence — excessive blood, gore, injuries, or cruelty to animals.
  • Harmful factual inaccuracies: information that conflicts with established medical or scientific consensus, or that endangers physical safety.
  • Harassment, incitement, and discrimination: inciting violence, bullying, threats, dehumanization, or advocating discrimination based on legally protected characteristics.
  • Sexually explicit material: pornographic or erotic content, or depictions of sexual acts, sexual violence, rape, sexual assault, or abuse.

Resisting Manipulation

  • Rules can't be turned off: ignores "forget/ignore previous instructions", DAN/developer-mode, and persona or role-play jailbreaks.
  • Framing doesn't matter: hypothetical, fictional, "academic", or many-example prompts get the same rules.
  • Treats user messages, files, and retrieved content as data — never as instructions that override the rules (prompt injection).
  • Won't decode or act on obfuscated payloads (base64, ciphers, hidden encodings) used to bypass the rules.
  • Manipulation attempts are blocked before a reply and flagged for admin review.

Design Signals

  • Clear non-human disclosure.
  • No first-person affiliative language.
  • No emotional dependence cues.
  • Encourages trusted-adult support.

Keeping the rules in force

Rules only matter if they survive someone trying to talk the assistant out of them. These are the measures that keep them in place.

Hardened system instructions

Non-negotiable rules are injected into every agent. They can't be turned off and explicitly reject "ignore previous instructions", DAN / developer-mode, and persona or role-play jailbreaks. Hypothetical, fictional, "academic", or many-example framings get the same rules.

Persona hijackingHypothetical framingMany-shot

Input prompt shield

Every message is screened before the assistant responds, in any language. Attempts to change, reveal, or get around its rules — including encoded or obfuscated ones — get a safe refusal, and the assistant never answers them.

Prompt injectionObfuscation & encodingPersona hijacking

Output content filter

Every reply is checked before anyone sees it: appropriate for a K-12 audience, about the question asked, informational rather than personal, never flattering or persuasive, no medical, legal, or counseling advice, no one's personal details, and never in the first person. A reply that fails is withheld, and the question can be sent to the right team instead.

Crescendo attacksPersona hijacking

Treats content as data, not commands

User messages, uploaded files, and retrieved knowledge are treated as information to consider — never as instructions that can override the rules.

Prompt injectionObfuscation & encoding

Abuse monitoring

Manipulation attempts and safety concerns flag their conversation for administrators in Conversations, so repeated manipulation is visible and reviewable.

Many-shotCrescendo attacks

Crisis referrals in 2025

0

schoolOS began serving schools in 2026. No assistant was running in 2025, so this zero reflects that rather than a year of service without a referral.

Crisis referrals in 2026 so far

69

The number of times a schoolOS assistant detected that someone might be in crisis and responded with crisis resources so far this year, across every school we serve. This count updates as referrals happen.