See how schoolAgent is doing

Admin › Agent › Analytics shows how your agent is doing: whether people find its answers helpful, what it sent to a person, and where its knowledge falls short. Check it after you change the agent's knowledge or prompt, and every few weeks.

You need Manage the agent to open it.

What it counts#

• Everything so far. There's no date range: the numbers cover every conversation since the agent started, except deleted ones.
• Your reach. A district-level role sees its district's conversations, and a school-level role its schools'.
• schoolAgent only. Website chats and signed-in chats, together. Otto conversations aren't counted.
• Ratings come from people. Answers only count as helpful or not helpful when someone selects thumbs up or down. Most people don't, so treat the percentages as a sample.

Refresh reloads the numbers.

Overview#

• Rated helpful. Of the answers someone rated, the share marked helpful, with how many that is. It shows a dash until anyone has rated an answer. It measures what people thought, not whether the answer was correct.
• Questions answered. How many answers the agent has given.
• Marked helpful. How many answers were rated helpful.
• Awaiting feedback. Answers nobody has rated.

Once someone has rated an answer, a bar under the numbers shows helpful against not helpful, with how many were rated and how many are still waiting.

By agent#

A table of each agent: the share of its rated answers marked helpful, how many were helpful and not helpful, and how many were rated. Select a column heading to sort. Empty until someone rates an answer.

Routed questions#

A table of the questions most often sent to a person, with the route they went to and how many times. Search it, or sort by any column. It shows the top 5, taken from your most recent 100 routed conversations. Empty until the agent has routed something. To change where questions go, see Route questions to a person.

Content safety flags#

Not shown for now. Flags would mark conversations as inappropriate for K-12, but schoolOS doesn't set them on conversations yet, so every count would be 0. The page will show them once it does.

To look for safety messages now, use the Safety Concern sentiment on Admin › Agent › Conversations, and check that the School Safety Team route has an email so crisis alerts reach someone. See School Safety Team.

Experience rating#

How people rated their whole chat, from 1 to 5 stars: the average, the number of ratings, and how many gave each score. Empty until someone rates a chat.

Knowledge suggestions#

A table of questions whose answer someone marked not helpful, most often first, up to 8, with similar questions people asked. Search it, or sort by how many times. Each is a sign the agent needs better knowledge on that topic. It's empty until someone marks an answer not helpful; questions nobody rated don't appear here.

The same question asked in different words shows up separately, because questions are grouped by their exact wording. A question stays on the list after you fix its knowledge: there's no date range, and ratings can't be dismissed.

To fix one, add or improve a source, or write a verified answer, on the Knowledge tab of Admin › Agent › Configuration. The Health part of that tab also lists questions the agent answered with low confidence.

Examples#

• In the week after winter break, Knowledge suggestions shows "When do report cards come out?" with a count of 3. The agent's knowledge has last year's grading calendar. Replace the source on the Knowledge tab. The question stays on this list afterwards, because old ratings are never cleared: judge the fix by whether its count stops growing, and by the Health part of the Knowledge tab.
• A district with 12 schools sees "My Chromebook won't turn on" at the top of Routed questions, sent to its IT route. That's routing doing its job. If it's most of what's routed, a verified answer with the first troubleshooting steps can save the IT team a ticket for the easy cases.