Iris failure modes
Most Iris responses are confident answers backed by a clear catalog tool. The interesting cases are at the edges — when Iris doesn’t know, refuses, or asks. This page maps each failure mode + the right operator response.
Failure mode 1 — “I don’t have a tool for this”
The most common edge case. Iris tells you it can’t answer that from the data it has, usually with a suggestion to rephrase. The wording comes from the model, not a fixed template.
When this fires: no tool fits the question. Routing first narrows the catalogue by wording, by the page you are on and, if neither matches, by meaning. If nothing narrows it, the model is offered every tool your role allows — and when none of those fits either, it should say so rather than guess. A role with fewer tools sees this more often.
The right operator response:
- Rephrase first. A surprising fraction of “I don’t know” responses are recoverable by rephrasing. “Tell me about user activity” might fail; “audit event volume in the last 7 days” succeeds because the second phrasing matches the canonical metric name closely.
- Ask about the data area directly. For example: “from user access, list users who logged in during the last 30 days”. If your role holds
iris_sql, Iris builds a structured read-only query over that one view — it does not run pasted SQL text — and shows it under Query run. - File a feature request if you ask the same question regularly + Iris can’t catch it. The catalog grows from observed gaps; feature requests are how the team picks the next 5 metrics to add.
Failure mode 2 — Iris asks a clarifying question
When your question could reasonably mean several things, Iris may ask rather than guess:
Do you mean:
- All access requests in the last 30 days?
- Pending access requests only?
- Access requests for a specific app?
- Access requests by status (count breakdown)?
Pick one or rephrase.
When this fires: this is the model’s judgement, not a fixed rule — it tends to ask when a missing piece materially changes the answer. Common triggers:
- Missing time-window (Iris asks because “last week” vs “all time” produces dramatically different numbers)
- Missing status filter (e.g., “show me access requests” — pending? approved? all?)
- Ambiguous between counts vs lists (e.g., “tell me about users” — count or names?)
- Multiple compatible metrics (e.g., “agents” — could mean all agents, high-risk agents or agents pending review)
The right operator response: pick from the offered options. The question itself reveals which dimension Iris saw as ambiguous, so use that as the disambiguating term in future similar queries.
This is a feature, not a failure. Iris asks for clarification specifically to avoid producing a confident-sounding-but-wrong answer. A response that says “247 access requests” when you wanted pending-only (which is 19) is worse than the clarifying question.
Failure mode 3 — Routing miss (rare, but real)
Iris picks a tool whose output doesn’t match your intent. Symptoms:
- The number returned feels wrong (much higher or lower than expected)
- The cited tool name doesn’t match what you asked
- The source citation queries a table you didn’t expect
When this fires: the model chose a similar-but-wrong tool. Iris’s self-check catches one common case — a count question answered with a listing tool — and makes one corrected call before answering. Other misses surface to you.
The right operator response:
- Re-ask with the disambiguating noun. “Access GRANTS” not “access” if the latter routed to access-requests. “Pending access REQUESTS” not “pending access” if the latter routed to something else.
- Verify against the source citation. Data answers carry a “Based on” footer naming the source tables (and, for
iris_sql, the exact query). If the source looks wrong for your intent, the routing missed. - File the miss. If a specific phrasing reliably misroutes, file a feature request with the canonical phrasing — routing is improved when patterns emerge.
Failure mode 4 — Tool execution failure (timeout / error)
Iris picked the right tool, but the tool’s SQL execution failed:
Iris reports that the query failed and suggests narrowing it.
When this fires: the tool ran but timed out OR returned an error from the database (rare — usually transient). iris_sql queries are stopped after 5 seconds; a whole Iris turn is limited to 60 seconds.
The right operator response:
- Add a tighter filter (time window, specific user/app).
- Ask for a count or breakdown instead of every row, or for a smaller sample (“the latest 20”).
- If the timeout persists with reasonable filters, file a request — there may be a missing index that engineering can add.
Failure mode 5 — Cross-tenant safeguard fired (P0 incident, very rare)
Iris’s runtime tripwire on iris_sql fired. Iris tells you that the safety layer detected cross-tenant data and blocked the response.
When this fires: the final check on iris_sql results (see How Iris stays accurate) found rows belonging to another tenant. This is a P0 incident — the cross-tenant defense worked, but a near-miss occurred that warrants engineering attention. It is recorded in the audit log as IRIS_SQL_TENANT_LEAK.
The right operator response:
- Note the time of the question
- Report to AtmanID support immediately — every occurrence is investigated as a P0
- The response is blocked; no cross-tenant data was returned to you
Failure mode 6 — Confirmation declined or expired
For actions, Iris shows a confirmation card before executing. If you cancel it, nothing runs. A proposal is signed and valid for 15 minutes; confirming it later is refused with “This action has expired. Please try the action again.” A proposal can be confirmed only once.
When this fires: any Iris-mediated action (e.g., “suspend user X”, “approve request REQ-0042”) requires an explicit click to confirm. Only executed actions write their own audit records; a cancelled card leaves no action record.
The right operator response: if you intended to perform the action, re-ask and confirm. If you didn’t, cancelling is the right outcome.
Failure mode 7 — Usage limit reached
Iris counts questions per person and per organisation. The defaults are 120 per person per hour, 1,000 per person per day and 5,000 per organisation per day (your deployment may set different limits). When you hit one, Iris answers “You have reached your Iris usage limit for this hour” (or day), or tells you your organisation has reached its limit for today. Wait and try again, or ask an administrator.
How to recognize a low-confidence response (vs a confident one)
Iris doesn’t currently surface a numeric confidence score on every response, but you can read confidence from the response shape:
| Signal | What it means |
|---|---|
| Specific number (“78 users”) + clear tool citation | High confidence — trust the answer |
| Specific number + tool citation BUT tool name doesn’t match your intent | Routing miss — re-ask with disambiguating noun |
| “Approximately N” or “around N” | Iris is hedging — verify before acting on it |
| Clarifying question | Iris detected ambiguity — pick from offered options |
| “I can’t answer that” | Out of catalog — try rephrasing or a view query |
| Long prose answer with no numeric claim or table | Iris is generating text instead of running a tool — verify by re-asking with explicit “give me a count” or “show me a list” |
The last row is the most important. If you ask for IAM data and get a paragraph of explanation instead of a number or table, Iris probably failed to route to a tool and is generating answer-shaped text instead of querying real data. Always re-ask with explicit aggregation language.
Operator best practices for handling failures
- Always check the source citation. Data answers carry a “Based on” footer (and the exact query for
iris_sql). A source that doesn’t match your intent is your earliest signal of a routing miss. - Trust empty results — don’t assume Iris missed. “0 pending access requests” might mean you actually have zero, not that Iris failed. Verify by checking from a different angle (e.g., “show me access requests by status” — if Pending = 0, then 0 was correct).
- Cross-check with a second route. If a catalog tool returns an unexpected number, ask for the same figure another way (a different metric, a view query, or the dashboard page the footer links to) and compare. Discrepancies indicate a catalog bug worth filing.
- Don’t rephrase indefinitely. If 3-4 rephrasings all fail, the catalog genuinely doesn’t cover the question. Try a view query or file a feature request rather than continuing to rephrase.
Adjacent reading
- How Iris stays accurate — the four safeguards behind these failure modes
- Iris tool catalog — full reference for what’s in the catalog
- Iris query patterns cookbook — canonical phrasings that route reliably
- Building on Iris — for developers handling Iris responses programmatically
© AtmanID · Privacy