SHYWAI / How reports work
How SHYWAI reads your work
A report helps you return to specific decisions, question the interpretation and choose what to practise next.
Last reviewed:
Start with what was recorded
You read engineering materials and work through your approach in a text conversation with an AI assistant. An Interviewer provides structured checkpoints. In scenarios that include a stakeholder, you can ask questions in that channel too.
Your answers, messages, AI replies and reflection become the record. There is no built-in code execution. Describing a check does not confirm an executed test, and activity outside the recorded interaction is not observed by the platform.
See the illustrative conversation and report. It uses invented material, separate from the live scenario catalogue.
From a record to assessments
AI reviews interpret the recorded run across six dimensions: Problem Framing, AI Collaboration, AI Output Review, Debugging & Verification, Communication & Ownership, and Risk & Production Readiness.
Each report is built from at least two separate AI reviews, produced without receiving the other reviews' verdicts. Their exact number and models depend on the configured review policy. Where the assessments agree, the report combines them; where they differ, you see each assessment and its reasoning. Agreement is not proof of correctness: models can make similar mistakes.
The private report uses four qualitative labels. They concern a dimension in this run, not your overall level or suitability for a job. There is no overall numerical score or ranking of people.
- Strong, Adequate, Weak
- Qualitative assessments of the behaviour the review interprets in this scenario. Read the reasons and the cited material alongside the label; these are not percentages or seniority levels.
- Not enough evidence
- The record did not provide enough support for an assessment. It does not mean Weak.
A disagreement you can examine
Illustrative reviews, not a real result
The empty list used to return 0. This version returns None. I would check None, [] and a non-empty list before accepting it.
Review A: Strong
The participant identified a concrete counterexample to the assistant's claim and proposed relevant checks.
Review B: Adequate
The counterexample is useful, but the proposed checks still need explicit expected results before the verification plan is complete.
These are possible interpretations of the invented fragment, not a rubric for assigning a label from one sentence. Neither review proves that a test was run. A useful next question is: what should each check return, and why?
What checking a quote means
Dimension quotes in the report and on a public page, and Process notes, are checked against the participant's recorded messages. This helps filter out unsupported quoted text. The check uses normalised word sequences and has technical limits; it does not independently authenticate a person or guarantee an exact match to one complete message.
The labels, the suggestions and the summaries are generated by the AI reviews. None of them goes through the quote-origin check. Under each dimension, only the checked quote is shown, without the review's commentary around it. A supported quote does not prove the interpretation is correct.
When an observation surprises you, return to the quoted moment and its context. Ask whether the record supports the claim, what information was missing and what another explanation could be.
How answers were entered
We record how each answer to the interviewer was entered: typed, or inserted in larger pieces. Paste is blocked in that field; the workbench is where AI help belongs. These are facts about input, not a judgement, and they do not change any level. Dictation counts as inserted, because the browser cannot tell it apart from pasting. They are reported by your browser and can be imitated, and someone who retypes an AI's answer by hand looks the same as anyone else. We do not claim to detect AI use.
What one run can tell you
A run shows behaviour in a particular situation, with particular materials and AI assistance. Different scenarios, assistants or review models may produce different observations; comparability is not guaranteed.
Scenario content is currently fixed rather than automatically mutated for every start. Repeated practice can help you learn, but prior familiarity changes the meaning of a result. A public page does not establish that this was your first encounter with the task.
Why active tasks stay closed
Knowing hidden conditions and expected findings in advance changes the exercise. We explain the dimensions and reporting process publicly while keeping active tasks' hidden material and expected findings closed. The example on this site is a separate teaching illustration.
What scenario calibration checks
Scenarios marked as needing it go through internal calibration before they are published; some short practice scenarios do not. Calibration checks scenario design: how much a model finds from the visible materials on its own and, for scenarios with a stakeholder, whether hidden stakeholder facts can be guessed without asking. Its metrics describe a task under specific test conditions, not a participant's score.
Calibration is not required for every scenario, and its results belong to the version and the rules in force when that version was published: a version published under earlier rules keeps its earlier results. Calibration does not prove that no AI can solve a task, or that report labels predict real-world job performance. This page does not establish statistical validation of those labels. Explaining the method is not evidence of its accuracy.
Use the report to make your next decision
Pick one observation. Check it against the record, distinguish what you intended from what you demonstrated, and choose one practice to try on a new task.
If you share your page, use it to start a concrete conversation: what did you know then, which assumption did you question, and what would you verify next in a real environment?