AI screening modes
Off, advisory, second reviewer or auto-exclude. What each mode does, the settings around it, and how to check the AI's performance before you rely on it.
On this page
Set the AI's role in screening on the review's AI tab.
The roles
| Role | What it does |
|---|---|
| Off | No AI screening |
| Advisory | The AI's suggestion is shown to screeners (see Reveal) but is not a vote |
| Second reviewer | The AI's decision counts as one of the required votes, so one person plus the AI completes dual screening |
| Auto-exclude | Titles and abstracts the AI excludes with a confidence above the threshold are excluded without a human vote; everything else needs people. PRISMA reports them as "records marked as ineligible by automation tools" |
The settings
- Reveal: when screeners see the AI's suggestion: always, only after their own vote (recommended: it avoids anchoring), or never.
- Threshold: the minimum confidence for an automatic exclusion. Higher is safer.
- Allow maybe: off by default. When on, the AI may answer "maybe" for records it cannot decide; when off, it must choose include or exclude and expresses doubt through a lower confidence.
- Mode: structured puts the records in the prompt and checks the JSON answer; agent lets the AI work through the site's API with a token that is valid for that job only (good for large batches). Full-text screening always uses the agent so the AI can read the PDFs.
Every AI judgement stores the model, the prompt's fingerprint, the confidence, the rationale and a verdict on each numbered criterion.
Check before you trust it
The AI tab compares the AI's decisions with the final human decisions on the same records:
- Sensitivity: the share of relevant records the AI included. A missed include is the costly error.
- Specificity: the share of irrelevant records it excluded.
- κ between the AI and the people.
"Maybe" counts as include here.
Before using auto-exclude, screen a sample by hand and check the AI's sensitivity on it. An exclusion the AI makes alone is reported as such in the paper.