periscope
A language model screened 832 research abstracts for a literature review. I built Periscope to audit its decisions — reviewing each verdict and deciding whether to keep or overturn it.
The model worked from titles and abstracts alone, returning a verdict, rationale, and supporting quote for each paper. My job was to decide which calls to trust.
When AI does the first pass, human attention becomes the scarce resource. The interface should direct it to the decisions most likely to need review.
One paper under audit — worklist context, model output, and the verdict controls in a single view.
The demo runs on a 30-paper sample. Every number on this page comes from the full 832-paper screening.
The problem
Reviewing 832 AI decisions one by one defeats much of the value of automating the first pass. The interface needed to help me decide where to look first and make each decision faster to verify.
The screening pipeline produced useful signals: confidence, rule violations, unresolved criteria, and cases where the model found evidence against its own verdict. Periscope uses those signals to organize the review around risk rather than sequence.
Start where the risk is
Rather than treating all 832 decisions equally, Periscope points me toward the ones most likely to need attention. At the top of the queue, it recommends starting with high-risk worklists such as Excluded, but model pushed back and Low-confidence rows.
The filter menu exposes additional signals, including Maybes still carrying a criterion and Linkage-rule violations. Each worklist shows its size and groups papers by why they may need closer review.
For example, 164 excluded papers contained evidence that pushed against the model’s own verdict. Instead of finding those conflicts manually, I could pull them into a review queue with one filter.
Checking the model against the source
Evidence and counter-evidence are paired for comparison, then highlighted in the abstract for verification. The pair makes conflicting evidence easy to compare; the highlights make each quote easy to check against its source.
Not every claim can be highlighted. Some exclusions depend on something being absent from the paper — for example, no user study or no visual component. In those cases, Periscope places the model’s absence claim beside the exclusion criterion, making it easy to check whether what the model says is missing actually matches the rule it applied.
The goal is to make verification almost as easy as accepting the model’s output, while keeping the limits of that verification visible.
Excluding asks for a reason
Including simply carries a paper forward. Excluding removes it from the review, so it requires a reason.
Choosing Exclude opens a criterion picker, pre-filled with the criterion the model applied. The reviewer can confirm it or choose another before saving the verdict.
The pre-fill keeps the interaction fast, while the required confirmation separates two decisions that are easy to collapse into one: Was the model’s verdict right? Was its reason right?
It adds friction only where the consequence is highest — when a paper leaves the review.
Signals can disagree
Periscope keeps confidence visible, but secondary to the evidence behind the prediction.
The screening prompt instructed the model to lower its confidence when it found counter-evidence. In practice, the outputs did not always follow that rule: 59 exclusions had both counter-evidence and high confidence.
Reviewing those cases revealed another problem. Some high-confidence exclusions were correct; the supposed counter-evidence was the weaker signal. In one case, the model flagged language about an “intuitive interface” as counter-evidence, but checking the full paper confirmed there was no user study or interface evaluation.
Rather than resolving these signals into a single score, Periscope exposes them separately. Confidence tells me how certain the model was. Evidence and counter-evidence let me inspect why. The final judgment remains mine.
Two views, different jobs
Reviewing these cases one at a time suggested that the exclusions were often correct. The grid made it possible to see whether that was a pattern.
Filtering for exclude + high confidence + counter-evidence brought the cases together. As I reviewed them, the verdict column began telling a consistent story: I was repeatedly confirming the exclusions.
This is why Periscope has two views over the same queue. The card brings together everything needed to inspect one decision. The grid makes patterns across decisions visible.
In this case, the grid shifted the question from Why is the model so confident despite counter-evidence? to Is the counter-evidence itself the unreliable signal?
Card to decide. Grid to diagnose.
When AI made the interaction worse
I prototyped a natural-language control for filtering the grid and choosing columns. It seemed like a good use of AI: describe the view you want instead of configuring it manually.
In practice, it was slower and less reliable than the controls it replaced. It sometimes missed conditions or interpreted them incorrectly, while simple filters and column toggles were predictable and immediate.
I removed it.
Natural language reduced the effort of specifying a view, but increased the effort of verifying that the system understood it.
