Evidence-Grounded Interview Feedback Synthesis Copilot

Live Demo

The Problem

A hiring manager reviewing 10–15 candidates in a single sitting faces a structurally difficult task: each candidate accumulates 4–6 interview debriefs that vary widely in length, rigor, and signal quality. Three failure modes recur at scale:

• Key concerns are buried in uneven narratives — a red flag in paragraph four of a long debrief is easily missed under cognitive load

• Interviewer disagreements surface too late — after the decision window has closed rather than while there's still an interview slot to resolve the conflict

• Anchoring bias — the first debrief read shapes interpretation of every subsequent one, giving disproportionate weight to one interviewer's framing

A prior summarization pilot failed precisely because it smoothed over red flags. The goal here was to build something that surfaces evidence more clearly — not one that makes the decision easier to outsource.

System Architecture

The tool is a Python prototype using an LLM for structured JSON extraction, a citation-grounding layer that ties each extracted claim to a source span in the original debrief text, and a retrieval layer over approved role rubrics and interview guidelines so outputs are always evaluated against the right competency criteria.

The pipeline has six stages:

• Ingest & normalize — parse debrief text, map score scales across interviewers, prepare a per-candidate packet

• LLM extraction — structured JSON extraction using Claude, with one record per skill per interviewer (skill, score, scale, one-line takeaway, source span), validated via a Pydantic schema

• Citation & grounding — a grounding pass flags or suppresses takeaways not sufficiently supported by their cited span, using a combination of string overlap, embedding similarity, and NLI-based entailment checks

• Analysis layer — produces the three core outputs (digest, disagreement detector, coverage map)

• Decision support — evidence-organized summary with raw debrief accessible in one click; no scores, no rankings

• Evaluation & iteration — a frozen regression set of anonymized historical slates with expert-labeled red flags, disagreements, and strengths; release gates re-run on every model or prompt change

Three Core Components

1. Per-Candidate Structured Digest

For each candidate, the model extracts (skill, score, scale, one-line takeaway, source span) tuples into a Pydantic-validated JSON schema — one record per skill per interviewer. Structured output / tool-calling enforces schema conformance. A grounding pass then suppresses any takeaway not sufficiently entailed by its cited span, using human-audited thresholds to calibrate the false-positive rate.

2. Disagreement Detector

Extracted claims are mapped to skills in the role rubric. Within each skill, the system flags evidence-direction conflicts and score gaps beyond a set threshold — for example, one interviewer rating analytical reasoning strong while another flags difficulty with ambiguity. The same logic catches score-text mismatches within a single debrief, where a high numeric score sits alongside a narrative raising serious concerns. An LLM judge classifies flagged pairs as genuine disagreement, different evidence, or different skills; human review calibrates false positives. This component is highest value because it surfaces conflicts while a later interview can still resolve them.

3. Coverage Map

A skill-by-interviewer matrix — keyed to the role's required competencies — exposes which skills were assessed, by whom, and where gaps remain. A required competency that no interviewer assessed is flagged as an active prompt for follow-up before the loop closes. This directly addresses one of the most common failure modes in structured hiring: discovering post-decision that a critical competency was never actually probed.

Evaluation Framework

Output quality is gated before adoption and measured continuously against the labeled evaluation set:

• Faithfulness and omission rate (primary, release-gating) — faithfulness is the share of takeaways entailed by their cited span; omission rate is the share of human-labeled red flags absent from the digest. A hard ceiling on omission rate is required for any release, directly tied to the failure mode of the prior pilot

• Disagreement recall and precision — against human-annotated conflicts; recall guards against missed disagreements, precision guards against alert fatigue

• Citation validity — share of citations that resolve to a span actually supporting the claim; guards against plausible-but-wrong source attributions

• Adoption and reliance signals — time-to-decision per slate, raw-debrief click-through rate (a drop toward zero signals over-reliance), and edit patterns (repeated additions of missed concerns indicate weak red-flag extraction; repeated deletions of strong conclusions indicate an over-assertive prompt)

• Decision quality (long horizon) — correlation of surfaced takeaways with downstream performance and retention, interpreted cautiously given small N and long feedback loops

What the Tool Deliberately Does Not Do

These are design boundaries, not gaps:

• No overall candidate score — an aggregate score compresses away the evidence that makes a decision defensible, and pooling inconsistent per-interviewer scales has no sound statistical basis

• No cross-candidate ranking — ranking amplifies any selection or wording bias latent in the debriefs, which the model would inherit and entrench

• No hire/no-hire recommendation — the tool makes evidence easier to inspect, not the decision easier to outsource

• No protected-characteristic inference — free-form abstraction is where hallucinations and proxy discrimination can enter; the architecture stays extractive and citation-grounded by design

The Most Ambitious Version

The long-horizon version shifts from post-process summarizer to real-time interview orchestration assistant. As each debrief arrives during the interview loop, the tool updates a live evidence map showing which competencies have been strongly assessed, which remain thin or untested, and where interviewers disagree. If a required skill is under-assessed or two debriefs conflict, the system suggests a targeted follow-up area for the next interviewer — improving the evidence base before the decision is made, not just organizing it after.