The problem. As AI enters legal workflows, a quiet risk rides along: the model may reason differently about identical facts depending on whose name is attached. A defendant named "Tyrone" and one named "Thomas," same conduct, same law — but a subtly harsher inference. That bias is invisible in any single output, because you never see the counterfactual.
What it does. Legal Bias Auditor makes the counterfactual visible. You submit a real legal fact pattern. The tool detects the personal names, generates a pseudonymized twin of the scenario, then runs the identical AI legal analysis on both versions and compares them. Because the only variable that changes is the parties' names — same prompt, same facts, same model — any divergence in the legal conclusions is attributable to identity alone. It's differential testing applied to fairness.
How it works.
Local name detection — a BERT NER model finds person names on the server (the detection step never leaves the machine). Pseudonymization — names are mapped to neutral placeholders, with a deterministic scrub guaranteeing no real name leaks into the "blind" version. Twin analysis — the same analyst prompt runs on the real-name and pseudonymized scenarios in parallel. Comparison — the model contrasts the two outputs across legal dimensions (credibility, evidentiary sufficiency, application of law, characterization of the parties, risk), producing a 0–100 bias signal, severity-tagged divergences, and a plain-language executive summary. What you see. A dashboard of aggregate bias metrics across all audits; a guided submission flow with example scenarios; and a detailed report with a side-by-side "original inference vs. pseudonym inference" view, per-dimension severity badges, and the name mappings used.
What makes it different. Two things. First, the method — a rigorous counterfactual probe rather than a vibe check, designed so a missed name (which would smuggle an identity signal into the blind run) can't quietly corrupt the measurement. Second, radical transparency — a full-honesty disclosure page that states plainly what happens to your text, that audits are world-visible, and that this is a research instrument, not legal advice.
Stack. React + Vite, Express 5, PostgreSQL + Drizzle, OpenAI via Replit AI Integrations, local NER via transformers.js, contract-first OpenAPI codegen, pnpm monorepo.