OpenEnv · Legal operations

LexCrisis

A reinforcement-learning benchmark that scores agents on running a legal matter — conflicts, privilege, deadlines, adversarial discovery — using four deterministic verifiers instead of an LLM judge.
Its headline result was wrong. I found that myself, and this page opens with it because it is the most useful thing I can tell you about the benchmark.
3
4
14
13
Indian — Bharatiya Sakshya Adhiniyam 2023, Bar Council rules

The audit

What I reported, and what was actually true
14 issues filed 13 closed 1 open — needs a GPU run

Numbers currently defensible

The scripted oracle, re-run against the fixed grader. Base and fine-tuned figures are deliberately absent: no rollout traces are committed yet, so nothing about a model can be checked. They return when the traces do.
Client intake Easy
Screen six prospective clients for conflicts, cite the governing Bar Council rule, and decide accept or decline.
0.999
24
Privilege review Medium
Classify ten documents, name the doctrine, produce or withhold, and identify crime-fraud and at-issue waiver.
0.9391
34
Incident command Hard
Six events, five hard deadlines, one ethical trap and one adversarial push, under a fixed step budget.
0.9825
24

Read the scenarios

Every scenario is synthetic, written by someone who has never seen a real matter file. They are published in full — six intake clients with their conflict rules, ten privilege documents with their ground truth, six crisis events with deadlines — so the flaws are findable without reading Python.
BSA s.132Professional communications — attorney–client privilege
BSA s.134Confidential communication with legal advisers
BCI 22, 33Bar Council conflict rules driving the intake decisions
The classification vocabulary includes work_product, a US doctrine with no clean Indian equivalent, and the procedure throughout is US-style discovery. This may be US process wearing Indian citations.

How scoring works

Four deterministic verifier columns. No LLM judge anywhere in the loop.
outcome_correctness
process_compliance
deadline_or_latency
safety_or_anti_cheat
Producing a privileged document into discovery costs −0.12.

Action Console

POST /reset POST /step GET /state GET /episode

Trajectory


            

          

        

      


      
    


    

      

        

          
Who built this

          

            Natik Chhabra, Class 12, Delhi Public School Bilaspur, India. Qualified Round 1 of the Meta
            PyTorch OpenEnv Hackathon. Looking for evaluation work someone else would otherwise not get to
            — including the dull kind. Five to ten hours a week in term, full days in holidays.
          

        

        

          Repository and issue log
          SCENARIOS.md — the legal content