The accuracy number everyone quotes — and what it doesn’t tell you

When CRE teams evaluate AI lease abstraction, the first question is almost always about accuracy. Vendors respond with numbers: 95%, 98%, sometimes 99%.

Those numbers are meaningful. But they answer the wrong question.

The real question is not “how often is AI right?” It’s “what happens when it isn’t — and how do I know the difference?”

A commercial lease is not a form. It’s a bespoke legal document with negotiated carve-outs, cross-references between sections, clauses that modify other clauses, and amendments that may partially override the original. In that environment, a missed termination right or a misread rent escalation schedule is not a data quality issue. It’s a financial exposure.

This is why human-in-the-loop review is not a workaround for imperfect AI. It is the design. The question is how well it’s implemented.

What “human-in-the-loop” actually means in lease abstraction

The phrase gets used loosely. In some systems, it means a reviewer receives a completed extraction and approves or rejects it wholesale. In better systems, it means something more specific: reviewers see exactly where each data point came from, which fields carry lower confidence, and where the AI models disagreed — before approving anything.

The difference matters enormously in practice.

Wholesale review

means the reviewer is doing a second read of the entire output. It is faster than manual abstraction, but it doesn’t change where attention goes. Complex, unusual, or ambiguous clauses get the same time as routine boilerplate.

Field-level review with source attribution

means the reviewer’s attention is directed. High-confidence fields can be accepted quickly. Low-confidence fields — the ones where extraction is genuinely uncertain — get closer scrutiny. The reviewer sees the extracted value and the exact clause it came from, in context, side by side.

This is the design that makes AI-assisted abstraction better than either manual abstraction or fully automated extraction. It combines AI’s ability to process the entire document without fatigue with a human expert’s ability to evaluate ambiguity, apply judgment, and catch what the models missed.

Where AI lease abstraction actually struggles

Human review is necessary in AI lease abstraction because lease documents often contain poor OCR quality, cross-referenced clauses, amendments, negotiated language, and inferred fields that cannot be extracted through simple keyword matching. In these cases, AI can surface the right data, assign confidence scores, and flag complex fields for validation — but expert reviewers are still needed to confirm context, resolve dependencies, and make sure structured lease data is accurate, traceable, and usable for lease administration.

How confidence scoring directs the review

Confidence scoring is the mechanism that makes human-in-the-loop review practical at scale.

In Basking’s system, three independent AI models read the same document and extract each field separately. Their outputs are then compared field by field.

When all three models agree on a value — and agree that it comes from the same clause — confidence is high. The reviewer sees the value, the source location, and a 100% confidence indicator. Accepting that field takes seconds.

When models disagree — one extracts a 12-month notice period, another extracts 9 months, a third finds no notice requirement at all — that disagreement is the signal. The field is flagged. The reviewer sees all three model outputs, the relevant document sections, and makes the determination.

This pattern means reviewers are not spending equal time on every field. They are spending the right time on the right fields. A 100-field lease document with high confidence across 90 fields and flags on 10 becomes a focused, structured review task rather than a full re-read.

Basking’s accuracy testing on real documents shows 90 to 98% accuracy depending on document type and field complexity. The human review layer is designed specifically to close that remaining gap — not by catching every error after the fact, but by directing expert attention to where uncertainty actually lives.

The audit trail: why it matters beyond compliance

Every edit a reviewer makes — and every acceptance — is recorded. Not at the document level. At the field level.

This means that when a question arises six months after abstraction (“why does our system show a 3% annual escalation when the lease says CPI?”), the answer is traceable. You can see the original extracted value, the reviewer who edited it, when the change was made, and what the source clause says.

This is not just a compliance requirement. It is a practical necessity for any lease administration process that involves multiple stakeholders.

CRE, legal, and finance teams often work from the same lease data but interpret it differently. A legal team may flag a clause as requiring landlord consent for subletting. Finance may be modeling costs based on a fixed escalation. When these assumptions diverge, the question becomes: what does the lease actually say, and who made the call on how to record it?

An audit trail answers that question without a document review. It also makes handoffs between team members, between fiscal years, and between systems far less risky.

In LeaseOps Flow, this traceability extends beyond abstraction into the workflow layer. Approval decisions, renewal assessments, and portfolio actions are linked to the data that informed them — creating a connected record from document to decision.

What this looks like in practice

The user journey in Basking’s human-in-the-loop interface illustrates how this works concretely.

After AI processing completes, the reviewer enters the review and approve interface. On the left is a reconstructed view of the original document — not a thumbnail, but a navigable representation with clause locations highlighted. On the right are the extracted fields, organized by category: general information, key dates, financial terms, options, obligations, events.

Each field shows its value, its source clause and line number, and its confidence score. Fields with 100% confidence and full model agreement can be accepted with a single click. Fields flagged for review show the conflicting model outputs and the relevant document context.

The reviewer can accept the extracted value, edit it, or override it entirely. Edits are tagged with the reviewer’s identity and timestamp. If an extracted value is correct but the source attribution is wrong — the AI found the right number in the wrong clause — that can be corrected too.

When review is complete, the verified data is available for import into the lease administration record, export as CSV, or integration via API. The output is not a draft. It is a validated, traceable dataset ready for operational use.

Why “fully automated” abstraction is not the right goal

Some vendors position their product as requiring no human review. This is worth examining carefully.

A system that achieves 99% accuracy on straightforward leases with clean text may genuinely require minimal review. But commercial real estate portfolios are not composed of straightforward leases with clean text. They include older documents, multilingual agreements, complex amendments, heavily negotiated terms, and scanned originals of varying quality.

In that environment, a fully automated system without review creates a specific kind of risk: confident errors. The system presents output without uncertainty signals, and reviewers — if they exist at all — are reviewing against a system that is not designed to surface doubt.

The goal of a well-designed human-in-the-loop system is not to maximize automation. It is to maximize the value of expert review by making it faster, more focused, and more reliable. The AI does the work that should be automated. The expert does the work that requires judgment.

As said during a recent product demo: “Lease is not the place to experiment with AI. You don’t want to be cavalier with AI — that’s an important part of the process. Having a human reviewing every output that matters, where there is a question about confidence scores, is what makes the data something you can rely on.”

What to look for when evaluating human-in-the-loop systems

If you are comparing AI lease abstraction vendors, the review workflow is one of the most important differentiators. These are the questions worth asking.

Does the system show source attribution?

For every extracted field, can you see exactly which clause and line it came from? Without this, reviewers are validating outputs, not reviewing extractions.

Does confidence scoring reflect model agreement or a single model’s self-assessment?

A system that uses multiple independent models and compares outputs is giving you a more reliable uncertainty signal than one that asks a single model to rate its own confidence.

Are inferred fields distinguished from extracted fields?

When AI reasons from context rather than finding explicit text, that distinction should be visible to reviewers.

Is the edit trail field-level or document-level?

Document-level versioning tells you something changed. Field-level audit trails tell you what changed, who changed it, when, and from what source.

How does the system handle amendments?

Can it process a base lease and multiple amendments as a package, identifying where later documents modify earlier terms?

What happens after review?

Does the validated data connect to a lease administration platform, or does it export to a spreadsheet that starts the fragmentation process all over again?

FAQ: Human-in-the-loop lease abstraction

Does human-in-the-loop review slow down the process?

It depends on how the review is structured. A well-designed system significantly reduces total time compared to manual abstraction, even with full human review included. Basking processes a document in approximately 30 minutes of AI extraction. Human review time depends on document complexity and confidence score distribution — but reviewers are working from structured outputs with source attribution, not reading the document from scratch.

Who should be doing the human review?

Review is typically done by lease administrators, in-house counsel, or asset management professionals familiar with the lease terms being abstracted. The review interface is designed for domain experts, not for data scientists.

What happens if the reviewer disagrees with the AI extraction?

The reviewer can edit any field. The original extracted value, the edited value, and the reviewer’s identity are all retained in the audit trail. This is not considered an AI failure — it is the system working as designed.

Can the system learn from reviewer corrections?

This depends on the vendor and their model update process. At minimum, corrections should be captured in a way that informs future model improvements for similar document types.

Is human review required for every document, or just flagged fields?

This is a policy decision for the team, not a system constraint. Some teams review every field. Others accept high-confidence fields automatically and review only flagged items. The right balance depends on document complexity, portfolio risk, and internal compliance requirements.

How does the audit trail support compliance?

The field-level change log records who extracted, who reviewed, who approved, and when. This supports internal audit requirements, supports IFRS 16 and ASC 842 compliance processes, and provides a clear chain of custody for lease data used in financial reporting.

Key takeaways

When evaluating vendors, the review workflow is as important as the extraction accuracy numbers.

• Human-in-the-loop review is not a fallback for AI limitations. It is the design that makes AI-assisted abstraction reliable at scale.

• Confidence scoring and source attribution direct expert attention to the fields that most need it — rather than distributing review time evenly across routine and complex content alike.

• The audit trail at the field level is what makes lease data operationally trustworthy: traceable, correctable, and auditable.

• The goal is not full automation. The goal is faster, more focused, more reliable expert review — where AI handles extraction and humans handle judgment.