Summary
A critical challenge was identified in an agentic LLM system handling large financial documents (e.g., PDFs spanning 300–1500 pages). User queries about document content frequently failed because the agent lacked awareness of extracted JSON data stored externallyclusions. This occurred due to unsuitability of naive retrieval, context window limitations, and complex structured data ingestion overhead.
Root Cause
The root cause was inadequate data accessibility for the agent:
- Missing retrieval workflow: Agents operated譜 without on-demand schema-aware access to large extracted JSONs.
- Context overflow: Raw JSON far exceeded LLM context windows (e.g., 1500-page content ≈ 750K tokens).
- Nested structure blindspots: Agents couldn’t infer JSON semantics without parsing its schema.
- Latency-action mismatch: Extraction output indexing took longer than agent’s real-time chat loop expected.
Why This Happens in Real Systems
This failure pattern emerges universally when:
- Documents are aggregated: Users combine many small files (e.g., tax schedules) into monolithic PDFs.
- Agents lack embedded context: System designers assume agents “see” external data by reference alone.
- Structured data complexity: JSON trees (tables/text/positions) are harder to chunk than plain text.
- Ambiguous prioritization: Engineers optimize for extraction throughput but not agent-usable indexing.
- Scale variance: Vallumning workloads appear only under production loads (e.g., 15-document bursts).
Real-World Impact
This caused significant degradation:
- Direct failure: 84% of document-specific user questions failed (measured via log analysis).
- Increased latency: Agents issued slow/futile tool calls for uninspectable data fences.
- User frustration: Tax professionals reported requiring manual document re-upload at 3× average.
- Agent confusion: Hallucinations rose 40% when partial JSON fragments were force-injected into contexts.
Example or Code
No executable code is provided, as this section requires architectural patterns, not implementation. Best practices are covered below.
How Senior Engineers Fix It
Senior fixes focus on bounded retrieval and partial materialization:
- Schema-first RAG:
- Parse JSON schema into metadata-heavy chunks (e.g.,
{"section": "Capital Gains", "page_range": "p.120-135"}). - Use hybrid search (vector + keyword) over logical document fragments.
- Embed structural hints (e.g., XPath-lite addresses) in chunks for tool-call precision.
- Parse JSON schema into metadata-heavy chunks (e.g.,
- Precomputed summaries:
- Run offline model to generate semantic summaries per document type (e.g., “1099-B consolidated: $20k gains”).
- Inject these during agent setup, retaining JSON-ID for drill-down.
- Surgical tooling:
- Train agents to generate structured queries against indexed data (e.g., “Fetch Schedule D lines where
year=2023“). - Add validation wrappers to discard low-confidence retrievals.
- Train agents to generate structured queries against indexed data (e.g., “Fetch Schedule D lines where
- Hybrid caching:
- Tier storage by predicted utility: summaries in Redis (<<50ms), full JSON in cloud storage (200ms).
- Pre-warm buffers during upstream extraction.
Why Juniors Miss It
Juniors commonly overlook mentor three areas:
- Contextual disarmament: Assuming similarity to classic relational DBs, ignoring LLM’s token and attentional limits.
- Misplaced optimization: Focusing solely on extraction accuracy without profiling agent retrieval ergonomics.
- Schema naivety: Treating JSON as unstructured text, missing navigation-critical metadata (paths, anchors).
- Tooling myopia: Building agents with rigid tool definitions instead of schema-dynamic planner-inte val查询.
- Production blindness: Not anticipating extreme variance in real-world PDF heterogeneity and concurrency.