Building a 'Grounded' Document QA Agent

A guide for building a document QA agent that provides verifiable answers with page numbers and source snippets. The pipeline uses LlamaParse for layout-aware parsing, local embeddings and LLM, and CitationQueryEngine for inline citations. The key insight is that parsing quality is the ceiling for answer accuracy.
Key insights
Technical Pipeline
The pipeline: cloud parsing with LlamaParse to generate layout-aware Markdown with page metadata, local embedding with nomic-embed-text via Ollama, LlamaIndex vector index, retrieval of top-4 chunks, local qwen3:4b-instruct for generation, and CitationQueryEngine for inline citations [1], [2]. The UI shows page numbers, retrieved text, query highlights, and relevance scores.
Key Technical Points
Parsing is the ceiling: basic text extraction fails on dual-column papers, financial tables, and scanned documents. Citation granularity is a trade-off: 512-character chunks balance precision and context. Citation does not guarantee correctness: if parsing misaligns table values, the answer is wrong despite correct page references. The verification UI makes the pipeline debuggable.