Decoding 1951 census: LLMs extract messy tables

Extracting structured data from historical documents sounds simple — until you encounter tables that change layout every few pages, handwritten annotations, inconsistent schemas, and OCR pipelines that completely lose the plot.

In this talk, Aryan Srivastava (Development Data Lab) walks through building an LLM-powered extraction pipeline for India’s 1951 Population Census handbooks — combining contextual reasoning, schema templates, rule-based systems, and evaluation frameworks to reliably parse messy, format-variant tables at scale.

The session covers:
• why traditional OCR pipelines fail on historical tabular data
• using LLMs to infer document structure and semantics
• balancing automation with rule-based validation
• evaluation strategies for extraction reliability
• lessons from processing large-scale census archives

A practical talk for engineers, researchers, and data practitioners working with unstructured documents, retrieval pipelines, or production LLM systems.

If you enjoy deep practitioner discussions on AI systems, data infrastructure, and production engineering, become part of The Fifth Elephant community:
https://hasgeek.com/fifthelephant#memberships

Видео Decoding 1951 census: LLMs extract messy tables канала Hasgeek TV