NASA Space Talks RAG Chat

Garbage in, garbage out: A RAG preprocessing cautionary tale.

Welcome to NASA Space Talks
Query the historical NASA mission archives across Apollo, Gemini & Mercury. All the conversations had in space are at your disposal.

Mission

Apollo 11 102:45:40 Ground Elapsed Time
GET
Apollo 11 Mission Parameters
Crew: Armstrong, Aldrin, Collins
Launch: 16 Jul 1969
Duration: 8d 03h 18m
View Sample
Navigate by Examples Diff Samples

Apollo Transcript

Transcript Preprocessing & RAG Pipeline PROCESS & METHODOLOGY

Deterministic Normalization, Not Generative Rewriting

Historical NASA transcripts contain OCR degradation inherited from scans, microfilm and photocopied source material. Space Talks uses deterministic, rule-based normalization rather than generative LLM rewriting. Each transformation is designed to remain inspectable and traceable back to the source record.

Three Confidence Tiers of Preprocessing
1. Safe Normalization
0xygen → Oxygen
passiva- tion → passivation
High-confidence deterministic character and dictionary repairs that heal obvious OCR digit/letter substitutions without modifying domain vocabulary.
2. Structural Reconstruction
broken columns → speaker / timestamp / utterance
Bounding-box column geometry extraction and multi-line de-hyphenation that separates tabular transcript metadata from dialogue speech.
3. Contextual Inference
ambiguous code → probable role
Position- and context-aware resolution of shorthand callsigns (e.g. SC → Commander (CDR)), tracked with explicit provenance.
Multi-Tier Lifecycle & Tooling Architecture
Tier Name & Target Methodology Tools & Libraries
Tier 0 Raw OCR
Primary record		 Bounding-box column geometry extraction from historical 1960s declassified PDF scans. PyMuPDF (fitz), pdfplumber, SQLite3, JSONL
Tier 1 Clean v1
Baseline heuristic		 Speaker callsign canonicalization & naive dictionary replacements (e.g. l0 → 10, SC → CDR). Python re (word-boundary \b), SQLite3
Tier 2 Clean v2
Context release		 Lookaround-guarded regex ((?<!['’\w])ll\b), contraction backreferences, multi-line de-hyphenation. Python re (lookbehinds), MiniLM-L6-v2, Qdrant Cloud
Tier 3 Clean v3 (HITL)
Human-Verified Review		 High-entropy anomaly extraction + side-by-side human domain verification & immutable audit override logging. Web Workbench (/turns Dial & Reel), overrides.jsonl
Why Preprocessing Matters Downstream
  • Retrieval — Can the right evidence be found? OCR corruption can alter tokenization and embeddings, reducing similarity between a query and otherwise relevant source material. Normalization is intended to improve retrieval consistency.
  • Grounding — What evidence does the model actually receive? Corrupted source text can introduce ambiguity into retrieved context. A generation model may then misinterpret or attempt to resolve that ambiguity, increasing the risk of unsupported output.
  • Provenance — Can we trace what happened between source and answer? Every transformation step is logged into immutable audit records (records_clean_v2.jsonl), preserving a clear lineage back to the original scanned transcripts.

RAG Search Modes & Output Formats RAG GUIDE

Search Modes
Semantic (MiniLM / BGE) Dense Vectors
Meaning-based retrieval. Converts the query into an embedding and retrieves transcript passages with similar meaning, even when they use different words. For example, “power loss” may retrieve “Main B bus undervolt.”
Best for: questions, concepts and events.
Keyword Exact Match Lexical Search
Text-based retrieval. Searches the transcript database for the literal text you enter.
Best for: procedure codes (P64), acronyms (SCE to AUX), names and known phrases.
Output Formats
Smart Answer AI Synthesis
AI-synthesized output. Retrieves relevant transcript passages and asks Gemini to produce a concise answer grounded in that evidence, with source citations.
Dialogue Scene Source-Only
Source-only output. Shows the retrieved historical transcript turns directly — no AI-generated synthesis (bypasses AI synthesis) — including mission, GET timestamp and speaker.
Mission Filter

Limit retrieval to one mission using its ID (e.g. as11, as13, gt04, ma09), or leave blank to search across the entire Mercury, Gemini and Apollo corpus.

What stays the same?

These controls change retrieval and output behavior. They do not change the underlying preprocessing or chunking of the NASA corpus.