Protected Module

Vector Database Foundations

Build the mental model for semantic search: embeddings, chunks, metadata, similarity search, filtering, reranking, and retrieval quality.

Lesson 1: Embeddings

An embedding is a numeric representation of meaning. Use it when exact keywords are too brittle, but do not treat similarity as truth.

Create sample sentences with similar and different meanings.
Compare nearest neighbors.
Write down surprising matches.

Lesson 2: Chunking

Chunking decides what evidence the model can retrieve. Bad chunks create missing context, duplicated context, and weak citations.

Compare paragraph chunks and fixed token chunks.
Track chunk ID, source title, and section.
Record which chunk size retrieves best.

Lesson 3: Metadata

Metadata makes retrieval controllable. Store source, date, type, owner, permission level, and topic.

{
  "source": "runbook.md",
  "section": "deploy",
  "topic": "oci",
  "permission": "course"
}

Lesson 4: Search

Start with top-k similarity search, then add filters and reranking when results are noisy.

Search by question.
Filter by topic.
Rerank top 20 down to top 5.

Lab

Build a local index from five markdown notes. Ask ten questions. For every bad answer, decide whether the problem was chunking, metadata, search, or the source material itself.