> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.nora.my/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunk & parse

> Break documents into retrievable pieces of the right size.

The **Normalize** stage — commonly the largest part of a pipeline — parses raw items into structured chunks the retrieval index can search.

## Why chunk?

Retrieval works by finding the smallest piece of content most relevant to a query. A whole 200-page PDF is too coarse (irrelevant sections drown the signal). A single sentence is too fine (loses context). Chunks are the goldilocks unit.

The right chunk size depends on your content. Docs pages: 200-500 tokens. Chat transcripts: whole conversations. Code: file or function. Nora has sensible defaults but every block is tunable.

## Chunkers

### Markdown chunker

For markdown, HTML, PDF-with-good-structure, and Notion pages.

* Splits on heading hierarchy first (never mid-section).
* Each chunk retains the heading path (`Guides > Auth > OAuth`) as metadata.
* Fallback: character split at \~1200 chars if a section is too long, with 100-char overlap.

### Semantic chunker

Uses a small model to find natural topic boundaries. Slower and more expensive but produces better chunks for prose without clear headings.

### PDF chunker

Handles PDFs with layout — columns, footnotes, tables. Preserves reading order. Extracts tables as HTML or CSV depending on your preference.

### Spreadsheet chunker

For CSV, XLSX, Google Sheets. Two modes:

* **Row-wise** — each row is a chunk. Good for record-like data.
* **Table-summary** — one chunk per sheet with a summary plus a link to the raw data. Good for large tables where retrieval is topical rather than record-lookup.

### Code chunker

Language-aware. Splits at function/class boundaries. Keeps imports and file-scope context attached to every function chunk.

Supported: JS/TS, Python, Rust, Go, Java, C++, C#, Ruby, PHP, SQL. Others fall back to character split.

### Custom chunker (inline)

Write a JS/Python function that takes the item and returns a list of chunk objects. Complete control.

## Chunk metadata

Every chunk gets:

* **Content** — the text.
* **Source ID** — the source item it came from.
* **Path** — hierarchical location (e.g., heading path, sheet name, function name).
* **Position** — chunk index within the source.
* **Custom fields** — any metadata you added upstream.

Retrieval can filter on any of these. So if you tag by department, you can restrict an Agent to its department's chunks.

## Overlap

Some chunkers support **overlap** — reusing the last N tokens of one chunk at the start of the next. Small overlaps (50-100 tokens) improve retrieval recall when a query straddles a chunk boundary.

## Testing

Every chunk block has a **Preview** button. Pick a sample source item, and see exactly how it gets split. Useful for tuning chunker settings.

## Change detection

If a source item changes, only the affected chunks are re-computed and re-indexed. Nora tracks chunk-to-source lineage.