> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.nora.my/llms.txt
> Use this file to discover all available pages before exploring further.

# Clean stage

> Redact PII, deduplicate, and remove noise before your data hits the index.

The **Clean** stage sits between Source and Normalize. Its job is to remove things you don't want stored — accidental PII, boilerplate, near-duplicates, or content that shouldn't leave your workspace.

## Clean blocks

### PII redaction

Automatically detects and replaces:

* **Emails** → `[email]`
* **Phone numbers** → `[phone]` (US, EU, JP, KR formats)
* **Credit card numbers** → `[cc]`
* **National IDs** (SSN, KRRN, MyKad) → `[id]`
* **Names** — optional, off by default. Uses NER + a small on-device model.
* **Custom regex** — add your own patterns.

Replacements are consistent within a document (same email replaced with the same placeholder) so the meaning is preserved.

### Boilerplate stripping

Removes common noise:

* Email headers and signatures.
* HTML tags and styling.
* Table of contents, page numbers, headers/footers.
* Legal footers and disclaimers.
* Cookie notices and nav menus (web content).

### Language filter

Drops items in unwanted languages. Set an allowlist (e.g., `en, ko, ja`). Everything else is filtered out and logged.

### Content filter

Drops items whose content matches a **denylist** — patterns, keywords, or a small classifier you supply. Useful for keeping test data or internal-only content out of a public-facing agent's knowledge.

### Near-duplicate detector

Removes items that are >95% similar to something already ingested. Uses MinHash sketches — fast and doesn't require re-embedding.

Tunable threshold. Set higher (>99%) to only catch exact-ish copies; lower (\~85%) to be aggressive.

### Custom transform (inline)

Run a small JS/Python function on each item. Full access to `content`, `metadata`, and helper libs (`re`, `bs4`, `markdown-it`). Return the transformed item, or return `null` to drop it.

## Order matters

Blocks in Clean run in the order you wire them. Common recipe:

1. Language filter (drop early — cheap).
2. Boilerplate stripping (clean up structure).
3. PII redaction (act on clean text).
4. Near-duplicate detector (dedup after everything else has normalized noise).

## Logging what got cleaned

The run log records per-item cleaning actions: how many PII hits, what was stripped, whether the item was dropped. Useful for auditing without keeping the pre-clean data around.

## Reversibility

Redactions are **not reversible** by design. If you need to un-redact, you have to re-source and re-run.

If you need the original for audit, keep the source untouched in its original store — Nora only redacts the derived copy in the pipeline.