Skip to main content
The Clean stage sits between Source and Normalize. Its job is to remove things you don’t want stored — accidental PII, boilerplate, near-duplicates, or content that shouldn’t leave your workspace.

​
Clean blocks

​
PII redaction

Automatically detects and replaces:
  • Emails → [email]
  • Phone numbers → [phone] (US, EU, JP, KR formats)
  • Credit card numbers → [cc]
  • National IDs (SSN, KRRN, MyKad) → [id]
  • Names — optional, off by default. Uses NER + a small on-device model.
  • Custom regex — add your own patterns.
Replacements are consistent within a document (same email replaced with the same placeholder) so the meaning is preserved.

​
Boilerplate stripping

Removes common noise:
  • Email headers and signatures.
  • HTML tags and styling.
  • Table of contents, page numbers, headers/footers.
  • Legal footers and disclaimers.
  • Cookie notices and nav menus (web content).

​
Language filter

Drops items in unwanted languages. Set an allowlist (e.g., en, ko, ja). Everything else is filtered out and logged.

​
Content filter

Drops items whose content matches a denylist — patterns, keywords, or a small classifier you supply. Useful for keeping test data or internal-only content out of a public-facing agent’s knowledge.

​
Near-duplicate detector

Removes items that are >95% similar to something already ingested. Uses MinHash sketches — fast and doesn’t require re-embedding. Tunable threshold. Set higher (>99%) to only catch exact-ish copies; lower (~85%) to be aggressive.

​
Custom transform (inline)

Run a small JS/Python function on each item. Full access to content, metadata, and helper libs (re, bs4, markdown-it). Return the transformed item, or return null to drop it.

​
Order matters

Blocks in Clean run in the order you wire them. Common recipe:
  1. Language filter (drop early — cheap).
  2. Boilerplate stripping (clean up structure).
  3. PII redaction (act on clean text).
  4. Near-duplicate detector (dedup after everything else has normalized noise).

​
Logging what got cleaned

The run log records per-item cleaning actions: how many PII hits, what was stripped, whether the item was dropped. Useful for auditing without keeping the pre-clean data around.

​
Reversibility

Redactions are not reversible by design. If you need to un-redact, you have to re-source and re-run. If you need the original for audit, keep the source untouched in its original store — Nora only redacts the derived copy in the pipeline.