Skip to main content
The Source stage is where a pipeline first touches the outside world. A Source block emits raw items — files, records, messages — into the pipeline for the next stages to work on.

​
Source block types

​
Cloud folder

Pulls files from a connected cloud drive (Google Drive, OneDrive, S3-compatible).
  • Folder — pick from the connected account.
  • Recurse — include subfolders. On by default.
  • File types — restrict to .pdf, .docx, .md, .csv, etc. Skip binaries you can’t process.
  • Since — only files modified after this timestamp. Combine with a run-payload timestamp to build incremental pipelines.
Emits one item per file, with content (bytes), path, mime_type, modified_at, and source_id (stable across renames).

​
Document store

Pulls from an existing Nora document collection. Useful when a downstream pipeline builds on data another pipeline already curated.

​
Database query

Runs a SQL query on a connected Postgres/MySQL/SQL Server database, streams rows.
  • Query — parameterized SQL. Reference run-payload with :key.
  • Batch size — rows per emitted item. Larger batches are more efficient; smaller batches let downstream stages parallelize better.
Emits one item per row (or one per batch, if you set batch > 1).

​
HTTP fetch

Custom HTTP GET/POST. Emits the response body. Configure:
  • URL (templatable with run-payload).
  • Method, headers, body.
  • Follow pagination — automatically loop until a stop condition. Cursor param, page count, or “until empty response”.

​
Webhook payload

Uses the payload from the Webhook trigger as the source. No external call — the payload is already in memory.

​
Manual upload

An empty Source that lets you upload files when running the pipeline manually. Useful for testing.

​
Idempotency & dedup

Every emitted item gets a stable content hash. Later runs against the same source recompute the hash — unchanged items are skipped end-to-end. This means a scheduled Foundry run over a large folder is cheap: it only touches what changed. You can force a full re-process via Run → Rerun (ignore cache) if the pipeline downstream changed.

​
Change detection

For Watch triggers, the Source block also emits delete events when a source item vanishes. Downstream stages should handle deletes — often by pushing a supersede to the Output stage rather than a fresh chunk. See Supersede.

​
Limits

  • Max items per run — default 50k. Larger runs should be split by date range.
  • Max total bytes — default 5 GB. Raise on Enterprise plans.
  • Per-item byte cap — default 100 MB per file. Larger items are skipped and logged.