> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sterndesk.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Collectors and Extractions
> Understand how collectors gather data and extractions process it
# Collectors and Extractions
Collectors are the entry points for data into the Sterndesk extraction pipeline. They define **how** and **from where** data is gathered before being processed by an [Extraction Schema](/concepts/02-extraction-schemas).
Each collector type represents a different method of data ingestion, allowing you to bring documents and content from various origins into your project for structured data extraction.
## Architecture Overview
All collectors feed into the same extraction pipeline, creating a unified data flow regardless of the origin:
## Collector Types
Sterndesk supports multiple collector types, each optimized for different data origins:
| Collector Type | Origin | Use Case |
| ------------------ | ------------ | ------------------------------- |
| **Upload** | File uploads | PDFs, images, documents |
| **Crawl** | Web URLs | Web pages, online content |
| *More coming soon* | — | Additional integrations planned |
## Organizing Collectors
You can create multiple collectors within a single project to organize different data streams. Each collector can have its own extraction schema, allowing you to process different document types with specialized schemas.
**Common patterns include:**
* **By document type** — Separate collectors for invoices, receipts, and contracts
* **By business unit** — One collector for finance documents, another for HR paperwork
* **By UI context** — Different parts of your application use different collectors (e.g., an invoice upload widget vs. a research paper ingestion form)
* **By data origin** — Distinguish between user-uploaded content and crawled web data
This separation provides clear organization, distinct extraction schemas per use case, and easier tracking of where data originated.
## Extraction Modes
Each collector supports two extraction modes that determine when and how data is processed:
### Direct Extraction
In **Direct Extraction** mode, documents are processed immediately upon collection. The extraction workflow runs automatically:
1. **Collect** — Data is ingested via the collector
2. **Convert** — Documents are converted to a processable format
3. **Structure** — An LLM extracts structured data according to your schema
4. **Deliver** — Results are available immediately via the API
Direct Extraction is ideal for:
* Real-time processing requirements
* Single document workflows
* Immediate data availability needs
To enable Direct Extraction, attach an [Extraction Schema](/concepts/02-extraction-schemas) to your collector when creating it.
### Bundled Extraction
In **Bundled Extraction** mode, collected data is persisted to long-term S3 storage before extraction runs as a separate step. Enable this mode by setting `bundle_enabled: true` when creating a collector.
When bundling is enabled:
1. **Collect** — Data is ingested via the collector
2. **Bundle** — Data is stored in S3 as a bundle with status `CREATED` → `STORED`
3. **Extract** — Extraction can run on the bundled data (when configured)
This allows for:
* Batch processing of multiple documents
* Re-extraction with updated schemas
* Data archival and audit trails
* Deferred processing workflows
#### Working with Bundles
List bundles for a project using the `ListBundles` RPC:
```bash theme={null}
curl "https://api.eu.sterndesk.com/r/bundles?project_id=proj_xyz789" \
-H "Authorization: Bearer YOUR_API_KEY"
```
Download bundled data using the `DownloadBundle` RPC, which returns pre-signed GET URLs:
```bash theme={null}
curl "https://api.eu.sterndesk.com/r/bundles/bnd_abc123/download" \
-H "Authorization: Bearer YOUR_API_KEY"
```
Each bundle tracks:
* **Source type** — Whether it originated from an upload (`BUNDLE_SOURCE_TYPE_UPLOAD`) or crawl (`BUNDLE_SOURCE_TYPE_CRAWL`)
* **Object count** — Number of objects stored in the bundle
* **Total size** — Total size in bytes of all bundled objects
***
## Upload Collectors
Upload Collectors enable you to ingest documents directly into Sterndesk via file uploads. They are ideal for processing PDFs, images, and other document formats.
### How Upload Collectors Work
1. **Create an Upload Collector** — Define a named collector within your project
2. **Request [Pre-signed URLs](/concepts/05-upload-urls)** — Get secure upload URLs for your files
3. **Upload Files** — PUT your files directly to the pre-signed URLs
4. **Automatic Processing** — If a schema is attached, extraction runs automatically
### Upload Lifecycle
Each upload goes through the following states:
| Status | Description |
| -------------------- | ----------------------------------------------- |
| `CREATED` | Upload initiated, awaiting file transfer |
| `TRANSFERRED` | Files successfully uploaded |
| `DIRECTLY_EXTRACTED` | Extraction complete (if schema attached) |
| `EXPIRED` | Upload window expired before transfer completed |
***
## Crawl Collectors
Crawl Collectors enable you to extract data from web pages by providing URLs. They automatically fetch and process web content.
### How Crawl Collectors Work
1. **Create a Crawl Collector** — Define a named collector within your project
2. **Submit URLs** — Provide URLs to crawl
3. **Automatic Fetching** — The crawler retrieves and processes the web content
4. **Extraction** — If a schema is attached, structured data is extracted
### Crawl Lifecycle
Each crawl goes through the following states:
| Status | Description |
| --------- | ---------------------------------- |
| `CREATED` | Crawl initiated, pending execution |
| `CRAWLED` | Web content successfully retrieved |
***
## API Reference
For detailed information on managing collectors, see the API Reference:
* [Upload API Reference](https://api.eu.sterndesk.com/r/#tag/upload)
* [Crawl API Reference](https://api.eu.sterndesk.com/r/#tag/crawl)
## Next Steps
Learn about pre-signed URLs for secure file uploads
Define how data is extracted and structured