> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sterndesk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl and Extract URLs

> Learn how to crawl web pages and extract structured data from them

# Crawl and Extract URLs

This guide walks you through crawling web pages and extracting structured data from them using Sterndesk's Crawl Collector.

## Prerequisites

Before you begin, ensure you have:

* An API key with appropriate permissions (see [Authentication](/get-started/02-authentication))
* An existing [organization and project](/concepts/01-organizations-and-projects) (see the [guide](/guides/01-create-projects-and-organizations))

For this guide, we'll assume you have a project ID available. We'll use `proj_xyz789` as an example.

## Step 1: Create an Extraction Schema

An [extraction schema](/concepts/02-extraction-schemas) defines the structure of data you want to extract from web pages. Create a schema that matches the information you're looking for.

For this example, we'll create a schema to extract article information from web pages:

```bash theme={null}
curl -X POST https://api.eu.sterndesk.com/r/extraction-schemas \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_xyz789",
    "name": "Article Extraction",
    "json_schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\",\"description\":\"The title of the article\"},\"author\":{\"type\":\"string\",\"description\":\"The author of the article\"},\"published_date\":{\"type\":\"string\",\"description\":\"The publication date\"},\"summary\":{\"type\":\"string\",\"description\":\"A brief summary of the article content\"}},\"required\":[\"title\"]}"
  }'
```

<Note>
  The `json_schema` field must be a JSON-encoded string, not a nested object. See [Extraction Schemas](/concepts/02-extraction-schemas#schema-encoding) for details on schema encoding.
</Note>

Response:

```json theme={null}
{
  "id": "exsc_abc123",
  "name": "Article Extraction"
}
```

Save the schema ID (`exsc_abc123`) for the next step.

## Step 2: Create a Crawl Collector

A Crawl Collector is a [collector](/concepts/03-collectors) that fetches and processes web pages. When you attach an extraction schema to it, crawled pages are automatically extracted upon completion (Direct Extraction mode).

```bash theme={null}
curl -X POST https://api.eu.sterndesk.com/r/crawl-collectors \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_xyz789",
    "name": "News Articles",
    "strategy": "CRAWL_STRATEGY_FIRECRAWL",
    "direct_extraction_schema_id": "exsc_abc123"
  }'
```

The `strategy` field specifies the crawling engine. Currently, `CRAWL_STRATEGY_FIRECRAWL` is the only supported strategy.

<Tip>
  To enable long-term storage of crawled content, add `"bundle_enabled": true` to your request. This stores the crawled data in S3 as a bundle that can be downloaded later. See [Bundled Extraction](/concepts/03-collectors#bundled-extraction) for details.
</Tip>

Response:

```json theme={null}
{
  "id": "crw_coll_def456",
  "name": "News Articles"
}
```

Save the collector ID (`crw_coll_def456`) for creating crawls.

## Step 3: Create a Crawl

To crawl a web page, create a crawl request specifying the URL you want to process:

```bash theme={null}
curl -X POST https://api.eu.sterndesk.com/r/crawls \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "crawl_collector_id": "crw_coll_def456",
    "url": "https://example.com/article/12345"
  }'
```

Response:

```json theme={null}
{
  "id": "crw_ghi789"
}
```

The crawl starts processing immediately in the background.

## Step 4: Poll for Results

<Warning>
  **Development in Progress**: The Crawl implementation is still being developed. While crawls will transition to `CRAWL_STATUS_CRAWLED`, extraction processing is not yet fully implemented. No extraction results will be returned at this time.
</Warning>

Poll the crawls endpoint to check the status of your crawl:

```bash theme={null}
curl "https://api.eu.sterndesk.com/r/crawls?crawl_collector_id=crw_coll_def456" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

Response:

```json theme={null}
{
  "items": [
    {
      "id": "crw_ghi789",
      "status": "CRAWL_STATUS_CREATED",
      "url": "https://example.com/article/12345"
    }
  ]
}
```

## Deleting a Crawl Collector

When you no longer need a crawl collector, you can delete it.

<Warning>
  Deleting a crawl collector permanently removes the collector and **all associated crawls and extractions**. This action cannot be undone.
</Warning>

```bash theme={null}
curl -X DELETE https://api.eu.sterndesk.com/r/crawl-collectors/crw_coll_def456 \
  -H "Authorization: Bearer YOUR_API_KEY"
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Extraction Schemas" icon="code" href="/concepts/02-extraction-schemas">
    Learn more about designing extraction schemas
  </Card>

  <Card title="Upload Documents" icon="upload" href="/guides/02-upload-and-extract-documents">
    Extract data from uploaded documents instead of web pages
  </Card>
</CardGroup>