Sitemap jobs
Full request and response reference for the asynchronous sitemap crawl endpoints.
Discovers every URL on a site from a single starting point. Crawls are asynchronous jobs: submit for a quote, approve, poll status, then page through the results. See Sitemap for the walkthrough.
| Method | Path | Description |
|---|---|---|
POST | /sitemap | Submit a crawl and get a quote. |
POST | /sitemap/{jobId}/approve | Approve the quote; starts the crawl. |
GET | /sitemap/{jobId} | Job status and progress. |
GET | /sitemap/{jobId}/urls | Paginated discovered URLs. |
GET | /sitemap | List your jobs. |
DELETE | /sitemap/{jobId} | Cancel a job. |
Every path is also served under /map — /map, /map/{jobId}/approve, and so on are the same endpoints with the
same request and response shapes.
Submit a crawl
POST https://request.usestring.ai/v1/sitemapRequest body
| Field | Type | Default | Description |
|---|---|---|---|
url | string (URL) | — | Required. Starting URL. The crawl stays on this hostname. |
maxPages | integer | 10 | Maximum pages to crawl (1–10,000). |
maxDepth | integer | 2 | Maximum link depth from the starting URL (1–100). |
pathPrefix | string | — | Only crawl URLs whose path starts with this prefix. |
budgetUsd | number | quote | Spend cap in USD (min 0.0001). Defaults to the quote itself, so every job is capped; the crawl stops with token_cap_exceeded before exceeding it. |
useSitemap | boolean | false | Also seed the crawl from the site's root /sitemap.xml (one extra quoted and billed page; finds pages internal links miss). |
Any other field is rejected with a 400, so a typo like maxPage fails loudly instead of quoting
the default of 10 pages.
Response — 202
The job is created in awaiting_approval; nothing is crawled or charged until approved. Quotes expire after one hour —
an unapproved job is marked failed with errorMessage: "quote_expired".
{
"jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
"status": "awaiting_approval",
"estimatedCostUsd": "0.0500000000",
"estimatedPages": 500
}| Status | Meaning |
|---|---|
202 | Quote produced; job awaiting approval. |
400 | Validation error. |
401 | Missing or invalid API key. |
403 | Destination not enabled for your organization. |
429 | Rate limit exceeded. |
500 | Internal error. |
Approve a job
POST https://request.usestring.ai/v1/sitemap/{jobId}/approveStarts the crawl. The estimated cost is held against your balance and settled when the job finishes. Approve within an
hour of submitting — expired quotes are failed and cannot be approved.
| Status | Meaning |
|---|---|
202 | Approved — { "jobId": "...", "status": "running" }. Re-approving a still-running job also returns 202. |
200 | Job already in a terminal state — returns its current status (e.g. failed for an expired quote). |
401 | Missing or invalid API key. |
402 | Insufficient balance for estimatedCostUsd (error: "insufficient_funds"); the job is marked failed and cannot be re-approved after a top-up. |
404 | Job not found. |
409 | The approval did not complete (status: "partial_state"). Retry the approve call. |
429 | Rate limit exceeded. |
500 | Approval failed. |
A canceled or token_cap_exceeded job can't be approved; submit a new crawl.
Get job status
GET https://request.usestring.ai/v1/sitemap/{jobId}The response shape depends on the job's status:
status | Additional fields |
|---|---|
awaiting_approval | estimatedCostUsd |
running | pending (queued URLs), processed (crawled so far) |
completed | pagesProcessed, plus urls (full results) shortly after completion or finishedAt |
failed | errorMessage, finishedAt |
canceled / token_cap_exceeded | pending, processed |
{
"jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
"status": "running",
"pending": 42,
"processed": 458
}| Status | Meaning |
|---|---|
200 | Status returned. |
401 | Missing or invalid API key. |
404 | Job not found or expired. |
409 | The approval did not complete (partial_state). Retry the approve call. |
Get discovered URLs
GET https://request.usestring.ai/v1/sitemap/{jobId}/urls?limit=1000&offset=0| Query param | Default | Description |
|---|---|---|
limit | 1000 | Page size (max 5,000). |
offset | 0 | Number of entries to skip. |
Response — 200
{
"jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
"total": 500,
"urls": [
{
"url": "https://example.com/blog/post-1",
"statusCode": 200,
"discoveredUrls": 12,
"depth": 2,
"isSitemap": false,
"parentUrl": "https://example.com/blog",
"sourceType": "html_link"
}
]
}| Field | Type | Description |
|---|---|---|
url | string | The discovered URL. |
statusCode | integer | HTTP status from fetching the URL; 0 when the fetch failed or the code wasn't retained. |
discoveredUrls | integer | Number of new URLs found on this page. Present while the job runs and shortly after it finishes; omitted after that. |
depth | integer | Link depth from the starting URL (0 = seed). |
isSitemap | boolean | Whether this URL is a sitemap file rather than a page. |
error | string | Present when fetching this URL failed. |
parentUrl | string | null | URL this one was discovered on. |
sourceType | string | How it was discovered: seed, sitemap_index, sitemap_urlset, or html_link. |
While the job runs, the endpoint returns the URLs discovered so far, each with its discoveredUrls count, and total
grows as the crawl progresses (check job status to know when the set is complete). Results stay available after the job
finishes, without discoveredUrls.
| Status | Meaning |
|---|---|
200 | Results page returned. |
401 | Missing or invalid API key. |
404 | Job not found or expired. |
List jobs
GET https://request.usestring.ai/v1/sitemap?limit=20&offset=0Returns your organization's jobs, most recent first. limit defaults to 20 (max 100).
{
"jobs": [
{
"jobId": "6f1c9f0a-8f3e-4a5d-9c2b-1e7d0a4b8c3d",
"domain": "example.com",
"startUrl": "https://example.com",
"status": "completed",
"pagesProcessed": 500,
"maxDepth": 3,
"maxPages": 500,
"estimatedCostUsd": "0.0500000000",
"createdAt": "2026-07-07T12:00:00.000Z",
"finishedAt": "2026-07-07T12:04:30.000Z",
"errorMessage": null
}
]
}Cancel a job
DELETE https://request.usestring.ai/v1/sitemap/{jobId}| Status | Meaning |
|---|---|
200 | Canceled — { "jobId": "...", "status": "canceled" }. |
401 | Missing or invalid API key. |
404 | Job not found. |
409 | Job already finished, failed, or canceled (returns current status). |