POST /fetch
Full request and response reference for the Web Access API fetch endpoint.
Executes an HTTP/HTTPS request to a URL and returns the result, deciding JavaScript rendering, proxy tier and captcha
handling from the url alone.
POST https://request.usestring.ai/v1/fetchRequest body
Unknown fields are rejected. url is the only required field, and a request with only url is the recommended
one. Pages that need JavaScript are rendered without executeJS.
| Field | Type | Default | Description |
|---|---|---|---|
url | string (URL) | — | Required. The http/https URL to fetch. |
method | GET POST PUT PATCH DELETE HEAD OPTIONS | GET | HTTP method, in any case (get and Post work). Any other method returns a 400 that names it. |
body | string | object | — | Request body. Forbidden on GET and HEAD. A string is sent as given; an object is sent as the exact JSON text you wrote. |
format | json raw markdown | json | Response format. |
markdownMode | readable full | full | Markdown preservation level. Only affects Markdown responses. |
mainContentOnly | boolean | false | Remove page chrome. Only affects Markdown responses. |
jsonSchema | object | — | JSON Schema to extract into. Requires format: json. |
executeJS | boolean | — | Render the page in a browser on every request. Pages that need it are rendered without this field. Required by captureXHR. Incompatible with headers. |
requireWSS | boolean | — | Require a browser WebSocket fetch path. Incompatible with headers. |
waitUntil | domcontentloaded load networkidle0 networkidle2 networkidle | — | Wait for this state before the page is read, or with actions before the first action runs. networkidle is the same as networkidle0. Requires executeJS: true, actions or screenshot. |
captureXHR | boolean | object | — | Return the page's XHR and fetch() requests as xhr. true captures all of them; { "include": [...] } only those matching a filter; { "iframes": true } adds the requests of cross-site iframes. Requires executeJS: true. |
includeIframes | boolean | — | Inline each iframe's content into data, in every format and for jsonSchema. Requires executeJS: true, actions or screenshot. |
captureIframes | boolean | object | — | List the page's iframes as iframes. true returns each one's HTML; { "include": [...] } only the HTML of iframes whose URL matches a filter. Requires executeJS: true, actions or screenshot. |
captureCookies | boolean | — | Enterprise. Return the cookies the browser would send to the final page URL, as cookies. Requires one of executeJS: true, actions or screenshot. Incompatible with jsonSchema. |
headers | object | — | Custom headers (max 50). Direct fetch only. |
countryCode | string(2) | — | ISO 3166-1 alpha-2 proxy country. |
solveCaptcha | boolean | true | Captcha solving. false fails on a challenge. |
ignoreCertificateErrors | boolean | false | Skip TLS certificate verification for this request. |
maxRedirects | integer (0–10) | — | Follow at most this many redirects, then return the redirect response. Cannot be combined with followRedirects: false, executeJS, requireWSS, captureXHR, actions or screenshot. |
followRedirects | boolean | true | false returns the destination's redirect response instead of following it, the same as maxRedirects: 0. false cannot be combined with maxRedirects, executeJS, requireWSS, captureXHR, actions or screenshot. |
screenshot | boolean | — | Capture a screenshot. Shorthand for one screenshot action. |
actions | array | — | Browser actions to run (1–50). |
Field compatibility
bodyis forbidden onGETandHEAD.jsonSchemarequiresformat: json.markdownModeandmainContentOnlyare accepted with any format, but are ignored unlessformat: markdown.headerscannot be combined withexecuteJSorrequireWSS.captureXHRrequiresexecuteJS: true, including withactionsorscreenshot, and cannot be combined withjsonSchema. Withoutactionsorscreenshotit also requiresformat: json.waitUntil,includeIframesandcaptureIframesrequireexecuteJS: true,actionsorscreenshot.captureIframescannot be combined withjsonSchema. Withoutactionsorscreenshotit also requiresformat: json.includeIframesworks with every format and withjsonSchema.captureCookiesrequires one ofexecuteJS: true,actionsorscreenshot, and cannot be combined withjsonSchema. Withoutactionsorscreenshotit also requiresformat: json.- A
waitaction withframerequiresselector. See Act inside an iframe. - At most one
waitForResponseaction can setasResult. That request requiresformat: jsonand cannot usejsonSchema. See Return a response as the result. screenshotandactionsare mutually exclusive. When either is set,format: markdown,markdownMode, andmainContentOnlyare supported, butformat: raw, a non-GETmethod,body,headers, andsolveCaptcha: falseare not.jsonSchemais supported with browser actions unless the request also asks for a screenshot.followRedirects: falseandmaxRedirectscannot be combined, and neither can be combined withexecuteJS,requireWSS,captureXHR,actionsorscreenshot.
Requests only some fetch paths can send
A request with a method other than GET, or with a body, is sent only on fetch paths that deliver its method and
body to the destination exactly as given. A browser path loads a page with a bodiless GET, so it never carries one.
When no fetch path available for the destination can send the request, it returns a 400 before anything is sent,
for example:
No fetch method available for https://example.com can send a POST request with a body; only a GET without a body is supported hereThe request is never sent as a GET instead. Send it without a body if the destination accepts that.
Redirects
By default a request follows the destination's redirects and returns the page they lead to. Set maxRedirects to
follow at most that many, or followRedirects: false to follow none. When the destination redirects more times than
that, the redirect the request stopped on is the response:
- The API responds with HTTP
200.statusCodein the JSON body, orx-status-codein every format, is the redirect's status, such as301or302. x-redirect-urlis the absolute URL the redirect points to. The redirect'sLocationheader is inheaderswithformat: json, and is a response header withformat: raw.x-final-urlandfinalUrlare the URL that answered with the redirect.
A redirect returned this way is billed like any other destination response.
With maxRedirects or followRedirects: false, the page is never rendered in a browser, so a page that builds itself
with JavaScript comes back as served. When no fetch path for the destination can follow the request's redirects this way, the request returns a
400 naming both fields. Send it without them to fetch the page.
TLS certificate verification
Certificates are verified by default: a target whose certificate is expired, issued for another hostname, self-signed,
or served with an incomplete chain fails with a TargetCertificateError. Set ignoreCertificateErrors: true to fetch
it anyway. It applies to that one request only, and is never inferred from a failure.
Without verification, nothing guarantees the response came from the destination
Certificate verification is what proves a response came from the destination. With it off, anything on the network path can answer in the destination's place, and this API returns that answer as the destination's response. Skip verification only for public pages on sites with neglected certificates, never for credentials, payments, or data you will act on as authoritative.
The flag applies to executeJS and requireWSS requests too. Browser sessions driven by actions or screenshot
always verify certificates.
HTTPS to HTTP fallback
When an https:// URL has no working TLS service, so the connection fails before any response arrives, a GET or
HEAD with no body, no headers, no actions or screenshot, and no port or credentials in the URL is retried
once over plain http:// on the same host. Other requests are never retried this way. A response served over that fallback carries an
x-protocol-fallback-url header with the http:// URL that was fetched, in every format; the header is absent when
the fetch stayed on https://. If you need the response to have come over TLS, treat that header as a failure.
HTTP to HTTPS upgrade
An http:// URL is requested over https:// first when it is a GET or HEAD with no body, no headers, no
actions or screenshot, no followRedirects: false or maxRedirects, and no port or credentials in
the URL. Other http:// requests are sent as they are. The https:// request always verifies the certificate, even
with ignoreCertificateErrors: true. If the destination answers over https://, that response is returned, whatever
its status. If it doesn't, or its certificate doesn't verify, the request is made over http:// as you sent it.
When the https:// request answers, the response carries an x-protocol-upgrade-url header with that https:// URL,
in every format, and x-final-url reports where it landed. The header is
absent when the response came over http://: the request went back to your http:// URL, or the https:// page
redirected to http://.
Example
curl https://request.usestring.ai/v1/fetch \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "url": "https://httpbin.org/json" }'Retries
A GET, HEAD or OPTIONS request may be tried on more than one fetch path until one of them returns the
destination's answer. Only a GET is ever raced: a slow one may be sent on a second path at the same time. HEAD and
OPTIONS run on one path at a time.
A write — POST, PUT, PATCH or DELETE — reaches the destination at most once:
- It runs on one fetch path at a time and is never raced against another.
- It is sent again on another path only when it provably never reached the destination (the proxy could not connect, or refused the connection before the request left), or when the destination answered with an anti-bot block.
- After an ambiguous failure, where the connection dropped, was reset, or timed out after the request may already have been sent, it is not retried and the request returns an error.
An error on a write therefore does not mean the destination did not act on it. Check the destination's state before you send the write again.
Responses
| Status | Meaning |
|---|---|
200 | Successful fetch. Body shape depends on format (and jsonSchema). |
400 | Validation error — bad input, or a countryCode requests can't be routed from (unsupported_geography). See Proxies & geolocation. |
401 | Invalid or malformed API key, or a request without a key was not accepted. Retry with an active key. |
402 | Insufficient account balance. |
403 | Destination not enabled for your organization, or a path that needs KYC. See Access control & KYC. Also captureCookies on an account without it; see Errors. |
429 | Rate limit exceeded, or XHR capture is at capacity (X-RateLimit-Scope: xhr-capture), or requests to this destination are failing right now (TargetFailingError), or over String's request limit for this destination (TargetRequestLimitError). Retry after Retry-After when present; without it, lower your request rate. |
502 | Upstream/destination error, including a destination we could not reach and a destination that is rate-limiting requests (OriginRateLimitedError). See Errors. |
503 | A feature the request uses is temporarily unavailable: XHR capture or waitForResponse, iframes, or waitUntil and waitForNetworkIdle. Also no exit in the requested countryCode was available for this request (geography_unavailable). Retry; honor Retry-After when present. |
504 | The destination did not respond in time (TargetTimeoutError). |
500 | Internal error. |
200 — JSON format
{
"statusCode": 200,
"headers": { "content-type": ["application/json"] },
"data": { "message": "Hello World" },
"finalUrl": "https://httpbin.org/json"
}For raw, the body is the gzip-compressed bytes with x-status-code set; for markdown, the body is text/markdown.
Every format sets x-final-url to the URL the response was served from, after any redirects or an
HTTPS upgrade, and every JSON body carries the same value as finalUrl. Both are also sent on error
responses. They are left out only when the request body has no valid url. See
Final URL.
Destination 304 Not Modified responses still use an HTTP 200 Web Access response and carry 304 in the JSON
envelope's statusCode or the pass-through format's x-status-code. See
Conditional revalidation.
A destination's own 5xx page, such as its 500 or 504, is its answer too: the API responds with HTTP 200, the
destination's status is in statusCode and x-status-code, and the request is billed like
any other destination response. Check statusCode before you use data.
With screenshot/actions, the body includes finalUrl, data (final HTML by default, or converted Markdown with
format: markdown), and an optional screenshot: { format, base64 }. When the sequence succeeded and its
waitForResponse action with asResult matched, data is that response's body instead and matchedResponse
describes it.
With captureXHR, the JSON body also carries xhr, the list of captured requests, and xhrTruncated: true when a
capture limit left requests or bodies out. See XHR capture for the entry
shape.
With jsonSchema, the body is the extracted object when extraction succeeds. A
fallback returns the JSON envelope above with the unextracted
page as data, plus extracted: false and a machine-readable reason. The x-json-schema-applied and
x-json-schema-fallback-reason headers mirror those body fields.
See Response formats, Screenshots and Browser actions for each shape.
Response fields
The JSON envelope's fields. A field marked "when true" is absent otherwise.
| Field | Present | Description |
|---|---|---|
statusCode | Always, except with actions when no page loaded | The destination's status. With actions, the final page's. |
headers | Always, except with actions when no page loaded | The destination's response headers. With actions, the final page's. |
data | Always | The page, in the requested format. |
finalUrl | Always | See Final URL. |
xhr | With captureXHR | The captured requests. Each carries frameUrl and frameId when known. |
xhrTruncated | When true | A capture limit left requests or bodies out of xhr, or the page's requests could not be captured for this response. |
iframes | With captureIframes | The final page's iframes. |
iframesTruncated | When true | includeIframes or captureIframes left an iframe or its content out. See Limits. |
cookies | With captureCookies | The cookies the browser would send to the final page URL; empty when they could not be read. |
cookiesTruncated | When true | The cookies could not be read for this response, so cookies is empty. |
waitUntilTimedOut | When true | The page was read, or the actions ran, before it reached the waitUntil state. |
screenshot | With a screenshot action or screenshot: true | { format, base64 }. |
matchedResponse | With a waitForResponse that sets asResult | The matched response, when the sequence succeeded and it matched. |
error | With actions, when the sequence failed | The failure, such as Browser action sequence failed. |
failedActionIndex | With actions, when an action failed | 0-based index into actions of the action that failed. |
timedOutActions | With actions, when a wait ran out | 0-based indexes into actions of the waitForNetworkIdle actions that ran out and let the sequence carry on. |
extracted, reason | With jsonSchema, on a fallback | extracted: false and the fallback reason. |
Response headers
Besides the headers every response carries, these report on the
features above. x-iframes-truncated, x-wait-until-timed-out and x-redirect-url are sent in every format, including raw,
markdown and an extracted jsonSchema response:
| Header | Value |
|---|---|
x-iframes-truncated | true when an iframe or its content was left out, as iframesTruncated reports. Absent otherwise. |
x-wait-until-timed-out | true when the waitUntil wait ran out, as waitUntilTimedOut reports. Absent otherwise. |
x-json-schema-applied | With jsonSchema: whether the body is the extracted object. |
x-json-schema-fallback-reason | With jsonSchema, on a fallback: the reason. |
x-redirect-url | With maxRedirects or followRedirects: false, when the response is the redirect the request stopped on: the absolute URL it points to. Absent otherwise. See Redirects. |