String docs

POST /fetch

Full request and response reference for the Web Access API fetch endpoint.

Executes an HTTP/HTTPS request to a URL and returns the result, deciding JavaScript rendering, proxy tier and captcha handling from the url alone.

POST https://request.usestring.ai/v1/fetch

Request body

Unknown fields are rejected. url is the only required field, and a request with only url is the recommended one. Pages that need JavaScript are rendered without executeJS.

FieldTypeDefaultDescription
urlstring (URL)—Required. The http/https URL to fetch.
methodGET POST PUT PATCH DELETE HEAD OPTIONSGETHTTP method, in any case (get and Post work). Any other method returns a 400 that names it.
bodystring | object—Request body. Forbidden on GET and HEAD. A string is sent as given; an object is sent as the exact JSON text you wrote.
formatjson raw markdownjsonResponse format.
markdownModereadable fullfullMarkdown preservation level. Only affects Markdown responses.
mainContentOnlybooleanfalseRemove page chrome. Only affects Markdown responses.
jsonSchemaobject—JSON Schema to extract into. Requires format: json.
executeJSboolean—Render the page in a browser on every request. Pages that need it are rendered without this field. Required by captureXHR. Incompatible with headers.
requireWSSboolean—Require a browser WebSocket fetch path. Incompatible with headers.
waitUntildomcontentloaded load networkidle0 networkidle2 networkidle—Wait for this state before the page is read, or with actions before the first action runs. networkidle is the same as networkidle0. Requires executeJS: true, actions or screenshot.
captureXHRboolean | object—Return the page's XHR and fetch() requests as xhr. true captures all of them; { "include": [...] } only those matching a filter; { "iframes": true } adds the requests of cross-site iframes. Requires executeJS: true.
includeIframesboolean—Inline each iframe's content into data, in every format and for jsonSchema. Requires executeJS: true, actions or screenshot.
captureIframesboolean | object—List the page's iframes as iframes. true returns each one's HTML; { "include": [...] } only the HTML of iframes whose URL matches a filter. Requires executeJS: true, actions or screenshot.
captureCookiesboolean—Enterprise. Return the cookies the browser would send to the final page URL, as cookies. Requires one of executeJS: true, actions or screenshot. Incompatible with jsonSchema.
headersobject—Custom headers (max 50). Direct fetch only.
countryCodestring(2)—ISO 3166-1 alpha-2 proxy country.
solveCaptchabooleantrueCaptcha solving. false fails on a challenge.
ignoreCertificateErrorsbooleanfalseSkip TLS certificate verification for this request.
maxRedirectsinteger (0–10)—Follow at most this many redirects, then return the redirect response. Cannot be combined with followRedirects: false, executeJS, requireWSS, captureXHR, actions or screenshot.
followRedirectsbooleantruefalse returns the destination's redirect response instead of following it, the same as maxRedirects: 0. false cannot be combined with maxRedirects, executeJS, requireWSS, captureXHR, actions or screenshot.
screenshotboolean—Capture a screenshot. Shorthand for one screenshot action.
actionsarray—Browser actions to run (1–50).

Field compatibility

  • body is forbidden on GET and HEAD.
  • jsonSchema requires format: json.
  • markdownMode and mainContentOnly are accepted with any format, but are ignored unless format: markdown.
  • headers cannot be combined with executeJS or requireWSS.
  • captureXHR requires executeJS: true, including with actions or screenshot, and cannot be combined with jsonSchema. Without actions or screenshot it also requires format: json.
  • waitUntil, includeIframes and captureIframes require executeJS: true, actions or screenshot.
  • captureIframes cannot be combined with jsonSchema. Without actions or screenshot it also requires format: json. includeIframes works with every format and with jsonSchema.
  • captureCookies requires one of executeJS: true, actions or screenshot, and cannot be combined with jsonSchema. Without actions or screenshot it also requires format: json.
  • A wait action with frame requires selector. See Act inside an iframe.
  • At most one waitForResponse action can set asResult. That request requires format: json and cannot use jsonSchema. See Return a response as the result.
  • screenshot and actions are mutually exclusive. When either is set, format: markdown, markdownMode, and mainContentOnly are supported, but format: raw, a non-GET method, body, headers, and solveCaptcha: false are not.
  • jsonSchema is supported with browser actions unless the request also asks for a screenshot.
  • followRedirects: false and maxRedirects cannot be combined, and neither can be combined with executeJS, requireWSS, captureXHR, actions or screenshot.

Requests only some fetch paths can send

A request with a method other than GET, or with a body, is sent only on fetch paths that deliver its method and body to the destination exactly as given. A browser path loads a page with a bodiless GET, so it never carries one. When no fetch path available for the destination can send the request, it returns a 400 before anything is sent, for example:

No fetch method available for https://example.com can send a POST request with a body; only a GET without a body is supported here

The request is never sent as a GET instead. Send it without a body if the destination accepts that.

Redirects

By default a request follows the destination's redirects and returns the page they lead to. Set maxRedirects to follow at most that many, or followRedirects: false to follow none. When the destination redirects more times than that, the redirect the request stopped on is the response:

  • The API responds with HTTP 200. statusCode in the JSON body, or x-status-code in every format, is the redirect's status, such as 301 or 302.
  • x-redirect-url is the absolute URL the redirect points to. The redirect's Location header is in headers with format: json, and is a response header with format: raw.
  • x-final-url and finalUrl are the URL that answered with the redirect.

A redirect returned this way is billed like any other destination response.

With maxRedirects or followRedirects: false, the page is never rendered in a browser, so a page that builds itself with JavaScript comes back as served. When no fetch path for the destination can follow the request's redirects this way, the request returns a 400 naming both fields. Send it without them to fetch the page.

TLS certificate verification

Certificates are verified by default: a target whose certificate is expired, issued for another hostname, self-signed, or served with an incomplete chain fails with a TargetCertificateError. Set ignoreCertificateErrors: true to fetch it anyway. It applies to that one request only, and is never inferred from a failure.

Without verification, nothing guarantees the response came from the destination

Certificate verification is what proves a response came from the destination. With it off, anything on the network path can answer in the destination's place, and this API returns that answer as the destination's response. Skip verification only for public pages on sites with neglected certificates, never for credentials, payments, or data you will act on as authoritative.

The flag applies to executeJS and requireWSS requests too. Browser sessions driven by actions or screenshot always verify certificates.

HTTPS to HTTP fallback

When an https:// URL has no working TLS service, so the connection fails before any response arrives, a GET or HEAD with no body, no headers, no actions or screenshot, and no port or credentials in the URL is retried once over plain http:// on the same host. Other requests are never retried this way. A response served over that fallback carries an x-protocol-fallback-url header with the http:// URL that was fetched, in every format; the header is absent when the fetch stayed on https://. If you need the response to have come over TLS, treat that header as a failure.

HTTP to HTTPS upgrade

An http:// URL is requested over https:// first when it is a GET or HEAD with no body, no headers, no actions or screenshot, no followRedirects: false or maxRedirects, and no port or credentials in the URL. Other http:// requests are sent as they are. The https:// request always verifies the certificate, even with ignoreCertificateErrors: true. If the destination answers over https://, that response is returned, whatever its status. If it doesn't, or its certificate doesn't verify, the request is made over http:// as you sent it.

When the https:// request answers, the response carries an x-protocol-upgrade-url header with that https:// URL, in every format, and x-final-url reports where it landed. The header is absent when the response came over http://: the request went back to your http:// URL, or the https:// page redirected to http://.

Example

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://httpbin.org/json" }'

Retries

A GET, HEAD or OPTIONS request may be tried on more than one fetch path until one of them returns the destination's answer. Only a GET is ever raced: a slow one may be sent on a second path at the same time. HEAD and OPTIONS run on one path at a time.

A write — POST, PUT, PATCH or DELETE — reaches the destination at most once:

  • It runs on one fetch path at a time and is never raced against another.
  • It is sent again on another path only when it provably never reached the destination (the proxy could not connect, or refused the connection before the request left), or when the destination answered with an anti-bot block.
  • After an ambiguous failure, where the connection dropped, was reset, or timed out after the request may already have been sent, it is not retried and the request returns an error.

An error on a write therefore does not mean the destination did not act on it. Check the destination's state before you send the write again.

Responses

StatusMeaning
200Successful fetch. Body shape depends on format (and jsonSchema).
400Validation error — bad input, or a countryCode requests can't be routed from (unsupported_geography). See Proxies & geolocation.
401Invalid or malformed API key, or a request without a key was not accepted. Retry with an active key.
402Insufficient account balance.
403Destination not enabled for your organization, or a path that needs KYC. See Access control & KYC. Also captureCookies on an account without it; see Errors.
429Rate limit exceeded, or XHR capture is at capacity (X-RateLimit-Scope: xhr-capture), or requests to this destination are failing right now (TargetFailingError), or over String's request limit for this destination (TargetRequestLimitError). Retry after Retry-After when present; without it, lower your request rate.
502Upstream/destination error, including a destination we could not reach and a destination that is rate-limiting requests (OriginRateLimitedError). See Errors.
503A feature the request uses is temporarily unavailable: XHR capture or waitForResponse, iframes, or waitUntil and waitForNetworkIdle. Also no exit in the requested countryCode was available for this request (geography_unavailable). Retry; honor Retry-After when present.
504The destination did not respond in time (TargetTimeoutError).
500Internal error.

200 — JSON format

{
  "statusCode": 200,
  "headers": { "content-type": ["application/json"] },
  "data": { "message": "Hello World" },
  "finalUrl": "https://httpbin.org/json"
}

For raw, the body is the gzip-compressed bytes with x-status-code set; for markdown, the body is text/markdown. Every format sets x-final-url to the URL the response was served from, after any redirects or an HTTPS upgrade, and every JSON body carries the same value as finalUrl. Both are also sent on error responses. They are left out only when the request body has no valid url. See Final URL. Destination 304 Not Modified responses still use an HTTP 200 Web Access response and carry 304 in the JSON envelope's statusCode or the pass-through format's x-status-code. See Conditional revalidation. A destination's own 5xx page, such as its 500 or 504, is its answer too: the API responds with HTTP 200, the destination's status is in statusCode and x-status-code, and the request is billed like any other destination response. Check statusCode before you use data.

With screenshot/actions, the body includes finalUrl, data (final HTML by default, or converted Markdown with format: markdown), and an optional screenshot: { format, base64 }. When the sequence succeeded and its waitForResponse action with asResult matched, data is that response's body instead and matchedResponse describes it.

With captureXHR, the JSON body also carries xhr, the list of captured requests, and xhrTruncated: true when a capture limit left requests or bodies out. See XHR capture for the entry shape.

With jsonSchema, the body is the extracted object when extraction succeeds. A fallback returns the JSON envelope above with the unextracted page as data, plus extracted: false and a machine-readable reason. The x-json-schema-applied and x-json-schema-fallback-reason headers mirror those body fields.

See Response formats, Screenshots and Browser actions for each shape.

Response fields

The JSON envelope's fields. A field marked "when true" is absent otherwise.

FieldPresentDescription
statusCodeAlways, except with actions when no page loadedThe destination's status. With actions, the final page's.
headersAlways, except with actions when no page loadedThe destination's response headers. With actions, the final page's.
dataAlwaysThe page, in the requested format.
finalUrlAlwaysSee Final URL.
xhrWith captureXHRThe captured requests. Each carries frameUrl and frameId when known.
xhrTruncatedWhen trueA capture limit left requests or bodies out of xhr, or the page's requests could not be captured for this response.
iframesWith captureIframesThe final page's iframes.
iframesTruncatedWhen trueincludeIframes or captureIframes left an iframe or its content out. See Limits.
cookiesWith captureCookiesThe cookies the browser would send to the final page URL; empty when they could not be read.
cookiesTruncatedWhen trueThe cookies could not be read for this response, so cookies is empty.
waitUntilTimedOutWhen trueThe page was read, or the actions ran, before it reached the waitUntil state.
screenshotWith a screenshot action or screenshot: true{ format, base64 }.
matchedResponseWith a waitForResponse that sets asResultThe matched response, when the sequence succeeded and it matched.
errorWith actions, when the sequence failedThe failure, such as Browser action sequence failed.
failedActionIndexWith actions, when an action failed0-based index into actions of the action that failed.
timedOutActionsWith actions, when a wait ran out0-based indexes into actions of the waitForNetworkIdle actions that ran out and let the sequence carry on.
extracted, reasonWith jsonSchema, on a fallbackextracted: false and the fallback reason.

Response headers

Besides the headers every response carries, these report on the features above. x-iframes-truncated, x-wait-until-timed-out and x-redirect-url are sent in every format, including raw, markdown and an extracted jsonSchema response:

HeaderValue
x-iframes-truncatedtrue when an iframe or its content was left out, as iframesTruncated reports. Absent otherwise.
x-wait-until-timed-outtrue when the waitUntil wait ran out, as waitUntilTimedOut reports. Absent otherwise.
x-json-schema-appliedWith jsonSchema: whether the body is the extracted object.
x-json-schema-fallback-reasonWith jsonSchema, on a fallback: the reason.
x-redirect-urlWith maxRedirects or followRedirects: false, when the response is the redirect the request stopped on: the absolute URL it points to. Absent otherwise. See Redirects.