Fetch and extract content from URLs

Fetches web pages, renders JavaScript-heavy pages when needed, and returns clean extracted content in your preferred format. Submit up to 10 URLs, get back structured content. Per-URL failures appear in errors[] and do not fail the entire request.

Per-URL error codes (in errors[].error):

• target_http_error — target server returned a non-2xx HTTP status other than 404/410; the raw status code is in errors[].status
• page_not_found — target URL returned HTTP 404 or 410; the raw status code is in errors[].status
• target_unreachable — connection refused, TLS failure, DNS failure, or other network error
• timeout — request timed out
• proxy_error — proxy tunnel failure
• bot_blocked — bot-challenge page detected (Cloudflare, etc.)
• empty_content — page loaded but no extractable text was found
• invalid_url — malformed URL or SSRF-blocked address
• invalid_redirect_url — redirect target rejected before fetch
• conditional_unsupported — conditional requests (if_none_match / if_modified_since) are supported on the fast path only; this URL requires browser rendering
• selector_not_matched — no elements matching any include_selectors entry remained after exclude_selectors was applied; the error carries unmatched_selectors plus candidate_selectors retry hints (a partial miss is not an error — it’s reported on the result’s unmatched_selectors)
• selector_unsupported — include_selectors / exclude_selectors sent for a URL that resolves to a direct PDF/CSV download (no HTML to scope)

POST /

Fetch and extract content from URLs

cURL

curl --request POST \
  --url https://api.fetch.tinyfish.ai/ \
  --header 'Content-Type: application/json' \
  --header 'X-API-Key: <api-key>' \
  --data '
{
  "urls": [
    "https://example.com",
    "https://example.org"
  ],
  "purpose": "Compare pricing tiers across vendors for a procurement report",
  "format": "markdown",
  "links": false,
  "image_links": false,
  "ttl": 0,
  "per_url_timeout_ms": 45000,
  "if_none_match": "W/\"abc123\"",
  "if_modified_since": "Wed, 21 Oct 2015 07:28:00 GMT",
  "include_etag_and_last_modified": true,
  "include_selectors": [
    "article"
  ],
  "exclude_selectors": [
    ".comments",
    ".newsletter-signup"
  ]
}
'

200

{
  "results": [
    {
      "url": "https://example.com",
      "final_url": "https://www.example.com",
      "title": "Example Domain",
      "description": "This domain is for use in illustrative examples.",
      "language": "en",
      "format": "markdown",
      "text": "<string>",
      "author": "John Doe",
      "published_date": "2024-01-15",
      "links": [
        "<string>"
      ],
      "image_links": [
        "<string>"
      ],
      "latency_ms": 1183.4,
      "not_modified": true,
      "etag": "W/\"abc123\"",
      "last_modified": "Wed, 21 Oct 2015 07:28:00 GMT",
      "unmatched_selectors": [
        "aside.related"
      ]
    }
  ],
  "errors": [
    {
      "url": "https://invalid.example.com",
      "error": "target_http_error",
      "status": 404,
      "unmatched_selectors": [
        "article",
        "#content"
      ],
      "candidate_selectors": [
        "main",
        "nav",
        "#content"
      ]
    }
  ],
  "request_id": "8f9dba20-e37b-4749-a919-2269e28b4a2c"
}

Authorizations

X-API-Key · string · header · required

API key for authentication. Get your key from the API Keys page.

Body

application/json

URLs to fetch and extraction options

urls · string<uri>[] · required

Array of URLs to fetch (1-10). All URLs are fetched in parallel. Each URL is processed independently — if one fails, others still return successfully. Errors are reported per-URL in the errors array.

Required array length: 1 - 10 elements

Example:

[
  "https://example.com",
  "https://example.org"
]

purpose · string

Why these URLs are being fetched — the underlying goal or task the content will be used for. Used to better tailor fetching and extraction to your intent.

Required string length: 1 - 2000

Example:

"Compare pricing tiers across vendors for a procurement report"

format · enum<string> · default: markdown

Output format for extracted content. "markdown" (default) is ideal for LLM consumption. "html" returns cleaned semantic HTML. "json" returns a structured document tree.

Available options: markdown, html, json

Example:

"markdown"

links · boolean · default: false

Extract all outbound links () from each page. Useful for discovering related pages or navigating to specific content. Links are returned as absolute URLs in the links array of each result. [blocked]

Example:

false

image_links · boolean · default: false

Extract all image URLs ([Image blocked: No description]) from each page. Useful for finding visual content or media assets. Image links are returned as absolute URLs in the image_links array of each result.

Example:

false

ttl · integer

Caller freshness tolerance in seconds for the cached entry. Omit (default) for unlimited tolerance — any cached entry is acceptable. Set to 0 to prefer a live fetch; a cached entry is still served if the origin's Cache-Control: max-age covers its age, or the host is in the small allowlist of operator-pinned never-expire domains. Set to N > 0 to accept a cached entry whose age is below N; the upstream Cache-Control: max-age and the never-expire allowlist may extend (never shorten) this tolerance.

Required range: x >= 0

Example:

0

per_url_timeout_ms · integer

Wall-clock timeout budget in milliseconds applied independently to each URL. If one URL exceeds this budget, it returns a per-URL timeout error while other URLs in the same request continue.

Required range: 1 <= x <= 110000

Example:

45000

if_none_match · string

ETag validator from a prior fetch of this URL, forwarded verbatim as the If-None-Match header on the origin request. Only valid with a single URL — combining with a batch of URLs returns a 400. tf-fetch does not persist validators; the caller owns replaying them.

Minimum string length: 1

Example:

"W/\"abc123\""

if_modified_since · string

Last-Modified validator from a prior fetch of this URL, forwarded verbatim as the If-Modified-Since header on the origin request. Only valid with a single URL — combining with a batch of URLs returns a 400. tf-fetch does not persist validators; the caller owns replaying them.

Minimum string length: 1

Example:

"Wed, 21 Oct 2015 07:28:00 GMT"

include_etag_and_last_modified · boolean

Opt-in to receiving etag / last_modified validators (and not_modified detection) on each result. Defaults to false — tf-fetch omits these fields unless requested. Independent of if_none_match / if_modified_since: works with a single URL or a batch.

Example:

true

include_selectors · string[]

Array of CSS selectors (1-20 entries, each 1-1000 characters) that scope extracted content (text, links, image_links) to elements matching ANY entry, concatenated in document order. Tag selectors cover semantic sections (main, article, nav); entries may themselves use CSS comma-grouping. Selected content is returned verbatim in the requested format (scripts/styles stripped) — automatic boilerplate removal is bypassed. Page-level metadata (title, description, language, author, published_date) still comes from the full document. When some entries match and others do not, the URL still succeeds and the misses are reported in the result's unmatched_selectors. When no entry matches anything, that URL fails with the per-URL error code selector_not_matched (carrying unmatched_selectors and candidate_selectors retry hints) — never a silent full-page fallback. URLs that resolve to direct PDF/CSV downloads fail with selector_unsupported. Invalid CSS selector syntax is rejected with a 422. Applied post-fetch: caching and routing are unchanged.

Required array length: 1 - 20 elements

Required string length: 1 - 1000

Example:

["article"]

exclude_selectors · string[]

Array of CSS selectors (1-20 entries, each 1-1000 characters) for elements to remove before extraction — applied before include_selectors scopes what remains, so it also prunes inside selected regions. Entries may themselves use CSS comma-grouping. Entries that match nothing are a no-op, never an error, but URLs that resolve to direct PDF/CSV downloads fail with selector_unsupported. Invalid CSS selector syntax is rejected with a 422. Applied post-fetch: caching and routing are unchanged.

Required array length: 1 - 20 elements

Required string length: 1 - 1000

Example:

[".comments", ".newsletter-signup"]

Response

200

application/json

Fetch completed. Check errors[] for any per-URL failures.

Fetch response with results and errors

results · object[] · required

Successfully fetched URLs

Show child attributes

errors · object[] · required

URLs that failed to fetch

Show child attributes

request_id · string

Request correlation ID, also returned as the X-Request-ID response header. Present only when errors is non-empty; include it when reporting failed URLs.

Example:

"8f9dba20-e37b-4749-a919-2269e28b4a2c"