Async Bulk Requests

Submit multiple runs and poll for results

Overview

The async API pattern is ideal when you want to submit multiple long-running tasks and check their status later. Instead of waiting for each run to complete, you submit all requests and get back run IDs that you can poll for completion.

How It Works

1. Submit requests to /v1/automation/run-async, which returns corresponding run_ids, which you will need if you want to check the status of a particular run.
2. Check individual runs with GET /v1/runs/:id to check status
3. Or fetch all runs with GET /v1/runs to monitor batch progress

Basic Example

Submit multiple TinyFish Web Agent runs and poll for completion:

Python

import asyncio
from tinyfish import AsyncTinyFish, RunStatus

async def wait_for_completion(client, run_id, poll_interval=2):
    """Poll a run until it completes"""
    while True:
        run = await client.runs.get(run_id)

        if run.status in (RunStatus.COMPLETED, RunStatus.FAILED, RunStatus.CANCELLED):
            return run

        await asyncio.sleep(poll_interval)

async def main():
    client = AsyncTinyFish()

    # Define your batch of tasks
    tasks_to_run = [
        {
            "url": "https://scrapeme.live/shop/",
            "goal": "Extract all available products on page two with their name, price, and review rating (if available)",
        },
        {
            "url": "https://books.toscrape.com/",
            "goal": "Extract all available books on page two with their title, price, and review rating (if available)",
        },
    ]

    # Step 1: Submit all tinyfish runs and collect run_ids
    print("Submitting tinyfish runs...")
    submit_tasks = [
        client.agent.queue(url=task["url"], goal=task["goal"])
        for task in tasks_to_run
    ]
    responses = await asyncio.gather(*submit_tasks)
    run_ids = [r.run_id for r in responses]
    print(f"Submitted {len(run_ids)} runs: {run_ids}")

    # Step 2: Wait for all runs to complete
    print("Waiting for completion...")
    completion_tasks = [
        wait_for_completion(client, run_id)
        for run_id in run_ids
    ]
    results = await asyncio.gather(*completion_tasks)

    # Step 3: Process results
    for i, run in enumerate(results):
        print(f"Run {i + 1} ({run.run_id}):")
        print(f"  Status: {run.status}")
        if run.status == RunStatus.COMPLETED:
            print(f"  Result: {run.result}")

# Run the async main function
asyncio.run(main())

Fire and Forget Pattern

Submit tasks without waiting for completion:

Python

async def main():
    client = AsyncTinyFish()

    tasks_to_run = [
        {"url": "https://example.com/page1", "goal": "Extract product info"},
        {"url": "https://example.com/page2", "goal": "Extract product info"},
        {"url": "https://example.com/page3", "goal": "Extract product info"},
    ]

    # Submit all tasks
    submit_tasks = [
        client.agent.queue(url=task["url"], goal=task["goal"])
        for task in tasks_to_run
    ]
    responses = await asyncio.gather(*submit_tasks)
    run_ids = [r.run_id for r in responses]

    print(f"Submitted {len(run_ids)} runs")
    print(f"Run IDs: {run_ids}")
    print("Check status later using client.runs.get(run_id)")

asyncio.run(main())

When to Use Async vs Sync

Use Case	API Pattern	Why
Quick tasks (<30s)	Sync /run	Simpler code, immediate results
Long-running tasks	Async /run-async	Don’t block, check later
Large batches	Async /run-async	Submit all at once, monitor progress
Fire and forget	Async /run-async	No need to wait
Real-time feedback	SSE /run-sse	Stream progress events

Best Practices

Polling Interval

• Short tasks (under 1 min): Poll every 2-3 seconds
• Medium tasks (1-5 min): Poll every 5-10 seconds
• Long tasks (over 5 min): Poll every 30-60 seconds

Error Handling

Always check run status and handle failures:

Python

async def process_completed_run(run):
    if run.status == RunStatus.COMPLETED:
        return run.result
    elif run.status == RunStatus.FAILED:
        print(f"Run {run.run_id} failed: {run.error}")
        return None
    elif run.status == RunStatus.CANCELLED:
        print(f"Run {run.run_id} was cancelled")
        return None

API Reference

Run Async

Start TinyFish Web Agent run asynchronously

List Runs

Get all runs

Get Run by ID

Check individual run status

Cancel Run

Cancel a running automation

Related

Concurrent Requests

Sync API for immediate results

Web Scraping

Extract data from pages