Export Data and Paginate Results

After a run completes, choose JSON when another program will process the rows and CSV when you need a file download.

Wait for a completed snapshot

Items can appear progressively while a run is in progress. For final counts and a stable result set, poll GET /v1/runs/{id} until status is completed before paging through the items.

The completed response includes item_count, page_count, credits_used, and the final billing breakdown.

Page through JSON items

GET /v1/runs/{id}/items uses limit and offset query parameters:

• limit is the number of items to return in one response. It defaults to 50 and can be at most 200.
• offset is the number of items to skip. It defaults to 0.

const allItems = []
const limit = 200
let offset = 0

while (true) {
  const response = await fetch(
    `https://api.gluecrawl.ai/v1/runs/RUN_ID/items?limit=${limit}&offset=${offset}`,
    { headers: { Authorization: `Bearer ${process.env.GLUECRAWL_API_KEY}` } },
  )
  const page = await response.json()

  if (!response.ok) throw new Error(page.error?.message ?? 'Request failed')

  allItems.push(...page.items)
  offset += page.items.length

  if (offset >= page.total || page.items.length === 0) break
}

console.log(allItems)

Each item has this shape:

{
  "data": {
    "product_name": "Wireless Keyboard",
    "price": 49.99
  },
  "page_number": 1,
  "item_index": 0
}

data contains the fields defined by the ready job's mapping. page_number and item_index let you trace a row back to its place in the scrape.

Download CSV

Use the CSV endpoint for a streamed file with headers derived from the scraped data:

curl -L "https://api.gluecrawl.ai/v1/runs/RUN_ID/items/csv" \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY" \
  -o export.csv

The CSV response is a stream, not JSON. See Download Items (CSV) for Node.js and Python download examples.

Do not confuse the two kinds of pagination

• max_pages on a job, run, or schedule limits how many source listing pages Gluecrawl crawls.
• limit and offset on the items endpoint paginate the already extracted rows returned by the API.

Set max_pages based on the source catalog or directory size. Use limit and offset only to retrieve the resulting rows efficiently.