Extract a Product Catalog

Use this guide when you need a repeatable catalog export: product names, prices, availability, URLs, or fields found on individual product pages.

Start with the listing page

Give Gluecrawl the category, collection, or search-results page that contains the repeating product cards. It maps the list, detects pagination, and can identify linked detail pages with additional fields.

For example, create a job from a collection page with a goal:

{
  "url": "https://example.com/collections/keyboards",
  "input": {
    "type": "goal",
    "value": "Extract each product's name, price, availability, and product URL"
  },
  "max_pages": 10
}

Send that body to POST /v1/jobs. Use max_pages to set the maximum number of listing pages Gluecrawl may crawl for this job.

Review the mapped job

Poll the job until it is ready. The response shows the fields Gluecrawl mapped under columns.listing and any extra product-page fields under columns.detail.

{
  "status": "ready",
  "columns": {
    "listing": [
      { "name": "product_name", "type": "text" },
      { "name": "price", "type": "number" },
      { "name": "product_url", "type": "url" }
    ],
    "detail": [
      { "name": "description", "type": "text" }
    ]
  },
  "protection_level": "light"
}

Gluecrawl chooses the protection level during processing. It also discovers detail-page extraction automatically when the catalog links to individual product pages; you do not configure a second crawler for those links.

Run the catalog scrape

Create a run only after the job is ready:

POST /v1/jobs/JOB_ID/runs

{ "max_pages": 10 }

Then poll GET /v1/runs/{id} until the run is completed. The completed run reports the final item_count, page_count, credits_used, and billing breakdown.

Retrieve or export rows

For programmatic processing, request a page of JSON items:

GET /v1/runs/RUN_ID/items?limit=50&offset=0

Each item stores extracted fields in data, plus its source page_number and item_index. Continue with larger offset values until you have read total items. See Export data and paginate results for a complete loop.

For a spreadsheet or data warehouse import, download the completed run as CSV.

Control scope and cost

• Keep max_pages close to the catalog size you need. It limits listing pages, not API result pagination.
• Detail-page fields add page visits. Their cost depends on the job's protection level.
• A run's final credits_used and billing fields show the work that was actually settled. Review Credits before scheduling large catalogs.

Once a job is ready, create another run whenever you need a fresh catalog snapshot. To automate that cadence, attach a schedule.