Jobs and export

A large job (a walk over thousands of creators, say) has more results than you want to page through. Export them instead: one request starts the export, polling it returns download links.

FlowCopy Anchor Link

POST /v1/jobs/{id}/export   -> 202, Location, Retry-After: 5   (state: pending)
GET  /v1/jobs/{id}/export   -> 202 while pending or running
GET  /v1/jobs/{id}/export   -> 200, state: done, parts[] with URLs
GET  <part url>             -> the gzipped NDJSON file (no API key)

The job must be finished first: done, failed or canceled, with finished_at set. A job still queued or running gets 409 job_not_finished.

Start an exportCopy Anchor Link

curl -s -X POST "https://scraper.riken.me/v1/jobs/685/export" -H "x-api-key: $RIKEN_API_KEY"

{"job_id": 685, "state": "pending", "format": "ndjson.gz", "poll_url": "/v1/jobs/685/export",
 "parts": [], "rows_total": 0, "bytes_total": 0, "credits_charged": "0", "attempts": 0}

Idempotent. POST again and you get the same export, not rebuilt and not charged. It is rebuilt only if the last export failed or the job finished again after it was built.

Poll and downloadCopy Anchor Link

curl -s "https://scraper.riken.me/v1/jobs/685/export" -H "x-api-key: $RIKEN_API_KEY"

{"job_id": 685, "state": "done", "format": "ndjson.gz",
 "sections": {"creators": 9720, "media": 299818, "walks": 9731},
 "parts": [
   {"section": "creators", "part": 0, "rows": 9720,  "bytes": 1350211, "url": "https://..."},
   {"section": "media",    "part": 0, "rows": 50000, "bytes": 6012345, "url": "https://..."}
 ],
 "rows_total": 319269, "urls_expire_at": "2026-11-05T23:41:00Z", "credits_charged": "0"}

The URL is the credential, so send no API key with it:

curl -s -o media-0000.ndjson.gz "<url>"
gunzip -c media-0000.ndjson.gz | head -1 | jq .

What is in a fileCopy Anchor Link

• One section per file, up to 50,000 rows each. A section with no rows has no file.
• One JSON object per line, identical to the object GET /v1/jobs/{id} returns in that list, so one parser reads both.
• Stored as gzip (Content-Type: application/gzip), so your client never unzips it silently.
• The whole export is read in one database snapshot: no row skipped or repeated, and no count changes between parts.

Paging insteadCopy Anchor Link

GET /v1/jobs/{id}?results=true pages 500 rows at a time, free.

• Creators and media come in id order, so every page costs the same however deep you go.
• Send sections= to read one list only, for example sections=walks with walk_cursor.
• Page a job only after it is done. While it runs, a new row can land behind your cursor.
• An old cursor gets 400 cursor_expired: start again without one.

ErrorsCopy Anchor Link

Only the key that submitted a job can read or export it.