Page through results
List endpoints return results in batches. There are two styles.
| Endpoints | Batch size | Next batch | Total |
|---|---|---|---|
/search.json |
limit |
offset, or page |
numFound |
/works/{id}/editions.json, /authors/{id}/works.json |
limit |
links.next in the response |
size |
Page through search results
Set limit to the batch size and offset to the index of the first result you want. offset starts at 0.
import time
import requests
HEADERS = {"User-Agent": "openlibrary-unofficial-docs/0.1"}
LIMIT = 3
for page in range(2):
params = {"q": "dune", "fields": "key,title", "limit": LIMIT, "offset": page * LIMIT}
response = requests.get(
"https://openlibrary.org/search.json", params=params, headers=HEADERS, timeout=10
)
response.raise_for_status()
data = response.json()
print(f"start={data['start']} of {data['numFound']}")
for book in data["docs"]:
print(" ", book["key"], book["title"])
if data["start"] + LIMIT >= data["numFound"]:
break
time.sleep(1) # stay under 1 request per second
Output:
start=0 of 48279
/works/OL893414W Dune
/works/OL893461W Dune Messiah
/works/OL893516W Children of Dune
start=3 of 48279
/works/OL893515W God Emperor of Dune
/works/OL893502W Heretics of Dune
/works/OL893508W Chapterhouse Dune
The sample stops after two pages on purpose. This search has 48,279 matches, so loop over all of them only if you need them.
page instead of offset
page counts from 1, in steps of limit. These two requests returned the same three results, and both had start: 3:
/search.json?q=dune&fields=key,title&limit=3&offset=3
/search.json?q=dune&fields=key,title&limit=3&page=2
With page, the response field offset is null. Read start to know where the batch begins.
Follow links.next on list endpoints
The editions and author works endpoints return a links object:
curl -s -A "openlibrary-unofficial-docs/0.1" \
"https://openlibrary.org/authors/OL26320A/works.json?limit=2"
Output (trimmed):
{
"links": {
"self": "/authors/OL26320A/works.json?limit=2",
"author": "/authors/OL26320A",
"next": "/authors/OL26320A/works.json?limit=2&offset=2"
},
"size": 359,
"entries": [
{"key": "/works/OL46165393W", "title": "Der Schmied von GroĆholzingen"},
{"key": "/works/OL46147739W", "title": "El Hobbit"}
]
}
Request https://openlibrary.org + links.next to get the next batch.
Tips
- Results can shift between requests as the catalog changes. Do not treat page boundaries as stable IDs.
- Send at most 1 request per second, or 3 per second if you identify your app. See Handle errors and rate limits.
- For large exports, Open Library asks you to use its monthly data dumps instead of paging through the API.