26 ms·

Any examples of how it handles pagination? Couldn't find it in the README.md

• simonw 9y ago
  Good catch, I haven't documented that yet. There's a hard limit of 1,000 rows allowed to be returned at any one time. This is because the overhead of serializing more JSON that than could potentially lock up the server. If you try to pull more than that back you get a warning that the data has been truncated, e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+*+from+%5Btwitter-ratio%2Fsenators%5D https://fivethirtyeight.datasettes.com/fivethirtyeight-2628d... - in the JSON that shows up as "truncated": true If that happens to you using raw SQL, it's up to you to implement offset/limit pagination. If you are looking at a view, the tool implements offset/limit pagination for you e.g. https://parlgov.datasettes.com/parlgov-25f9855/view_election?_next=100 https://parlgov.datasettes.com/parlgov-25f9855/view_election... For iterating through a raw table, a more efficient mechanism is used: cursor-based pagination. The table is ordered by its primary key (or its rowid if no primary key was specified) and the "next" links say "get the next 100 items where the primary key is greater than this value" - e.g. https://parlgov.datasettes.com/parlgov-25f9855/cabinet_party?_next=100 https://parlgov.datasettes.com/parlgov-25f9855/cabinet_party... This means even a table with millions of records can be efficiently paginated through - unlike offset/limit which tends to get slow after a few hundred thousand records.