3 ms·
Edit: https://gist.github.com/alecco/9976dab8fda8256ed403054ed0a65d7b https://gist.github.com/alecco/9976dab8fda8256ed403054ed0a65... I think using a range of
by alecco 6y ago
Edit: https://gist.github.com/alecco/9976dab8fda8256ed403054ed0a65d7b https://gist.github.com/alecco/9976dab8fda8256ed403054ed0a65...
I think using a range of rows is overkill, at least for row-stores. And also in the majority of cases random rows are preferred than a range.
In the case where the table has a simple Primary Key the query is easier. Select all the valid PKs (rows) ordered by random and then limit.
SQLite gives access to the rowid making this query even simpler and likely faster (no need for PK and the query works on tables without a PK).
SELECT * FROM test
WHERE rowid IN
(SELECT rowid FROM test
ORDER BY random() LIMIT 10);
or with a more verbose JOIN:
SELECT * FROM test JOIN
(SELECT rowid as rid
FROM test ORDER BY random() LIMIT 10) AS srid
ON test.rowid = srid.rid;
The database engine tracks existing rows by some sort of id with its own internal rowid/PK index structure. Materializing these IDs should not be that expensive and as it's sequential access it should be pretty fast. The expensive part is the ORDER BY random().
If your table is truly big, say billions of rows, this could be improved by reducing the list of rowids with a WHERE clause.
But don't overdo it or you'll affect the truer randomness. For most cases just reduce to hundreds of thousands.
For whatever reason, using this filtered (WHERE), the JOIN query to generates a seemingly better SQLite plan.
SELECT * FROM test JOIN
(SELECT rowid as rid FROM test
WHERE random() % 10 = 0 -- Reduce rowids
ORDER BY random() LIMIT 10) AS srid
ON test.rowid = srid.rid;
The manual '% 10' filter could be improved with some calculation of the table's row count, minding small tables. Left as exercise.
- tpetry 6y agoFor every pf your examples the database will do a full table scan. If i am wrong feel free to correct me. But i tried multiple approaches more than the ones discussed and as soon as you start using random() the database will scan every record.
- alecco 6y ago1. Picking a random sub-range is not the same thing as random sampling. See other comments here and all over the Internet. 2. To truly random sample k out of a set of n rows it's needed to know the set first. 3. Walking the rowid B-Tree should be relatively fast (and necessary because of 2.) It may be the case a sub-range is good enough (e.g. the table is quite disordered relatively to the properties looked after). But this is very unusual and you should warn the users of this data because later on they might change Imagine your sub-range picks rows created over the weekend or some other particular time-frame. Or an import from some other legacy system. This will not be representative of the full set in many ways. If the system needs to do a lot of sampling queries, perhaps it would be better to make overnight an auxiliary table. You can make many sample tables and compare which ones deviate less from the actual table. This table will be small and could even have materialized views pre-computed with the most expensive computations.