3 ms·
Ten million items is a lot for real-time on-demand code. I rarely deal with more than a few hundred items. Let's compare: dataset_1mil = range(1000000)
by shazow 15y ago
Ten million items is a lot for real-time on-demand code. I rarely deal with more than a few hundred items. Let's compare:
dataset_1mil = range(1000000)
dataset_1k = range(1000)
dataset_100 = range(100)
[random.shuffle(d) for d in (dataset_1mil, dataset_1k, dataset_100)]
%timeit sorted(dataset_1mil, reverse=True)[:10]
# 1 loops, best of 3: 1.12 s per loop
%timeit heapq.nlargest(10, dataset_1mil)
# 1 loops, best of 3: 330 ms per loop
%timeit heapSearch(dataset_1mil, 10)
# 1 loops, best of 3: 268 ms per loop
%timeit sorted(dataset_1k, reverse=True)[:10]
# 1000 loops, best of 3: 302 us per loop
%timeit heapq.nlargest(10, dataset_1k)
# 1000 loops, best of 3: 248 us per loop
%timeit heapSearch(dataset_1k, 10)
# 1000 loops, best of 3: 346 us per loop
%timeit sorted(dataset_100, reverse=True)[:10]
# 10000 loops, best of 3: 22 us per loop
%timeit heapq.nlargest(10, dataset_100)
# 10000 loops, best of 3: 46.9 us per loop
%timeit heapSearch(dataset_100, 10)
# 10000 loops, best of 3: 128 us per loop
There you have it. Sorted is best for small datasets in the hundreds, break-even is in the thousands, heapSearch wins in the millions.
Datastructures are great, but be aware of the strengths and weaknesses of their implementations. Most recently I discovered that checking collisions in many small sets is faster with tuples (creating sets is rather slow). Test things with your data, not just "big data".
Bonus tip: Amortize when you can. If you're sorting your data anyways, then even the cleverest heap won't undo work already done.