3 ms·

Lets not kid ourselves, a few million rows is not small either and if not done properly can quickly become a system's bottleneck. But it's not humongous either.

Lets not kid ourselves, a few million rows is not small either and if not done properly can quickly become a system's bottleneck. But it's not humongous either. Depending on ones perspective it could be seen as medium or large but certainly not small.

nomel 5y ago
I think it's safe to call anything that fits within the memory of a mid range notebook computer "small". I guess it depends on how wide those tables are.
- bigiain 5y ago
  The data scientist I've worked with that I most respect once told me "It's not 'big data' if it fits on your laptop". Anything that fits in ram, at least in the context of "a table" in a RDBM probably counts as 'micro data'...
bigiain 5y ago
Anybody who has a "bottleneck" caused by a table with a few million rows is functionally incompetent. From any half-modern database's perspective that's "tiny". Any reasonable production machine would be able to fit (at least the important parts of) a million row table in memory. Even if this is your hobby project running on a RaspberryPi Zero, you'd have at least the index in ram. Either you don't care about that query's runtime (a perfectly valid approach if your use case is "this creates a report after COB on Friday that needs to have completed before 9am Monday morning"), or you need someone who knows what they're doing take over from you (also a valid approach if this is in your tech demo or POC, and you're the Technical Founder who's now out of their depth and needs a proper engineer to build a scalable product for you).
hnthrowaway0315 5y ago
I mean Excel can handle a few million if use Power Pivot.
pdpi 5y ago
A million 1k-wide rows is still just 1GB. You can go double that width and triple the rows, and you still comfortably fit in memory on a raspberry pi. I don’t know about “big” or “small”, but the threshold for “done properly” has to be really low for that to become a bottleneck.

nomel 5y ago

I think it's safe to call anything that fits within the memory of a mid range notebook computer "small". I guess it depends on how wide those tables are.

bigiain 5y ago
The data scientist I've worked with that I most respect once told me "It's not 'big data' if it fits on your laptop". Anything that fits in ram, at least in the context of "a table" in a RDBM probably counts as 'micro data'...

bigiain 5y ago

Anybody who has a "bottleneck" caused by a table with a few million rows is functionally incompetent. From any half-modern database's perspective that's "tiny". Any reasonable production machine would be able to fit (at least the important parts of) a million row table in memory. Even if this is your hobby project running on a RaspberryPi Zero, you'd have at least the index in ram. Either you don't care about that query's runtime (a perfectly valid approach if your use case is "this creates a report after COB on Friday that needs to have completed before 9am Monday morning"), or you need someone who knows what they're doing take over from you (also a valid approach if this is in your tech demo or POC, and you're the Technical Founder who's now out of their depth and needs a proper engineer to build a scalable product for you).

hnthrowaway0315 5y ago

I mean Excel can handle a few million if use Power Pivot.

pdpi 5y ago

A million 1k-wide rows is still just 1GB. You can go double that width and triple the rows, and you still comfortably fit in memory on a raspberry pi. I don’t know about “big” or “small”, but the threshold for “done properly” has to be really low for that to become a bottleneck.