3 ms·

10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.

by gaius 8y ago

Groxx 8y ago
Yeah, or R. When I need to do stuff bigger than RAM on a laptop I just reach for sqlite (assuming it can even fit on disk).
- gaius 8y ago
  The funny thing is we didn’t even consider our 50Tb to be particularly big, we knew guys running 200Tb... BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.
  - Groxx 8y ago
    Oh yeah, the default dataframe is practically only for toy problems / very-already-filtered data (which accounts for a LOT of useful scenarios!). It's more that I've run into "big data" people who (╯°□°)╯︵ ┻━┻ when the default settings + dataframe choke, which is around 200MB of heap.
Swinx43 8y ago
This, a million times this!!! I see people doing consulting promoting the most over engineered solutions I have ever seen. Doing it for data that is a few hundred GB or maybe at best a TB. It makes me want to cry, knowing we handled that with a single server and a relational database 10 years ago. Lets also not forget that everyone today forgets that the majority of data actually has some sort of structure. There is no point in pretending that every piece of data is a BLOB or a JSON document. I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises.
- gaius 8y ago
  I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises I sometimes wish I was unscrupulous enough to cash in on these trends, I’d be a millionaire now. Instead I’m just a sucker who tries to make solid engineering decisions on behalf of my employers and clients. It’s depressing to think what professionalism has cost me in cash terms. But you’ve got to be able to look at yourself in the mirror.
- therealdrag0 8y ago
  Yep. I left consulting after working on a project that was a Ferrari when the customer only needed a Honda. Our architect kept being like "it needs to be faster", and I'm like "our current rate would process the entire possible dataset (everyone on earth) in a night, do we really need a faster Apache Storm cluster?" :S "Resume driven development" is a great term.
- attaboyjon 8y ago
  It's important to remember what drives this - employers often like to think their problems are 'big data' and by god, they need the over-engineered solution. Your peers who interview you will toss your resume in the trash if you are not buzzword compliant. Hate the game not the player.
sgt101 8y ago
There's a big difference between a 50Tb data warehouse and a data warehouse which has a number of 10Tb tables. Our data warehouse used to have 20k tables and 18Tb data. Our big data instance has 4k tables and 800Tb data
geezerjay 8y ago
> Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it. IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.
- gaius 8y ago
  large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data What about too large for Excel, is that “big data” too? (Neither are)
- samuel_ms 8y ago
  That definition might be a bit loose. If you have an existing application designed to run on a Raspberry Pi, it'll hit that limit pretty quickly. This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.

10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.

by gaius 8y ago

10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.

Groxx 8y ago

Yeah, or R. When I need to do stuff bigger than RAM on a laptop I just reach for sqlite (assuming it can even fit on disk).

gaius 8y ago
The funny thing is we didn’t even consider our 50Tb to be particularly big, we knew guys running 200Tb... BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.
- Groxx 8y ago
  Oh yeah, the default dataframe is practically only for toy problems / very-already-filtered data (which accounts for a LOT of useful scenarios!). It's more that I've run into "big data" people who (╯°□°)╯︵ ┻━┻ when the default settings + dataframe choke, which is around 200MB of heap.

Swinx43 8y ago

This, a million times this!!! I see people doing consulting promoting the most over engineered solutions I have ever seen. Doing it for data that is a few hundred GB or maybe at best a TB. It makes me want to cry, knowing we handled that with a single server and a relational database 10 years ago. Lets also not forget that everyone today forgets that the majority of data actually has some sort of structure. There is no point in pretending that every piece of data is a BLOB or a JSON document. I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises.

gaius 8y ago
I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises I sometimes wish I was unscrupulous enough to cash in on these trends, I’d be a millionaire now. Instead I’m just a sucker who tries to make solid engineering decisions on behalf of my employers and clients. It’s depressing to think what professionalism has cost me in cash terms. But you’ve got to be able to look at yourself in the mirror.
therealdrag0 8y ago
Yep. I left consulting after working on a project that was a Ferrari when the customer only needed a Honda. Our architect kept being like "it needs to be faster", and I'm like "our current rate would process the entire possible dataset (everyone on earth) in a night, do we really need a faster Apache Storm cluster?" :S "Resume driven development" is a great term.
attaboyjon 8y ago
It's important to remember what drives this - employers often like to think their problems are 'big data' and by god, they need the over-engineered solution. Your peers who interview you will toss your resume in the trash if you are not buzzword compliant. Hate the game not the player.

sgt101 8y ago

There's a big difference between a 50Tb data warehouse and a data warehouse which has a number of 10Tb tables. Our data warehouse used to have 20k tables and 18Tb data. Our big data instance has 4k tables and 800Tb data

geezerjay 8y ago

> Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it. IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.

gaius 8y ago
large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data What about too large for Excel, is that “big data” too? (Neither are)
samuel_ms 8y ago
That definition might be a bit loose. If you have an existing application designed to run on a Raspberry Pi, it'll hit that limit pretty quickly. This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.