4 ms·

> - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen fir

> - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.

_dps 7y ago
I remember distinctly a conversation I had in early 2012 about what I jokingly called "big memory" as killing off the (then ubiquitous) "big data" trend. Most people's sensibilities have not adjusted to a world where getting 64GB of memory in a server is a triviality, and you can 8x that without too much effort or cost. Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.
danpalmer 7y ago
I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I've heard of companies doing extensive work to set up Hadoop to process a few GB. Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.
- geezerjay 7y ago
  > I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. From Heroku's site, Heroku's 4GB database plan goes for 50$/(instance.month) while Heroku's 8GB plan goes for 200$/(instance.month). Therefore, it isn't a question of if it makes finantial sense (it does) but how long the startup plans to operate to recover their investment. > I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I find it rather strange how someone believes that it's a decent idea to conduct a company's data analysis work on what an employee manages to fit on an external HD, as it creates a whole lot of hard problems both legal and technical. I mean, how do you ensure the data's provenance is tracked and other data analysts can access the data? Who in their right mind would put himself on a situation where a minor lapse or misfortune (losing/getting the HD stolen) could put the company at risk? > Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though. That's a rather naive and missinformed take on Hadoop. Hadoop might be conflated with big data but it's actually a distributed system designed to reliably process data shards without having to incur a penalty to move data around. It makes absolutely no sense to base your assertion on data volumes alone. What matters if it the performance increase justifies setting up a hadoop cluster with the resourses available to a company.

_dps 7y ago

I remember distinctly a conversation I had in early 2012 about what I jokingly called "big memory" as killing off the (then ubiquitous) "big data" trend. Most people's sensibilities have not adjusted to a world where getting 64GB of memory in a server is a triviality, and you can 8x that without too much effort or cost. Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.

danpalmer 7y ago

I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I've heard of companies doing extensive work to set up Hadoop to process a few GB. Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.

geezerjay 7y ago
> I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. From Heroku's site, Heroku's 4GB database plan goes for 50$/(instance.month) while Heroku's 8GB plan goes for 200$/(instance.month). Therefore, it isn't a question of if it makes finantial sense (it does) but how long the startup plans to operate to recover their investment. > I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I find it rather strange how someone believes that it's a decent idea to conduct a company's data analysis work on what an employee manages to fit on an external HD, as it creates a whole lot of hard problems both legal and technical. I mean, how do you ensure the data's provenance is tracked and other data analysts can access the data? Who in their right mind would put himself on a situation where a minor lapse or misfortune (losing/getting the HD stolen) could put the company at risk? > Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though. That's a rather naive and missinformed take on Hadoop. Hadoop might be conflated with big data but it's actually a distributed system designed to reliably process data shards without having to incur a penalty to move data around. It makes absolutely no sense to base your assertion on data volumes alone. What matters if it the performance increase justifies setting up a hadoop cluster with the resourses available to a company.