5 ms·
> 1.75GB > 3.46GB These will fit in memory on modest hardware. No reason to use Hadoop. The title could be: "Using tools suitable for a problem can be 235x f
by mej10 10y ago
> 1.75GB
> 3.46GB
These will fit in memory on modest hardware. No reason to use Hadoop.
The title could be: "Using tools suitable for a problem can be 235x faster than tools unsuitable for the problem"
- TickleSteve 10y agoThis is exactly the point he was making. People have a desire to use the 'big' tools instead of trying to solve the real problem. People both underestimate the power of their desktop machine and the 'old' tools and overestimate the size of their task.
- cloudjacker 10y agoI love it when clients think they need a server workstation Specs be damned! I need to start selling boxes
- sotojuan 10y agoOccasionally designers seem to seek credit merely for possessing a new technology, rather than using it to make better designs. Computers and their affiliated apparatus can do powerful things graphically, in part by turning out the hundreds of plots necessary for good data analysis. But at least a few computer graphics only evoke the response "Isn’t it remarkable that the computer can be programmed to draw like that?" instead of "My, what interesting data". - Edward Tufte Applies to more than just design.
- cheez 10y agoYeah, someone was telling me they need big data for a million rows. I laughed and said SQLite handles that...
- matt_wulfeck 10y agoI would not want to be the one on-call for a million row SQLite database!
- regularfry 10y agoI would. That's a pager that's never going off.
- qwertyuiop924 10y agoYeah. No significant software is bug-free, but if there was one, SQLite would be a good candidate.
- jschwartzi 10y agoEspecially having looked at how thorough their test suite is.
- qwertyuiop924 10y agoThat's what I'm talking about. What other database tests against power outages and hardware failure?
- cheez 10y agoI'm sure many do...
- qwertyuiop924 10y agoDo they do it for every release? What about OOM error testing? IO error tests? fuzz tests? UB tests? Fuzz tests? Malformed DB tests? Valgrind analysis? Memory leak checking? Regression tests? SQLite does all of that and more with every release (https://sqlite.org/testing.html https://sqlite.org/testing.html). There's a reason I consider their test coverage impressive...
- cheez 10y agoI don't know. I remember reading that other DBs use SQLite's test suite
- mrweasel 10y ago>People have a desire to use the 'big' tools Not only that, people seems to love claiming that they're "big data", perhaps because it makes them sound impressive and bigger than they are. Very few of us will ever do projects that justifies using tools like Hadoop and to few us are willing to accept that our data fits in SQLite.
- taf2 10y ago1TB will fit in memory too: https://aws.amazon.com/blogs/aws/ec2-instance-update-x1-sap-hana-t2-nano-websites/ https://aws.amazon.com/blogs/aws/ec2-instance-update-x1-sap-...
- coredog64 10y agoI'd rather pay for 7 c4.large* instances ($.70/hour for all of them) compared to an x1 ($13.38/hour). *The original article is from 2014 and references c1.medium which are no longer available. c4.large is the closest you can get without dipping down into the t2 class.
- carlesfe 10y agoLast paragraph of the article: "Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools" That was exactly his point.
- deleted 10y ago[deleted]
- cle 10y agoNot necessarily true. Depending on your use cases, it often still makes sense to use Hadoop. A really common scenario is that you'll implement your 3.5 GB job on one box, then you'll need to schedule it to run hourly. Then you'll need to schedule 3 or 4 to run hourly. Then your boss will want you to join it with some other dataset, and run that too with the other jobs. You'll eventually implement retries, timeouts, caching, partitioning, replication, resource management, etc. You'll end up with a half assed, homegrown Hadoop implementation, wishing you had just used Hadoop from the beginning.