3 ms·
Not really sure what you are talking about. Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as e
by threeseed 11y ago
Not really sure what you are talking about.
Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as easy/hard as setting up HDFS and installing Hive. In all cases people who are at big data scale are buying OTS solutions e.g. Cloudera appliance. They aren't rolling their own.
And if you are using Hive then I can understand why you are not feeling the buzz. But play around for Spark for a while and it's easy to see the future. Being able to write Scala/Python/SQL/R against a data set that can be anywhere from 100MB to 100PB without any changes is pretty compelling.
- angrybits 11y agoYou are either being dishonest or just patently out of your mind if you think that you can query 100MB and 100PB in the same way. That's not even reasonable by HN standards of hyperbole. Do you have any idea how many orders of magnitude that is?
- MichaelGG 11y agoHe's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
- parasubvert 11y agoHe's talking about writing it for Apache Spark. Which really isn't intended for 100 MB (I bet I could write a unix pipe & filter script that's faster than Spark), but is intended for 10 TB through several PB.
- parasubvert 11y agoHe's talking about Apache Spark. While it can handle 100 MB easily there probably are faster ways to handle that small amount of data. But yes, Spark can handle many PB and doesn't require a ton of changes in the code as you scale up from say 10 TB to 100 PB. The underlying cluster would change, and the performance profile would change a lot (10 TB can be done in-memory ... many PB, not so much)
- dang 11y agoPlease don't make technical comments (or any comments) in this inflammatory, nasty way. We're lucky that you didn't spark a horrible flamewar, but instead got patient, factual replies.
- angrybits 11y agoI know. And I'm sorry, bitterly sorry, but I know that... no apologies I can make can alter the fact that in our thread you have been given a dirty, filthy, smelly piece of technical argument.
- dang 11y agoI'm not following you here. The point is that you have a history of doing this and if you won't stop doing it you can't comment on HN.
- angrybits 11y agoWhat can I say, being an asshole is one of my diversity dimensions. We just had that training at work, it was very helpful. Also, I hope you aren't a Python dev in your spare time. Guido would be appalled. https://www.youtube.com/watch?v=EdzqTGmEcZE https://www.youtube.com/watch?v=EdzqTGmEcZE