4 ms·

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these d

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain scale. What system do you turn to next? Nosql? Will that scale infinitely? Will any system scale infinitely?

raverbashing 7y ago
The issue is that 80% of "big data" isn't https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- misterman0 7y ago
  That's not at all what my issue is though. I have what you might call "big data problems" and I'm so turned-off by Hadoop that I'm actually rolling my own distributed DB. But what principles other than a DFS should one look into, is the question, really. At least to me.
  - vonmoltke 7y ago
    It's a reaction to your statement > if you want to search over big data, which should be a pretty common procedure these days It's not a common procedure by any stretch because the vast majority of datasets aren't really "big".
mattmanser 7y ago
What's common about big data? My guess is that almost all programming jobs are in fields that produce no more data than a gb or two a month. I'm happy to be proven wrong, but I would guess that there are far more companies making project management software, time tracking apps, invoicing software, etc. than there are facebooks, googles or reddits obsessively logging every user mouse twitch. And that's data that's much better sitting in a nice, normal, relational database.
- misterman0 7y ago
  >> facebooks, googles or reddits Yes, it definitely seems with market leadership comes big data. Seems to me big data is highly relevant. More relevant now than ever.
  - TeMPOraL 7y ago
    But the point is, vast majority of companies are nowhere close to Facebook/Google/Reddit wrt. data scale, and have no need for tools applicable at Facebook/Google/Reddit scale.
arnon 7y ago
Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight. Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.
- misterman0 7y ago
  Which of those indexes/databases should I be using if I want to query over big data using natural language?
  - kthejoker2 7y ago
    This question makes no sense. First, NLP based search can be executed on top of any engine (APIs are very handy), relational, kv, graph, filesystem .. so that part is totally irrelevant. Assuming "big data" in this context is still relational data, then any of those systems would suffice, within their own particular tradeoffs and features.
  - arnon 7y ago
    There is no direct relation between the query language and the underlying infrastructure. If you're talking about taking some questions and getting graphs from them, ThoughtSpot does a good job.
- fmajid 7y ago
  There's also CitusDB, which performs much better than Hadoop because it doesn't have the bloat of the JVM or the inefficiency of brain-dead HDFS. You can write your queries in a few lines of SQL rather than reams of convoluted Java code.
wongarsu 7y ago
Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.
- nimrody 7y ago
  Whether Postgres (or similar) can handle the data depends not only on the data size but also on what you want to do with the data. I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines. Now Hadoop may not be a particularly efficient solution unless you need 100's of machines. But there is a limit to what a non-parallel single machine database can do. There are other solutions in-between. And you really don't have to be twitter or google to handle significant amount of data these days. People are recording much more data in the hope of generating new insights and do need tools to process that data.
  - anarazel 7y ago
    > I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines. Aggregating 100s of GB isn't much of a problem for PG these days. Yes, you can be faster - obviously - but it works quite well. And the price for separate systems (duplicated infrastructure, duplicated data, out-of-sync systems, ...) is noticable as well. But yea, for many petabytes of data you either have to go to an entirely different system, or use something like Citus. Disclaimer: I work on PG, and I used to work for Citus. So I'm definitely biased.
    - throwaway082729 7y ago
      I'm genuinely curious. Can PG handle hundreds of users querying 100s of GB, sometimes the same set of tables, at the same time?
nwmcsween 7y ago
There is no such thing as 'big data' in terms of data, there is proper and improperly designed programs, Hadoop is in the latter.

raverbashing 7y ago

The issue is that 80% of "big data" isn't https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...

misterman0 7y ago
That's not at all what my issue is though. I have what you might call "big data problems" and I'm so turned-off by Hadoop that I'm actually rolling my own distributed DB. But what principles other than a DFS should one look into, is the question, really. At least to me.
- vonmoltke 7y ago
  It's a reaction to your statement > if you want to search over big data, which should be a pretty common procedure these days It's not a common procedure by any stretch because the vast majority of datasets aren't really "big".

mattmanser 7y ago

What's common about big data? My guess is that almost all programming jobs are in fields that produce no more data than a gb or two a month. I'm happy to be proven wrong, but I would guess that there are far more companies making project management software, time tracking apps, invoicing software, etc. than there are facebooks, googles or reddits obsessively logging every user mouse twitch. And that's data that's much better sitting in a nice, normal, relational database.

misterman0 7y ago
>> facebooks, googles or reddits Yes, it definitely seems with market leadership comes big data. Seems to me big data is highly relevant. More relevant now than ever.
- TeMPOraL 7y ago
  But the point is, vast majority of companies are nowhere close to Facebook/Google/Reddit wrt. data scale, and have no need for tools applicable at Facebook/Google/Reddit scale.

arnon 7y ago

Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight. Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

misterman0 7y ago
Which of those indexes/databases should I be using if I want to query over big data using natural language?
- kthejoker2 7y ago
  This question makes no sense. First, NLP based search can be executed on top of any engine (APIs are very handy), relational, kv, graph, filesystem .. so that part is totally irrelevant. Assuming "big data" in this context is still relational data, then any of those systems would suffice, within their own particular tradeoffs and features.
- arnon 7y ago
  There is no direct relation between the query language and the underlying infrastructure. If you're talking about taking some questions and getting graphs from them, ThoughtSpot does a good job.
fmajid 7y ago
There's also CitusDB, which performs much better than Hadoop because it doesn't have the bloat of the JVM or the inefficiency of brain-dead HDFS. You can write your queries in a few lines of SQL rather than reams of convoluted Java code.

wongarsu 7y ago

Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.

nimrody 7y ago
Whether Postgres (or similar) can handle the data depends not only on the data size but also on what you want to do with the data. I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines. Now Hadoop may not be a particularly efficient solution unless you need 100's of machines. But there is a limit to what a non-parallel single machine database can do. There are other solutions in-between. And you really don't have to be twitter or google to handle significant amount of data these days. People are recording much more data in the hope of generating new insights and do need tools to process that data.
- anarazel 7y ago
  > I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines. Aggregating 100s of GB isn't much of a problem for PG these days. Yes, you can be faster - obviously - but it works quite well. And the price for separate systems (duplicated infrastructure, duplicated data, out-of-sync systems, ...) is noticable as well. But yea, for many petabytes of data you either have to go to an entirely different system, or use something like Citus. Disclaimer: I work on PG, and I used to work for Citus. So I'm definitely biased.
  - throwaway082729 7y ago
    I'm genuinely curious. Can PG handle hundreds of users querying 100s of GB, sometimes the same set of tables, at the same time?

nwmcsween 7y ago

There is no such thing as 'big data' in terms of data, there is proper and improperly designed programs, Hadoop is in the latter.