22 ms·

I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is.

I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is. Now making (big)money with its ecosystem is another question.

nell 7y ago
Solutions to hard problems of big enterprises should be the most juicy way to make money no?
- vkazanov 7y ago
  I am not a certified specialist in Hadoop-related things but there's definitely an industry of people working with Hadoop/HDFS/Spark/etc. It's almost a standard in Business Intelligence where I work these days. So there should be some money there.
  - anticensor 7y ago
    HDFS is Hadoop, other parts being MapReduce, YARN and HDDS.
- HillRat 7y ago
  Big problems mean big projects, big risks, big staffing plans, big costs, and smaller margins. If you can bear the cost of sales, ten $10mm projects beat a single $100mm in my experience.
Proven 7y ago
Is that a joke? As an earlier comment noted great majority of Cloudera/Hortonworks paying customers have just a handful of nodes that usually work slower than a properly designed single system. As far as large customers with HDFS are concerned, many have realized that HDFS is shitty to manage and usually creates yet another data island and have been moving to generic single-host and scale-out file systems, as well as S3. In my estimate about 10-15 percent of users really need Hadoop with HDFS, the rest have wasted time and money.
notacoward 7y ago
...except that HDFS has never been a distributed filesystem, despite the name. It's an object store with just enough hackery so that a particular set of apps with very limited needs could be modified in a short time to use it instead of a filesystem. Arguably that's still a big deal, but nowhere near as impressive or generally useful as a distributed filesystem. To me it's a bit amazing that anyone involved in its design still has a job in this industry.
- vkazanov 7y ago
  ...ok, then what are the alternatives? And why is it so bad? Something that can handle hundreds of terabytes on hundreds of machines and provides useful tools on top of the whole thing (Spark, Hive, etc)?
  - notacoward 7y ago
    Well, rather topically, there's MapR. Depending on the desired scale and level of fidelity to real-filesystem semantics, one might also count Gluster, Ceph, and Lustre. Don't know if PVFS is still around. There are more proprietary offerings from IBM (GPFS/SpectrumScale), EMC, NetApp, etc. plus a plethora of startups. There are other object stores that don't make a pretense of being filesystems. Alternatives abound. BTW, "hundreds of terabytes on hundreds of machines" isn't interesting territory any more. Most people's needs are far smaller, so HDFS isn't much help. Those who need more generally need much more, so HDFS isn't much help again. Richer semantics are nice either way. Imagine thousands of machines with dozens of terabytes each and you might start to see the problems with HDFS's design (though you'll still be far short of the domain I work in).
    - vkazanov 7y ago
      The thing is that this is exactly my company's scale, and hdfs is perfectly fine here. :-) And I am curious about possible alternatives: open source, about 100-200 machines, with good support for analytics and SQlish systems.

nell 7y ago

Solutions to hard problems of big enterprises should be the most juicy way to make money no?

vkazanov 7y ago
I am not a certified specialist in Hadoop-related things but there's definitely an industry of people working with Hadoop/HDFS/Spark/etc. It's almost a standard in Business Intelligence where I work these days. So there should be some money there.
- anticensor 7y ago
  HDFS is Hadoop, other parts being MapReduce, YARN and HDDS.
HillRat 7y ago
Big problems mean big projects, big risks, big staffing plans, big costs, and smaller margins. If you can bear the cost of sales, ten $10mm projects beat a single $100mm in my experience.

Proven 7y ago

Is that a joke? As an earlier comment noted great majority of Cloudera/Hortonworks paying customers have just a handful of nodes that usually work slower than a properly designed single system. As far as large customers with HDFS are concerned, many have realized that HDFS is shitty to manage and usually creates yet another data island and have been moving to generic single-host and scale-out file systems, as well as S3. In my estimate about 10-15 percent of users really need Hadoop with HDFS, the rest have wasted time and money.

notacoward 7y ago

...except that HDFS has never been a distributed filesystem, despite the name. It's an object store with just enough hackery so that a particular set of apps with very limited needs could be modified in a short time to use it instead of a filesystem. Arguably that's still a big deal, but nowhere near as impressive or generally useful as a distributed filesystem. To me it's a bit amazing that anyone involved in its design still has a job in this industry.

vkazanov 7y ago
...ok, then what are the alternatives? And why is it so bad? Something that can handle hundreds of terabytes on hundreds of machines and provides useful tools on top of the whole thing (Spark, Hive, etc)?
- notacoward 7y ago
  Well, rather topically, there's MapR. Depending on the desired scale and level of fidelity to real-filesystem semantics, one might also count Gluster, Ceph, and Lustre. Don't know if PVFS is still around. There are more proprietary offerings from IBM (GPFS/SpectrumScale), EMC, NetApp, etc. plus a plethora of startups. There are other object stores that don't make a pretense of being filesystems. Alternatives abound. BTW, "hundreds of terabytes on hundreds of machines" isn't interesting territory any more. Most people's needs are far smaller, so HDFS isn't much help. Those who need more generally need much more, so HDFS isn't much help again. Richer semantics are nice either way. Imagine thousands of machines with dozens of terabytes each and you might start to see the problems with HDFS's design (though you'll still be far short of the domain I work in).
  - vkazanov 7y ago
    The thing is that this is exactly my company's scale, and hdfs is perfectly fine here. :-) And I am curious about possible alternatives: open source, about 100-200 machines, with good support for analytics and SQlish systems.