12 ms·

Let's Build a Modern Hadoop

146 points by jaz46 12y ago

sytse 12y ago
Sounds awesome, do you have any performance stats?
- jdoliner 12y ago
  Hi! We unfortunately don't right now. It's surprisingly hard to get benchmarks that give you a true apples to apples comparison between us and Hadoop. However it seems from the comments in this thread that there's a lot of interest in those though so we'll try to put those together ASAP.
  - sytse 12y ago
    Thanks!
ericfrenkiel 12y ago
I find it very odd that the author completely glossed over the recent Spark developments in the Hadoop ecosystem. In many ways, Spark is meant to replace the Map-Reduce paradigm to enable easier access to the data.
- quadrature 12y ago
  Its not that odd, spark is very similar to the hadoop way of doing things and there are a few other projects like it that are attempting to be a better hadoop. So I dont think that much was lost by not mentioning it.
  - threeseed 12y ago
    How is Spark similar to the Hadoop way of doing things ? It operates very differently from Hadoop for us. SparkSQL allows third parties apps e.g. analytics to use JDBC/ODBC rather than going HDFS. And the in memory model and ease of caching data from HDFS allows for different use cases. We do most work now via SQL. Combining Spark with Storm, ElasticSearch etc also permits a true real time ingestion and searching architecture.
  - ianburrell 12y ago
    Spark is a more general data processing framework than Hadoop. It can do map-reduce, can run on top of Hadoop clusters, and can use Hadoop data. It can also do streaming, interactive queries, machine learning, and graph processing.
    - justicezyx 12y ago
      hadoop isnt hadoop mapreduce
- jdoliner 12y ago
  Hi, co-founder This is something we probably should have spoken directly to in the article because of how popular spark is. The main reason we didn't is that we like Spark and don't feel it needs to be replaced. It doesn't seem to have most of the ecosystem problems we discuss because it's got a company behind it. My understanding of Spark is that it's designed to be used with different storage backends and I'm very curious to see what would happen if we got it talking to pfs. I think it could work very well because Spark's notion of immutable RDDs seems very similar to the way pfs handles snapshots. Hope this clears a few things up and apologizes if I've mischaracterized Spark here I've only used it a little bit.
trhway 12y ago
Writing new implementation of a batch processing misses the main point that batch processing in general is just a trade-off made in order to get things started. This is why new generations of systems are going the way of online and/or stream processing.
- threeseed 12y ago
  Have you actually seen this happening ? I have terabytes of new data arriving per day that needs to be verified, ingested and translated. There simply is no way to do this via online or stream processing. If you ingesting clickstream or Twitter data then sure it will work. But more often than not you need to work with sets of data. And for that batch processing is the only option.
  - jdoliner 12y ago
    JD @ Pachyderm here. We try to talk to as many people as possible about their data infrastructure so I have a decent sample size on this. My experience is that there a specific use cases for which streaming solutions are gaining ground but there's still a lot for which it isn't and probably never will. Streaming is great but there are a lot of situations where it just doesn't fit. I think what might be going on is that there isn't a lot of chatter surrounding batch based processing so it seems like it's going out of style but the use cases are still real
  - angersock 12y ago
    So, I've got to ask...what do you actually get for that data? Like, what does your business actually use it for? Are you folks doing sensor analysis or something, or eating logfiles, or what?
    - threeseed 12y ago
      Financial Services with millions of customers (and growing rapidly). What we can get for that data is a major competitive advantage. We can offer much cheaper financial products since we model risk individually rather than as a cohort. It also allows us to have a single customer view despite reselling other companies products. Building a single customer view with lots of disparate data sets is a big trend right now.
jandrewrogers 12y ago
The article makes many good points but also misses on a few in my opinion. Hadoop was designed to solve a pretty narrow problem a long time ago; modern use cases tend to be far outside its intended design envelope. Being more explicitly Linux-centric and dropping the JVM paves the way for some major performance optimizations. In practice, virtually everyone uses Linux for these kinds of applications so portability is not a significant concern. Efficiency in the data center is becoming a major concern, both in terms of CapEx and OpEx, and the basic Hadoop stack can be exceedingly suboptimal in this regard. Free software is not cost effective if you require 10-100x the hardware (realistic) of a more optimal implementation. However, I would use a different architecture generally rather than reimplementing the Hadoop one. The Achille's Heel of Hadoop (and Spark) is its modularity. Throughput for database-like workloads, which matches a lot of modern big data applications, is completely dependent on tightly scheduling execution, network, and storage operations without any hand-offs to other subsystems. If the architecture requires network and storage I/O to be scheduled by other processes, performance falls off a cliff for well-understood reasons (see also: mmap()-ed databases). People know how to design I/O intensive data processing engines, it was just outside the original architectural case for Hadoop. We can do better. A properly designed server kernel running on cheap, modern hardware and a 10 GbE network can do the following concurrently on the same data model: - true stream/event processing on the ingest path at wire speed (none of that "fast batch" nonsense) - drive that ingest stream all the way through indexing and disk storage with the data fully online - execute hundreds of concurrent ad hoc queries on that live data model These are not intrinsically separate systems, it is just that we've traditionally designed data platforms to be single purpose. However, it does require a single scheduler across the complete set of operations required in order to deliver that workload. From a software engineering standpoint it is inconvenient that good database kernels are essentially monolithic and incredibly dense but it is unavoidable if performance matters.
- jdoliner 12y ago
  Hi. Co-founder and lead Pachyderm dev here. This is a really interesting comment and raises some questions that I haven't fully considered. Building the infrastructure in this decoupled way has let us move quickly early on and made it really easy to scale out horizontally. We haven't paid as much attention to what you talk about here. However I think there might be some optimizations we can make within our current design that would help a lot. Probably the biggest source of slowness right now is that we post files in to jobs via HTTP, this is nice because it gives you a simple API for implementing new jobs but it's reaaally slow. A much more performant solution would be to give the data directly to the job as a shared volume (a feature Docker offers) this would lead to very performant i/o because the data could be read directly off disk by the container that's doing the processing.
lmm 12y ago
The JVM runs every language I care about, in a mature system with a simple, stable interface. It has its problems (startup time), but for the kind of jobs you use Hadoop for it's rarely an issue. Switching to Docker feels like a real step backwards. Rather than a jar containing strictly-specified bytecode, cluster jobs are going to be random linux binaries that could do any combination of system calls. You need another layer of build/management system to make your docker containers useable. Worst of all, rather than defining common shared services for e.g. job scheduling, this is going to encourage users to put any random component in their docker image. We'll end up with a more fragmented ecosystem, not less.
- eropple 12y ago
  I'm not really sure why this post got downvoted, because I think there's some merit to it. In a spherical-cow universe I think there's an argument for opening up the universe to all tools, but I think in practice this makes good practices much harder to maintain and harder to proof against issues and failures at a glance. Personally, I dig Spark on EMR or Spark on Mesos (with Scala as a domain language), and I'm not sure how this plays with the rest of the world.
  - sgt101 12y ago
    I think Spark on Yarn on Mesos co habiting with Docker on Mesos managed by Twill is the future. I just need to take a big breath every time I say it though.
- jdoliner 12y ago
  Hi, JD @ Pachyderm here. Our feeling isn't that Java is bad it's that it shouldn't be the only option for big data processing. There are a lot of people that want to do this type of computing and a lot of them hate Java. We want to build a solution that can serve them better and Docker is actually an incredibly high leverage way to do it. I feel like I don't fully understand your criticism of choosing Docker as platform. > cluster jobs are going to be random linux binaries that could do any combination of system calls Could you give an example of what could go wrong here? Docker prevents you from talking to the outside world so you can't call out to an outside service in the middle of a job or anything like that. > Worst of all, rather than defining common shared services for e.g. job scheduling In Pachyderm job scheduling happens outside the user submitted containers. The job scheduler is what actually spins up the containers and pushes data through them. I'm afraid I don't understand what it would even mean for one of those containers to then do job scheduling itself. Hope this clears up our thoughts on this a bit.
batbomb 12y ago
> Second, Etcd and Fleet are themselves designed to be modular, so it’s easy to support a variety of other deployment methods as well. Zookeeper isn't Modular? No mention of HTCondor+Dagman? No mention of Spark? I've written a general DAG processor/workflow engine/metascheduler/whatever you want to call it. It's used by various physics and astronomy experiments. I've interfaced it with the Grid/DIRAC, Amazon, LSF, Torque, PBS, SGE, and Crays. There's nothing in it that precludes Docker jobs from running. I've implemented a mixed-mode (Streaming+batch processing + more DAG) version of it which just uses job barriers to do the setup and ZeroMQ for inter-job communication. I think something like Spark's resilient distributed dataset would be nice here as well. We don't use hadoop because, as was said, because it's narrow in scope and it's not a good general purpose DAG/Workflow Engine/Pipeline. I think this is a small improvement, but I don't really see it being much better than hadoop, or HTCondor, or Spark. HTCondor is pretty amazing. I think somebody should be building a modern HTCondor, not a modern Hadoop.
- a8da6b0c91d 12y ago
  > somebody should be building a modern HTCondor A bug-free, cross platform, rock-solid HTCondor would be great. Sadly the existing HTCondor is a buggy turd. I've fixed some stuff in it and a lot of the code is eye-roll worthy. I can practically guarantee you hours of wondering why something that by any reasonable expectation should be working isn't working, and not giving any meaningful error message. But like you said, a really solid HTCondor rewrite with the Stork stuff baked in would pretty much be all anyone should need, I think. It's not like you'd even really need to dramatically re-architect the thing. The problems with HTCondor really are mostly just code quality and the user experience.
  - lrm242 12y ago
    I use HTCondor every day. I find it to be rock solid with top notch documentation. It is cross-platform today as well, so I'm not sure where you're coming from on this.
    - a8da6b0c91d 12y ago
kev009 12y ago
No disclosure of inspiration from Manta https://github.com/joyent/manta https://github.com/joyent/manta? Also, why yet another DFS that makes excuses for shitty local filesystems instead of leveraging ZFS? Use ZFS as the base for integrity, snapshots, send/receive and make a small service on top for managing cluster metadata.
- jdoliner 12y ago
  Hi, JD @ Pachyderm here. We think Manta is pretty cool but it wasn't actually an inspiration for us. I'm not sure I fully understand your criticism of the filesystem. Pfs was actually initially conceived as almost exactly what you're describing, a small service on top of ZFS. But we realized early on that ZFS was going to be a pain because of its patent status so we used btrfs instead. I think btrfs is worse than ZFS but I'm not sure I'd call it a shitty local filesystem.
CurtMonash 12y ago
This sounds bizarre. Spark is the replacement for MapReduce, with Map and Reduce being just 2 of its 15 (?) primitives. Who needs another MapReduce engine? The bit about "We'll get around to Hive and Pig eventually" is also odd, as they're approaching legacy status too. (Albeit not there yet; please recall that I'm somewhat of an Impala skeptic, and of course there's the Tez booster to Hive.) Also bizarre are some of the history claims in the article, e.g. that YARN is part of more or less original Hadoop.
- jdoliner 12y ago
  It's been talked about a few times in this thread already so I'll try not to rehash too much. We think Spark is way cool but disagree that it's a complete replacement for MapReduce. There were people else where in this thread who described pretty well their use cases that didn't work with Spark. My experience is that Hive is pretty widely used, I have a lot fewer data points about Pig. We're not particularly attached to either of these. That sentence was more to indicate that we'd be adding some sort of interfaces for specific data types. Seems like we got our history wrong here. When was YARN added we'll add a correction. Hope this helps clear things up.
  - thinkmassive 12y ago
    YARN was introduced with Hadoop 2.0, by MAPREDUCE-279: https://issues.apache.org/jira/browse/MAPREDUCE-279 https://issues.apache.org/jira/browse/MAPREDUCE-279 Note that YARN also supports Docker containers as of Hadoop 2.6... YARN-1964: Create Docker analog of the LinuxContainerExecutor in YARN https://issues.apache.org/jira/browse/YARN-1964 https://issues.apache.org/jira/browse/YARN-1964
  - CurtMonash 12y ago
    Thanks. I'll stick with http://www.dbms2.com/2014/04/30/spark-on-fire/ http://www.dbms2.com/2014/04/30/spark-on-fire/ however, for the most part. (The mid-2014 PR blitz I thought I was engineering didn't actually happen.)
michaelochurch 12y ago
This sounds promising. I'd love to see a Haskell-centric (performance and robust code) but ultimately language-agnostic alternative to the JVM-heavy stack that's currently in vogue. (Spark is a promising and powerful technology.) I choose Haskell because pretty much anything that the JVM does well (e.g. concurrency) Haskell also does well and, while I like Clojure and can enjoy Scala when written by competent people, Java is ugly and its community has worse aesthetic sense than a taint mole.
- necubi 12y ago
  You may be interested in Cloud Haskell[0], which is an erlang-like distributed computation engine. The main thing that's missing in Haskell-land is an alternative to HDFS. [0] https://wiki.haskell.org/Cloud_Haskell https://wiki.haskell.org/Cloud_Haskell
  - enigmo 12y ago
    Why not use Ceph? There are solid Haskell bindings for RADOS: https://github.com/anchor/rados-haskell https://github.com/anchor/rados-haskell
- deleted 12y ago
  [deleted]
spennant 12y ago
Funny... I just put up a hadoop cluster last week and named it pachyderm.mydomain.com (i'm probably not the first to do that either)
bipin_nag 12y ago
I agree with your findings on Hadoop. It is a very complicated ecosystem. The core is tied to JVM and abstractions of map-reduce are heavy and difficult to use. There are multiple domain specific languages/api to do the same thing : pig, cascading. Yet if you pick one you cant use other. Hadoop feels like tower of babel. But I am not in the favour of creating another stack as Hadoop replacement. Spark fills the role of distributed computation framework very well. It uses scala a functional language which provides neat abstractions for map,reduce,filter and other operations. It introduces RDD to abstract data. All the components graphx, sparksql, blinkdb use scala. Also building a new map-reduce engine is a difficult task. The computation engine will lie in the middle of stack not on top as shown. Libraries will sit on top of that, so if you build an engine you will have to build everything on top of that. See the Spark ecosystem : https://amplab.cs.berkeley.edu/software/ https://amplab.cs.berkeley.edu/software/
thinkmoore 12y ago
Still trying to wrap my head around the idea that 10 year old software is not "modern." That said, I have no opinion on this article.
bra-ket 12y ago
fantastic idea, unfortunate name (Pachyderm, seriously?)
pjmlp 12y ago
It is not modern if GNU/Linux is the only deployment option. I love the ability to run the cluster in any OS that has a JVM available, from big commercial UNIXes, Windows, *BSD and GNU/Linux distributions.
KaiserPro 12y ago
Sorry, but using HTTP for transport is just terrible. Hadoop is a poor imitation of a mainframe. When you are processing large lumps of data you need the following: A task scheduler A message passing system(although in some workloads this can be proxied by the scheduler or filesystem) File storage In mainframe land, you have a large machine with many processors all linked to the same kernel. As its one large machine, there is a scheduler that's so good you don't even know its there. However mainframes require specialist skills. If you want shared memory cluster, you're going to need inifiband (which is actually quite cheap now, and has 40gig ethernet type majigger in it aswell) but people are scared of that, so that leaves us with your standard commodity cluster. In VFX we have this nailed. before leaving the industry I was looking after a ~25k CPU cluster backed to 15pb of storage. No fancy map reduce bollocks, just plain files and scripts. Everything was connected by ten gigs ethernet. Sustained write rate of 25 gigabytes a second. firstly task placement is dead simple: https://github.com/mikrosimage/OpenRenderManagement/wiki/Introduction-to-job-submission https://github.com/mikrosimage/OpenRenderManagement/wiki/Int... look how simple that task mapping is. It is legitimate to point out that there are no primitives for IPC, but then your IPC needs are varied. Most log churn apps don't really need IPC, they just need a catcher to aggregate the results. Map:reduce has fooled people into thinking that everything is must be that way. This is designed to run on bared metal. Anything else is a waste of CPU/IO. We used cgroups to limit process memory, so that we could run concurrent tasks on the same box. But everything ran the same image, anything else is just a plain arse ache. The last item is file storage. HDFS is just terrible. the reason that its able to scale is that it provides literally nothing. Its just a key:value store, based on a bunch of key:value stores. If you want fast super scalable posix filesystem, then use GPFS(or elastic storage). The new release has a kind of software raid that allows you to recover from a dead disk in less than an hour. (think ZFS with shards) failing that you can use lustre (although that requires decent hardware raids, as its a glorified network raid0) or if you are lucky you can split your file system into name spaces that reside on different NFS servers. Either way, Hadoop, and hadoop inspired infrastructures are normally not the right answer. Measos is also probably not the answer unless you want a shared memory cluster. but then if you want this you need inifinband...
rkwasny 12y ago
Scheduling jobs by distributing docker container? Looks really heavyweight. There is a good reason most of Map-Reduce frameworks are written on top of JVM - it is VERY easy to serialize Java/Scala code ship it to remote host and execute there. This is not trivial in compiled languages ( I believe some hacks for Haskell exists ) What we need in Big Data space is more people that understand how computers really work. Some effort must be put into supporting TCP/IP Stacks in Userspace using new 10GBe cards or support for IB verbs. Also the long forgotten art of writing zero-copy code.
- why-el 12y ago
  > Some effort must be put into supporting TCP/IP Stacks in Userspace using new 10GBe cards or support for IB verbs Obviously my understanding of networks is not at that level yet. What would a layman do to become somewhat vested in this? Any books or resrouces?
  - chaotic-good 12y ago
    This technique usually called 'kernel bypass'. http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html http://highscalability.com/blog/2013/5/13/the-secret-to-10-m... and http://lukego.github.io/blog/2013/01/04/kernel-bypass-networking/ http://lukego.github.io/blog/2013/01/04/kernel-bypass-networ...
    - why-el 12y ago
      Thank you. :)
- codygman 12y ago
  I wouldn't consider StaticPointers a hack. I believe this is the feature you are talking about.
TeMPOraL 12y ago
Excuse me for going off-topic, but heck, Hadoop itself was released 3 years ago. It's a testament to how insanely fast things change in web that Hadoop is no longer considered "modern"...
amelius 12y ago
Personally, I'd like to run purely functional jobs, and get progress information while they are running (this is not as easy as it sounds). Automatic memoization of jobs. And automatic killing of jobs and removal of data when results are no longer required (this should work transitively/recursively).
- sp332 12y ago
  You want a purely functional job with a side effect of updating progress info?
  - scott_s 12y ago
    It's a reasonable request. The job is purely functional in the computation on its incoming data. That means you don't need to worry about persistent state management, which is a problem if you want to easily move jobs around a distributed system. The job (or part of the job) can still send metrics about what its doing to some other service, and still remain functional with respect to the data it's processing.
    - sp332 12y ago
      Functional doesn't just mean repeatable. If you modify global state that is not represented in the "input" and "output" of the job, then it's not purely functional.
      - scott_s 12y ago
        Yes, I am aware of what "functional" means. But I think you are being too literal. If you view the computation through the lens of the actual data involved in the computation, then it is functional. That it also has some internal state that gets updated and reported as part of the monitoring of the computation - but not part of the computation - then it is still functional with respect to the actual data in the job. The key distinction is that this internal metrics data that is just used for monitoring is incidental, and can be thrown away without impacting the computation's result. It's just convenient information that humans might want to have so they can reason about the internal state of the overall system.
eaxitect 12y ago
TL,DR; Although I'm not a fan of entire Hadoop system, some judgements about it is totally wrong. 1) Do one thing well - Hadoop is so stacked up because of this (which has pros and cons). Every single component of Hadoop is designed specific to a job (there are many overlapped components, though) 2) Hadoop's main idea is to be data-local, yet this Pachy thing not to be: "Data is POSTed to the container over HTTP and the results are stored back in the file system". Data-locality (almost) always win. 3) There is no scheduler like YARN specified, yet fleet and etcd are not schedulers. 4) You may not like Java or JVM-based things (like I do), but there is no problem of being Java-based or JVM-based. …

146 points by jaz46 12y ago

sytse 12y ago

Sounds awesome, do you have any performance stats?

jdoliner 12y ago
Hi! We unfortunately don't right now. It's surprisingly hard to get benchmarks that give you a true apples to apples comparison between us and Hadoop. However it seems from the comments in this thread that there's a lot of interest in those though so we'll try to put those together ASAP.
- sytse 12y ago
  Thanks!

ericfrenkiel 12y ago

I find it very odd that the author completely glossed over the recent Spark developments in the Hadoop ecosystem. In many ways, Spark is meant to replace the Map-Reduce paradigm to enable easier access to the data.

quadrature 12y ago
Its not that odd, spark is very similar to the hadoop way of doing things and there are a few other projects like it that are attempting to be a better hadoop. So I dont think that much was lost by not mentioning it.
- threeseed 12y ago
  How is Spark similar to the Hadoop way of doing things ? It operates very differently from Hadoop for us. SparkSQL allows third parties apps e.g. analytics to use JDBC/ODBC rather than going HDFS. And the in memory model and ease of caching data from HDFS allows for different use cases. We do most work now via SQL. Combining Spark with Storm, ElasticSearch etc also permits a true real time ingestion and searching architecture.
- ianburrell 12y ago
  Spark is a more general data processing framework than Hadoop. It can do map-reduce, can run on top of Hadoop clusters, and can use Hadoop data. It can also do streaming, interactive queries, machine learning, and graph processing.
  - justicezyx 12y ago
    hadoop isnt hadoop mapreduce
jdoliner 12y ago
Hi, co-founder This is something we probably should have spoken directly to in the article because of how popular spark is. The main reason we didn't is that we like Spark and don't feel it needs to be replaced. It doesn't seem to have most of the ecosystem problems we discuss because it's got a company behind it. My understanding of Spark is that it's designed to be used with different storage backends and I'm very curious to see what would happen if we got it talking to pfs. I think it could work very well because Spark's notion of immutable RDDs seems very similar to the way pfs handles snapshots. Hope this clears a few things up and apologizes if I've mischaracterized Spark here I've only used it a little bit.

trhway 12y ago

Writing new implementation of a batch processing misses the main point that batch processing in general is just a trade-off made in order to get things started. This is why new generations of systems are going the way of online and/or stream processing.

threeseed 12y ago
Have you actually seen this happening ? I have terabytes of new data arriving per day that needs to be verified, ingested and translated. There simply is no way to do this via online or stream processing. If you ingesting clickstream or Twitter data then sure it will work. But more often than not you need to work with sets of data. And for that batch processing is the only option.
- jdoliner 12y ago
  JD @ Pachyderm here. We try to talk to as many people as possible about their data infrastructure so I have a decent sample size on this. My experience is that there a specific use cases for which streaming solutions are gaining ground but there's still a lot for which it isn't and probably never will. Streaming is great but there are a lot of situations where it just doesn't fit. I think what might be going on is that there isn't a lot of chatter surrounding batch based processing so it seems like it's going out of style but the use cases are still real
- angersock 12y ago
  So, I've got to ask...what do you actually get for that data? Like, what does your business actually use it for? Are you folks doing sensor analysis or something, or eating logfiles, or what?
  - threeseed 12y ago
    Financial Services with millions of customers (and growing rapidly). What we can get for that data is a major competitive advantage. We can offer much cheaper financial products since we model risk individually rather than as a cohort. It also allows us to have a single customer view despite reselling other companies products. Building a single customer view with lots of disparate data sets is a big trend right now.

jandrewrogers 12y ago

The article makes many good points but also misses on a few in my opinion. Hadoop was designed to solve a pretty narrow problem a long time ago; modern use cases tend to be far outside its intended design envelope. Being more explicitly Linux-centric and dropping the JVM paves the way for some major performance optimizations. In practice, virtually everyone uses Linux for these kinds of applications so portability is not a significant concern. Efficiency in the data center is becoming a major concern, both in terms of CapEx and OpEx, and the basic Hadoop stack can be exceedingly suboptimal in this regard. Free software is not cost effective if you require 10-100x the hardware (realistic) of a more optimal implementation. However, I would use a different architecture generally rather than reimplementing the Hadoop one. The Achille's Heel of Hadoop (and Spark) is its modularity. Throughput for database-like workloads, which matches a lot of modern big data applications, is completely dependent on tightly scheduling execution, network, and storage operations without any hand-offs to other subsystems. If the architecture requires network and storage I/O to be scheduled by other processes, performance falls off a cliff for well-understood reasons (see also: mmap()-ed databases). People know how to design I/O intensive data processing engines, it was just outside the original architectural case for Hadoop. We can do better. A properly designed server kernel running on cheap, modern hardware and a 10 GbE network can do the following concurrently on the same data model: - true stream/event processing on the ingest path at wire speed (none of that "fast batch" nonsense) - drive that ingest stream all the way through indexing and disk storage with the data fully online - execute hundreds of concurrent ad hoc queries on that live data model These are not intrinsically separate systems, it is just that we've traditionally designed data platforms to be single purpose. However, it does require a single scheduler across the complete set of operations required in order to deliver that workload. From a software engineering standpoint it is inconvenient that good database kernels are essentially monolithic and incredibly dense but it is unavoidable if performance matters.

jdoliner 12y ago
Hi. Co-founder and lead Pachyderm dev here. This is a really interesting comment and raises some questions that I haven't fully considered. Building the infrastructure in this decoupled way has let us move quickly early on and made it really easy to scale out horizontally. We haven't paid as much attention to what you talk about here. However I think there might be some optimizations we can make within our current design that would help a lot. Probably the biggest source of slowness right now is that we post files in to jobs via HTTP, this is nice because it gives you a simple API for implementing new jobs but it's reaaally slow. A much more performant solution would be to give the data directly to the job as a shared volume (a feature Docker offers) this would lead to very performant i/o because the data could be read directly off disk by the container that's doing the processing.

lmm 12y ago

The JVM runs every language I care about, in a mature system with a simple, stable interface. It has its problems (startup time), but for the kind of jobs you use Hadoop for it's rarely an issue. Switching to Docker feels like a real step backwards. Rather than a jar containing strictly-specified bytecode, cluster jobs are going to be random linux binaries that could do any combination of system calls. You need another layer of build/management system to make your docker containers useable. Worst of all, rather than defining common shared services for e.g. job scheduling, this is going to encourage users to put any random component in their docker image. We'll end up with a more fragmented ecosystem, not less.

eropple 12y ago
I'm not really sure why this post got downvoted, because I think there's some merit to it. In a spherical-cow universe I think there's an argument for opening up the universe to all tools, but I think in practice this makes good practices much harder to maintain and harder to proof against issues and failures at a glance. Personally, I dig Spark on EMR or Spark on Mesos (with Scala as a domain language), and I'm not sure how this plays with the rest of the world.
- sgt101 12y ago
  I think Spark on Yarn on Mesos co habiting with Docker on Mesos managed by Twill is the future. I just need to take a big breath every time I say it though.
jdoliner 12y ago
Hi, JD @ Pachyderm here. Our feeling isn't that Java is bad it's that it shouldn't be the only option for big data processing. There are a lot of people that want to do this type of computing and a lot of them hate Java. We want to build a solution that can serve them better and Docker is actually an incredibly high leverage way to do it. I feel like I don't fully understand your criticism of choosing Docker as platform. > cluster jobs are going to be random linux binaries that could do any combination of system calls Could you give an example of what could go wrong here? Docker prevents you from talking to the outside world so you can't call out to an outside service in the middle of a job or anything like that. > Worst of all, rather than defining common shared services for e.g. job scheduling In Pachyderm job scheduling happens outside the user submitted containers. The job scheduler is what actually spins up the containers and pushes data through them. I'm afraid I don't understand what it would even mean for one of those containers to then do job scheduling itself. Hope this clears up our thoughts on this a bit.

batbomb 12y ago

> Second, Etcd and Fleet are themselves designed to be modular, so it’s easy to support a variety of other deployment methods as well. Zookeeper isn't Modular? No mention of HTCondor+Dagman? No mention of Spark? I've written a general DAG processor/workflow engine/metascheduler/whatever you want to call it. It's used by various physics and astronomy experiments. I've interfaced it with the Grid/DIRAC, Amazon, LSF, Torque, PBS, SGE, and Crays. There's nothing in it that precludes Docker jobs from running. I've implemented a mixed-mode (Streaming+batch processing + more DAG) version of it which just uses job barriers to do the setup and ZeroMQ for inter-job communication. I think something like Spark's resilient distributed dataset would be nice here as well. We don't use hadoop because, as was said, because it's narrow in scope and it's not a good general purpose DAG/Workflow Engine/Pipeline. I think this is a small improvement, but I don't really see it being much better than hadoop, or HTCondor, or Spark. HTCondor is pretty amazing. I think somebody should be building a modern HTCondor, not a modern Hadoop.

a8da6b0c91d 12y ago
> somebody should be building a modern HTCondor A bug-free, cross platform, rock-solid HTCondor would be great. Sadly the existing HTCondor is a buggy turd. I've fixed some stuff in it and a lot of the code is eye-roll worthy. I can practically guarantee you hours of wondering why something that by any reasonable expectation should be working isn't working, and not giving any meaningful error message. But like you said, a really solid HTCondor rewrite with the Stork stuff baked in would pretty much be all anyone should need, I think. It's not like you'd even really need to dramatically re-architect the thing. The problems with HTCondor really are mostly just code quality and the user experience.
- lrm242 12y ago
  I use HTCondor every day. I find it to be rock solid with top notch documentation. It is cross-platform today as well, so I'm not sure where you're coming from on this.
  - a8da6b0c91d 12y ago

kev009 12y ago

No disclosure of inspiration from Manta https://github.com/joyent/manta https://github.com/joyent/manta? Also, why yet another DFS that makes excuses for shitty local filesystems instead of leveraging ZFS? Use ZFS as the base for integrity, snapshots, send/receive and make a small service on top for managing cluster metadata.

jdoliner 12y ago
Hi, JD @ Pachyderm here. We think Manta is pretty cool but it wasn't actually an inspiration for us. I'm not sure I fully understand your criticism of the filesystem. Pfs was actually initially conceived as almost exactly what you're describing, a small service on top of ZFS. But we realized early on that ZFS was going to be a pain because of its patent status so we used btrfs instead. I think btrfs is worse than ZFS but I'm not sure I'd call it a shitty local filesystem.

CurtMonash 12y ago

This sounds bizarre. Spark is the replacement for MapReduce, with Map and Reduce being just 2 of its 15 (?) primitives. Who needs another MapReduce engine? The bit about "We'll get around to Hive and Pig eventually" is also odd, as they're approaching legacy status too. (Albeit not there yet; please recall that I'm somewhat of an Impala skeptic, and of course there's the Tez booster to Hive.) Also bizarre are some of the history claims in the article, e.g. that YARN is part of more or less original Hadoop.

jdoliner 12y ago
It's been talked about a few times in this thread already so I'll try not to rehash too much. We think Spark is way cool but disagree that it's a complete replacement for MapReduce. There were people else where in this thread who described pretty well their use cases that didn't work with Spark. My experience is that Hive is pretty widely used, I have a lot fewer data points about Pig. We're not particularly attached to either of these. That sentence was more to indicate that we'd be adding some sort of interfaces for specific data types. Seems like we got our history wrong here. When was YARN added we'll add a correction. Hope this helps clear things up.
- thinkmassive 12y ago
  YARN was introduced with Hadoop 2.0, by MAPREDUCE-279: https://issues.apache.org/jira/browse/MAPREDUCE-279 https://issues.apache.org/jira/browse/MAPREDUCE-279 Note that YARN also supports Docker containers as of Hadoop 2.6... YARN-1964: Create Docker analog of the LinuxContainerExecutor in YARN https://issues.apache.org/jira/browse/YARN-1964 https://issues.apache.org/jira/browse/YARN-1964
- CurtMonash 12y ago
  Thanks. I'll stick with http://www.dbms2.com/2014/04/30/spark-on-fire/ http://www.dbms2.com/2014/04/30/spark-on-fire/ however, for the most part. (The mid-2014 PR blitz I thought I was engineering didn't actually happen.)

michaelochurch 12y ago

This sounds promising. I'd love to see a Haskell-centric (performance and robust code) but ultimately language-agnostic alternative to the JVM-heavy stack that's currently in vogue. (Spark is a promising and powerful technology.) I choose Haskell because pretty much anything that the JVM does well (e.g. concurrency) Haskell also does well and, while I like Clojure and can enjoy Scala when written by competent people, Java is ugly and its community has worse aesthetic sense than a taint mole.

necubi 12y ago
You may be interested in Cloud Haskell[0], which is an erlang-like distributed computation engine. The main thing that's missing in Haskell-land is an alternative to HDFS. [0] https://wiki.haskell.org/Cloud_Haskell https://wiki.haskell.org/Cloud_Haskell
- enigmo 12y ago
  Why not use Ceph? There are solid Haskell bindings for RADOS: https://github.com/anchor/rados-haskell https://github.com/anchor/rados-haskell
deleted 12y ago
[deleted]

spennant 12y ago

Funny... I just put up a hadoop cluster last week and named it pachyderm.mydomain.com (i'm probably not the first to do that either)

bipin_nag 12y ago

I agree with your findings on Hadoop. It is a very complicated ecosystem. The core is tied to JVM and abstractions of map-reduce are heavy and difficult to use. There are multiple domain specific languages/api to do the same thing : pig, cascading. Yet if you pick one you cant use other. Hadoop feels like tower of babel. But I am not in the favour of creating another stack as Hadoop replacement. Spark fills the role of distributed computation framework very well. It uses scala a functional language which provides neat abstractions for map,reduce,filter and other operations. It introduces RDD to abstract data. All the components graphx, sparksql, blinkdb use scala. Also building a new map-reduce engine is a difficult task. The computation engine will lie in the middle of stack not on top as shown. Libraries will sit on top of that, so if you build an engine you will have to build everything on top of that. See the Spark ecosystem : https://amplab.cs.berkeley.edu/software/ https://amplab.cs.berkeley.edu/software/

thinkmoore 12y ago

Still trying to wrap my head around the idea that 10 year old software is not "modern." That said, I have no opinion on this article.

bra-ket 12y ago

fantastic idea, unfortunate name (Pachyderm, seriously?)

pjmlp 12y ago

It is not modern if GNU/Linux is the only deployment option. I love the ability to run the cluster in any OS that has a JVM available, from big commercial UNIXes, Windows, *BSD and GNU/Linux distributions.

KaiserPro 12y ago

Sorry, but using HTTP for transport is just terrible. Hadoop is a poor imitation of a mainframe. When you are processing large lumps of data you need the following: A task scheduler A message passing system(although in some workloads this can be proxied by the scheduler or filesystem) File storage In mainframe land, you have a large machine with many processors all linked to the same kernel. As its one large machine, there is a scheduler that's so good you don't even know its there. However mainframes require specialist skills. If you want shared memory cluster, you're going to need inifiband (which is actually quite cheap now, and has 40gig ethernet type majigger in it aswell) but people are scared of that, so that leaves us with your standard commodity cluster. In VFX we have this nailed. before leaving the industry I was looking after a ~25k CPU cluster backed to 15pb of storage. No fancy map reduce bollocks, just plain files and scripts. Everything was connected by ten gigs ethernet. Sustained write rate of 25 gigabytes a second. firstly task placement is dead simple: https://github.com/mikrosimage/OpenRenderManagement/wiki/Introduction-to-job-submission https://github.com/mikrosimage/OpenRenderManagement/wiki/Int... look how simple that task mapping is. It is legitimate to point out that there are no primitives for IPC, but then your IPC needs are varied. Most log churn apps don't really need IPC, they just need a catcher to aggregate the results. Map:reduce has fooled people into thinking that everything is must be that way. This is designed to run on bared metal. Anything else is a waste of CPU/IO. We used cgroups to limit process memory, so that we could run concurrent tasks on the same box. But everything ran the same image, anything else is just a plain arse ache. The last item is file storage. HDFS is just terrible. the reason that its able to scale is that it provides literally nothing. Its just a key:value store, based on a bunch of key:value stores. If you want fast super scalable posix filesystem, then use GPFS(or elastic storage). The new release has a kind of software raid that allows you to recover from a dead disk in less than an hour. (think ZFS with shards) failing that you can use lustre (although that requires decent hardware raids, as its a glorified network raid0) or if you are lucky you can split your file system into name spaces that reside on different NFS servers. Either way, Hadoop, and hadoop inspired infrastructures are normally not the right answer. Measos is also probably not the answer unless you want a shared memory cluster. but then if you want this you need inifinband...

rkwasny 12y ago

Scheduling jobs by distributing docker container? Looks really heavyweight. There is a good reason most of Map-Reduce frameworks are written on top of JVM - it is VERY easy to serialize Java/Scala code ship it to remote host and execute there. This is not trivial in compiled languages ( I believe some hacks for Haskell exists ) What we need in Big Data space is more people that understand how computers really work. Some effort must be put into supporting TCP/IP Stacks in Userspace using new 10GBe cards or support for IB verbs. Also the long forgotten art of writing zero-copy code.

why-el 12y ago
> Some effort must be put into supporting TCP/IP Stacks in Userspace using new 10GBe cards or support for IB verbs Obviously my understanding of networks is not at that level yet. What would a layman do to become somewhat vested in this? Any books or resrouces?
- chaotic-good 12y ago
  This technique usually called 'kernel bypass'. http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html http://highscalability.com/blog/2013/5/13/the-secret-to-10-m... and http://lukego.github.io/blog/2013/01/04/kernel-bypass-networking/ http://lukego.github.io/blog/2013/01/04/kernel-bypass-networ...
  - why-el 12y ago
    Thank you. :)
codygman 12y ago
I wouldn't consider StaticPointers a hack. I believe this is the feature you are talking about.

TeMPOraL 12y ago

Excuse me for going off-topic, but heck, Hadoop itself was released 3 years ago. It's a testament to how insanely fast things change in web that Hadoop is no longer considered "modern"...

amelius 12y ago

Personally, I'd like to run purely functional jobs, and get progress information while they are running (this is not as easy as it sounds). Automatic memoization of jobs. And automatic killing of jobs and removal of data when results are no longer required (this should work transitively/recursively).

sp332 12y ago
You want a purely functional job with a side effect of updating progress info?
- scott_s 12y ago
  It's a reasonable request. The job is purely functional in the computation on its incoming data. That means you don't need to worry about persistent state management, which is a problem if you want to easily move jobs around a distributed system. The job (or part of the job) can still send metrics about what its doing to some other service, and still remain functional with respect to the data it's processing.
  - sp332 12y ago
    Functional doesn't just mean repeatable. If you modify global state that is not represented in the "input" and "output" of the job, then it's not purely functional.
    - scott_s 12y ago
      Yes, I am aware of what "functional" means. But I think you are being too literal. If you view the computation through the lens of the actual data involved in the computation, then it is functional. That it also has some internal state that gets updated and reported as part of the monitoring of the computation - but not part of the computation - then it is still functional with respect to the actual data in the job. The key distinction is that this internal metrics data that is just used for monitoring is incidental, and can be thrown away without impacting the computation's result. It's just convenient information that humans might want to have so they can reason about the internal state of the overall system.

eaxitect 12y ago

TL,DR; Although I'm not a fan of entire Hadoop system, some judgements about it is totally wrong. 1) Do one thing well - Hadoop is so stacked up because of this (which has pros and cons). Every single component of Hadoop is designed specific to a job (there are many overlapped components, though) 2) Hadoop's main idea is to be data-local, yet this Pachy thing not to be: "Data is POSTed to the container over HTTP and the results are stored back in the file system". Data-locality (almost) always win. 3) There is no scheduler like YARN specified, yet fleet and etcd are not schedulers. 4) You may not like Java or JVM-based things (like I do), but there is no problem of being Java-based or JVM-based. …