15 ms·

Billions wasted on Hadoop startups, the same will eventually be true of Docker

227 points by quicksilver03 7y ago

metaphor 7y ago
> ...but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. What are these supposed "hard problems" the author speaks of?
- saagarjha 7y ago
  Sandboxed, consistent environments to run code in?
  - Just1689 7y ago
    and environments as code
  - jsty 7y ago
    Although with recent adventures in speculative execution, that sandboxing isn't quite as 'solved' a problem as previously thought
  - 7373737373 7y ago
    All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems, instead of creating another inner-platform effect. I hope WebAssembly goes in this direction, instead of trying to adapt to current programming language paradigms.
    - AmericanChopper 7y ago
      >All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems But this isn’t the problem Docker is trying to solve. It’s just a problem that Docker needed to solve in order for their product to be useful, this is completely transparent to Docker users. Docker abstracts away a whole bunch of work you’d otherwise have to do to implement repeatable builds, it makes those builds widely distributable, and (depending on how you choose to use containers) can also simplify some capacity planning problems.
      - robfig 7y ago
        I thought Docker builds are not generally repeatable, since they often `apt-get update && apt-get install`, which depends on the current state of external package management? They are definitely not reproducible in the sense of building bit-for-bit identical containers, unless you use Bazel. That being said, I've found Dockerfiles to be a much more reliable build process than most others (recently struggled to get through LogDevice's cmake-based build.. ugh).
        AmericanChopper 7y ago
        You’re correct, but how reproducible you’re builds end up being depends on how you use it, and how reproducible you need them to be depends on your use case. Maybe a particular use case wouldn’t fit in very well, maybe it would be better served by using something like packer, maybe your dependency management requirements mean you should use something like artifactory. No technology is going to be suitable for everybody’s needs, but Docker provides enough value to enough people that it’s found a place in the market. If it Docker dies, I’d imagine it would be because it was replaced by something better, not because people suddenly realized that they weren’t getting any value out of it.
kumarvvr 7y ago
I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machine is just as sufficient for the task. Is that the reality of today? Personally I too feel that distributed computing is an overkill for most 'big' data problems.
- dkersten 7y ago
  I've heard people say that a huge portion of the "big" data is garbage, throwing more garbage at something won't make it better, so you'd get much more out of it by cutting it down to a more manageable amount of good data, which you can then easily process in postgres or whatever. But even if you "collect everything and sort it out later", in my own personal experience and in what I've read here on HN, you can go a long long way before you need to reach for the power tools. What most companies call "big data" is typically not that much (in quantity and in velocity). Most companies don't have tens or hundreds of terabytes of data. For example, I'm currently processing timeseries data in postgres using the timescaledb extension, which makes it perform very well. Still too early to state numbers, but its looking promising so far and if their claims are true, then I won't need anything else. We will see :)
- hobofan 7y ago
  It really depends on how "big" your data is. If you have data that is actually big (= doesn't fit onto a machine), then Hadoop is a reasonable candidate. Otherwise you are fine with a lot simpler tools. This has probably been true for quite some time, and companies are just slowly realizing now that their relevant data isn't actually that big. On the other hand computing power and storage has still grown in the last years, and less resource intensive ETL has become more acessible, so the bar for "big" data has been raised quite a bit.
  - more-coffee 7y ago
    About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did. Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.
chime 7y ago
> The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, I bought a Raspberry Pi and using a few commands, installed pre-configured Docker ARM images for 8-9 different media applications that would've taken me days to setup and manage individually. I didn't have to worry about dependencies or compilations. It just worked.
- deleted 7y ago
  [deleted]
- shatnersbassoon 7y ago
  Exactly. "other VMs"? The whole point of Docker is that it's not a VM...
- Sharlin 7y ago
  The problem of installing prepackaged applications was solved decades before Docker was a thing.
  - LoSboccacc 7y ago
    even considering puppet etc things weren't this simple, the base image depends on the provider so you'd get variants depending on the kernel and glibc version, especially with openvz based host now, on bare metal server and premium providers where you're able to specify the boot image that wouldn't have been a problem, but there are lot more people dealing with cheap housing giving you whatever image they baked last than people doing bare metal deployments and for the former docker is an order of magnitude easier to deploy
  - IshKebab 7y ago
    Evidently not or Docker wouldn't exist. Yes, Docker is a shitty solution to just packaging applications, but it exists because Linux developers keep saying "apt exists. It's solved! What's the problem? Static linking? Why would you want that? Portable binaries? But apt is the only place you need to put your app so you don't need them!"
- 7y ago
imtringued 7y ago
I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better basically boils down to saying which brand of hammer is better.
- pytester 7y ago
  >Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when containing and deploying somewhat badly written applications that have weirdly specific, non-standard system level dependencies (e.g. oracle) that you don't want polluting other applications' dependencies. I've compiled and installed postgres in separate folders lots of times (super easy) and I've lost count of the number of times people have said "why don't you just dockerize that?" as if that was simpler and/or necessary in some way. That's the effect of "docker hype" talking.
  - sz4kerto 7y ago
    The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour. The two primary use cases for Docker is, as far as I can see, is simplifying deployment on varying environments. Variations can happen because of many reasons. Sometimes you have clusters of various sizes in production. Sometimes the environment is a developer laptop. And so on.
    - pytester 7y ago
bryanrasmussen 7y ago
Isn't this the way it normally works though, a bunch of investments don't work out - those are wasted - those that do work out, the people who did the investing get more money back. Or to put it another way: There must have been some few Hadoop investments that worked out, the same will eventually be true of Docker.
- huffmsa 7y ago
  Shhhh get out of our ivory tower with your statistics about how the real world works. /s But yes, there's no reason to think that the distribution of successful, neutral and failure returns for Docker centric startups won't follow the usual distribution.
- gdulli 7y ago
  I think what they've noticed is a similar arc where both products got initially sold as being universally required and applicable and generational which justified enough investment to spawn a whole industry. And instead of fulfilling such dramatic hype they're both just good tools that are far from universally needed, not objectively superior to all other options, and there's nothing special about them that will keep them from getting supplanted by newer tools, which is the norm for even the industry even if the tools are good.
bayareanative 7y ago
Yup. Tech faabionabilism is akin to Big4/MBB FUD flavor of the quarter... populism/marketing doesn't a necessity make. Docker, Kubernetes, gulp, Hadoop, mosh, Nix, SmartOS, serverless, cloud, virtualization, [insert tech fashion hype > utility here]. Speaking of Hadoop: my vehicle is parked outside one of the HQ's of another top 10 Hadoop startup. It's one of the most expensive, nearly empty buildings in the highest rent areas of the Valley. (Money flushing sound here.) Fun fact: one of the enterprise Hadoop CTOs is a broney.
- threeseed 7y ago
  I love how the "cloud" is a tech fad in your eyes as well as technologies like Kubernetes which are core to Google and virtualisation which is core to every VPS/cloud provider in existence. Be a pretty awful world if everyone took your advice.
  - AmericanChopper 7y ago
    Isn’t it obvious? Everybody who doesn’t operate their own datacenters are mindless technology hipsters. Whenever a new big piece of tech comes out, most of the detractors seem to either a) find a use case that isn’t fit for purpose or b) try to use it without bothering to learn how, and the proceed to say ‘see, it’s not all it’s cracked up to be’.
- CraftThatBlock 7y ago
  How is mosh in there? Mosh is something to make ssh more reliable, how would that be a trend?
sasavilic 7y ago
You can't really compare VM with Docker. Managing containers with Docker + Kubernetes is far easier then managing VMs. Docker might be replaced with something else in future (i.e. rkt), but basic concept IMHO is here to stay.
- aduitsis 7y ago
  What about setup and maintenance of the infrastructure that hosts either VMs or containers? Given a mediocre number of physical machines (say, 40), which is easier, install, setup and maintain a VMWare cluster or a Kubernetes cluster? If someone has any insight, preferably backed by actual experience, it'll be most appreciated.
  - kklimonda 7y ago
    Assuming unlimited budget for hardware and licenses, then building a team that can deploy and manage VMWare on 40 nodes will be much easier - however raw VM is not really comparable to what k8s is giving you, so you'll be also solving reproducible deployments, load balancing traffic to your cluster etc. Still, with enough money those can be solved by purchasing more hardware&software, and you'll have easier time finding people who can maintain that over on-premise kubernetes.
    - aduitsis 7y ago
      Many thanks!
  - jarfil 7y ago
    VMs are application agnostic, while Kubernetes is a service manager. If you want to run 200 different services on 40 machines, you may find hand crafted VMs easier to create and forget. If you want to run 10 services with different levels of replication and redundancy on 40 machines, then Kubernetes will do that for you.
- raesene9 7y ago
  small aside, but it looks like rkt is not going to be that replacement. Recent disclosure of vulnerabilities in rkt that Redhat are apparently not going to bother patching indicates they don't currently see it as an active project.
viraptor 7y ago
> and number three – well it’s not even worth staying in business because there’s no money to be made, I guess tell that to the 10s of companies doing PaaS. Or another 10s doing app monitoring / logging. They're successful companies, far from just "breaking even".
threeseed 7y ago
Cloudera earnt $145m last quarter and grew by 37% over the previous quarter. Other Hadoop startups like Databricks are doing well and Docker has gone from a 2 digit revenue to a 3 digit revenue company from 2017-2018. How has billions been wasted when we have successful companies doing well against the toughest competitors ever i.e. Google, Microsoft and Amazon.
- geodel 7y ago
  Well that include revenues from Hortonworks also. With revenue rise losses are also rising to $85.5M. I feel in couple of year they will fold up or brought by some big cloud vendor.
villgax 7y ago
What is even a Docker startup? Docker the tool is a relief in every way & so is Kubernetes.
mrosett 7y ago
The article just quotes the news about MapR and then asserts the same will happen with Docker. That may be true, but there’s no evidence here.
hardwaresofton 7y ago
I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (using the word "docker" for everything isn't right either) is an isolation + sandboxing mechanism, NOT a virtual machine. Even if you talk about things like LXC (orchestrated by LXD), that's basically just the addition of the user namespacing feature. A docker container is not a VM, it is a regular process, isolated with the use of cgroups and namespaces, possibly protected (like any other process) with selinux/apparmor/etc. Containerization is almost objectively a better way of running applications -- there's only one question, do you want your process to be isolated, or not. All the other stuff (using Dockerfiles, pulling images, the ease of running languages that require their own interpreters since you package the filesystem) is on top of this basic value propostion. An easy way to tell that someone doesn't know what they're talking about when speaking about containerization is if they call it a VM (and don't qualify/note that they're being fast and loose with terminology). All this said -- I do think Docker will die, and it should die because Docker is no longer the only game in town for reasonably managing (see: podman,crictl) and running containers (see: containerd/cri-o, libcontainer which turned into runc) . [EDIT] - I want to point out that I do not mean the Docker the company or Docker the project will "die" -- they have done amazing things for the community and development as a whole that will literally go down in history as a paradigm shift. What I should have written was that "docker <x>" where x is "image", "container", "registry", etc should be replaced by "container <x>".
- chrisdone 7y ago
  Deployment of Docker containers is nice, but deploying a VM as in Vagrant was also fine. I avoided learning anything about Docker for about 2 years because I thought it was just a fad. However, I would add that for my own personal use, it's invaluable for development work. All that work that you do _before_ your CI or deployment. 1) When I'm working with a collection of tools that I need but are a complete mess with lots of state (think: compiler tools, LaTeX, things like that), then docker image build with its incremental way of running each command piece by piece, and saving the state after each RUN, is actually a life saver. You follow the steps of some instructions, and of course, as usual, there's one extra step not documented in the manual, so you add that to your Dockerfile. You make a mistake, no big deal, just change the command, the bad state is discarded, and you get to try again. You don't have to run the whole thing all over again. And it's instantaneous. 2) When I have to work with a client's codebase, as a consultant, you'd be surprised how many projects do not have a reproducible build, with Docker or anything else. So I end up building my own Dockerfile. The number of times I've heard "but you just have to run this setup script once" -- well, those scripts never work (why would they? nobody runs them anymore). Especially when it begins with `npm` or `pip` -- almost guaranteed to fail catastrophically, with some g++ compile error, or a segfault, or just a backtrace that means nothing. For example, I recently had to run an `npm` install command and it failed with `npm ERR! write after end`. I re-ran the container again, and again once more, and then it succeeded (https://gist.github.com/chrisdone/ea6e4ba3d8bf2d02f491b4a17fdc4433 https://gist.github.com/chrisdone/ea6e4ba3d8bf2d02f491b4a17f...). npm has a race condition (https://github.com/npm/npm/issues/19989; https://github.com/npm/npm/issues/19989; fixed in the latest version). I wouldn't have been able to confidently share with my client this situation unless I had that reproducibility. 3) It's trivial to share my Dockerfile with anyone else and they can run the same build process. I don't have to share an opaque VM that's hundreds of megs and decide where to put it and how long I want to keep it there, etc. 4) It's a small one; but speed. Spinning up or resuming a VirtualBox machine is just slow. I can run docker containers like scripts, there isn't the same overhead. 5) Popularity is a blessing; the fact that I _can_ share my Dockerfile with someone is a network effect. Like using Git, instead of e.g. darcs or bzr. By the way, you can also do nice container management with systemd and git. There's nothing inherently technologically _new_ about Docker; it's the workflow and network effects; it lets me treat a system's state like a Git repo. Nix has similar advantages, but I see Docker as one small layer above Nix.
Jonnax 7y ago
Who is this commenter? Containerised applications are commonly used. At this point it's a proven technology with clear use cases.
skc 7y ago
I suppose I haven't worked in environments scary enough that Docker was a necessity. As a result I find it difficult understand the hype.
jgalt212 7y ago
My beef with Hadoop, and other big data tools, is that for pretty much any task other than outlier detection, sampling works as good, and is cheaper and easy to manage and reason about. Even Google, the king of big data, will sample your hits on Google Analytics if your site gets too much traffic.
deleted 7y ago
[deleted]
mananvaghasiya 7y ago
> I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs Oh boy
dio123 7y ago
How about “service mesh”?
harimau777 7y ago
What is a "Hadoop business"? Hadoop is a tool not an industry or product. Can someone explain what the author might be trying to say?
- dijksterhuis 7y ago
  Businesses that based their SaaS data analysis products on Hadoop. Preconfigured clusters, integration with your existing AWS deployments. All that sort of jazz.
some_random 7y ago
Seriously, who reads this kind of garbage? It's painfully clear to anyone with any experience with Docker that the author hasn't even skimmed the wikipedia page. Reminds me of the idiots blasting out blog posts about how BITCOIN IS THE FUTURE one month, then BITCOIN IS A SCAM the next.
alfiedotwtf 7y ago
The difference being that people actually use Docker
jressey 7y ago
Billions are "wasted" on startups in every industry. That's the point. The rich guys gamble and some of their gambles pay off. Most of them fail.
xenospn 7y ago
So I've never used Docker, but it sounds an awful lot like FreeBSD Jails. How is it unique?
- kkapelon 7y ago
  Much more userfriendly. Works on all major platforms. Big public repository (dockerhub). Extensive tooling for everything. Graphical and CLI tools for all kinds of users.

227 points by quicksilver03 7y ago

> ...but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. What are these supposed "hard problems" the author speaks of?

saagarjha 7y ago
Sandboxed, consistent environments to run code in?
- Just1689 7y ago
  and environments as code
- jsty 7y ago
  Although with recent adventures in speculative execution, that sandboxing isn't quite as 'solved' a problem as previously thought
- 7373737373 7y ago
  All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems, instead of creating another inner-platform effect. I hope WebAssembly goes in this direction, instead of trying to adapt to current programming language paradigms.
  - AmericanChopper 7y ago
    >All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems But this isn’t the problem Docker is trying to solve. It’s just a problem that Docker needed to solve in order for their product to be useful, this is completely transparent to Docker users. Docker abstracts away a whole bunch of work you’d otherwise have to do to implement repeatable builds, it makes those builds widely distributable, and (depending on how you choose to use containers) can also simplify some capacity planning problems.
    - robfig 7y ago
      I thought Docker builds are not generally repeatable, since they often `apt-get update && apt-get install`, which depends on the current state of external package management? They are definitely not reproducible in the sense of building bit-for-bit identical containers, unless you use Bazel. That being said, I've found Dockerfiles to be a much more reliable build process than most others (recently struggled to get through LogDevice's cmake-based build.. ugh).
      - AmericanChopper 7y ago
        You’re correct, but how reproducible you’re builds end up being depends on how you use it, and how reproducible you need them to be depends on your use case. Maybe a particular use case wouldn’t fit in very well, maybe it would be better served by using something like packer, maybe your dependency management requirements mean you should use something like artifactory. No technology is going to be suitable for everybody’s needs, but Docker provides enough value to enough people that it’s found a place in the market. If it Docker dies, I’d imagine it would be because it was replaced by something better, not because people suddenly realized that they weren’t getting any value out of it.

kumarvvr 7y ago

I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machine is just as sufficient for the task. Is that the reality of today? Personally I too feel that distributed computing is an overkill for most 'big' data problems.

dkersten 7y ago
I've heard people say that a huge portion of the "big" data is garbage, throwing more garbage at something won't make it better, so you'd get much more out of it by cutting it down to a more manageable amount of good data, which you can then easily process in postgres or whatever. But even if you "collect everything and sort it out later", in my own personal experience and in what I've read here on HN, you can go a long long way before you need to reach for the power tools. What most companies call "big data" is typically not that much (in quantity and in velocity). Most companies don't have tens or hundreds of terabytes of data. For example, I'm currently processing timeseries data in postgres using the timescaledb extension, which makes it perform very well. Still too early to state numbers, but its looking promising so far and if their claims are true, then I won't need anything else. We will see :)
hobofan 7y ago
It really depends on how "big" your data is. If you have data that is actually big (= doesn't fit onto a machine), then Hadoop is a reasonable candidate. Otherwise you are fine with a lot simpler tools. This has probably been true for quite some time, and companies are just slowly realizing now that their relevant data isn't actually that big. On the other hand computing power and storage has still grown in the last years, and less resource intensive ETL has become more acessible, so the bar for "big" data has been raised quite a bit.
- more-coffee 7y ago
  About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did. Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.

chime 7y ago

> The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, I bought a Raspberry Pi and using a few commands, installed pre-configured Docker ARM images for 8-9 different media applications that would've taken me days to setup and manage individually. I didn't have to worry about dependencies or compilations. It just worked.

deleted 7y ago
[deleted]
shatnersbassoon 7y ago
Exactly. "other VMs"? The whole point of Docker is that it's not a VM...
Sharlin 7y ago
The problem of installing prepackaged applications was solved decades before Docker was a thing.
- LoSboccacc 7y ago
  even considering puppet etc things weren't this simple, the base image depends on the provider so you'd get variants depending on the kernel and glibc version, especially with openvz based host now, on bare metal server and premium providers where you're able to specify the boot image that wouldn't have been a problem, but there are lot more people dealing with cheap housing giving you whatever image they baked last than people doing bare metal deployments and for the former docker is an order of magnitude easier to deploy
- IshKebab 7y ago
  Evidently not or Docker wouldn't exist. Yes, Docker is a shitty solution to just packaging applications, but it exists because Linux developers keep saying "apt exists. It's solved! What's the problem? Static linking? Why would you want that? Portable binaries? But apt is the only place you need to put your app so you don't need them!"
7y ago

imtringued 7y ago

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better basically boils down to saying which brand of hammer is better.

pytester 7y ago
>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when containing and deploying somewhat badly written applications that have weirdly specific, non-standard system level dependencies (e.g. oracle) that you don't want polluting other applications' dependencies. I've compiled and installed postgres in separate folders lots of times (super easy) and I've lost count of the number of times people have said "why don't you just dockerize that?" as if that was simpler and/or necessary in some way. That's the effect of "docker hype" talking.
- sz4kerto 7y ago
  The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour. The two primary use cases for Docker is, as far as I can see, is simplifying deployment on varying environments. Variations can happen because of many reasons. Sometimes you have clusters of various sizes in production. Sometimes the environment is a developer laptop. And so on.
  - pytester 7y ago

bryanrasmussen 7y ago

Isn't this the way it normally works though, a bunch of investments don't work out - those are wasted - those that do work out, the people who did the investing get more money back. Or to put it another way: There must have been some few Hadoop investments that worked out, the same will eventually be true of Docker.

huffmsa 7y ago
Shhhh get out of our ivory tower with your statistics about how the real world works. /s But yes, there's no reason to think that the distribution of successful, neutral and failure returns for Docker centric startups won't follow the usual distribution.
gdulli 7y ago
I think what they've noticed is a similar arc where both products got initially sold as being universally required and applicable and generational which justified enough investment to spawn a whole industry. And instead of fulfilling such dramatic hype they're both just good tools that are far from universally needed, not objectively superior to all other options, and there's nothing special about them that will keep them from getting supplanted by newer tools, which is the norm for even the industry even if the tools are good.

bayareanative 7y ago

Yup. Tech faabionabilism is akin to Big4/MBB FUD flavor of the quarter... populism/marketing doesn't a necessity make. Docker, Kubernetes, gulp, Hadoop, mosh, Nix, SmartOS, serverless, cloud, virtualization, [insert tech fashion hype > utility here]. Speaking of Hadoop: my vehicle is parked outside one of the HQ's of another top 10 Hadoop startup. It's one of the most expensive, nearly empty buildings in the highest rent areas of the Valley. (Money flushing sound here.) Fun fact: one of the enterprise Hadoop CTOs is a broney.

threeseed 7y ago
I love how the "cloud" is a tech fad in your eyes as well as technologies like Kubernetes which are core to Google and virtualisation which is core to every VPS/cloud provider in existence. Be a pretty awful world if everyone took your advice.
- AmericanChopper 7y ago
  Isn’t it obvious? Everybody who doesn’t operate their own datacenters are mindless technology hipsters. Whenever a new big piece of tech comes out, most of the detractors seem to either a) find a use case that isn’t fit for purpose or b) try to use it without bothering to learn how, and the proceed to say ‘see, it’s not all it’s cracked up to be’.
CraftThatBlock 7y ago
How is mosh in there? Mosh is something to make ssh more reliable, how would that be a trend?

sasavilic 7y ago

You can't really compare VM with Docker. Managing containers with Docker + Kubernetes is far easier then managing VMs. Docker might be replaced with something else in future (i.e. rkt), but basic concept IMHO is here to stay.

aduitsis 7y ago
What about setup and maintenance of the infrastructure that hosts either VMs or containers? Given a mediocre number of physical machines (say, 40), which is easier, install, setup and maintain a VMWare cluster or a Kubernetes cluster? If someone has any insight, preferably backed by actual experience, it'll be most appreciated.
- kklimonda 7y ago
  Assuming unlimited budget for hardware and licenses, then building a team that can deploy and manage VMWare on 40 nodes will be much easier - however raw VM is not really comparable to what k8s is giving you, so you'll be also solving reproducible deployments, load balancing traffic to your cluster etc. Still, with enough money those can be solved by purchasing more hardware&software, and you'll have easier time finding people who can maintain that over on-premise kubernetes.
  - aduitsis 7y ago
    Many thanks!
- jarfil 7y ago
  VMs are application agnostic, while Kubernetes is a service manager. If you want to run 200 different services on 40 machines, you may find hand crafted VMs easier to create and forget. If you want to run 10 services with different levels of replication and redundancy on 40 machines, then Kubernetes will do that for you.
raesene9 7y ago
small aside, but it looks like rkt is not going to be that replacement. Recent disclosure of vulnerabilities in rkt that Redhat are apparently not going to bother patching indicates they don't currently see it as an active project.

viraptor 7y ago

> and number three – well it’s not even worth staying in business because there’s no money to be made, I guess tell that to the 10s of companies doing PaaS. Or another 10s doing app monitoring / logging. They're successful companies, far from just "breaking even".

threeseed 7y ago

Cloudera earnt $145m last quarter and grew by 37% over the previous quarter. Other Hadoop startups like Databricks are doing well and Docker has gone from a 2 digit revenue to a 3 digit revenue company from 2017-2018. How has billions been wasted when we have successful companies doing well against the toughest competitors ever i.e. Google, Microsoft and Amazon.

geodel 7y ago
Well that include revenues from Hortonworks also. With revenue rise losses are also rising to $85.5M. I feel in couple of year they will fold up or brought by some big cloud vendor.

villgax 7y ago

What is even a Docker startup? Docker the tool is a relief in every way & so is Kubernetes.

mrosett 7y ago

The article just quotes the news about MapR and then asserts the same will happen with Docker. That may be true, but there’s no evidence here.

hardwaresofton 7y ago

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (using the word "docker" for everything isn't right either) is an isolation + sandboxing mechanism, NOT a virtual machine. Even if you talk about things like LXC (orchestrated by LXD), that's basically just the addition of the user namespacing feature. A docker container is not a VM, it is a regular process, isolated with the use of cgroups and namespaces, possibly protected (like any other process) with selinux/apparmor/etc. Containerization is almost objectively a better way of running applications -- there's only one question, do you want your process to be isolated, or not. All the other stuff (using Dockerfiles, pulling images, the ease of running languages that require their own interpreters since you package the filesystem) is on top of this basic value propostion. An easy way to tell that someone doesn't know what they're talking about when speaking about containerization is if they call it a VM (and don't qualify/note that they're being fast and loose with terminology). All this said -- I do think Docker will die, and it should die because Docker is no longer the only game in town for reasonably managing (see: podman,crictl) and running containers (see: containerd/cri-o, libcontainer which turned into runc) . [EDIT] - I want to point out that I do not mean the Docker the company or Docker the project will "die" -- they have done amazing things for the community and development as a whole that will literally go down in history as a paradigm shift. What I should have written was that "docker <x>" where x is "image", "container", "registry", etc should be replaced by "container <x>".

chrisdone 7y ago
Deployment of Docker containers is nice, but deploying a VM as in Vagrant was also fine. I avoided learning anything about Docker for about 2 years because I thought it was just a fad. However, I would add that for my own personal use, it's invaluable for development work. All that work that you do _before_ your CI or deployment. 1) When I'm working with a collection of tools that I need but are a complete mess with lots of state (think: compiler tools, LaTeX, things like that), then docker image build with its incremental way of running each command piece by piece, and saving the state after each RUN, is actually a life saver. You follow the steps of some instructions, and of course, as usual, there's one extra step not documented in the manual, so you add that to your Dockerfile. You make a mistake, no big deal, just change the command, the bad state is discarded, and you get to try again. You don't have to run the whole thing all over again. And it's instantaneous. 2) When I have to work with a client's codebase, as a consultant, you'd be surprised how many projects do not have a reproducible build, with Docker or anything else. So I end up building my own Dockerfile. The number of times I've heard "but you just have to run this setup script once" -- well, those scripts never work (why would they? nobody runs them anymore). Especially when it begins with `npm` or `pip` -- almost guaranteed to fail catastrophically, with some g++ compile error, or a segfault, or just a backtrace that means nothing. For example, I recently had to run an `npm` install command and it failed with `npm ERR! write after end`. I re-ran the container again, and again once more, and then it succeeded (https://gist.github.com/chrisdone/ea6e4ba3d8bf2d02f491b4a17fdc4433 https://gist.github.com/chrisdone/ea6e4ba3d8bf2d02f491b4a17f...). npm has a race condition (https://github.com/npm/npm/issues/19989; https://github.com/npm/npm/issues/19989; fixed in the latest version). I wouldn't have been able to confidently share with my client this situation unless I had that reproducibility. 3) It's trivial to share my Dockerfile with anyone else and they can run the same build process. I don't have to share an opaque VM that's hundreds of megs and decide where to put it and how long I want to keep it there, etc. 4) It's a small one; but speed. Spinning up or resuming a VirtualBox machine is just slow. I can run docker containers like scripts, there isn't the same overhead. 5) Popularity is a blessing; the fact that I _can_ share my Dockerfile with someone is a network effect. Like using Git, instead of e.g. darcs or bzr. By the way, you can also do nice container management with systemd and git. There's nothing inherently technologically _new_ about Docker; it's the workflow and network effects; it lets me treat a system's state like a Git repo. Nix has similar advantages, but I see Docker as one small layer above Nix.

Jonnax 7y ago

Who is this commenter? Containerised applications are commonly used. At this point it's a proven technology with clear use cases.

skc 7y ago

I suppose I haven't worked in environments scary enough that Docker was a necessity. As a result I find it difficult understand the hype.

jgalt212 7y ago

My beef with Hadoop, and other big data tools, is that for pretty much any task other than outlier detection, sampling works as good, and is cheaper and easy to manage and reason about. Even Google, the king of big data, will sample your hits on Google Analytics if your site gets too much traffic.

deleted 7y ago

[deleted]

mananvaghasiya 7y ago

> I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs Oh boy

dio123 7y ago

How about “service mesh”?

harimau777 7y ago

What is a "Hadoop business"? Hadoop is a tool not an industry or product. Can someone explain what the author might be trying to say?

dijksterhuis 7y ago
Businesses that based their SaaS data analysis products on Hadoop. Preconfigured clusters, integration with your existing AWS deployments. All that sort of jazz.

some_random 7y ago

Seriously, who reads this kind of garbage? It's painfully clear to anyone with any experience with Docker that the author hasn't even skimmed the wikipedia page. Reminds me of the idiots blasting out blog posts about how BITCOIN IS THE FUTURE one month, then BITCOIN IS A SCAM the next.

alfiedotwtf 7y ago

The difference being that people actually use Docker

jressey 7y ago

Billions are "wasted" on startups in every industry. That's the point. The rich guys gamble and some of their gambles pay off. Most of them fail.

xenospn 7y ago

So I've never used Docker, but it sounds an awful lot like FreeBSD Jails. How is it unique?

kkapelon 7y ago
Much more userfriendly. Works on all major platforms. Big public repository (dockerhub). Extensive tooling for everything. Graphical and CLI tools for all kinds of users.