6 ms·

> But maybe I envision my side project turning into full-time startup some day. The state of the art for cluster management will probably something completely

> But maybe I envision my side project turning into full-time startup some day. The state of the art for cluster management will probably something completely different by then. Better to build a good product now and if you really want to turn it into a startup, productionize it then. > Maybe I see all the news about Kubernetes and think it would be cool to be more familiar with it. If learning Kubernetes _is_ your side project, then perfect go do that. Otherwise its just a distraction, taking more time away from actually building your side project and putting it into building infrastructure around your side project. If what you really wanted to build is infrastructure, then great, you're doing swell, but if you were really trying to build some other fun side app, Kubernetes is just a time/money sink in almost all cases IMO.

tomnipotent 8y ago
> The state of the art for cluster management will probably something completely different by then Has it not only changed twice in the prior two decades between first VMs and now containers? Don't think this is something you have to worry about long term.
- llama052 8y ago
  Uh I don't think vms or containers are cluster management in itself. They are just technologies. What you use to orchestrate them is entirely different, and yes that has changed over the years many times.
  - tomnipotent 8y ago
    For example? Most early cluster management was found in products doing general fleet provisioning, HPC/HTC in academia, proprietary commercials offerings like relational databases, or proprietary in-house solutions like Google's Borg. I'd describe cluster management as four things - provisioning + configuration, resource management (CPU+RAM+Disk), scheduling (when & where to put things based on availability/resources), and deployment (running things). At least, these are the things I'm concerned about when managing a product that requires a cluster (besides monitoring). Early cluster management tools were often just doing provisioning and configuration (CFEngine). We see Puppet, Chef, and eventually Ansible further refine the solution as we enter the "Bronze Era" where standing up new servers take a fraction of the time. Now we didn't even bother to name each server by hand when we were going through the installation process after booting up the OS - servers had become cattle, and they were tagged appropriately. Around the same time (2003-2006) we see virtual machines begin to catch on, culminating in the 2006 debut of Amazon Elastic Compute Cloud (EC2) and the birth of modern cloud computing. We now had a general purpose mechanism (VMs) to spin up isolated compute units that could be provisioned, configured, and managed using tools like CFEngine & Puppet. IT departments begin spending less on the SANs that dominated the early aughts and shift budgets to AWS or VMWare ESXi. Then in 2006 we see Hadoop spin off from Nutch, and the MapReduce paradigm popularized by Google leads to an optimization of the resource and scheduling problems thanks to YARN and Mesos and related tools. Non-trivial programs and workloads can now be scheduled across a cluster with vastly improved confidence. Distributed programs are now in reach of a much larger audience. Suddenly new Hadoop clusters are springing up everywhere and the "Silver Era" begins. Tools enabling greater efficiency hit the scene like RPC serialization frameworks (Thrift, Protobuf), improved on-disk storage (O/RCFile, Parquet), distributed queues (RabbitMQ, SQS), clustered databases (Cassandra, HBase), and event streaming and processing (Storm, Kafka). Coordination becomes more complicated and essential, so we get ZooKeeper and etcd and consul. Managing passwords and other secure data leads to tools like Vault. Logstash and Graylog make log management less of a nightmare. Solr leads to Elasticsearch leads Kibana and we now have logging and visualization for both server and application monitoring. Developers also begin to take advantage of VMs and it's not long before tools like Vagrant help bridge the gap between development, staging, and production. Our profession takes a collective sigh of relief as getting our distributed programs started on a new developers machine went from three days of trial-and-error to minutes thanks to well-maintained Vagrantfiles. Still, deployment is a big pain point. A single cluster could be home to Hadoop (which benefits from YARN) and thirty other things being run by teams across the organization. One day you discover that your mission critical web app is sharing resources with your BI team after your CEO calls you in a panic to tell you the website is down. Turns out a DBA transferred a 20TB backup across the same switch shared with your customer-facing website because somebody forgot to isolate a VLAN to prevent backups from interfering with other traffic. This doesn't even take into the consideration the abominations we call deployment scripts that developers and DevOps wrangle together to get our programs deployed. Then Docker and containers become a thing. Developers are now able to setup incredibly complex distributed programs with little more than text files. No more coordinating with DevOps to make sure your servers have the latest version of libpng, or spending months fighting to get Java upgraded to the latest version. I shout in glee as I delete Chef from my machine because progress. Then the beer goggles dissipate and the love affair ends and we realize things are shitty in a different way. Docker Swarm and Kubernetes emerge, and that brings us to today which I'll call the "Golden Era". We now have incredible tooling that deals with provisioning and configuration, resource management, scheduling, and deployment. But like any new era there are rough spots, but I'm positive incredible new tech will popup. Throughout all of this, virtual machines and containers were the fundamental building blocks that enabled improved clustering and cluster management. They're inextricably tied together. But all-in-all, things have changed VERY little (couhg LXC cough) in 20 years compared to the rest of the landscape. We're solid for another 10 years before anything like this is going to happen again.
dunk010 8y ago
A million times yes. You have to pick your battles.
carlisle_ 8y ago
> taking more time away from actually building your side project and putting it into building infrastructure around your side project. I generally dislike this way of thinking. Infrastructure is a core component of whatever it is you're building, not an afterthought. Maybe you can defer things until a little bit later, but if you can build with infrastructure in mind you'll be saving yourself so many headaches down the road. You don't need to build with the entire future of your project's infrastructure in mind, but deploying your project shouldn't be "ok now what?" when you're ready, like it was a big surprise.
- theptip 8y ago
  > Infrastructure is a core component of whatever it is you're building That's true in some sense -- but you can get surprisingly far using a PaaS like Heroku to abstract that infrastructure away. I'm a big fan of Kubernetes, and use it in production at my company, but I would not recommend using k8s in a prototype/early-stage startup unless you're already very familiar with the tool. The complexity overhead of k8s is non-trivial, and switching from Heroku to something like k8s doesn't involve undoing much work, since setting up Heroku is trivial.
  - hosh 8y ago
    I am a big fan of K8S too, not only use it in production, but I was also the one that set it up for my team. I agree that, unless you are already familiar with it, it is not always useful for protyping stage. There is something to be said about having the infrastructure in mind though. That's why I'm inclined to use something like Elixir/Phoenix for web-based projects. Some (not all) of the ideas that K8S brings to the table are already built into the Erlang/OTP platform. As for Heroku, there was a recent announcement that I think shifts things quite a bit: https://blog.heroku.com/buildpacks-go-cloud-native https://blog.heroku.com/buildpacks-go-cloud-native ... having standardized container images that runs buildpacks. The ecosystem and tooling is not quite there yet, but I can see this as significantly reducing the investment to put into Dockerizing your app for K8S. At that point, for the hobbyist, it might be: Prototype -> Heroku -> K8S with an Operator that can run the buildpack K8S is really a toolset for building your own PAAS. If there were a self-driving PAAS (using Operators) targeting small hobbyists that will run cloud native buildpacks, then the barriers of entry for a hobbyist using K8S is much lower.
drb91 8y ago
> Better to build a good product now and if you really want to turn it into a startup, productionize it then. Well, you gotta deploy with something. Why not k8s? Obviously if you're developing locally, k8s would not help in the least.

tomnipotent 8y ago

> The state of the art for cluster management will probably something completely different by then Has it not only changed twice in the prior two decades between first VMs and now containers? Don't think this is something you have to worry about long term.

llama052 8y ago
Uh I don't think vms or containers are cluster management in itself. They are just technologies. What you use to orchestrate them is entirely different, and yes that has changed over the years many times.
- tomnipotent 8y ago
  For example? Most early cluster management was found in products doing general fleet provisioning, HPC/HTC in academia, proprietary commercials offerings like relational databases, or proprietary in-house solutions like Google's Borg. I'd describe cluster management as four things - provisioning + configuration, resource management (CPU+RAM+Disk), scheduling (when & where to put things based on availability/resources), and deployment (running things). At least, these are the things I'm concerned about when managing a product that requires a cluster (besides monitoring). Early cluster management tools were often just doing provisioning and configuration (CFEngine). We see Puppet, Chef, and eventually Ansible further refine the solution as we enter the "Bronze Era" where standing up new servers take a fraction of the time. Now we didn't even bother to name each server by hand when we were going through the installation process after booting up the OS - servers had become cattle, and they were tagged appropriately. Around the same time (2003-2006) we see virtual machines begin to catch on, culminating in the 2006 debut of Amazon Elastic Compute Cloud (EC2) and the birth of modern cloud computing. We now had a general purpose mechanism (VMs) to spin up isolated compute units that could be provisioned, configured, and managed using tools like CFEngine & Puppet. IT departments begin spending less on the SANs that dominated the early aughts and shift budgets to AWS or VMWare ESXi. Then in 2006 we see Hadoop spin off from Nutch, and the MapReduce paradigm popularized by Google leads to an optimization of the resource and scheduling problems thanks to YARN and Mesos and related tools. Non-trivial programs and workloads can now be scheduled across a cluster with vastly improved confidence. Distributed programs are now in reach of a much larger audience. Suddenly new Hadoop clusters are springing up everywhere and the "Silver Era" begins. Tools enabling greater efficiency hit the scene like RPC serialization frameworks (Thrift, Protobuf), improved on-disk storage (O/RCFile, Parquet), distributed queues (RabbitMQ, SQS), clustered databases (Cassandra, HBase), and event streaming and processing (Storm, Kafka). Coordination becomes more complicated and essential, so we get ZooKeeper and etcd and consul. Managing passwords and other secure data leads to tools like Vault. Logstash and Graylog make log management less of a nightmare. Solr leads to Elasticsearch leads Kibana and we now have logging and visualization for both server and application monitoring. Developers also begin to take advantage of VMs and it's not long before tools like Vagrant help bridge the gap between development, staging, and production. Our profession takes a collective sigh of relief as getting our distributed programs started on a new developers machine went from three days of trial-and-error to minutes thanks to well-maintained Vagrantfiles. Still, deployment is a big pain point. A single cluster could be home to Hadoop (which benefits from YARN) and thirty other things being run by teams across the organization. One day you discover that your mission critical web app is sharing resources with your BI team after your CEO calls you in a panic to tell you the website is down. Turns out a DBA transferred a 20TB backup across the same switch shared with your customer-facing website because somebody forgot to isolate a VLAN to prevent backups from interfering with other traffic. This doesn't even take into the consideration the abominations we call deployment scripts that developers and DevOps wrangle together to get our programs deployed. Then Docker and containers become a thing. Developers are now able to setup incredibly complex distributed programs with little more than text files. No more coordinating with DevOps to make sure your servers have the latest version of libpng, or spending months fighting to get Java upgraded to the latest version. I shout in glee as I delete Chef from my machine because progress. Then the beer goggles dissipate and the love affair ends and we realize things are shitty in a different way. Docker Swarm and Kubernetes emerge, and that brings us to today which I'll call the "Golden Era". We now have incredible tooling that deals with provisioning and configuration, resource management, scheduling, and deployment. But like any new era there are rough spots, but I'm positive incredible new tech will popup. Throughout all of this, virtual machines and containers were the fundamental building blocks that enabled improved clustering and cluster management. They're inextricably tied together. But all-in-all, things have changed VERY little (couhg LXC cough) in 20 years compared to the rest of the landscape. We're solid for another 10 years before anything like this is going to happen again.

dunk010 8y ago

A million times yes. You have to pick your battles.

carlisle_ 8y ago

> taking more time away from actually building your side project and putting it into building infrastructure around your side project. I generally dislike this way of thinking. Infrastructure is a core component of whatever it is you're building, not an afterthought. Maybe you can defer things until a little bit later, but if you can build with infrastructure in mind you'll be saving yourself so many headaches down the road. You don't need to build with the entire future of your project's infrastructure in mind, but deploying your project shouldn't be "ok now what?" when you're ready, like it was a big surprise.

theptip 8y ago
> Infrastructure is a core component of whatever it is you're building That's true in some sense -- but you can get surprisingly far using a PaaS like Heroku to abstract that infrastructure away. I'm a big fan of Kubernetes, and use it in production at my company, but I would not recommend using k8s in a prototype/early-stage startup unless you're already very familiar with the tool. The complexity overhead of k8s is non-trivial, and switching from Heroku to something like k8s doesn't involve undoing much work, since setting up Heroku is trivial.
- hosh 8y ago
  I am a big fan of K8S too, not only use it in production, but I was also the one that set it up for my team. I agree that, unless you are already familiar with it, it is not always useful for protyping stage. There is something to be said about having the infrastructure in mind though. That's why I'm inclined to use something like Elixir/Phoenix for web-based projects. Some (not all) of the ideas that K8S brings to the table are already built into the Erlang/OTP platform. As for Heroku, there was a recent announcement that I think shifts things quite a bit: https://blog.heroku.com/buildpacks-go-cloud-native https://blog.heroku.com/buildpacks-go-cloud-native ... having standardized container images that runs buildpacks. The ecosystem and tooling is not quite there yet, but I can see this as significantly reducing the investment to put into Dockerizing your app for K8S. At that point, for the hobbyist, it might be: Prototype -> Heroku -> K8S with an Operator that can run the buildpack K8S is really a toolset for building your own PAAS. If there were a self-driving PAAS (using Operators) targeting small hobbyists that will run cloud native buildpacks, then the barriers of entry for a hobbyist using K8S is much lower.

drb91 8y ago

> Better to build a good product now and if you really want to turn it into a startup, productionize it then. Well, you gotta deploy with something. Why not k8s? Obviously if you're developing locally, k8s would not help in the least.