9 ms·

Flynn: first preview release

216 points by bencevans 12y ago

Artemis2 12y ago
Flynn is very interesting! I've been following its development since the project was made public and the team behind it made great progress during the last months.
- danielsiders 12y ago
  Thanks! Let us know if you have any questions!
  - deleted 12y ago
    [deleted]
zakelfassi 12y ago
About time. One of the very few projects I'm following closely and excited about Can't wait to compare it side-by-side with Dokku.
andrewmunsell 12y ago
It's great to see the project making progress. Is there any instructions for starting up a Flynn cluster on something other than Vagrant?
- Titanous 12y ago
  There are currently no tutorial-style instructions for using anything but Vagrant, but you should be able to just to run these commands in whatever environment you are using: https://github.com/flynn/flynn-demo/blob/26fcd98cd76461995133f69e5bceb574e1235470/Vagrantfile#L22-L34 https://github.com/flynn/flynn-demo/blob/26fcd98cd7646199513...
  - andrewmunsell 12y ago
    Kind of what I figured-- trying it out now. Thanks!
  - dpritchett 12y ago
    PSA to anyone trying this on Ubuntu 14: the docker command seems to be "docker.io" now.
    - bigonlogn 12y ago
      FYI, the "docker.io" repository in trusty points to an older version of docker (0.9.1). I believe the script at http://get.docker.io/ubuntu http://get.docker.io/ubuntu will install the latest version of docker.
    - shykes 12y ago
      There is an ongoing conundrum in the downstream Debian/Ubuntu packaging of Docker. Our friends at Debian and Ubuntu are doing their best to solve it. In the meantime, you should either a) install the latest docker from the official docker APT repos http://docs.docker.io/installation/ http://docs.docker.io/installation/ or b) do something like ln /usr/bin/docker{.io,} In any case, don't leave your Docker install without a docker binary in your path. That will universally break all tutorials and scripts ever written for Docker, and is unsupported by us (Docker maintainers).
      - dpritchett 12y ago
        Yeah, I went with the `ln -s` method. Thanks Solomon!
        shykes 12y ago
        another thing to watch out for: downstream packages (the one you have installed) are out-of-date. The current version of Docker is 0.10.0, and ships a ton of improvements. For that reason alone you should consider installing from upstream repos, at least until downstream catches up. Just keep it in a separate file in /etc/apt/sources.list.d to easily remove it later.
deltron 12y ago
Was the codename Walt Jr?
- Goranek 12y ago
  Not sure why is this downvoted..but it made my day.
  - pcmonk 12y ago
    It was likely downvoted since it was essentially content-free. While we can all appreciate a good joke, they lower the signal-to-noise ratio. This isn't reddit. A good comment says something constructive about the product.
toisanji 12y ago
Im looking forward to playing with this!
akhatri_aus 12y ago
I've been waiting ages for flynn its an awesome project. I used Dokku and its lovely & its so simple. Is there a way to install it manually on a server with stuff already on it? Also is the source of the aws flynn.cupcake.io tool on github, it would be nice to see whats being installed onto an aws cluster.
- ghayes 12y ago
  On that note, it would be great to know what AWS `User Policy` Flynn needs / expects so I could grant it a unique IAM key with limited access for creating servers, etc.
  - notdonspaulding 12y ago
    FWIW, the default PowerUser role that can be assigned to an IAM account was sufficient for my tests. I don't know if it could have worked with fewer permissions.
  - Titanous 12y ago
    We haven't nailed down the exact API calls that we're going to need long-term so there isn't a policy we're recommending. Currently EC2 and Route53 full access will work, but we recommend creating a new AWS account for security reasons if you have anything else running.
- danielsiders 12y ago
  Running it on your own server: We're currently only providing step by step instructions for vagrant, but you should be able to use the script in the vagrant file on your own host with some modifications on your own hardware. Happy to help in IRC (#flynn on freenode) Flynn Dashboard: At least some part of this will end up as open source in the not distant future, we're trying to figure out the best strategy for additional tools like the dashboard and what parts should be packaged.
cheez 12y ago
How do people build up the experience + the time to develop this kind of thing?
- danielsiders 12y ago
  We actually did a crowdfunding campaign[1] last summer to pay for the development time. [1] https://news.ycombinator.com/item?id=6058662 https://news.ycombinator.com/item?id=6058662
  - thu 12y ago
    Actually after Jeff Lindsay latest blog post[0] I thought that the project was put on the back burner. What is the relationship between cupcake and the initial author(s) ? [0]: http://progrium.com/blog/2014/02/06/the-start-of-the-age-of-flynn/ http://progrium.com/blog/2014/02/06/the-start-of-the-age-of-...
    - danielsiders 12y ago
      Cupcake (Apollic Software) has been the company behind Flynn (and Tent) since day 0 (we tend to downplay the branding because we think both projects can stand on their own). Most of the development on Flynn (nearly all of the 2014 work) was done by Jonathan Rudenberg (@titanous), one of the founders of Cupcake. We brought in Jeff as a contractor and paid him out of the crowdfunding campaign. He contributed to the initial architecture and prototypes but stepped back in December. I don't want to speak for Jeff (@progrium) but my understanding is that he'll keep evangelizing the project at conferences and may be working on a proprietary version/components at Digital Ocean. The Cupcake team (especially Jonathan) will keep developing Flynn (of course with open source contributions) full time in the future.
      - thu 12y ago
        Thanks a lot for your answer!
      - progrium 12y ago
sdegutis 12y ago
Images containing text that I can't select? But.. why?
dominotw 12y ago
Could someone point out to me (non superficial) differences between various service discovery projects that have popped up recently like, [1]. etcd [2] skydock [3] consul [4] zookeper ( ok this is not new) [5] flynn They all seem to be doing the exact same to me. 1. https://github.com/coreos/etcd https://github.com/coreos/etcd 2. https://github.com/crosbymichael/skydock https://github.com/crosbymichael/skydock 3. https://github.com/hashicorp/consul/ https://github.com/hashicorp/consul/ 5.https://flynn.io/ https://flynn.io/ Edit: flynn uses discoverd (https://github.com/flynn/discoverd https://github.com/flynn/discoverd)
- andrewmunsell 12y ago
  Flynn isn't a service discovery library by itself. It's more of a self hosted Heroku. Though, it does use etcd for service discovery.
  - dominotw 12y ago
    Are you sure flynn uses etcd for service discovery instead of its own[1]? Looks like it uses Google Omega instead of RAFT which etcd uses. https://github.com/flynn/discoverd https://github.com/flynn/discoverd
    - andrewmunsell 12y ago
      You can read more about service discovery on the Flynn site: https://flynn.io/docs/architecture#service-discovery https://flynn.io/docs/architecture#service-discovery discoverd is an API on top of another system. Right now, Flynn uses discoverd backed by etcd, though they indicate that this could be replaced with Zookeeper, etc. due to the mdoular design.
    - thu 12y ago
      Very first line of discoverd's README: It's currently backed by etcd[...]
      - dominotw 12y ago
StavrosK 12y ago
Are these sorts of services good enough? I want somewhere to deploy my Django apps, ideally I should be able to get a beefy server and deploy every app as a single thing on that server, completely compartmentalized. The problem I keep hitting is that these things have a tradeoff between ease of use and power. I want to use Docker, but it has no easy way to say "take this file that contains instructions and make everything". You can write Dockerfiles, but you can only use one part of the stack in them, otherwise you run into trouble. I'm hoping Flynn will allow me to use a database, Redis, uwsgi, nginx, etc with a simple deployment command. Does anyone know of anything else like this?
- danielsiders 12y ago
  Built-in database appliances are a big part of what Flynn will offer in the next few months (right now it's only Postgres).
  - davidcelis 12y ago
    One thing I've been wondering about with Flynn is data persistence. I have an app already running that I'd like to move to Flynn for deployment; will it be easy to bring over a populated database and have it persist, shared, across instances of the app?
    - Titanous 12y ago
      Currently there is a very alpha Postgres appliance[0] (including automatic cluster orchestration and a database provisioning API) included with Flynn and we'll be standardizing the appliance model and adding more in the future. Flynn datastore appliances run inside of containers and are managed by Flynn just like everything else and will include support for backups, HA/replication, etc. out of the box with close to zero configuration. [0] https://github.com/flynn/flynn-postgres https://github.com/flynn/flynn-postgres
      - davidcelis 12y ago
        Thanks. It's been pretty tough to wait for Flynn's release. Great job so far, can't wait for it to get to stability so I can migrate!
gales 12y ago
Does anyone know if Flynn components can be run on CoreOS? Also, does Fylnn offer the ability to use sub-domains for accessing apps, with either a random string or custom, eg. //dev.example.com? Thx
- danielsiders 12y ago
  Flynn is designed to run on any modern Linux kernel. We haven't tested CoreOS yet, but once we put out a few fires we'll explore more fully. You can add any route to any app in Flynn, so different subdomains, domains, and TCP ports are pretty easy.
necubi 12y ago
Can anybody discuss how Flynn compares to Mesos? Superficially, it seems to be solving the same sort of problems.
- hendzen 12y ago
  It seems to partially duplicate the functionality of Mesos, as they are writing their own task scheduling framework [0] based on Google's Omega [1]. The Omega authors claim that Mesos' system of application specific schedulers accepting resource offers from the Mesos master is well suited toward short-lived jobs (think ephemeral Map/Reduce or MPI type workloads) but is not well suited for long lived 'service' jobs (like a Rails app or DB server). As this seems to be an important use-case for Flynn, it would seem like a valid architecture decision to not use Mesos. [0] - https://github.com/flynn/flynn-host/tree/master/sampi https://github.com/flynn/flynn-host/tree/master/sampi [1] - http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/S...
  - necubi 12y ago
    I haven't read the Omega paper yet, but plenty of people are running long-running tasks in Mesos (Marathon [0] is a framework for doing just that). [0] https://github.com/mesosphere/marathon https://github.com/mesosphere/marathon
    - hendzen 12y ago
      From the Omega paper (section 4.2): Mesos achieves fairness by alternately offering all available cluster resources to different schedulers, predicated on assumptions that resources become available frequently and scheduler decisions are quick. As a result, a long scheduler decision time means that nearly all cluster resources are locked down for a long time, inaccessible to other schedulers. The only resources available for other schedulers in this situation are the few becoming available while the slow scheduler is busy. These are often insufficient to schedule an above-average size batch job, meaning that the batch scheduler cannot make progress while the service scheduler holds an offer. It nonetheless keeps trying, and as a consequence, we find that a number of jobs are abandoned because they did not finish scheduling their tasks by the 1,000-attempt retry limit in the Mesos case (Figure 7c). This pathology occurs because of Mesos’s assumption of quick scheduling decisions, small jobs and high re- source churn, which do not hold for our service jobs. Mesos could be extended to make only fair-share offers, although this would complicate the resource allocator logic, and the quality of the placement decisions for big or picky jobs would likely decrease, since each scheduler could only see a smaller fraction of the available resources. We have raised this point with the Mesos team; they agree about the limitation and are considering to address it in future work. Its worth noting that Andy Konwinski was a coauthor on both Mesos & Omega, so I'd hope they (Omega authors) represented Mesos' capabilties accurately. I don't have any personal experience running Mesos in production, I'm just going of what was written.
cellis 12y ago
Can anyone from the Flynn project comment on this versus Deis? I've tried Dokku and thought that was pretty slick, but ultimately decided to move on to a Deis/Chef devops setup, because dokku was too lightweight for my needs and Flynn wasn't ready.
- danielsiders 12y ago
  Flynn definitely isn't "ready" yet. This is a preview release, production grade stability is still a few months away. I haven't looked closely at Deis in several months, but Flynn doesn't use Chef (or anything similar) anywhere and don't plan to.
  - bacongobbler 12y ago
    Deis dev here. We've removed the Chef dependency as of v0.7.0 in favour of CoreOS and fleet for container/machine job scheduling. That ship set sail only recently in master. :)
    - fizx 12y ago
      Good riddance. Oh, and Chef is still in your website's meta tags (and therefore google results).
      - bacongobbler 12y ago
        We'll be updating the website in preparation for our next release. Most users are still on v0.7.0 or earlier. Once it's been released then our website should reflect the changes in master.
    - anentropic 12y ago
      oh good... time to have another look at Deis then :)
groby_b 12y ago
Very nit-picky comment, but: Can you explain in one or two sentences what the value proposition is? I might be dense, but it took me quite a bit of clicking around, reading HN comments, and poking at the Omega paper to understand what you want to do their. (Well, I think I understand. The "private Heroku" line points in the right direction, at least)
- danielsiders 12y ago
  Flynn is the product ops should provide to developers. Flynn is a single platform that runs all your services from databases to applications to individual Linux processes. With Flynn ops teams can stop being consultants and start providing a single product to their internal "customers". Basically you can deploy and scale whatever you want all in one place on any infrastructure without having to think about individual hosts.
  - groby_b 12y ago
    Yes. I read the site, as I said. Reposting the text from there is kind of pointless, no? I don't ultimately care one way or another - it was meant as friendly feedback that it's unclear right now. You'll notice that quite a few of the comments also said something to the effect of "I guess it does this". Which means it's not entirely clear what it does to more people than just me.
ochoseis 12y ago
How does Flynn compare to OpenShift or Cloud Foundry? Is it trying to solve the same problems, perhaps leapfrogging them in some ways? [Edit] I would also ask the same about Apache Mesos since that seems like it's coming up a lot in this discussion as well.
- milkmiruku 12y ago
  I too am interested in the comparison with such projects and the differing requirements they cater for. I'm also working on some notes on https://wiki.thingsandstuff.org/Stack https://wiki.thingsandstuff.org/Stack if anyone is interested in bettering the order/outline/etc.
Jemaclus 12y ago
I'm not a DevOps person. Can someone explain in less jargony terms what this is? Is it like Heroku? I have no idea.
- jmspring 12y ago
  Even with experience in ops, my brain hurts reading this. At first quick glance, I thought it was another HBO silicon valley spoof.
  - Jemaclus 12y ago
    It seems to me that a startup of any kind should have some sort of tagline or side bar that explains it like I'm five. If I'm not an Ops person and I come across, this, then if I can't grasp what it is or how useful it might be right off the bat, I won't recommend it to my Ops person at work, right? I understand the target audience is DevOps, but I think every startup should have a simple-yet-accurate layman's explanation on the front page. If you can't do that, maybe your service is too complicated? This is something I think most startups miss when they do landing pages, unfortunately. Even as a frequent Hacker News reader, I have no idea what's going on... and that really shouldn't happen. :( /rant. :)
    - jmspring 12y ago
      The landing page hasn't dislodged me from my dinosaur ways. Possible weekend project idea -- startup landing pages for the common man. A service to crowdsource/automate (and then index) the translation of buzz to something at least your or I understand in 5-10 seconds, a more lay person, say 30 seconds.
      - Jemaclus 12y ago
        Do it.
- erkkie 12y ago
  In short yes, it's like self-hosted heroku. They use docker and the heroku buildpack interface and many different components to coordinate container management, logging, etc to provide a whole system.
deleted 12y ago
[deleted]
sirsar 12y ago
I feel rather silly now, but... What does Flynn do? The word "ops" is so central to the pitch and so ambiguous to me that I have absolutely no idea what the function of this software (it is software, right?) is.
- benatkin 12y ago
  It's a project to build something like https://github.com/progrium/dokku https://github.com/progrium/dokku Its goals are more ambitious than Dokku but it's too early to tell if it will be any good.
  - malandrew 12y ago
    Or this: http://deis.io/ http://deis.io/
auvi 12y ago
My first impression was that it has to do something with TRON's Kevin Flynn. Quoting him: "I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? Motorcycles? Were the circuits like freeways?"
RivieraKid 12y ago
How does this relate to AWS or OpenStack? Is it somethin' like Heroku – i.e. somethin' that can run on top of AWS?

216 points by bencevans 12y ago

Flynn is very interesting! I've been following its development since the project was made public and the team behind it made great progress during the last months.

danielsiders 12y ago
Thanks! Let us know if you have any questions!
- deleted 12y ago
  [deleted]

zakelfassi 12y ago

About time. One of the very few projects I'm following closely and excited about Can't wait to compare it side-by-side with Dokku.

andrewmunsell 12y ago

It's great to see the project making progress. Is there any instructions for starting up a Flynn cluster on something other than Vagrant?

Titanous 12y ago
There are currently no tutorial-style instructions for using anything but Vagrant, but you should be able to just to run these commands in whatever environment you are using: https://github.com/flynn/flynn-demo/blob/26fcd98cd76461995133f69e5bceb574e1235470/Vagrantfile#L22-L34 https://github.com/flynn/flynn-demo/blob/26fcd98cd7646199513...
- andrewmunsell 12y ago
  Kind of what I figured-- trying it out now. Thanks!
- dpritchett 12y ago
  PSA to anyone trying this on Ubuntu 14: the docker command seems to be "docker.io" now.
  - bigonlogn 12y ago
    FYI, the "docker.io" repository in trusty points to an older version of docker (0.9.1). I believe the script at http://get.docker.io/ubuntu http://get.docker.io/ubuntu will install the latest version of docker.
  - shykes 12y ago
    There is an ongoing conundrum in the downstream Debian/Ubuntu packaging of Docker. Our friends at Debian and Ubuntu are doing their best to solve it. In the meantime, you should either a) install the latest docker from the official docker APT repos http://docs.docker.io/installation/ http://docs.docker.io/installation/ or b) do something like ln /usr/bin/docker{.io,} In any case, don't leave your Docker install without a docker binary in your path. That will universally break all tutorials and scripts ever written for Docker, and is unsupported by us (Docker maintainers).
    - dpritchett 12y ago
      Yeah, I went with the `ln -s` method. Thanks Solomon!
      - shykes 12y ago
        another thing to watch out for: downstream packages (the one you have installed) are out-of-date. The current version of Docker is 0.10.0, and ships a ton of improvements. For that reason alone you should consider installing from upstream repos, at least until downstream catches up. Just keep it in a separate file in /etc/apt/sources.list.d to easily remove it later.

deltron 12y ago

Was the codename Walt Jr?

Goranek 12y ago
Not sure why is this downvoted..but it made my day.
- pcmonk 12y ago
  It was likely downvoted since it was essentially content-free. While we can all appreciate a good joke, they lower the signal-to-noise ratio. This isn't reddit. A good comment says something constructive about the product.

toisanji 12y ago

Im looking forward to playing with this!

akhatri_aus 12y ago

I've been waiting ages for flynn its an awesome project. I used Dokku and its lovely & its so simple. Is there a way to install it manually on a server with stuff already on it? Also is the source of the aws flynn.cupcake.io tool on github, it would be nice to see whats being installed onto an aws cluster.

ghayes 12y ago
On that note, it would be great to know what AWS `User Policy` Flynn needs / expects so I could grant it a unique IAM key with limited access for creating servers, etc.
- notdonspaulding 12y ago
  FWIW, the default PowerUser role that can be assigned to an IAM account was sufficient for my tests. I don't know if it could have worked with fewer permissions.
- Titanous 12y ago
  We haven't nailed down the exact API calls that we're going to need long-term so there isn't a policy we're recommending. Currently EC2 and Route53 full access will work, but we recommend creating a new AWS account for security reasons if you have anything else running.
danielsiders 12y ago
Running it on your own server: We're currently only providing step by step instructions for vagrant, but you should be able to use the script in the vagrant file on your own host with some modifications on your own hardware. Happy to help in IRC (#flynn on freenode) Flynn Dashboard: At least some part of this will end up as open source in the not distant future, we're trying to figure out the best strategy for additional tools like the dashboard and what parts should be packaged.

cheez 12y ago

How do people build up the experience + the time to develop this kind of thing?

danielsiders 12y ago
We actually did a crowdfunding campaign[1] last summer to pay for the development time. [1] https://news.ycombinator.com/item?id=6058662 https://news.ycombinator.com/item?id=6058662
- thu 12y ago
  Actually after Jeff Lindsay latest blog post[0] I thought that the project was put on the back burner. What is the relationship between cupcake and the initial author(s) ? [0]: http://progrium.com/blog/2014/02/06/the-start-of-the-age-of-flynn/ http://progrium.com/blog/2014/02/06/the-start-of-the-age-of-...
  - danielsiders 12y ago
    Cupcake (Apollic Software) has been the company behind Flynn (and Tent) since day 0 (we tend to downplay the branding because we think both projects can stand on their own). Most of the development on Flynn (nearly all of the 2014 work) was done by Jonathan Rudenberg (@titanous), one of the founders of Cupcake. We brought in Jeff as a contractor and paid him out of the crowdfunding campaign. He contributed to the initial architecture and prototypes but stepped back in December. I don't want to speak for Jeff (@progrium) but my understanding is that he'll keep evangelizing the project at conferences and may be working on a proprietary version/components at Digital Ocean. The Cupcake team (especially Jonathan) will keep developing Flynn (of course with open source contributions) full time in the future.
    - thu 12y ago
      Thanks a lot for your answer!
    - progrium 12y ago

sdegutis 12y ago

Images containing text that I can't select? But.. why?

dominotw 12y ago

Could someone point out to me (non superficial) differences between various service discovery projects that have popped up recently like, [1]. etcd [2] skydock [3] consul [4] zookeper ( ok this is not new) [5] flynn They all seem to be doing the exact same to me. 1. https://github.com/coreos/etcd https://github.com/coreos/etcd 2. https://github.com/crosbymichael/skydock https://github.com/crosbymichael/skydock 3. https://github.com/hashicorp/consul/ https://github.com/hashicorp/consul/ 5.https://flynn.io/ https://flynn.io/ Edit: flynn uses discoverd (https://github.com/flynn/discoverd https://github.com/flynn/discoverd)

andrewmunsell 12y ago
Flynn isn't a service discovery library by itself. It's more of a self hosted Heroku. Though, it does use etcd for service discovery.
- dominotw 12y ago
  Are you sure flynn uses etcd for service discovery instead of its own[1]? Looks like it uses Google Omega instead of RAFT which etcd uses. https://github.com/flynn/discoverd https://github.com/flynn/discoverd
  - andrewmunsell 12y ago
    You can read more about service discovery on the Flynn site: https://flynn.io/docs/architecture#service-discovery https://flynn.io/docs/architecture#service-discovery discoverd is an API on top of another system. Right now, Flynn uses discoverd backed by etcd, though they indicate that this could be replaced with Zookeeper, etc. due to the mdoular design.
  - thu 12y ago
    Very first line of discoverd's README: It's currently backed by etcd[...]
    - dominotw 12y ago

StavrosK 12y ago

Are these sorts of services good enough? I want somewhere to deploy my Django apps, ideally I should be able to get a beefy server and deploy every app as a single thing on that server, completely compartmentalized. The problem I keep hitting is that these things have a tradeoff between ease of use and power. I want to use Docker, but it has no easy way to say "take this file that contains instructions and make everything". You can write Dockerfiles, but you can only use one part of the stack in them, otherwise you run into trouble. I'm hoping Flynn will allow me to use a database, Redis, uwsgi, nginx, etc with a simple deployment command. Does anyone know of anything else like this?

danielsiders 12y ago
Built-in database appliances are a big part of what Flynn will offer in the next few months (right now it's only Postgres).
- davidcelis 12y ago
  One thing I've been wondering about with Flynn is data persistence. I have an app already running that I'd like to move to Flynn for deployment; will it be easy to bring over a populated database and have it persist, shared, across instances of the app?
  - Titanous 12y ago
    Currently there is a very alpha Postgres appliance[0] (including automatic cluster orchestration and a database provisioning API) included with Flynn and we'll be standardizing the appliance model and adding more in the future. Flynn datastore appliances run inside of containers and are managed by Flynn just like everything else and will include support for backups, HA/replication, etc. out of the box with close to zero configuration. [0] https://github.com/flynn/flynn-postgres https://github.com/flynn/flynn-postgres
    - davidcelis 12y ago
      Thanks. It's been pretty tough to wait for Flynn's release. Great job so far, can't wait for it to get to stability so I can migrate!

gales 12y ago

Does anyone know if Flynn components can be run on CoreOS? Also, does Fylnn offer the ability to use sub-domains for accessing apps, with either a random string or custom, eg. //dev.example.com? Thx

danielsiders 12y ago
Flynn is designed to run on any modern Linux kernel. We haven't tested CoreOS yet, but once we put out a few fires we'll explore more fully. You can add any route to any app in Flynn, so different subdomains, domains, and TCP ports are pretty easy.

necubi 12y ago

Can anybody discuss how Flynn compares to Mesos? Superficially, it seems to be solving the same sort of problems.

hendzen 12y ago
It seems to partially duplicate the functionality of Mesos, as they are writing their own task scheduling framework [0] based on Google's Omega [1]. The Omega authors claim that Mesos' system of application specific schedulers accepting resource offers from the Mesos master is well suited toward short-lived jobs (think ephemeral Map/Reduce or MPI type workloads) but is not well suited for long lived 'service' jobs (like a Rails app or DB server). As this seems to be an important use-case for Flynn, it would seem like a valid architecture decision to not use Mesos. [0] - https://github.com/flynn/flynn-host/tree/master/sampi https://github.com/flynn/flynn-host/tree/master/sampi [1] - http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/S...
- necubi 12y ago
  I haven't read the Omega paper yet, but plenty of people are running long-running tasks in Mesos (Marathon [0] is a framework for doing just that). [0] https://github.com/mesosphere/marathon https://github.com/mesosphere/marathon
  - hendzen 12y ago
    From the Omega paper (section 4.2): Mesos achieves fairness by alternately offering all available cluster resources to different schedulers, predicated on assumptions that resources become available frequently and scheduler decisions are quick. As a result, a long scheduler decision time means that nearly all cluster resources are locked down for a long time, inaccessible to other schedulers. The only resources available for other schedulers in this situation are the few becoming available while the slow scheduler is busy. These are often insufficient to schedule an above-average size batch job, meaning that the batch scheduler cannot make progress while the service scheduler holds an offer. It nonetheless keeps trying, and as a consequence, we find that a number of jobs are abandoned because they did not finish scheduling their tasks by the 1,000-attempt retry limit in the Mesos case (Figure 7c). This pathology occurs because of Mesos’s assumption of quick scheduling decisions, small jobs and high re- source churn, which do not hold for our service jobs. Mesos could be extended to make only fair-share offers, although this would complicate the resource allocator logic, and the quality of the placement decisions for big or picky jobs would likely decrease, since each scheduler could only see a smaller fraction of the available resources. We have raised this point with the Mesos team; they agree about the limitation and are considering to address it in future work. Its worth noting that Andy Konwinski was a coauthor on both Mesos & Omega, so I'd hope they (Omega authors) represented Mesos' capabilties accurately. I don't have any personal experience running Mesos in production, I'm just going of what was written.

cellis 12y ago

Can anyone from the Flynn project comment on this versus Deis? I've tried Dokku and thought that was pretty slick, but ultimately decided to move on to a Deis/Chef devops setup, because dokku was too lightweight for my needs and Flynn wasn't ready.

danielsiders 12y ago
Flynn definitely isn't "ready" yet. This is a preview release, production grade stability is still a few months away. I haven't looked closely at Deis in several months, but Flynn doesn't use Chef (or anything similar) anywhere and don't plan to.
- bacongobbler 12y ago
  Deis dev here. We've removed the Chef dependency as of v0.7.0 in favour of CoreOS and fleet for container/machine job scheduling. That ship set sail only recently in master. :)
  - fizx 12y ago
    Good riddance. Oh, and Chef is still in your website's meta tags (and therefore google results).
    - bacongobbler 12y ago
      We'll be updating the website in preparation for our next release. Most users are still on v0.7.0 or earlier. Once it's been released then our website should reflect the changes in master.
  - anentropic 12y ago
    oh good... time to have another look at Deis then :)

groby_b 12y ago

Very nit-picky comment, but: Can you explain in one or two sentences what the value proposition is? I might be dense, but it took me quite a bit of clicking around, reading HN comments, and poking at the Omega paper to understand what you want to do their. (Well, I think I understand. The "private Heroku" line points in the right direction, at least)

danielsiders 12y ago
Flynn is the product ops should provide to developers. Flynn is a single platform that runs all your services from databases to applications to individual Linux processes. With Flynn ops teams can stop being consultants and start providing a single product to their internal "customers". Basically you can deploy and scale whatever you want all in one place on any infrastructure without having to think about individual hosts.
- groby_b 12y ago
  Yes. I read the site, as I said. Reposting the text from there is kind of pointless, no? I don't ultimately care one way or another - it was meant as friendly feedback that it's unclear right now. You'll notice that quite a few of the comments also said something to the effect of "I guess it does this". Which means it's not entirely clear what it does to more people than just me.

ochoseis 12y ago

How does Flynn compare to OpenShift or Cloud Foundry? Is it trying to solve the same problems, perhaps leapfrogging them in some ways? [Edit] I would also ask the same about Apache Mesos since that seems like it's coming up a lot in this discussion as well.

milkmiruku 12y ago
I too am interested in the comparison with such projects and the differing requirements they cater for. I'm also working on some notes on https://wiki.thingsandstuff.org/Stack https://wiki.thingsandstuff.org/Stack if anyone is interested in bettering the order/outline/etc.

Jemaclus 12y ago

I'm not a DevOps person. Can someone explain in less jargony terms what this is? Is it like Heroku? I have no idea.

jmspring 12y ago
Even with experience in ops, my brain hurts reading this. At first quick glance, I thought it was another HBO silicon valley spoof.
- Jemaclus 12y ago
  It seems to me that a startup of any kind should have some sort of tagline or side bar that explains it like I'm five. If I'm not an Ops person and I come across, this, then if I can't grasp what it is or how useful it might be right off the bat, I won't recommend it to my Ops person at work, right? I understand the target audience is DevOps, but I think every startup should have a simple-yet-accurate layman's explanation on the front page. If you can't do that, maybe your service is too complicated? This is something I think most startups miss when they do landing pages, unfortunately. Even as a frequent Hacker News reader, I have no idea what's going on... and that really shouldn't happen. :( /rant. :)
  - jmspring 12y ago
    The landing page hasn't dislodged me from my dinosaur ways. Possible weekend project idea -- startup landing pages for the common man. A service to crowdsource/automate (and then index) the translation of buzz to something at least your or I understand in 5-10 seconds, a more lay person, say 30 seconds.
    - Jemaclus 12y ago
      Do it.
erkkie 12y ago
In short yes, it's like self-hosted heroku. They use docker and the heroku buildpack interface and many different components to coordinate container management, logging, etc to provide a whole system.

deleted 12y ago

[deleted]

sirsar 12y ago

I feel rather silly now, but... What does Flynn do? The word "ops" is so central to the pitch and so ambiguous to me that I have absolutely no idea what the function of this software (it is software, right?) is.

benatkin 12y ago
It's a project to build something like https://github.com/progrium/dokku https://github.com/progrium/dokku Its goals are more ambitious than Dokku but it's too early to tell if it will be any good.
- malandrew 12y ago
  Or this: http://deis.io/ http://deis.io/

auvi 12y ago

My first impression was that it has to do something with TRON's Kevin Flynn. Quoting him: "I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? Motorcycles? Were the circuits like freeways?"

RivieraKid 12y ago

How does this relate to AWS or OpenStack? Is it somethin' like Heroku – i.e. somethin' that can run on top of AWS?