3 ms·

What I really want to know is how fly.io has recruited such talent. @tptacek (who wrote this article), @chrismccord (creator of elixir Phoenix/Liveview), and ot

What I really want to know is how fly.io has recruited such talent. @tptacek (who wrote this article), @chrismccord (creator of elixir Phoenix/Liveview), and others. Very impressive, but also slightly worrying since some parts of fly.io have such terrible DX/documentation. I've only recently started using them, but the documentation around billing and databases is shockingly bad for a team that is clearly so deeply stacked.

chubot 3y ago
Part of it is probably that smart people want to work on harder and more ambitious problems, which YC advisors seem to say a lot. And also smart people want to work together As I understand it, fly.io built a cloud from scratch, from the hardware up. That's HARD. It's typically something only well-funded companies like Amazon / Google / Microsoft do, with billions of dollars in products to justify it. Cloudflare is another company that pretty much did it, and it seems to have taken similarly ambitious engineering (although maybe theirs isn't fully general purpose) https://fly.io/docs/about/security/ https://fly.io/docs/about/security/ - We run on our own hardware deployed in secure data centers like Equinix Most cloud companies in the 2010's built on top of other clouds like AWS. For example, Heroku and I'm pretty sure dotCloud, the company that became Docker. I always thought that was a weird design, but it definitely saves you a lot of time and hassle. At the cost of a lot of software complexity and platform risk. Even so, there were at least 20 or 30 of these companies, and most of them didn't make it. Google App Engine (one of the first PaaS like Heroku, circa ~2007) also built on top of Google's data centers and Borg of course. They didn't manage their own hardware -- they used another cloud for that! From the outside, "hosting" may all look kind of similar, but there are worlds of difference under the hood. --- I'll also say that the subject of this blog post -- cloud security -- is extremely hard. FWIW I joined the team that published the Macaroons paper ~10 years ago, and I tried to implement Macaroons as a side project then (for about a week, it wasn't very serious). I was looking for something simpler and more elegant, but there's no real magic bullet, and I think this post kinda concedes that toward the end. But when you have really hard problems, it's not surprising if the solution is really complex! The problem can actually be infinitely hard, because you have adversaries that are both more powerful, and that adapt. Google learned this the hard way ~10 years ago when it learned that BOTH the United States government and the Chinese government had successfully attacked it :-/ People seem to forget about these incidents: https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/MUSCULAR https://en.wikipedia.org/wiki/MUSCULAR https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html https://www.washingtonpost.com/world/national-security/nsa-i... So basically, if you are successful, then it earns you more problems :)
- hinkley 3y ago
  I think they've stopped talking about it but there used to be commentary from Google's "Better to turn down a good person than hire a bad one" logic (Or was it Facebook?) about how a C-employee can't evaluate an A-employee, and so they hire B-employees and it becomes a race to mediocrity. My last hiring cycle I just missed working at a place with a couple guys who I dove pretty deep with on some esoteric topics over beers at a handful of tech meetups. The job req got weird and I had already accepted a position by the time it got sorted out. But lots of people have stories like that. It's one of the reasons people go to those meetups. I think for a lot of us it's being able to talk to someone who can keep up with your stream of technical talk, but the business contacts are a lovely bonus.
dartos 3y ago
Out of curiosity, what database documentation was lacking? IIRC they don’t provide a managed database, just like a template
- elbasti 3y ago
  Right. They provide a convenience function that spins up a machine running postgres and adds the right environment variables to any machines you already have running. It's "just another machine," but that's not obvious when you're setting it up! Two bits of documentation that are lacking here are: 1. How does this affect billing, especially if you're on the hobby plan? (Answer, as far as I can tell: since it's just another machine, it can be free if and only if you resource it such that it fits in your free "allowances." The defaults are NOT these, however, so you have to be careful if you want to mess around for free). 2. Once the db machine is created...how can I get credentials for it? A connection string is generated upon machine creation but...what if you don't write it down?
  - verelo 3y ago
    Yeah i feel your pain here. Fly has a lot of things that feel like they're hidden features, or at least, undocumented. Like, its trivial to get all the env vars from a machine, but i couldn't find this in the documentation anywhere. Someone else i was working with told me how and now..i know? fly ssh console --command env To be fair, it is mentioned in their docs, i just couldnt find it searching things like "how to find my environment variables".
    - arccy 3y ago
      part of it is common sense: fly ssh console == ssh, and then you run env to print the environment variables in your current session (which may not be the exact set of envs your app sees).
    - tptacek 3y ago
      There's a whole backstory to this, because we got a big influx of Heroku developers a couple years ago, and Heroku developers all expect to be able to dump all their environment variables, including secrets, from the command line or whatever. We don't do that: our API is like a diode for secrets, and once they're set, they're sealed into a cryptographic vault on separate hardware. It is and has to be the case that you can shell into a machine and get the secrets, which apps all expect to receive in environment variables, but we don't like that you can do that, and wouldn't want to encourage it. Environment variables are hazmat!
deleted 3y ago
[deleted]

chubot 3y ago

Part of it is probably that smart people want to work on harder and more ambitious problems, which YC advisors seem to say a lot. And also smart people want to work together As I understand it, fly.io built a cloud from scratch, from the hardware up. That's HARD. It's typically something only well-funded companies like Amazon / Google / Microsoft do, with billions of dollars in products to justify it. Cloudflare is another company that pretty much did it, and it seems to have taken similarly ambitious engineering (although maybe theirs isn't fully general purpose) https://fly.io/docs/about/security/ https://fly.io/docs/about/security/ - We run on our own hardware deployed in secure data centers like Equinix Most cloud companies in the 2010's built on top of other clouds like AWS. For example, Heroku and I'm pretty sure dotCloud, the company that became Docker. I always thought that was a weird design, but it definitely saves you a lot of time and hassle. At the cost of a lot of software complexity and platform risk. Even so, there were at least 20 or 30 of these companies, and most of them didn't make it. Google App Engine (one of the first PaaS like Heroku, circa ~2007) also built on top of Google's data centers and Borg of course. They didn't manage their own hardware -- they used another cloud for that! From the outside, "hosting" may all look kind of similar, but there are worlds of difference under the hood. --- I'll also say that the subject of this blog post -- cloud security -- is extremely hard. FWIW I joined the team that published the Macaroons paper ~10 years ago, and I tried to implement Macaroons as a side project then (for about a week, it wasn't very serious). I was looking for something simpler and more elegant, but there's no real magic bullet, and I think this post kinda concedes that toward the end. But when you have really hard problems, it's not surprising if the solution is really complex! The problem can actually be infinitely hard, because you have adversaries that are both more powerful, and that adapt. Google learned this the hard way ~10 years ago when it learned that BOTH the United States government and the Chinese government had successfully attacked it :-/ People seem to forget about these incidents: https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/MUSCULAR https://en.wikipedia.org/wiki/MUSCULAR https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html https://www.washingtonpost.com/world/national-security/nsa-i... So basically, if you are successful, then it earns you more problems :)

hinkley 3y ago
I think they've stopped talking about it but there used to be commentary from Google's "Better to turn down a good person than hire a bad one" logic (Or was it Facebook?) about how a C-employee can't evaluate an A-employee, and so they hire B-employees and it becomes a race to mediocrity. My last hiring cycle I just missed working at a place with a couple guys who I dove pretty deep with on some esoteric topics over beers at a handful of tech meetups. The job req got weird and I had already accepted a position by the time it got sorted out. But lots of people have stories like that. It's one of the reasons people go to those meetups. I think for a lot of us it's being able to talk to someone who can keep up with your stream of technical talk, but the business contacts are a lovely bonus.

dartos 3y ago

Out of curiosity, what database documentation was lacking? IIRC they don’t provide a managed database, just like a template

elbasti 3y ago
Right. They provide a convenience function that spins up a machine running postgres and adds the right environment variables to any machines you already have running. It's "just another machine," but that's not obvious when you're setting it up! Two bits of documentation that are lacking here are: 1. How does this affect billing, especially if you're on the hobby plan? (Answer, as far as I can tell: since it's just another machine, it can be free if and only if you resource it such that it fits in your free "allowances." The defaults are NOT these, however, so you have to be careful if you want to mess around for free). 2. Once the db machine is created...how can I get credentials for it? A connection string is generated upon machine creation but...what if you don't write it down?
- verelo 3y ago
  Yeah i feel your pain here. Fly has a lot of things that feel like they're hidden features, or at least, undocumented. Like, its trivial to get all the env vars from a machine, but i couldn't find this in the documentation anywhere. Someone else i was working with told me how and now..i know? fly ssh console --command env To be fair, it is mentioned in their docs, i just couldnt find it searching things like "how to find my environment variables".
  - arccy 3y ago
    part of it is common sense: fly ssh console == ssh, and then you run env to print the environment variables in your current session (which may not be the exact set of envs your app sees).
  - tptacek 3y ago
    There's a whole backstory to this, because we got a big influx of Heroku developers a couple years ago, and Heroku developers all expect to be able to dump all their environment variables, including secrets, from the command line or whatever. We don't do that: our API is like a diode for secrets, and once they're set, they're sealed into a cryptographic vault on separate hardware. It is and has to be the case that you can shell into a machine and get the secrets, which apps all expect to receive in environment variables, but we don't like that you can do that, and wouldn't want to encourage it. Environment variables are hazmat!

deleted 3y ago

[deleted]