11 ms·

Serving 250k developers with one support engineer

215 points by dban 4y ago

prh8 4y ago
I think it's interesting that their whole shtick is "all to serve the developer" but they just kept overloading their own engineers, moving from new solution to new solution, before they just decided to hire more people.
americoo 4y ago
[flagged]
trynewideas 4y ago
> As a team of one, we were able to support 250k users, and now that we've crossed that bridge, our goal is to support two million users with our 2-person team for a 1M-to-1 ratio. Is it weird then that they have an open req for a support engineer: https://railway.app/careers/support-engineer https://railway.app/careers/support-engineer
- ndneighbor 4y ago
  heyo Angelo (author) here- This is actually the original blogpost that we used to make our first hire! > You will be working with our existing Support Engineer, Angelo - to build out processes, formalize policies, and build out integrations between systems to make it so that we can track and record issues. I accidentally left that one open... but the Ashby interface is so confusing I don't know where to toggle the visibility. I will remove that and then wait for our jobs page to rebuild.
  - lotsofpulp 4y ago
    FYI, not disclosing at least minimum pay means applicants should assume pay will be below average. Also, it says “remote - anywhere” at the top, and if you are hiring remote in California, Colorado, New York City, or Washington state, employers are legally required to advertise pay range on job listings and possibly other compensation details. > At Railway, we provide best in class benefits. Great salary, full health benefits including dependents, strong equity grants, equipment stipend, and much more. For more details, check back on the main careers page.
    - peter_l_downs 4y ago
      > FYI, not disclosing at least minimum pay means applicants should assume pay will be below average. I don't think this is true.
  - deleted 4y ago
    [deleted]
  - abhikp 4y ago
    Hey! Ashby co-founder here, you can either close the job (if you’re no longer hiring) or unpublished the job posting, more here (https://www.notion.so/How-do-I-set-up-jobs-089229a8631f4b9fb1d4e5b75dad9988#09c324bc137e4356a8378b57d4b8611f https://www.notion.so/How-do-I-set-up-jobs-089229a8631f4b9fb...). Feel free to reach out to support@ashbyhq.com and we can chat there (if you haven’t already)
swyx 4y ago
extremely impressive, love the chart showing the major developments. congrats Railway! (disclosure: am small angel)
swatcoder 4y ago
Congratulations on achieving this milestone and probably keeping your customers very satisfied along the way, but I eagerly look forward to the future where everyone is so sick of virtual assistants, bots, and knowledge bases that a low ratio is what gets celebrated as a growth milestone rather than a high ratio. Automated triage is great for digital companies that want to scale cheaply, but not so great for the people who need to figure out how to usefully navigate them, with the brick wall support experience of Google/Facebook/etc being the already vivid example. It helped those companies grow and court techno-optimist investors, but nobody outside of their enterprise customers is happy about the customer service experience. Eventually, that becomes dirt on their reputation and works against their continued growth and success.
- ndneighbor 4y ago
  Honestly, you hit the nail on the head on what drives me. Before this I was a PM, and way even before that- I was an Infra Engineer. I think that brick-wall of contact when working on some Enterprise tool was what got me. ESPECIALLY so when there was a bug blocking my company and it felt like no one cared. When I first joined- this is why I would take great pains to respond to everyone extremely quickly. When I realized that was not sustainable, the focus then shifted to really making sure that our users are heard and their concerns are acted upon. Sure first contact SLA of 30 mins is great but- doesn't mean shit if we don't solve your problem. So I think yes, tooling should be first. I think there is just a way for that tooling to serve the developer, not some OKR.
  - kazanz 4y ago
    Agreed. Where I work now (Hivelocity) is a breath of fresh air. There are so many support techs, that every ticket is answered within minutes. It's definitely not cheap to do, but the customers are always happy when they can get almost instant support for their dedicated servers.
- daydream 4y ago
  When I have a problem with a service my ultimate goal is to get it solved - as fast as possible. If I can solve my problem though an automated support channel that’s actually better! Because I don’t need to wait for a human to do something. Computers don’t sleep, take lunch breaks, etc. The problem with google/Facebook/etc isn’t that they don’t have human support. It’s that when your account gets locked out there’s no way of getting your problem solved. If a working useful automated support system existed few would complain.
nbrempel 4y ago
Very impressive! That growth curve is amazing.
fishtoaster 4y ago
Seems like good tips for any support org trying to keep a handle on load: 1. Get a support tool (or in this case, roll your own, then buy one when you outgrow that) 2. Build/buy tooling to direct users to existing documentation that might answer their questions 3. Move any routine tasks that require a human to automated self-serve tools. 4. Rope your community into answering eachother's questions.
mkl95 4y ago
Any SLAs worth mentioning? Did you ever struggle with them?
- ndneighbor 4y ago
  One thing that didn't make it into the article (I ramble and rant) is how I set it so at one point that Teams were entitled to a reply from me or the founder 24/7. This broke nearly instantly after 50 Teams. At the moment we don't but I spoke to the fmr. Head of Support at Stripe and he told me we should think about our product coverage like an API, some parts of the product (builds for example) should be more critical on response times than others. My main goal this year is to publish an SLA table so that we can work with ever larger customers setting the right expectation.
fuddle 4y ago
I was impressed by their pricing, how do they keep costs so low?
- paxys 4y ago
  I have spent the last 10 minutes on their site and am unable to figure out what the pricing even is. Going by the top of their pricing page (https://railway.app/pricing https://railway.app/pricing): $10/GB per month RAM & $20/vCPU per month is actually pretty expensive when compared to services like fly.io and even EC2/Azure. Apart from that, what vCPUs are these exactly? What are the bandwidth costs? Custom domains? Certificates? Load balancing? Disks? Databases? Clicking "Choose Plan" takes you to a login page, but then if you log in the pricing page disappears altogether.
revskill 4y ago
With bad documentation on buildpack, it's hard for me to go on with Railway. (For example, there's no sample or docs at all for how to build image with multiple buildpacks like in heroku). And these days, good documentation is 1st criteria to trust a platform / framework / library.
- justjake 4y ago
  Railway team member here: So there is one but, if you can't find it, it's as good as there not being one! It's currently located at https://nixpacks.com/docs/guides/configuring-builds#change-what-providers-are-run https://nixpacks.com/docs/guides/configuring-builds#change-w... Is there some place you'd expect it to be instead? All ears on feedback!
nitsuaeekcm 4y ago
What an impressive company. A good reminder that, even in a market with umpteen competitors, success comes down to execution. Hope they keep up the good work.
- ryanSrich 4y ago
  Railway does look impressive. However, as someone in the market for a Heroku alternative - I’d say there really aren’t any Heroku competitors. At least nothing directly comparable. The thing that keeps me (and I’m sure other small business owners) tied to Heroku is just how much they take care of for you. I literally never ever want to think about my infrastructure beyond initial setup. Heroku, to my knowledge, is the only provider that allows me to do this while servicing more than 100,000 users.
  - benmartin 4y ago
    Oh there are numerous alternatives to Heroku. Just look for "PaaS" and your favourite software components (programming language, DB, CDN etc) and you'll find a list. I find Platform.sh is an excellent alternative to Heroku, dare I say better. Same thing - configure once and operate forever, scale as you need... Really Heroku used to be a big name but it is now declining, and lets face it - many used it just because it had a tiny plan for free.
  - hankmh 4y ago
    Like others have said there are a number of Heroku alternatives out there that are more or less viable to meet your requirements, which I've gathered to be: - Serving >100k users - Helping you to achieve SOC 2 In my capacity as one of the leaders at a Heroku alternative PaaS, I've studied the PaaS market a bit to understand the space and available alternatives to Heroku. Here's what I've found about some of the most popular: - Fly is managing its own infrastructure allowing it to be extremely competitive on cost. But on the flip side, its heavy focus on infrastructure is missing the “managed” options that make PaaS so valuable, such as a true Managed Database offering. This IMO makes it less of a viable alternative to Heroku. - Render is offering a more truly “managed” alternative, and is innovating on cost as well. But it’s early and is still missing some table stakes reliability features that you'd probably expect from a Heroku alternative. - Railway has a blockbuster FTUX, I love deploying and using PostgreSQL databases in the UI without even signing up. Coincidentally probably how it got to its count of 250k developers. But it’s own docs caution that it’s not exactly production ready, especially its databases. - Platform.sh has grown really well by focusing on enterprise marketing teams and their use cases. I think this is a great niche for them and has paid off well. In their capacity of working with enterprises I'm sure they could handle SOC 2, etc. - Aptible has run critical web apps and APIs dealing with sensitive data for hundreds of companies and has helped a few go public or get to billion dollar acquisitions. I am certainly biased, but Aptible seems to be the only non-Heroku PaaS focused on product/engineering teams that has repeatedly handled true "production" requirements, like your larger user base (many of our customers fit this description) and SOC 2 (most of our customers use our security & compliance dashboard for this). But that comes at a cost: Aptible is typically more expensive than the others, perhaps save for Platform.sh. Disclaimer: I'm one of the leaders at Aptible, which in my (admittedly biased) view is the best positioned alternative to Heroku for product/engineering teams that have any sort of scale or production use case.
theonething 4y ago
Can anyone comment on the security implications of using a service like this, render.com, fly.io, etc vs. rolling your own infra on AWS EC2 and friends?
- mmcclure 4y ago
  Is there something specific you've got in mind? There are a ton of "it depends" type questions in here, but generally speaking I'd argue that rolling your own infra has some parallels with rolling your own crypto. There are countless mistakes you can make when doing it yourself, and a reputable entity that's been doing it for a bit has probably already made those mistakes or, at least, considered them. Quadruply so if they've got certifications of any kind. At scale, or with a seasoned ops team/team member, the scenario is a bit different, but if you're asking "is it more secure for a few full-stack developers hacking at a startup to roll their own infra or use something like fly.io" I feel preeetty comfortable saying the latter is going to be more secure.
- fancyremarker 4y ago
  It's possible for a PaaS to improve your security posture by implementing many of the security controls you'd otherwise be responsible for yourself. Every PaaS provider has a Shared Responsibility Model, like this one from AWS [0], and a good PaaS can eat up much of what would otherwise be your responsibility as an AWS customer: network architecture, secure configuration, IAM, system access (and auditing), intrusion detection, etc. On the other hand, many PaaS providers obfuscate their security implementation, and ultimately your data could be compromised by their mistakes. Things you should look for when evaluating PaaS providers: - How are resources, networks, etc. separated/isolated per customer? - What are YOUR security responsibilities on the platform? - How transparent is the provider about their security controls? Do they have security whitepapers, SOC 2 reports, etc. that are transparent and legit? Better yet, can they prove to you in the product how security controls are being implemented? Disclaimer: I'm the CEO and founder of Aptible [1], a PaaS specifically built to meet and prove security requirements for companies in regulated/high-compliance environments. [0] https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode... [1] https://www.aptible.com/ https://www.aptible.com/
urbandw311er 4y ago
Not sure this is something I’d be bragging about. What happens when you do eventually have a major outage and need to rapidly provide reassurance and tailored support to hundreds of people. Like when Atlassian deleted thousands of JIRA tickets by mistake and seemed completely unprepared to scale up their support.
- trevor-e 4y ago
  Is that something you can reasonably prepare for as a company? Like just keep a bunch of extra support people on payroll in case there's a catastrophic event?
  - speakfreely 4y ago
    It's about not optimizing for a single variable. If you're trying to reduce headcount (and payroll cost) as a single goal, your business becomes a brittle shell subject to disruption by any number of circumstances. Investing in some redundancy is responsible to your shareholders and respectful to your customers.
  - urbandw311er 4y ago
    What a great question. Perhaps the answer is “not entirely” but draw up some best effort contingency plans (pool of contractors, insurance cover, etc). I still wouldn’t be highlighting how few support staff I have per customer like this in either case!
    - trevor-e 4y ago
      Ha I definitely agree with you there. While impressive from the company's standpoint, I'm not sure it's something I'd highlight either. If anything, learning this would steer me away as a prospective client.
ipaddr 4y ago
I checked out the open positions. There were more positions open than total employees. Are those real positions you are looking to fill or always open positions that rarely result in an actual job? I asked this because the interview process has this step where you ask the candidates to do a project and than go over it with someone. One of these little projects is design our api or build a github issue viewer frontend/backend. You still have 4 more rounds after someone invested all that time. The amount of effort to work here seems high and the chances of the position existing and you filling it is so low. I feel bad for the many looking for work.. do you invest 10+ hours in a process where a job may not exist in the first place and the company brags about the goal of one person servicing a million
- glenngillen 4y ago
  I can’t speak for the Railway team, but I am hiring at an early stage startup myself. There is absolutely no way we’d extend offers for all of the roles we have open if a candidate arrived for all of them next week. Growing that much so quickly would be a disastrous onboarding experience for most of the new hires and potentially risk our ability to build a consistent culture with the team. So we’ll hire whichever candidate(s) successfully complete the process first and then pause the other roles until we’re ready to onboard more people. Then there’s also the reality at this stage you need people who can wear multiple hats. And there’s a bunch of roles where your ideal candidate doesn’t have a neat pre-established label. So sometimes we’ll post the exact same role with different titles to try and make sure it gets the attention of someone who most strongly aligns with one of those job titles. At a previous place we worked remotely, and so the same job would be posted as both remote and then also as a dozen different specific city locations too. But if we were hiring multiple of any roles you wouldn’t multiply that, you’d just post the one listing(s) and keep it open until all the positions were filled. A jobs page is a marketing artefact for potential hires, it’s not a financial reporting/forecasting tool. TLDR: don’t make any assumptions from a job board about how many roles a company is realistically hiring for.
  - moomoo11 4y ago
    How did you find your core/initial team? Thanks
iLoveOncall 4y ago
Those numbers are absolutely meaningless. I work at Amazon and most teams do not have a single support engineer despite serving hundreds of millions of customers. The number of employees in the position X is absolutely meaningless.
- jorts 4y ago
  It depends on the type of product. If you're selling to consumers versus businesses, engineers versus HR, how mission critical your product is, etc... Given they're selling a mission critical piece of infrastructure it's a high ratio.
e12e 4y ago
Nice article - but given the title I wonder what "Community Champions" are? Sounds like free Labour?
- trungdq88 4y ago
  Having the same question. I guess the “Community Chanpions” are rewarded somehow.
  - supriyo-biswas 4y ago
    It's typically in the form of access to features that are reserved for higher paying "enterprise" customers. Cloudflare's MVP program does this for example.
fijiaarone 4y ago
Pretty sure AWS hasn’t given support to anywhere near 250k developers.
mawadev 4y ago
Unbelievable how a single developer... or a handful runs an entire organization. I hope the developer gets paid handsomely and not just in exposure through this blog :)
klabb3 4y ago
How did you measure that 250k? That’s the population of like Iceland. Sounds like a lot, but perhaps my “how many golfballs fit in a schoolbus” guesstimate game is off.
canto 4y ago
What do you guys use to run databases? Zalando operator for postgres perhaps? And for mysql and redis?
nevon 4y ago
Assuming that the support experience for the customers is actually good, I'm amazed they've managed to scale their support efforts that far. I work on a similar type of product, except in-house in an organization with around 2000-2500 engineers, and handling support has by far been the hardest thing to scale as we've grown. Even with documentation available, internal Stack Overflow with plenty of common questions answered, unified search, Slack integrations, error messages linking directly to documentation, pre-defined observability dashboards and monitors, we still get so many questions that it's a full time job staying on top of it. Can't imagine having 100x that amount of users.
chrismorgan 4y ago
> I am convinced that Software Engineering is more of an art than an engineering practice. Back in 2010, Monash University’s Bachelor of Software Engineering had a unit ENG1061 Engineering Profession mandatory for the degree (for a terrible reason: Engineers Australia accreditation of the degree, which no one that I ever talked to cared about in the slightest). The first lecture listed The 17 Branches of Engineering. Guess what wasn’t one of them. It went downhill from there.

215 points by dban 4y ago

prh8 4y ago

I think it's interesting that their whole shtick is "all to serve the developer" but they just kept overloading their own engineers, moving from new solution to new solution, before they just decided to hire more people.

americoo 4y ago

[flagged]

trynewideas 4y ago

> As a team of one, we were able to support 250k users, and now that we've crossed that bridge, our goal is to support two million users with our 2-person team for a 1M-to-1 ratio. Is it weird then that they have an open req for a support engineer: https://railway.app/careers/support-engineer https://railway.app/careers/support-engineer

ndneighbor 4y ago
heyo Angelo (author) here- This is actually the original blogpost that we used to make our first hire! > You will be working with our existing Support Engineer, Angelo - to build out processes, formalize policies, and build out integrations between systems to make it so that we can track and record issues. I accidentally left that one open... but the Ashby interface is so confusing I don't know where to toggle the visibility. I will remove that and then wait for our jobs page to rebuild.
- lotsofpulp 4y ago
  FYI, not disclosing at least minimum pay means applicants should assume pay will be below average. Also, it says “remote - anywhere” at the top, and if you are hiring remote in California, Colorado, New York City, or Washington state, employers are legally required to advertise pay range on job listings and possibly other compensation details. > At Railway, we provide best in class benefits. Great salary, full health benefits including dependents, strong equity grants, equipment stipend, and much more. For more details, check back on the main careers page.
  - peter_l_downs 4y ago
    > FYI, not disclosing at least minimum pay means applicants should assume pay will be below average. I don't think this is true.
- deleted 4y ago
  [deleted]
- abhikp 4y ago
  Hey! Ashby co-founder here, you can either close the job (if you’re no longer hiring) or unpublished the job posting, more here (https://www.notion.so/How-do-I-set-up-jobs-089229a8631f4b9fb1d4e5b75dad9988#09c324bc137e4356a8378b57d4b8611f https://www.notion.so/How-do-I-set-up-jobs-089229a8631f4b9fb...). Feel free to reach out to support@ashbyhq.com and we can chat there (if you haven’t already)

swyx 4y ago

extremely impressive, love the chart showing the major developments. congrats Railway! (disclosure: am small angel)

swatcoder 4y ago

Congratulations on achieving this milestone and probably keeping your customers very satisfied along the way, but I eagerly look forward to the future where everyone is so sick of virtual assistants, bots, and knowledge bases that a low ratio is what gets celebrated as a growth milestone rather than a high ratio. Automated triage is great for digital companies that want to scale cheaply, but not so great for the people who need to figure out how to usefully navigate them, with the brick wall support experience of Google/Facebook/etc being the already vivid example. It helped those companies grow and court techno-optimist investors, but nobody outside of their enterprise customers is happy about the customer service experience. Eventually, that becomes dirt on their reputation and works against their continued growth and success.

ndneighbor 4y ago
Honestly, you hit the nail on the head on what drives me. Before this I was a PM, and way even before that- I was an Infra Engineer. I think that brick-wall of contact when working on some Enterprise tool was what got me. ESPECIALLY so when there was a bug blocking my company and it felt like no one cared. When I first joined- this is why I would take great pains to respond to everyone extremely quickly. When I realized that was not sustainable, the focus then shifted to really making sure that our users are heard and their concerns are acted upon. Sure first contact SLA of 30 mins is great but- doesn't mean shit if we don't solve your problem. So I think yes, tooling should be first. I think there is just a way for that tooling to serve the developer, not some OKR.
- kazanz 4y ago
  Agreed. Where I work now (Hivelocity) is a breath of fresh air. There are so many support techs, that every ticket is answered within minutes. It's definitely not cheap to do, but the customers are always happy when they can get almost instant support for their dedicated servers.
daydream 4y ago
When I have a problem with a service my ultimate goal is to get it solved - as fast as possible. If I can solve my problem though an automated support channel that’s actually better! Because I don’t need to wait for a human to do something. Computers don’t sleep, take lunch breaks, etc. The problem with google/Facebook/etc isn’t that they don’t have human support. It’s that when your account gets locked out there’s no way of getting your problem solved. If a working useful automated support system existed few would complain.

nbrempel 4y ago

Very impressive! That growth curve is amazing.

fishtoaster 4y ago

Seems like good tips for any support org trying to keep a handle on load: 1. Get a support tool (or in this case, roll your own, then buy one when you outgrow that) 2. Build/buy tooling to direct users to existing documentation that might answer their questions 3. Move any routine tasks that require a human to automated self-serve tools. 4. Rope your community into answering eachother's questions.

mkl95 4y ago

Any SLAs worth mentioning? Did you ever struggle with them?

ndneighbor 4y ago
One thing that didn't make it into the article (I ramble and rant) is how I set it so at one point that Teams were entitled to a reply from me or the founder 24/7. This broke nearly instantly after 50 Teams. At the moment we don't but I spoke to the fmr. Head of Support at Stripe and he told me we should think about our product coverage like an API, some parts of the product (builds for example) should be more critical on response times than others. My main goal this year is to publish an SLA table so that we can work with ever larger customers setting the right expectation.

fuddle 4y ago

I was impressed by their pricing, how do they keep costs so low?

paxys 4y ago
I have spent the last 10 minutes on their site and am unable to figure out what the pricing even is. Going by the top of their pricing page (https://railway.app/pricing https://railway.app/pricing): $10/GB per month RAM & $20/vCPU per month is actually pretty expensive when compared to services like fly.io and even EC2/Azure. Apart from that, what vCPUs are these exactly? What are the bandwidth costs? Custom domains? Certificates? Load balancing? Disks? Databases? Clicking "Choose Plan" takes you to a login page, but then if you log in the pricing page disappears altogether.

revskill 4y ago

With bad documentation on buildpack, it's hard for me to go on with Railway. (For example, there's no sample or docs at all for how to build image with multiple buildpacks like in heroku). And these days, good documentation is 1st criteria to trust a platform / framework / library.

justjake 4y ago
Railway team member here: So there is one but, if you can't find it, it's as good as there not being one! It's currently located at https://nixpacks.com/docs/guides/configuring-builds#change-what-providers-are-run https://nixpacks.com/docs/guides/configuring-builds#change-w... Is there some place you'd expect it to be instead? All ears on feedback!

nitsuaeekcm 4y ago

What an impressive company. A good reminder that, even in a market with umpteen competitors, success comes down to execution. Hope they keep up the good work.

ryanSrich 4y ago
Railway does look impressive. However, as someone in the market for a Heroku alternative - I’d say there really aren’t any Heroku competitors. At least nothing directly comparable. The thing that keeps me (and I’m sure other small business owners) tied to Heroku is just how much they take care of for you. I literally never ever want to think about my infrastructure beyond initial setup. Heroku, to my knowledge, is the only provider that allows me to do this while servicing more than 100,000 users.
- benmartin 4y ago
  Oh there are numerous alternatives to Heroku. Just look for "PaaS" and your favourite software components (programming language, DB, CDN etc) and you'll find a list. I find Platform.sh is an excellent alternative to Heroku, dare I say better. Same thing - configure once and operate forever, scale as you need... Really Heroku used to be a big name but it is now declining, and lets face it - many used it just because it had a tiny plan for free.
- hankmh 4y ago
  Like others have said there are a number of Heroku alternatives out there that are more or less viable to meet your requirements, which I've gathered to be: - Serving >100k users - Helping you to achieve SOC 2 In my capacity as one of the leaders at a Heroku alternative PaaS, I've studied the PaaS market a bit to understand the space and available alternatives to Heroku. Here's what I've found about some of the most popular: - Fly is managing its own infrastructure allowing it to be extremely competitive on cost. But on the flip side, its heavy focus on infrastructure is missing the “managed” options that make PaaS so valuable, such as a true Managed Database offering. This IMO makes it less of a viable alternative to Heroku. - Render is offering a more truly “managed” alternative, and is innovating on cost as well. But it’s early and is still missing some table stakes reliability features that you'd probably expect from a Heroku alternative. - Railway has a blockbuster FTUX, I love deploying and using PostgreSQL databases in the UI without even signing up. Coincidentally probably how it got to its count of 250k developers. But it’s own docs caution that it’s not exactly production ready, especially its databases. - Platform.sh has grown really well by focusing on enterprise marketing teams and their use cases. I think this is a great niche for them and has paid off well. In their capacity of working with enterprises I'm sure they could handle SOC 2, etc. - Aptible has run critical web apps and APIs dealing with sensitive data for hundreds of companies and has helped a few go public or get to billion dollar acquisitions. I am certainly biased, but Aptible seems to be the only non-Heroku PaaS focused on product/engineering teams that has repeatedly handled true "production" requirements, like your larger user base (many of our customers fit this description) and SOC 2 (most of our customers use our security & compliance dashboard for this). But that comes at a cost: Aptible is typically more expensive than the others, perhaps save for Platform.sh. Disclaimer: I'm one of the leaders at Aptible, which in my (admittedly biased) view is the best positioned alternative to Heroku for product/engineering teams that have any sort of scale or production use case.

theonething 4y ago

Can anyone comment on the security implications of using a service like this, render.com, fly.io, etc vs. rolling your own infra on AWS EC2 and friends?

mmcclure 4y ago
Is there something specific you've got in mind? There are a ton of "it depends" type questions in here, but generally speaking I'd argue that rolling your own infra has some parallels with rolling your own crypto. There are countless mistakes you can make when doing it yourself, and a reputable entity that's been doing it for a bit has probably already made those mistakes or, at least, considered them. Quadruply so if they've got certifications of any kind. At scale, or with a seasoned ops team/team member, the scenario is a bit different, but if you're asking "is it more secure for a few full-stack developers hacking at a startup to roll their own infra or use something like fly.io" I feel preeetty comfortable saying the latter is going to be more secure.
fancyremarker 4y ago
It's possible for a PaaS to improve your security posture by implementing many of the security controls you'd otherwise be responsible for yourself. Every PaaS provider has a Shared Responsibility Model, like this one from AWS [0], and a good PaaS can eat up much of what would otherwise be your responsibility as an AWS customer: network architecture, secure configuration, IAM, system access (and auditing), intrusion detection, etc. On the other hand, many PaaS providers obfuscate their security implementation, and ultimately your data could be compromised by their mistakes. Things you should look for when evaluating PaaS providers: - How are resources, networks, etc. separated/isolated per customer? - What are YOUR security responsibilities on the platform? - How transparent is the provider about their security controls? Do they have security whitepapers, SOC 2 reports, etc. that are transparent and legit? Better yet, can they prove to you in the product how security controls are being implemented? Disclaimer: I'm the CEO and founder of Aptible [1], a PaaS specifically built to meet and prove security requirements for companies in regulated/high-compliance environments. [0] https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode... [1] https://www.aptible.com/ https://www.aptible.com/

urbandw311er 4y ago

Not sure this is something I’d be bragging about. What happens when you do eventually have a major outage and need to rapidly provide reassurance and tailored support to hundreds of people. Like when Atlassian deleted thousands of JIRA tickets by mistake and seemed completely unprepared to scale up their support.

trevor-e 4y ago
Is that something you can reasonably prepare for as a company? Like just keep a bunch of extra support people on payroll in case there's a catastrophic event?
- speakfreely 4y ago
  It's about not optimizing for a single variable. If you're trying to reduce headcount (and payroll cost) as a single goal, your business becomes a brittle shell subject to disruption by any number of circumstances. Investing in some redundancy is responsible to your shareholders and respectful to your customers.
- urbandw311er 4y ago
  What a great question. Perhaps the answer is “not entirely” but draw up some best effort contingency plans (pool of contractors, insurance cover, etc). I still wouldn’t be highlighting how few support staff I have per customer like this in either case!
  - trevor-e 4y ago
    Ha I definitely agree with you there. While impressive from the company's standpoint, I'm not sure it's something I'd highlight either. If anything, learning this would steer me away as a prospective client.

ipaddr 4y ago

I checked out the open positions. There were more positions open than total employees. Are those real positions you are looking to fill or always open positions that rarely result in an actual job? I asked this because the interview process has this step where you ask the candidates to do a project and than go over it with someone. One of these little projects is design our api or build a github issue viewer frontend/backend. You still have 4 more rounds after someone invested all that time. The amount of effort to work here seems high and the chances of the position existing and you filling it is so low. I feel bad for the many looking for work.. do you invest 10+ hours in a process where a job may not exist in the first place and the company brags about the goal of one person servicing a million

glenngillen 4y ago
I can’t speak for the Railway team, but I am hiring at an early stage startup myself. There is absolutely no way we’d extend offers for all of the roles we have open if a candidate arrived for all of them next week. Growing that much so quickly would be a disastrous onboarding experience for most of the new hires and potentially risk our ability to build a consistent culture with the team. So we’ll hire whichever candidate(s) successfully complete the process first and then pause the other roles until we’re ready to onboard more people. Then there’s also the reality at this stage you need people who can wear multiple hats. And there’s a bunch of roles where your ideal candidate doesn’t have a neat pre-established label. So sometimes we’ll post the exact same role with different titles to try and make sure it gets the attention of someone who most strongly aligns with one of those job titles. At a previous place we worked remotely, and so the same job would be posted as both remote and then also as a dozen different specific city locations too. But if we were hiring multiple of any roles you wouldn’t multiply that, you’d just post the one listing(s) and keep it open until all the positions were filled. A jobs page is a marketing artefact for potential hires, it’s not a financial reporting/forecasting tool. TLDR: don’t make any assumptions from a job board about how many roles a company is realistically hiring for.
- moomoo11 4y ago
  How did you find your core/initial team? Thanks

iLoveOncall 4y ago

Those numbers are absolutely meaningless. I work at Amazon and most teams do not have a single support engineer despite serving hundreds of millions of customers. The number of employees in the position X is absolutely meaningless.

jorts 4y ago
It depends on the type of product. If you're selling to consumers versus businesses, engineers versus HR, how mission critical your product is, etc... Given they're selling a mission critical piece of infrastructure it's a high ratio.

e12e 4y ago

Nice article - but given the title I wonder what "Community Champions" are? Sounds like free Labour?

trungdq88 4y ago
Having the same question. I guess the “Community Chanpions” are rewarded somehow.
- supriyo-biswas 4y ago
  It's typically in the form of access to features that are reserved for higher paying "enterprise" customers. Cloudflare's MVP program does this for example.

fijiaarone 4y ago

Pretty sure AWS hasn’t given support to anywhere near 250k developers.

mawadev 4y ago

Unbelievable how a single developer... or a handful runs an entire organization. I hope the developer gets paid handsomely and not just in exposure through this blog :)

klabb3 4y ago

How did you measure that 250k? That’s the population of like Iceland. Sounds like a lot, but perhaps my “how many golfballs fit in a schoolbus” guesstimate game is off.

canto 4y ago

What do you guys use to run databases? Zalando operator for postgres perhaps? And for mysql and redis?

nevon 4y ago

Assuming that the support experience for the customers is actually good, I'm amazed they've managed to scale their support efforts that far. I work on a similar type of product, except in-house in an organization with around 2000-2500 engineers, and handling support has by far been the hardest thing to scale as we've grown. Even with documentation available, internal Stack Overflow with plenty of common questions answered, unified search, Slack integrations, error messages linking directly to documentation, pre-defined observability dashboards and monitors, we still get so many questions that it's a full time job staying on top of it. Can't imagine having 100x that amount of users.

chrismorgan 4y ago

> I am convinced that Software Engineering is more of an art than an engineering practice. Back in 2010, Monash University’s Bachelor of Software Engineering had a unit ENG1061 Engineering Profession mandatory for the degree (for a terrible reason: Engineers Australia accreditation of the degree, which no one that I ever talked to cared about in the slightest). The first lecture listed The 17 Branches of Engineering. Guess what wasn’t one of them. It went downhill from there.