22 ms·

11 years of hosting a SaaS

334 points by ghiculescu 3y ago

oaiey 3y ago
> At our first meeting our [AWS] account manager brought along his solutions architect. I had never met a solutions architect, so I didn’t really know what they did. > Eventually I realised that I wasn’t the problem. I had the same experience. These free "Solution Architects" are a disgrace to the architecture profession. They are sales person and are paid/rated/motivated to sell you up. Not what is best for you. Ours went over the in-house architect with zero feedback and told our CTO/CIO how shitty the architecture is. Fun. Stay away from them.
- 123pie123 3y ago
  If something is free etc... I've always taken the advice of any prepaid 'experts' with a huge grain of salt, not just AWS
pdimitar 3y ago
It really makes me super bitter how people like the author of the article who basically fumble their way through stuff are still successful, even though they seem to have done a lot of stuff the wrong way. Calling out D.O. for a fairly transparent process is very wrong. They did what they had to and OP was oblivious to a fairly standard process of "hey, your CC expired, let's do something about it before we turn off your servers" which OP seems to have ignored several times in a row. All of the story felt like he was just stumbling from one hype to another, and from one easy to avoid mistake to another. Even when I had only 7 years of experience in total (now I have 21.5) I was paranoid enough to look for DB ID turn-over and absolutely would have made sure to upgrade the DB server when I start noticing it hits 80% load consistently. But that's life for you. A technically excellent guy still lives paycheck to paycheck, meanwhile an absent-minded guy who is easily hyped has a successful business. [sighs deeply] ...Moving on. Finally, choosing AWS but still only using EC2 is kinda non-intuitive for me; why do that at all? Maybe because they allow transparent upgrading that makes for less sysadmin work? If so then fair, but that's still like buying a Ferrari to drive on long empty roads but never go above 90 km/h.
amake 3y ago
> The team had several people in every time zone Kind of hard to believe
- ghiculescu 3y ago
  Crap sentence, but people in Australia, US, and UK can cover every time zone.
jarebear6expepj 3y ago
I'm sure its interesting but I am making a conscious choice to not read or linger on user hostile articles. As soon as I want to scroll I am interrupted by a page takeover requiring action and subsequently compelling me to lose interest. Did you want me to read it or click buttons for you? Too bad your UI gets in the way of the U.
- SPBS 3y ago
  Substack is definitely scummy in this regard, it's not really the author's choice (although a conscious decision was made to host it on Substack).
  - andai 3y ago
    I read a Substack article almost every day and still consistently get a little jolt of rage when the overlay fades in. I guess they do it because it works, and if the average person felt the same way, they wouldn't be doing it?
    - gwd 3y ago
      > if the average person felt the same way, they wouldn't be doing it? You might think this, but I tend to doubt it's true... people making these decisions end up being in their own little bubble, and not really have a good idea what people's actual response is. That said, I find Substack's pop-up tolerable: I am after all getting someone's work for free, and I'd rather have a simple pop-up that I know I can get rid of than ads or other aggressive forms of pop-ups.
      - ghiculescu 3y ago
        Yeah I'm aware of it. But it's less annoying than Medium.
- ValtteriL 3y ago
  Firefox's reader view gets rid of all that on top of reducing other distractions
awayto 3y ago
> We started on Heroku, because in 2012 if you did any Ruby on Rails tutorial that included deploying your app, you ended up with a Heroku account. Couldn't be more accurate. Heroku and Rails is almost as much of a throwback as Node and Express. You just had to be there. And it was great. Web dev was always a hobby for me as a teen, but then I turned to the rails book as a means of learning a professionally designed system, when I wanted to get serious as a dev. It definitely served me well. Heroku, at the time, offered a very streamlined and accessible way to integrate rails. It was a great time to learn.
- SenHeng 3y ago
  I sometimes wonder how my career (and life) would've differed if 11 years ago, I had chosen to learn rails instead of laravel because of familiarity with php via Wordpress. From there it went on to C# (windows), JS (frontend), scala/go/node.js backend + JS front end. Lots of JS. Here being Japan, there were a lot of opportunities to pick up ruby/rails along the way but I stuck on the JS trajectory partially because I didn't want to 'start over' with rails _now_. There were only a few times when not knowing ruby/rails meant I was limited to specific tasks so it was never really career limiting. It did mean that I actively avoided working in ruby shops, for better or worse. If I had picked rails instead then, would I have transitioned into more of a backend engineer with some frontend duties instead of the reverse? Not that it matters, but there definitely is a tendency in our industry to 'look down' on frontenders as not real engineers and thus not consider them for leadership positions.
  - yurishimo 3y ago
    It's not too late to go back! Rails and Laravel are still insanely popular and productive for getting work done. I'm also disappointed that real frontend expertise is not valued as highly as a traditional backend engineer. Think about someone like an Architect. You could say that they are glorified artists (yes, I know they do more than that) and they are highly valued for their artistic input as well as their professional advice; but it's also not unheard of to have an architect solely provide artistic direction and a structural engineer provide support.
  - 3y ago
esafak 3y ago
So what are the new Herokus? What do you recommended for starting a serious project? AWS AppRunner? How about something open source?
- flybayer 3y ago
  https://Flightcontrol.dev https://Flightcontrol.dev is a relatively new one that provides a PaaS-like experience on your own AWS account *I'm cofounder & CEO
- zoomzoom 3y ago
  I'am a cofounder of withcoherence.com, a PaaS developer experience on top of your own cloud. We are not open-source, but we support both AWS and GCP and provide a free tier of usage. Happy to answer any questions, feel free to get in touch!
- atomicnumber3 3y ago
  Can it be that between docker and digital ocean giving you a gig of ram for $6/mo, people are just DIYing it much more often these days?
- ghiculescu 3y ago
  fly.io is the least bad. But honestly nothing these days is as good as Heroku was 10 years ago, if you just want to put something online and don't care how it gets there.
  - babyshake 3y ago
    Vercel for serverless apps is very easy to do.
- kkapelon 3y ago
  Check railway.app, fly.io, render.com Possibly qovery.com as well.
- mstocks 3y ago
  FL0.com is new and looks v solid
irsagent 3y ago
A recent observation, and this could be a biased one, you mentioned: > we’ve spent a bit of time learning more about moving off the cloud to a managed data center. But the nice thing is not feeling like we need to. That SaaS's are moving/thinking/completed to running their own infustructure away from managed services like digital ocean and aws, like 37Signal(Basecamp & Hey).
aetherspawn 3y ago
Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though we have redundant everything, nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two). (1): How? Well instead of running small containers/micro boxes/services that we could scale up and down, since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore. We don't really care how much we utilize our own HW, so we just smooshed everything onto huge VMs in two huge 100 core boxes with affectionate names. We used super basic networking features (VLANs, firewall port forward) to setup the network. To simplify dev ops even further, there is only access to the servers from specific ethernet ports in the building.
- olive247 3y ago
  Does geolocation of the server not matter for your application? I am curious if there are any latency issues with this solution as opposed to a hosted option.
- ghiculescu 3y ago
  It's around 150k USD/month at the moment. Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.
  - aetherspawn 3y ago
    Our deployment is much smaller than yours! Our upfront hardware cost including carpentry and HVAC was around $80k, our monthly cost in cloud was around $5k/mo and now we pay only about $1k/mo in energy (which includes the existing $500/mo bill from the office). We could reduce this further with solar panels, but we share the tall-ish building with 13 other suites and we couldn't get approval to take up most of the roof.
  - Sai_ 3y ago
rozenmd 3y ago
> Use managed services for as long as possible Big agree here. Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or you can pay someone else and keep building your product: https://onlineornot.com/self-hosting-vs-managed-services-deciding-how-host-your-database#benefits-of-managed-services https://onlineornot.com/self-hosting-vs-managed-services-dec... This isn't to say, "don't bother learning how to do it yourself", but more "learn to pick your battles".
- jordiburgos 3y ago
  Which one do you recommend? Those get quite expensive.
  - rozenmd 3y ago
    I personally use RDS
- rldjbpin 3y ago
  having no skin in the game, I feel this is a very interesting view, especially when you are offering a managed service building on top of another. go through enough levels of dependencies, and someone who could do a few layers of the onion in-house can sweep in with competitive pricing/offerings a la sherlocking.
- Radim 3y ago
  It may be a generational thing, a matter of familiarity with computing and computers. For someone who's lived through the 80s, "handrolling postgres" doesn't sound nearly as scary as you imagine. I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different teams.
  - tacker2000 3y ago
    Yea i also dont see the point of having everything “managed”. RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB. If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine. But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.
ksajadi 3y ago
the progression from Heroku to AWS is very common as SaaS businesses grow. However one thing that’s missing from all “alternatives to Heroku” is what I call life after deployment or Day-2 operations. Many solutions show you a sleek demo of how easily you can deploy a Hello World app to their platform but leave you in a straight jacket when it the app is deployed and you need tools to keep it running, like changing an Nginx (or equivalent) setting or using persistent storage or a Postgres contrib. Making decisions about infrastructure is as much about day-2 as it is about the initial deployment. I see fly.io and render in this camp. Shiny day-1 docs and demos and then wishing you good luck when you need a `rails c` to see something in the DB
- flippy_flops 3y ago
  I hate what Salesforce has done to Heroku but after a year of running our saas on aws, render, and fly - here I am, back on Heroku. It sucks, but slightly less than the alternatives. Fly has potential, but they've changed/grown so much that most docs are out of date, everything is buggy, support is not very responsive, and their security posture leaves a lot to be desired. Not a fun place to be production issues pop up.
  - suryao 3y ago
    I'm building Argonaut to be a layer on top of AWS/GCP to make them less sucky. The primary use case is startups that have a bunch of services and have outgrown heroku etc. I'd love to chat if we can help - you get the stability of AWS and a push to deploy dev experience for your apps.
  - friism 3y ago
    (I work at Heroku) Do you have more details on what sucks? Anything we're not already tracking to fix in our public roadmap? https://github.com/heroku/roadmap/issues https://github.com/heroku/roadmap/issues
    - flippy_flops 3y ago
      Glad to hear there's a roadmap. In my PAAS thrashing I've got quite a few notes: - no wildcard subdomain ssl - poor metrics, nothing per instance - poor dyno granularity - (jumps from 2.5GB RAM to 14, no cpu/storage control) - no transparency on what each dyno actually is - external postgres replication disabled (deal breaker) - no first class postgres metrics and access logs/alerts (kibana recommended, but not great) - no external postgres backups (e.g. S3) - no deletion locks on dynos and add-ons, esp databases!!! - no warning when add-ons like databases are being deleted as a result of apps being deleted - deleting an add-on also irreversibly deletes all replicas and backups - painfully inconsistent naming for databases through their connection str env - dyno types for build processes use Perf-M? Not configurable - No lambda or github action style computing - No scheduled scaling - Unable to choose aws-us regions - Hard 30s timeout limit - Limited to 1 api key per user. No labels, configurable permissions, usage logging - no http/2 - frustrating enterprise offering. massive over-sell, near zero value
domlebo70 3y ago
Alex! Great to see you on HN.
- ghiculescu 3y ago
  Hey mate!!
l5870uoo9y 3y ago
> We didn’t take advantage of any other platform features, we just treated AWS like any other VPS. This is a lot of switching around for a plain VPS.
- joshstrange 3y ago
  Yeah, if you aren’t going to use AWS managed services then what are you doing on AWS? EC2 is great but if that’s all your using your probably overpaying.
polote 3y ago
Well I do not feel any of the 3 advice he gives, good advice. First the longer you stay in heroku ,the most complex is it to exit it the time you really need it and the less flexible you are in the time being. Second, wish he had pay for a pit team sooner, but could this money better used investing in marketing or sales like he probably did ? The guy has obviously succeeded as a business owner, would it still be the case if he had implemented these advice ? We will never know, but what we know for sure is not implementing these advice made him successful
- codetiger 3y ago
  I think he meant, to stay with managed services as long as possible. Which makes sense, as in the early stage, the focus has to be the functionality rather than cost/performance optimisation.
- ghiculescu 3y ago
  I think our success is despite a lot of decisions (specifically the ones in the post), not because of them.
- kakwa_ 3y ago
  > We will never know, but what we know for sure is not implementing these advice made him successful "Made him successful" implies a causality that is a bit too strong. Indeed, we will never know for sure. Maybe implementing these advice would have impeded development of other critical areas of his business. Maybe it would have would have helped make is business more successful as he would have had a more reliable product. Or maybe the business impact would have been neutral, but would have resulted in better quality of life/less stress for him and his employees. But in general, the way I read this article is: they made good decisions overall, but as everything in the world, it was not optimal (switching platform too early, making some big mistakes like the credit card one, etc). It's a very interesting read nonetheless, with clear take away: * chose boring tech you know and focus on your product, not the tech, specially in the early days * grow your infrastructure and complexity with your product needs * accept you will mess-up but properly learn from it, and grow your organizational knowledge, structure and processes accordingly.
dom_hutton 3y ago
If I could have my time again I'd focus on enablement of the broader engineering group through an incident management process, readiness exercises, some aggregate incident analyses for the org to learn from and leaning into observability. Infrastructure is kind of a solved problem for common use cases today, just requires the expertise. With our ducks in a row I'd next have look to a GRC function for the compliance bits whilst splitting the platform engineers time between embedding engagements and tooling investments. You're on the right path man, I'd love to know what I know now back then but unfortunately time doesn't work like that.
- fullstackchris 3y ago
  > Infrastructure is kind of a solved problem for common use cases today, just requires the expertise. This is the problem however for many (older) companies. They either don't care, or quite literally don't know the infrastructure solutions out there which can save literally thousands of hours per year of headache. Sure, for many companies with legacy systems they have a "dont fix what isn't broken" mindset, but from what I've seen, I always ask, if shipping and modifying new versions of a system takes hours or even days to complete, is the system really not 'broken'? I guess I never realized it, but having automated and clean infrastructure with tests and uptime metrics is a must-have for me on anything I build going forward. Take 2-3 weeks to save months of headache.
- ghiculescu 3y ago
  Thanks. Means a lot, particularly coming from you.
shri_krishna 3y ago
Get a 403 Forbidden when I visit https://www.tanda.co/ https://www.tanda.co/
- rathish_g 3y ago
  It seems that the new PIT team was unable to manage the HN traffic.
  - djvu97 3y ago
    They should switch back to digitalocean when things were manangeable
AndroTux 3y ago
> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to SSH. This wasn’t totally their fault, but at the time it just all felt like a dodgy, shaky setup. This wasn’t *at all* their fault and there’s nothing dodgy about it. This was entirely your fault, and even mentioning this as a „partly“ negative for DO is imo very wrong. I imagine any other service provider, including Heroku, would have acted the same way. You can’t expect to mooch off a month of service without paying and without consequences. If you don’t pay your bills, service will be interrupted. It’s as simple as that.
- tommica 3y ago
  I read the "dodgy, shaky" parts as their own setup was that, not digital oceans
- alberth 3y ago
  But it kind of is. Merchant payment services exist that can automatically obtain the updated card-on-file credential when they are about to expire. Both Visa and Mastercard offer this directly, or you can get the same service from a PSP.
  - TheNewsIsHere 3y ago
    It’s definitely not the merchant’s responsibility to catch when your card changes. We have this at work thanks to Stripe, and it’s wildly inconsistent. But beyond that, it won’t do anything for a closed/canceled card/cardholder account. That situation was fully on OP. This could have happened at GCP, AWS, Azure, Linode, OVH, Rackspace, Oracle Cloud, you name it.
    - TacticalCoder 3y ago
      > It’s definitely not the merchant’s responsibility to catch when your card changes. Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration. I mean: we're literally talking about credit cards often keeping the exact same owner name, the very same number and just having their MM/YY of expiration changed. It's not rocket science to update in a DB after an API call. And what's the catch on at least trying? If it works, you saved everybody time. If it doesn't work, it's not worse than your current "solution". I understand politicians and lawyer-minded people saying: "technically it's not our responsibility" but they're wasting everybody's time, leaving money on the table and wasting money on support. It's just poor judgment to react like that instead of thinking as to how life could be made better for everybody, starting with your paying customers (which, btw, are the reason you exist).
alexpls 3y ago
This really resonates. It's so easy to get caught up in chasing trends and overlook delivering customer value/stability. Good luck on the next 11 years :)
nicbou 3y ago
This is one of the best ops writeup I've read. People are rarely so vulnerable and relatable. Usually it's all "here's how we do things perfectly at our unicorn company". The most important lesson was about reducing the alerts and sleeping better. It's important to be able to leave your laptop at the office and not worry about things going wrong. This post gave me flashbacks of when I made websites for others and hosted them myself. Never again!
fullstackchris 3y ago
Great post, I've used all these services once or more at various points in my career. I'm still on the Digital Ocean stage, as I've found AWS is often overly complicated (or at least too much for me as a primarily solo founder) I'm curious about this line: > We were doubling the size of our customer base every 9 months, and pretty soon this meant we needed more servers. I've found that a moderately sized 4 core 'droplet' from Digital Ocean handles nearly anything I can throw at it (at least in terms of the web application world) - I see you product is doing employee management / scheduling - is there heavy compute / resources associated? Or is it more due to privacy requirements (i.e. you need a siloed instance / server per client?). Not trying to sound smug at all, just genuinely curious where the load comes from.
- ghiculescu 3y ago
  Great questions. There is some heavy compute - most of the real value is in our pay calculations and compliance features which are all based on algorithms that calculate correct pay based on legal requirements. But mostly it's just having lots of concurrent users. For a while a few droplets was plenty, and then one day we hit a tipping point where it was overwhelming.
- spiffytech 3y ago
  It seems like server performance has improved a lot over the last decade. The move from HDD -> SSD -> NVMe alone makes a huge difference for anything that touches disk. And my memory is that it was pricier to buy machines with more cores or RAM vs today. It wouldn't surprise me if today's budget VPS does the work of several machines from 10/11 years ago, and costs less too!
nologic01 3y ago
Something that would be interesting to compare and contrast is actually using your own hardware to host the service(s). Servers are super-reliable these days, it doesn't feel like a huge step from the VPS approach to simply have a bunch of them. At least one tradeoff is that you'd have to be physically in a location with adequate bandwidth / latency but for various use cases it might be very cost effective during some phase of the startup. I wonder also how the AI induced focus on specialized hardware might change the calculus
anonyfox 3y ago
After 20 years of dev, having devops skills nearly maximized (with a AWS specialization), built/maintained/scaled many products, ... can completely relate. I really do see hostings costs differently now compared to earlier in my career, when I was more junior/midlevel where one still argues about "cool" tech/languages/architectures. What you want from day one is a completely managed hosting for tech stack you work with, but are not hard tied to forever. Right now I am starting some new/modern meteor.js stuff again, there is meteor galaxy for hosting that "just works" for that stack, you bring your own mongodb (I prefer atlas itself, they even have serverless now), and everything is taken care of, including CI/CD/Monitoring/... . Its like a few minutes of initial configuration, and never think about it again, and if done correctly you shold have horizontal autoscale of some kind automatically nowadays. Yes, this is significantly more expensive than directly using AWS (which they use under the hood), and even though I am personally highly trained in this stuff+terraform/cdk/..., I don't want to have all this work anymore when I also can shell out a few hundred bucks per month instead. Just in case Galaxy becomes a problem for some reason, I _still_ can deploy the app stack to some VPS provider, but I'd use some already existing automation (like meteor-up in this case) instead of really digging into typical devops topics for it. There is a bad feeling in the mouth as an engineer to shell out "more than needed" for infrastructure, but my rule of thumb now is that I am happy to eat that frog as long as the potential cost saving is less than 2 infrastructure engineer FTEs, thats my trigger to _maybe_ discussing it.
nektro 3y ago
great writeup and congrats on 11 years!
jwr 3y ago
I read this article with interest, having run a SaaS for the last 8 years or so (solo-founder). I find it intriguing that the OP needed so much compute power. Computers are really powerful these days. As a counterpoint of sorts, here's my "hosting journey": * run everything on a single physical server rented at Hetzner (DE) * [... several years pass, business grows ...] * switch to ansible, learn it, spend a week or so to write automation for a 3-server setup, also learn terraform and write terraform configs for setting up a Digital Ocean system from scratch * run production on a 3-server setup at Hetzner DE, run a staging system at Hetzner FI, also serving as a possible quick manual failover, for a total of 6 physical servers, test re-initializing systems from scratch regularly, test setting up a Digital Ocean system from scratch regularly * [... several years pass ...] * that's it — I really can't see a need for more in the near future. But then, my software is not in Ruby on Rails, and I have no experience with that platform. I use Clojure and ClojureScript and I was careful to design everything to be rather client-heavy at the start. I also never wanted to depend on PaaS systems, mostly to avoid lock-in, but also because I don't buy the "just use our magic database offering and forget about database problems forever" selling pitch. You can sweep possible problems and complexity under the rug and hide them, but you can't run away from them. I also do not use Postgres (collective gasp in the audience), because having a single centralized point of failure is not something I want in my setup. I also never needed significant sysadmin/devops support. Granted, I do have some experience, but these things do not require dedicated teams, unless you are YouTube. A little ansible+terraform goes a long way, so does buying an hour of two of consulting from an experienced sysadmin. Those physical servers that I use are significantly faster than the over-subscribed cloud VM instances that you usually get from AWS and the like. And they have 64GB of RAM, not some measly amounts. If I need more servers, it takes on the order of hours to get additional ones, but I'm not sure what I'd use them for. My total hosting bill is on the order of 350€/month and is boringly predictable.
- gen220 3y ago
  Just curious, if you feel comfortable sharing, what do you use as your database? I'd agree Postgres is not the right answer to every problem, but the documentation of its failure modes and the mitigations thereof make it a "good" answer to "most" problems. I'm curious what the "problem" (in access pattern terms, doesn't have to be business terms) and "solution" (i.e. persistence technology) is in your setup!
RandomWorker 3y ago
This story resonates with me — apparently we are on phase two. And we need dedicated team members to take care of this. Thanks for sharing your lessons this was hugely valuable. Don’t let the negative comments get you down, when building things it’s hard and it’s all a big learning game. Congratulations on building a great piece of software.
- ghiculescu 3y ago
  Thanks! Reach out if you ever need a hand.
lakomen 3y ago
Those are such noob mistakes. No bigint OID, or worse, not UUIDv4, because even int64 will overflow, maybe not on their platform, but on a platform with millions of new inserts a day, despite being a large number, isn't future proof. If anyone knows a better 128bit alternative to UUIDv4, please respond. Side remark, that's what I was trying to tell the dgraph guy. If you have an int64 ID for ALL your transactions and all your IDs it will eventually overflow. It's a single incrementing value for ALL actions. Inset a new record, new ID, retrieve this record, new ID etc. Everything this guy writes about are newbie mistakes, stemming from a lack of "far sight", aka thinking 5 steps ahead. Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these. What annoys me is that I'm actually seeking a job now and that I can't find one that fits my needs, despite being way past those Kinderkrankheiten (child diseases), because those jobs nowadays require you to be an AWS zealot, which I'm not. I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part. I know this article is about Ruby. But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer, I would NOT go AWS lambda and complicate my life and, most importantly, the development process or as I call it the solution delivery process. This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations. There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters. The split between backend and frontend happened around 2013. Having a classic website with low performing Ruby, that also has such a bad workflow, e.g. Blog, you can't start with Comment, you have to start with Blog, then Post, then Comment. This is archaic. With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client. If I need SEO I'm back to the scaling problem, but the Go backend won't be the one that needs scaling. And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle. Doing this for the SSR JS frontend is where it gets expensive. So either try to static render with Svelte, if possible, to solve this problem and load balance or get locked into complex auto scaling cloud nonsense. And this is where it gets REALLY expensive. Your Ruby on Rails app might be good enough for a PoC, but that's about it. If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure. And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent. You can have your cake and eat it too. Just think it through from the start to the end.

334 points by ghiculescu 3y ago

> At our first meeting our [AWS] account manager brought along his solutions architect. I had never met a solutions architect, so I didn’t really know what they did. > Eventually I realised that I wasn’t the problem. I had the same experience. These free "Solution Architects" are a disgrace to the architecture profession. They are sales person and are paid/rated/motivated to sell you up. Not what is best for you. Ours went over the in-house architect with zero feedback and told our CTO/CIO how shitty the architecture is. Fun. Stay away from them.

123pie123 3y ago
If something is free etc... I've always taken the advice of any prepaid 'experts' with a huge grain of salt, not just AWS

pdimitar 3y ago

It really makes me super bitter how people like the author of the article who basically fumble their way through stuff are still successful, even though they seem to have done a lot of stuff the wrong way. Calling out D.O. for a fairly transparent process is very wrong. They did what they had to and OP was oblivious to a fairly standard process of "hey, your CC expired, let's do something about it before we turn off your servers" which OP seems to have ignored several times in a row. All of the story felt like he was just stumbling from one hype to another, and from one easy to avoid mistake to another. Even when I had only 7 years of experience in total (now I have 21.5) I was paranoid enough to look for DB ID turn-over and absolutely would have made sure to upgrade the DB server when I start noticing it hits 80% load consistently. But that's life for you. A technically excellent guy still lives paycheck to paycheck, meanwhile an absent-minded guy who is easily hyped has a successful business. [sighs deeply] ...Moving on. Finally, choosing AWS but still only using EC2 is kinda non-intuitive for me; why do that at all? Maybe because they allow transparent upgrading that makes for less sysadmin work? If so then fair, but that's still like buying a Ferrari to drive on long empty roads but never go above 90 km/h.

amake 3y ago

> The team had several people in every time zone Kind of hard to believe

ghiculescu 3y ago
Crap sentence, but people in Australia, US, and UK can cover every time zone.

jarebear6expepj 3y ago

I'm sure its interesting but I am making a conscious choice to not read or linger on user hostile articles. As soon as I want to scroll I am interrupted by a page takeover requiring action and subsequently compelling me to lose interest. Did you want me to read it or click buttons for you? Too bad your UI gets in the way of the U.

SPBS 3y ago
Substack is definitely scummy in this regard, it's not really the author's choice (although a conscious decision was made to host it on Substack).
- andai 3y ago
  I read a Substack article almost every day and still consistently get a little jolt of rage when the overlay fades in. I guess they do it because it works, and if the average person felt the same way, they wouldn't be doing it?
  - gwd 3y ago
    > if the average person felt the same way, they wouldn't be doing it? You might think this, but I tend to doubt it's true... people making these decisions end up being in their own little bubble, and not really have a good idea what people's actual response is. That said, I find Substack's pop-up tolerable: I am after all getting someone's work for free, and I'd rather have a simple pop-up that I know I can get rid of than ads or other aggressive forms of pop-ups.
    - ghiculescu 3y ago
      Yeah I'm aware of it. But it's less annoying than Medium.
ValtteriL 3y ago
Firefox's reader view gets rid of all that on top of reducing other distractions

awayto 3y ago

> We started on Heroku, because in 2012 if you did any Ruby on Rails tutorial that included deploying your app, you ended up with a Heroku account. Couldn't be more accurate. Heroku and Rails is almost as much of a throwback as Node and Express. You just had to be there. And it was great. Web dev was always a hobby for me as a teen, but then I turned to the rails book as a means of learning a professionally designed system, when I wanted to get serious as a dev. It definitely served me well. Heroku, at the time, offered a very streamlined and accessible way to integrate rails. It was a great time to learn.

SenHeng 3y ago
I sometimes wonder how my career (and life) would've differed if 11 years ago, I had chosen to learn rails instead of laravel because of familiarity with php via Wordpress. From there it went on to C# (windows), JS (frontend), scala/go/node.js backend + JS front end. Lots of JS. Here being Japan, there were a lot of opportunities to pick up ruby/rails along the way but I stuck on the JS trajectory partially because I didn't want to 'start over' with rails _now_. There were only a few times when not knowing ruby/rails meant I was limited to specific tasks so it was never really career limiting. It did mean that I actively avoided working in ruby shops, for better or worse. If I had picked rails instead then, would I have transitioned into more of a backend engineer with some frontend duties instead of the reverse? Not that it matters, but there definitely is a tendency in our industry to 'look down' on frontenders as not real engineers and thus not consider them for leadership positions.
- yurishimo 3y ago
  It's not too late to go back! Rails and Laravel are still insanely popular and productive for getting work done. I'm also disappointed that real frontend expertise is not valued as highly as a traditional backend engineer. Think about someone like an Architect. You could say that they are glorified artists (yes, I know they do more than that) and they are highly valued for their artistic input as well as their professional advice; but it's also not unheard of to have an architect solely provide artistic direction and a structural engineer provide support.
- 3y ago

esafak 3y ago

So what are the new Herokus? What do you recommended for starting a serious project? AWS AppRunner? How about something open source?

flybayer 3y ago
https://Flightcontrol.dev https://Flightcontrol.dev is a relatively new one that provides a PaaS-like experience on your own AWS account *I'm cofounder & CEO
zoomzoom 3y ago
I'am a cofounder of withcoherence.com, a PaaS developer experience on top of your own cloud. We are not open-source, but we support both AWS and GCP and provide a free tier of usage. Happy to answer any questions, feel free to get in touch!
atomicnumber3 3y ago
Can it be that between docker and digital ocean giving you a gig of ram for $6/mo, people are just DIYing it much more often these days?
ghiculescu 3y ago
fly.io is the least bad. But honestly nothing these days is as good as Heroku was 10 years ago, if you just want to put something online and don't care how it gets there.
- babyshake 3y ago
  Vercel for serverless apps is very easy to do.
kkapelon 3y ago
Check railway.app, fly.io, render.com Possibly qovery.com as well.
mstocks 3y ago
FL0.com is new and looks v solid

irsagent 3y ago

A recent observation, and this could be a biased one, you mentioned: > we’ve spent a bit of time learning more about moving off the cloud to a managed data center. But the nice thing is not feeling like we need to. That SaaS's are moving/thinking/completed to running their own infustructure away from managed services like digital ocean and aws, like 37Signal(Basecamp & Hey).

aetherspawn 3y ago

Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though we have redundant everything, nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two). (1): How? Well instead of running small containers/micro boxes/services that we could scale up and down, since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore. We don't really care how much we utilize our own HW, so we just smooshed everything onto huge VMs in two huge 100 core boxes with affectionate names. We used super basic networking features (VLANs, firewall port forward) to setup the network. To simplify dev ops even further, there is only access to the servers from specific ethernet ports in the building.

olive247 3y ago
Does geolocation of the server not matter for your application? I am curious if there are any latency issues with this solution as opposed to a hosted option.
ghiculescu 3y ago
It's around 150k USD/month at the moment. Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.
- aetherspawn 3y ago
  Our deployment is much smaller than yours! Our upfront hardware cost including carpentry and HVAC was around $80k, our monthly cost in cloud was around $5k/mo and now we pay only about $1k/mo in energy (which includes the existing $500/mo bill from the office). We could reduce this further with solar panels, but we share the tall-ish building with 13 other suites and we couldn't get approval to take up most of the roof.
- Sai_ 3y ago

rozenmd 3y ago

> Use managed services for as long as possible Big agree here. Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or you can pay someone else and keep building your product: https://onlineornot.com/self-hosting-vs-managed-services-deciding-how-host-your-database#benefits-of-managed-services https://onlineornot.com/self-hosting-vs-managed-services-dec... This isn't to say, "don't bother learning how to do it yourself", but more "learn to pick your battles".

jordiburgos 3y ago
Which one do you recommend? Those get quite expensive.
- rozenmd 3y ago
  I personally use RDS
rldjbpin 3y ago
having no skin in the game, I feel this is a very interesting view, especially when you are offering a managed service building on top of another. go through enough levels of dependencies, and someone who could do a few layers of the onion in-house can sweep in with competitive pricing/offerings a la sherlocking.
Radim 3y ago
It may be a generational thing, a matter of familiarity with computing and computers. For someone who's lived through the 80s, "handrolling postgres" doesn't sound nearly as scary as you imagine. I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different teams.
- tacker2000 3y ago
  Yea i also dont see the point of having everything “managed”. RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB. If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine. But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.

ksajadi 3y ago

the progression from Heroku to AWS is very common as SaaS businesses grow. However one thing that’s missing from all “alternatives to Heroku” is what I call life after deployment or Day-2 operations. Many solutions show you a sleek demo of how easily you can deploy a Hello World app to their platform but leave you in a straight jacket when it the app is deployed and you need tools to keep it running, like changing an Nginx (or equivalent) setting or using persistent storage or a Postgres contrib. Making decisions about infrastructure is as much about day-2 as it is about the initial deployment. I see fly.io and render in this camp. Shiny day-1 docs and demos and then wishing you good luck when you need a `rails c` to see something in the DB

flippy_flops 3y ago
I hate what Salesforce has done to Heroku but after a year of running our saas on aws, render, and fly - here I am, back on Heroku. It sucks, but slightly less than the alternatives. Fly has potential, but they've changed/grown so much that most docs are out of date, everything is buggy, support is not very responsive, and their security posture leaves a lot to be desired. Not a fun place to be production issues pop up.
- suryao 3y ago
  I'm building Argonaut to be a layer on top of AWS/GCP to make them less sucky. The primary use case is startups that have a bunch of services and have outgrown heroku etc. I'd love to chat if we can help - you get the stability of AWS and a push to deploy dev experience for your apps.
- friism 3y ago
  (I work at Heroku) Do you have more details on what sucks? Anything we're not already tracking to fix in our public roadmap? https://github.com/heroku/roadmap/issues https://github.com/heroku/roadmap/issues
  - flippy_flops 3y ago
    Glad to hear there's a roadmap. In my PAAS thrashing I've got quite a few notes: - no wildcard subdomain ssl - poor metrics, nothing per instance - poor dyno granularity - (jumps from 2.5GB RAM to 14, no cpu/storage control) - no transparency on what each dyno actually is - external postgres replication disabled (deal breaker) - no first class postgres metrics and access logs/alerts (kibana recommended, but not great) - no external postgres backups (e.g. S3) - no deletion locks on dynos and add-ons, esp databases!!! - no warning when add-ons like databases are being deleted as a result of apps being deleted - deleting an add-on also irreversibly deletes all replicas and backups - painfully inconsistent naming for databases through their connection str env - dyno types for build processes use Perf-M? Not configurable - No lambda or github action style computing - No scheduled scaling - Unable to choose aws-us regions - Hard 30s timeout limit - Limited to 1 api key per user. No labels, configurable permissions, usage logging - no http/2 - frustrating enterprise offering. massive over-sell, near zero value

domlebo70 3y ago

Alex! Great to see you on HN.

ghiculescu 3y ago
Hey mate!!

l5870uoo9y 3y ago

> We didn’t take advantage of any other platform features, we just treated AWS like any other VPS. This is a lot of switching around for a plain VPS.

joshstrange 3y ago
Yeah, if you aren’t going to use AWS managed services then what are you doing on AWS? EC2 is great but if that’s all your using your probably overpaying.

polote 3y ago

Well I do not feel any of the 3 advice he gives, good advice. First the longer you stay in heroku ,the most complex is it to exit it the time you really need it and the less flexible you are in the time being. Second, wish he had pay for a pit team sooner, but could this money better used investing in marketing or sales like he probably did ? The guy has obviously succeeded as a business owner, would it still be the case if he had implemented these advice ? We will never know, but what we know for sure is not implementing these advice made him successful

codetiger 3y ago
I think he meant, to stay with managed services as long as possible. Which makes sense, as in the early stage, the focus has to be the functionality rather than cost/performance optimisation.
ghiculescu 3y ago
I think our success is despite a lot of decisions (specifically the ones in the post), not because of them.
kakwa_ 3y ago
> We will never know, but what we know for sure is not implementing these advice made him successful "Made him successful" implies a causality that is a bit too strong. Indeed, we will never know for sure. Maybe implementing these advice would have impeded development of other critical areas of his business. Maybe it would have would have helped make is business more successful as he would have had a more reliable product. Or maybe the business impact would have been neutral, but would have resulted in better quality of life/less stress for him and his employees. But in general, the way I read this article is: they made good decisions overall, but as everything in the world, it was not optimal (switching platform too early, making some big mistakes like the credit card one, etc). It's a very interesting read nonetheless, with clear take away: * chose boring tech you know and focus on your product, not the tech, specially in the early days * grow your infrastructure and complexity with your product needs * accept you will mess-up but properly learn from it, and grow your organizational knowledge, structure and processes accordingly.

dom_hutton 3y ago

If I could have my time again I'd focus on enablement of the broader engineering group through an incident management process, readiness exercises, some aggregate incident analyses for the org to learn from and leaning into observability. Infrastructure is kind of a solved problem for common use cases today, just requires the expertise. With our ducks in a row I'd next have look to a GRC function for the compliance bits whilst splitting the platform engineers time between embedding engagements and tooling investments. You're on the right path man, I'd love to know what I know now back then but unfortunately time doesn't work like that.

fullstackchris 3y ago
> Infrastructure is kind of a solved problem for common use cases today, just requires the expertise. This is the problem however for many (older) companies. They either don't care, or quite literally don't know the infrastructure solutions out there which can save literally thousands of hours per year of headache. Sure, for many companies with legacy systems they have a "dont fix what isn't broken" mindset, but from what I've seen, I always ask, if shipping and modifying new versions of a system takes hours or even days to complete, is the system really not 'broken'? I guess I never realized it, but having automated and clean infrastructure with tests and uptime metrics is a must-have for me on anything I build going forward. Take 2-3 weeks to save months of headache.
ghiculescu 3y ago
Thanks. Means a lot, particularly coming from you.

shri_krishna 3y ago

Get a 403 Forbidden when I visit https://www.tanda.co/ https://www.tanda.co/

rathish_g 3y ago
It seems that the new PIT team was unable to manage the HN traffic.
- djvu97 3y ago
  They should switch back to digitalocean when things were manangeable

AndroTux 3y ago

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to SSH. This wasn’t totally their fault, but at the time it just all felt like a dodgy, shaky setup. This wasn’t *at all* their fault and there’s nothing dodgy about it. This was entirely your fault, and even mentioning this as a „partly“ negative for DO is imo very wrong. I imagine any other service provider, including Heroku, would have acted the same way. You can’t expect to mooch off a month of service without paying and without consequences. If you don’t pay your bills, service will be interrupted. It’s as simple as that.

tommica 3y ago
I read the "dodgy, shaky" parts as their own setup was that, not digital oceans
alberth 3y ago
But it kind of is. Merchant payment services exist that can automatically obtain the updated card-on-file credential when they are about to expire. Both Visa and Mastercard offer this directly, or you can get the same service from a PSP.
- TheNewsIsHere 3y ago
  It’s definitely not the merchant’s responsibility to catch when your card changes. We have this at work thanks to Stripe, and it’s wildly inconsistent. But beyond that, it won’t do anything for a closed/canceled card/cardholder account. That situation was fully on OP. This could have happened at GCP, AWS, Azure, Linode, OVH, Rackspace, Oracle Cloud, you name it.
  - TacticalCoder 3y ago
    > It’s definitely not the merchant’s responsibility to catch when your card changes. Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration. I mean: we're literally talking about credit cards often keeping the exact same owner name, the very same number and just having their MM/YY of expiration changed. It's not rocket science to update in a DB after an API call. And what's the catch on at least trying? If it works, you saved everybody time. If it doesn't work, it's not worse than your current "solution". I understand politicians and lawyer-minded people saying: "technically it's not our responsibility" but they're wasting everybody's time, leaving money on the table and wasting money on support. It's just poor judgment to react like that instead of thinking as to how life could be made better for everybody, starting with your paying customers (which, btw, are the reason you exist).

alexpls 3y ago

This really resonates. It's so easy to get caught up in chasing trends and overlook delivering customer value/stability. Good luck on the next 11 years :)

nicbou 3y ago

This is one of the best ops writeup I've read. People are rarely so vulnerable and relatable. Usually it's all "here's how we do things perfectly at our unicorn company". The most important lesson was about reducing the alerts and sleeping better. It's important to be able to leave your laptop at the office and not worry about things going wrong. This post gave me flashbacks of when I made websites for others and hosted them myself. Never again!

fullstackchris 3y ago

Great post, I've used all these services once or more at various points in my career. I'm still on the Digital Ocean stage, as I've found AWS is often overly complicated (or at least too much for me as a primarily solo founder) I'm curious about this line: > We were doubling the size of our customer base every 9 months, and pretty soon this meant we needed more servers. I've found that a moderately sized 4 core 'droplet' from Digital Ocean handles nearly anything I can throw at it (at least in terms of the web application world) - I see you product is doing employee management / scheduling - is there heavy compute / resources associated? Or is it more due to privacy requirements (i.e. you need a siloed instance / server per client?). Not trying to sound smug at all, just genuinely curious where the load comes from.

ghiculescu 3y ago
Great questions. There is some heavy compute - most of the real value is in our pay calculations and compliance features which are all based on algorithms that calculate correct pay based on legal requirements. But mostly it's just having lots of concurrent users. For a while a few droplets was plenty, and then one day we hit a tipping point where it was overwhelming.
spiffytech 3y ago
It seems like server performance has improved a lot over the last decade. The move from HDD -> SSD -> NVMe alone makes a huge difference for anything that touches disk. And my memory is that it was pricier to buy machines with more cores or RAM vs today. It wouldn't surprise me if today's budget VPS does the work of several machines from 10/11 years ago, and costs less too!

nologic01 3y ago

Something that would be interesting to compare and contrast is actually using your own hardware to host the service(s). Servers are super-reliable these days, it doesn't feel like a huge step from the VPS approach to simply have a bunch of them. At least one tradeoff is that you'd have to be physically in a location with adequate bandwidth / latency but for various use cases it might be very cost effective during some phase of the startup. I wonder also how the AI induced focus on specialized hardware might change the calculus

anonyfox 3y ago

After 20 years of dev, having devops skills nearly maximized (with a AWS specialization), built/maintained/scaled many products, ... can completely relate. I really do see hostings costs differently now compared to earlier in my career, when I was more junior/midlevel where one still argues about "cool" tech/languages/architectures. What you want from day one is a completely managed hosting for tech stack you work with, but are not hard tied to forever. Right now I am starting some new/modern meteor.js stuff again, there is meteor galaxy for hosting that "just works" for that stack, you bring your own mongodb (I prefer atlas itself, they even have serverless now), and everything is taken care of, including CI/CD/Monitoring/... . Its like a few minutes of initial configuration, and never think about it again, and if done correctly you shold have horizontal autoscale of some kind automatically nowadays. Yes, this is significantly more expensive than directly using AWS (which they use under the hood), and even though I am personally highly trained in this stuff+terraform/cdk/..., I don't want to have all this work anymore when I also can shell out a few hundred bucks per month instead. Just in case Galaxy becomes a problem for some reason, I _still_ can deploy the app stack to some VPS provider, but I'd use some already existing automation (like meteor-up in this case) instead of really digging into typical devops topics for it. There is a bad feeling in the mouth as an engineer to shell out "more than needed" for infrastructure, but my rule of thumb now is that I am happy to eat that frog as long as the potential cost saving is less than 2 infrastructure engineer FTEs, thats my trigger to _maybe_ discussing it.

nektro 3y ago

great writeup and congrats on 11 years!

jwr 3y ago

I read this article with interest, having run a SaaS for the last 8 years or so (solo-founder). I find it intriguing that the OP needed so much compute power. Computers are really powerful these days. As a counterpoint of sorts, here's my "hosting journey": * run everything on a single physical server rented at Hetzner (DE) * [... several years pass, business grows ...] * switch to ansible, learn it, spend a week or so to write automation for a 3-server setup, also learn terraform and write terraform configs for setting up a Digital Ocean system from scratch * run production on a 3-server setup at Hetzner DE, run a staging system at Hetzner FI, also serving as a possible quick manual failover, for a total of 6 physical servers, test re-initializing systems from scratch regularly, test setting up a Digital Ocean system from scratch regularly * [... several years pass ...] * that's it — I really can't see a need for more in the near future. But then, my software is not in Ruby on Rails, and I have no experience with that platform. I use Clojure and ClojureScript and I was careful to design everything to be rather client-heavy at the start. I also never wanted to depend on PaaS systems, mostly to avoid lock-in, but also because I don't buy the "just use our magic database offering and forget about database problems forever" selling pitch. You can sweep possible problems and complexity under the rug and hide them, but you can't run away from them. I also do not use Postgres (collective gasp in the audience), because having a single centralized point of failure is not something I want in my setup. I also never needed significant sysadmin/devops support. Granted, I do have some experience, but these things do not require dedicated teams, unless you are YouTube. A little ansible+terraform goes a long way, so does buying an hour of two of consulting from an experienced sysadmin. Those physical servers that I use are significantly faster than the over-subscribed cloud VM instances that you usually get from AWS and the like. And they have 64GB of RAM, not some measly amounts. If I need more servers, it takes on the order of hours to get additional ones, but I'm not sure what I'd use them for. My total hosting bill is on the order of 350€/month and is boringly predictable.

gen220 3y ago
Just curious, if you feel comfortable sharing, what do you use as your database? I'd agree Postgres is not the right answer to every problem, but the documentation of its failure modes and the mitigations thereof make it a "good" answer to "most" problems. I'm curious what the "problem" (in access pattern terms, doesn't have to be business terms) and "solution" (i.e. persistence technology) is in your setup!

RandomWorker 3y ago

This story resonates with me — apparently we are on phase two. And we need dedicated team members to take care of this. Thanks for sharing your lessons this was hugely valuable. Don’t let the negative comments get you down, when building things it’s hard and it’s all a big learning game. Congratulations on building a great piece of software.

ghiculescu 3y ago
Thanks! Reach out if you ever need a hand.

lakomen 3y ago

Those are such noob mistakes. No bigint OID, or worse, not UUIDv4, because even int64 will overflow, maybe not on their platform, but on a platform with millions of new inserts a day, despite being a large number, isn't future proof. If anyone knows a better 128bit alternative to UUIDv4, please respond. Side remark, that's what I was trying to tell the dgraph guy. If you have an int64 ID for ALL your transactions and all your IDs it will eventually overflow. It's a single incrementing value for ALL actions. Inset a new record, new ID, retrieve this record, new ID etc. Everything this guy writes about are newbie mistakes, stemming from a lack of "far sight", aka thinking 5 steps ahead. Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these. What annoys me is that I'm actually seeking a job now and that I can't find one that fits my needs, despite being way past those Kinderkrankheiten (child diseases), because those jobs nowadays require you to be an AWS zealot, which I'm not. I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part. I know this article is about Ruby. But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer, I would NOT go AWS lambda and complicate my life and, most importantly, the development process or as I call it the solution delivery process. This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations. There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters. The split between backend and frontend happened around 2013. Having a classic website with low performing Ruby, that also has such a bad workflow, e.g. Blog, you can't start with Comment, you have to start with Blog, then Post, then Comment. This is archaic. With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client. If I need SEO I'm back to the scaling problem, but the Go backend won't be the one that needs scaling. And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle. Doing this for the SSR JS frontend is where it gets expensive. So either try to static render with Svelte, if possible, to solve this problem and load balance or get locked into complex auto scaling cloud nonsense. And this is where it gets REALLY expensive. Your Ruby on Rails app might be good enough for a PoC, but that's about it. If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure. And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent. You can have your cake and eat it too. Just think it through from the start to the end.