16 ms·

Shopify Ruby on Rails distributed monolith runs 19M queries per second on MySQL

204 points by HatchedLake721 3y ago

ssd532 3y ago
What does distribute monolith mean? Is it just a monolith app deployment with distributed (master and replica) db servers in this case?
- pmontra 3y ago
  You might want to read this post from Shopify's engineering blog https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity https://shopify.engineering/deconstructing-monolith-designin...
- deleted 3y ago
  [deleted]
- CharlieDigital 3y ago
  I'm sure there are many variants and definitions, but the company I'm at runs one. The gist of it is that there's one codebase with multiple separate "modules". This codebase is packed and linked as a library and then we build different super slim hosts that load different parts of the monolith in production containers. Usually just different environment variables or config. But locally, we can run the whole thing in one process. We're.using .NET so `dotnet run` brings up the whole app. Whereas we might run parts of the app in different console hosts in prod, locally they are hosted background services in-process. From a debug perspective, this is super awesome since you can just launch and trace one codebase. If we broke it out into 3-4 separate services, we'd have to run 3 processes and 3 debuggers. 3 sets of configuration, 3 sets of CI/CD, 3 sets of testing. Terrible for productivity. We have parts of the system connected to SQS for processing events and if we need more throughout, we simply start more instances of the container all running the same monolith. I think GCP is probably one of the best platforms for building modular monoliths because of its tight orientation around HTTP push.
  - figassis 3y ago
    I implemented this in one of my past startup jobs. Basically a core banking system that implemented multiple roles of the system, including open banking. Depending on config, it would act as a bank, as an OB service provider, as an OB registry, as a merchant, etc. In addition, it was built in a way that instances, if allowed could talk to each other in 2 ways: 1. As part of the same entity, so you could scale your operation. 2. As part of an ecosystem, so you could for example create an entire open banking network or just a regular network with bank transfers and card payments using proper protocols such as ACH, ISO8583 and ISO2022 for example.
  - pshirshov 3y ago
    We call it "multi-tenant applications" or "role-based applications": https://www.youtube.com/live/Zk0Il6I5MQI?si=BFVL3JkHaj1hcGrZ https://www.youtube.com/live/Zk0Il6I5MQI?si=BFVL3JkHaj1hcGrZ And there is a separate concept of a configurable application which can completely reshape its component graph according to some high level configuration flags (like database=prod|dummy), we call it "multi-modal applications". And we created the perfect tool for wiring them: https://izumi.7mind.io/distage/index.html https://izumi.7mind.io/distage/index.html
  - alserio 3y ago
    I'm looking to break an old java monolith application in something like that: modular code base, single deployment artifact, multiple configurable use cases. I've yet to find a good existing tool to compose the application at build time, other than using a godawful lot of maven profiles combinations
    - CharlieDigital 3y ago
      Hmm; at least in the more recent versions of .NET, Microsoft has really cleaned up the runtime host paradigm so that it's consistent across console (think background services, timer jobs, pull-oriented processing) and web (classic HTTP). For us, it just becomes a matter of configuring the correct construction of the dependency injection container at host startup using some flag (usually environment variable) to pick the right bits and pieces to load into the container and which services to run from the monolith. Then each of the host "partitions" gets its own Dockerfile.
- konart 3y ago
  Often this is view as counter microservices pattern. Microservice architecture assumes each service has its own data storage and is logically independable from other services. Monolith on the other hand is a single application with all of the logic in one place. Distibuted monolith is a set of applications\services like in microservices pattern but they can share common data storage and depend on each other.
  - mlrtime 3y ago
    Data Storage and Compute should be separate orthogonal issues, it's not needed in this comparison. Stateful vs Stateless. Your monolith is a binary that gets distributed to hosts to perform some function. The binary has multiple entry points that can be envoked. Most calls are via internal library call. Microserverices (also stateless) have a different artifacts for each component, services call other services via a private API (often grpc/httprc).
    - konart 3y ago
      Good point. I was just trying to (awkwardly) say that monolith can also be split into a separate parts communicating via a private API.
aunwick 3y ago
My Master's thesis was on high performance distributed computing. And my conclusion was you likely don't have a problem that is hard enough to justify it. Thank you for promoting reality!
- ajmurmann 3y ago
  How is that the case? This example uses a distributed MySQL cluster which was of course tuned for high performance. Similarly the Rails app is distributed as well. Arguably the Rails app likely wouldn't qualify as "high performance", but it's distributed.
  - ericb 3y ago
    Aren't the full set of these numbers definitionally "high performance"?
    - deleted 3y ago
      [deleted]
    - ajmurmann 3y ago
      Oh, I read the parent comment to thank them for confirming that "you likely don't have a problem that is hard enough to justify it". But reading it again, it could be read both ways. Edit: To be clear, I agree that this is an example of distributed, high-performance which is why the comment made little sense to me.
  - gls2ro 3y ago
    It amazes me that even when we have numbers people still dismiss Rails. Nothing will run at that scale on a single VPS. All companies will have a wide range of languages used. If this is not Rails supporting high traffic then what do we need more?
    - chris12321 3y ago
      For some reason, some people with discount any example of Rails scalability as not counting.
jamil7 3y ago
I guess the more interesting data point is not how many queries it served but a what the cost was for serving them.
- pmontra 3y ago
  Ultimately the interesting thing to know is why they decided to stay on Rails instead of rewriting everything in a more performant language.
  - aaronbrethorst 3y ago
    Joel Spolsky wrote about it, actually: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
  - ericb 3y ago
    CPU cost is the obvious driver of who will win in this space, ergo, the correct optimization is to rewrite in assembly.
  - iimblack 3y ago
    Don’t they also have a ton of custom ruby stuff like their own jit?
    - campbel 3y ago
      They employee the creator of yjit https://shopify.engineering/yjit-just-in-time-compiler-cruby https://shopify.engineering/yjit-just-in-time-compiler-cruby
    - riffraff 3y ago
      Shopify did develop a jit, but it's part of the main ruby runtime (they did other performance improvement too, all available in the standard ruby).
  - abhibeckert 3y ago
    Modern CPUs are fast, and it's trivial to scale compute instances if you need that. The database would be the bottleneck and that's got nothing to do with rails - it's all mysql.
whalesalad 3y ago
Would love to see the infrastructure bill. Anything is possible if you throw enough $$$ at it. I’m not a ruby hater, but the average joe can’t accomplish this. If you consider each store an individual instance with an isolated/shard of a db it makes sense. But the underlying foundation is immense. Partitions are usually the key to scale. I’d imagine parts of their infrastructure would be better served by different runtimes. They’d save a lot of money. But if your entire team is hyper focused on ruby there is something to be said for a huge monolith.
- MuffinFlavored 3y ago
  > If you consider each store an individual instance with an isolated/shard of a db it makes sense. But the underlying foundation is immense. Each instance is probably just a k8s pod /s
  - deleted 3y ago
    [deleted]
- SkyPuncher 3y ago
  Most software organizations (remember most are not FAANG) will be better served by paying the $$$ to have their engineers focused on building more/better features for customers.
  - whalesalad 3y ago
    True up until a point. Once you plateau and your infra spend is $25-50k+ per month it’s wise to start re architecting.
    - aledalgrande 3y ago
      Rearchitecting doesn't necessarily entail changing technology/platform
- davidw 3y ago
  > the average joe can’t accomplish this The average joe does not need this. If you have decent margins on the pages you're serving, Rails is fine. Where you might want to investigate other things is if you're, say, an ad driven business with really thin margins and you want to minimize costs. Or if you've got things dialed and you're just not changing things much any more and you want to eke out some savings. And in any event, Rails is a good choice to figure out the problem space you're working in. Even Twitter started with it, and objectively, Twitter is very much not the sweet spot for Rails.
oopsthrowpass 3y ago
blasphemy, this should immediately be moved to microservices and AWS and cost optimized and each component should be scaled separately (maybe some components only need 0.5 CPU, think of the cost saving t3.micro could bring and you have a highly available 3AZ potato to make sure it never goes down!!) Then to handle any load we need to build autoscaling and spin up the toaster to medium potato and 20 instances (this costs 60k a month, but no worries we only pay for what we use so we will only run this for 27 minutes during our big sale). Oh what wonderful world we live in and the pain we inflict on ourselves. GJ Shopify for running a sane (tm) tech stack :) (btw of course Rails scales it's shared nothing setup, spin up infinite app servers as long as the db can handle it. It's pretty expensive for compute though)
- stevebmark 3y ago
  Maybe you haven't worked on a large production Rails app? Rails takes 3-4x the infrastructure hosting costs as other languages/frameworks. Not to mention the insane testing costs from Ruby test bloat.
  - briantakita 3y ago
    I got burnt out on Rails after the third app in a row that I was responsible for upgrading. I appreciate Rails' contribution to web development. It took about a decade for the front end framework-library ecosystem to figure out that MPAs are more effective than SPAs for most apps. Fortunately, Astrojs is part of the ecosystem. It's sortof like the fullstack JS version of Rails/Sinatra...without the ORM which IMO adds incidental complexity. If you have a thriving business already using Rails, it's difficult to justify moving off of Rails...now that painful upgrades seem to be mostly in the past. However, I do find isomorphic JS components & state management to be a pleasure to develop & maintain compared developing an app in two languages.
    - throwaway2990 3y ago
      I had to upgrade a project from rails 4.2 to 5.0 and it took like 3 weeks because things just broke randomly. Upgraded a .NET 4.7 project to .NET 6 in 3 days and haven’t touched it since. And the .NET project was larger. I haven’t done rails since that upgrade and I hope i never have to touch it again.
Reptur 3y ago
They may be bragging about that, but they still make people use Liquid and tell their customers that features were released when they're in beta for another 3-6 months.
- stanislavb 3y ago
  What's wrong with Liquid?
  - Reptur 3y ago
    It can't loop arrays with indices from a variable for one. Its basically a 90s scripting language that has had no growth.
- softwaredoug 3y ago
  Liquid is pretty phenomenal piece of engineering IMO
8f2ab37a-ed6c 3y ago
My impression is that it's simply harder to get promoted as an engineer in the industry by using boring, sustainable, unexciting solutions that have been used by everybody else and their dog. How do you even stand out that way? Looks bad on the resume, like you didn't even try. Great for the business, but maybe terrible for one's career? You could turn that one Rails app into a complex microservices architecture and do a conference talk about it, and get a promotion. Then you can undo the microservices architecture, write a blog post about returning to the majestic monolith, and do another conference talk about it, maybe get another promotion. Abstract, de-abstract, bundle, unbundle, rinse and repeat. It feels like a tragic situation that's nobody's fault, just the reality of human psychology being wired to reward the wrong things.
- ankitrgadiya 3y ago
  I absolutely agree with this. I sometimes use the phrase optimising for the resume.
  - 000ooo000 3y ago
    I like to think I coined 'Resume Driven Development' but chances are I'm dev #3747 to say that phrase
    - arp242 3y ago
      #628 https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%22Resume%20Driven%20Development%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
      - 000ooo000 3y ago
        Hey #628 is way better! Thanks!
  - stuxnet79 3y ago
    > optimising for the resume. Otherwise known as RDD (Resume Driven Development) - https://rdd.io https://rdd.io We value: - Specific technologies over working solutions - Hiring buzzwords over proven track records - Creative job titles over technical experience - Reacting to trends over more pragmatic options
ipsum2 3y ago
Neat, it seems like Shopify's product design choices make it easy to shard across database servers: https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale https://shopify.engineering/mysql-database-shard-balancing-t...
zombiwoof 3y ago
imagine what they could do with a real stack and programming language ;-)
- AHTERIX5000 3y ago
  Such as?
darth_avocado 3y ago
This is great, but it would also be great if they accept other solutions than monolith Ruby on Rails on MySQL for their systems design interview rounds. I’ve interviewed twice and there seems to be a resistance from engineers both the times I argued that MySQL wasn’t the right approach. Sure, MySQL could possibly run any use case possible in the world if you throw enough engineering resources at it and highly optimize for that use case, but why do that if there is another engine specifically designed for your use case. The argument was “Shopify runs on MySQl and we can handle millions of queries… blah… blah…”.
- pas 3y ago
  what do they use to shard MySQL? are they using upstream? fb's fork? what storage engine? innodb? myrocks? at millions of q/s "mySQL" is a bit meaningless :D
- vidarh 3y ago
  I dislike MySQL as much as the next guy, but if you go into an interview-situation and argue that their technology choices are wrong then that is a big red flag not so much because you're wrong, because you might be 100% right, but because it is likely to indicate to interviewers that you will be difficult to work with, possibly pushing for changes they don't want to happen, and that you won't read social and political situations within the team well - for starters by arguing with an interviewer. It's fine to argue with interviewers, but the threshold where it indicates to both sides that you're probably wrong for that job is pretty low.
  - intelVISA 3y ago
    As much as producing 'technically sound' decisions is paramount so is being able to work with other people, namely your boss whose decisions are always technically sound if asked. ;)
    - vidarh 3y ago
      I think if asked it's fine to probe how much disagreement your boss has the stomach for. Sometimes you're being brought on because they know they need to fix things. But, yeah, don't start to rip apart technical decisions they appear wedded to from the first moment. Not least because a lot of the time as you say 'technically sound' decisions aren't everything. A whole lot of technical decisions we - me included - have very strong opinions about don't really have that much of a measurable effect on technical outcomes. I once, many years ago, had a conflict with someone reporting to me because I refused to entertain rewriting our entire frontend in Rails. This was just after Rails was released, and we had a lot of PHP code. We had PHP code because PHP frontend devs were "cheap" and plentiful, not because any of us liked PHP. I agreed with him about the preference for Ruby, and we used Ruby for other things (ironically all our Ruby use was on the backend), but he kept pushing on the basis of no insight into the relative market conditions for hiring at that time and on the basis of making assumptions about how "trivial" a rewrite was because he had no insight into why our then-current frontend had all of the capabilities it had which he'd chosen to ignore because he didn't know the roadmap. He went to my boss - the CEO and co-founder - and tried to get me fired. The CEO went to me and asked if I wanted to fire him instead. I didn't, but I did have a rather serious chat with said developer about our respective roles, and how it was about more than technical preferences, and how he might get a lot further if he actually tried to work more constructively with me instead of thinking he had the clout to get me fired. Said CEO hired me to run a development department again in my previous job, while he to my knowledge has never again worked with that developer - looking like a troublemaker to the wrong person can have long term effects.
imheretolearn 3y ago
> But Rails doesn't scale so what are we even doing This without context is meaningless. What is the cost in $$ and engineering time to scale to that level? Would a native image be able to scale to the same level at half the total cost?
- gls2ro 3y ago
  You seem to imply that you know better than the CEO if Rails is a good fit both from product and cost perspective. Unless you are the CFO or have inside information I will be skeptical of the idea that I - an outsider of Shopify - would know better than them what would works best for them in terms of tech stack and costs savings. Even if I would be a consultant for them I will first try to understand the current situation and then imply a native image will have half the total cost is a good solution. What if reducing the hosting costs will actually damage their speed of pivoting and adapting to changes?
  - charcircuit 3y ago
    The ability to scale is independent from the cost to build / change that thing.
wg0 3y ago
Oh no I want my pizza size teams with five thousand Microservices written in Clojure, Haskel, Erlang and Scala. The cost argument about this monolith is just a straw to clutch at. Microservices are not cheaper than a monolith. Operationally or infrastructure wise. Logs, monitoring, tracing and what not for each Microservice.
- pas 3y ago
  microservices are a solution to certain problems. if the stack is already pretty diverse, needs a lot of separate teams, hiring is hard, coordinating deployments is hard, etc, then it makes sense. Shopify probably looks more like what you ridiculed than not. we can guess that it's not one big team, and it's not hundreds of identical copies of this big monolith (but configured during deployment to run in different roles).
  - gls2ro 3y ago
    > if the stack is already pretty diverse, needs a lot of separate teams, hiring is hard, coordinating deployments is hard, etc, then it makes sense. So what you are saying is that microservices are not a solution to a technical problem but a solution to organizational problems?
    - pas 3y ago
      I'm saying that microservices as a tech-cultural phenomena (or even era) was a response to a very specific set of constraints. The whole cloud thing was new, hiring was hard, expertise was scarce, business was booming so loud people got deaf from simply thinking about it, the church of true scalability was omnipotent, tools were crude, marvelously maintained monorepos were only artifacts of seriously wet [FAAN]G-fueled dreams, etc. It made sense for Netflix, because they had a big Cassandra cluster, too much money, and a very picky organizational/hiring culture, and so on. https://www.infoq.com/presentations/microservices-netflix-industry/ https://www.infoq.com/presentations/microservices-netflix-in... (https://www.youtube.com/watch?v=TOM6UhCetQ0 https://www.youtube.com/watch?v=TOM6UhCetQ0)
firefoxd 3y ago
I worked at a fortune 10, backing up our mysql db daily took 8 hours. We had a job that processed data that also took 8 hours. We had a small window of time during work hours to fetch the data, if you happened to query when either service was running, you'd probably not get any data back. Few months in, both services reached 12 hours to run. We had to change the start time just so they'd happen back to back. I rolled up my sleeves and got to work on the db. A week later, the job's runtime was down to 2 hours. I was nearly promoted out of the job. I kept making improvements to the database, reorganizing the application, and optimizing. One day I manually ran the job, then went to get coffee. When i came back, prompt was awaiting my next command. I thought it had silently failed. I ran it several times throughout the day and checked the response. It ran in ~17 minutes. The backup was also reduced to less than 20 minutes. This was all on mysql 5.7 We literally gained 23 hours of availability. We had no idea what to do with it. I was fired shortly after.
- appplication 3y ago
  Ha, that was quite a twist at the end there. Was this due to your self-created redundancy?
  - soks86 3y ago
    or embarrassment and concern on the part of the people who created this system before him. I've made similar improvements (40x speedup, 8x reduction in RAM footprint) and had been shown the door. This was for software that was a bit over 30 years old too so it was a bit involved. The team winning does not help a narcissist feel better. You'd be better off getting nothing done while stroking their ego daily.
    - alserio 3y ago
      it seems like they did you a solid showing you that you were wasted working in that environment
  - thinkingemote 3y ago
    "job security through database complexity"
OfficeChad 3y ago
Gigachad Monolith vs virgin micro service.
herbst 3y ago
Was running a rails monolith with up to 300 concurrent users and a huge growing database on a $20 VPS and free cloudflare. Sure it only had highly optimized queries (something rails makes kinda easy too), but that's what you are supposed to do anyway. There wasn't ever a need to recode to run it on a $10 instance. Nether do I think 99% of all database driven projects, including mine, are ever going to reach as many concurrent users. IMO: talking about performance for ANY major coding stack is just premature optimization
TheRealDunkirk 3y ago
Why do so many people on this forum have kittens whenever someone points to a success story about Rails and Ruby? It's like the hamburger people can't stand it when someone enjoys a plain hotdog with ketchup and mustard. It's tiring. Get a life, and just enjoy your bland, overpriced, gourmet, too-big-to-fit-in-your-mouth, Instagrammed burgers.
- ek750 3y ago
  heh, I know right? I see all those comments shitting on ruby and rails, and it's like, yeah, I get it dude. They hate rails, but luckily nobody is forcing them to use it. The more choices the better. Some people hate dynamic types, some people hate static types. There's room for both, imo. It's not like it's a secret there are faster or more efficient stacks. I guess they can't help themselves and need to make their position known. Personally, I love ruby, and rails is pretty good too.
mortallywounded 3y ago
Cool, MySQL scales. What does RoR have to do with it?
quazar987 3y ago
A while ago I read shopify hosts stores in pods. Pods are self contained instances that runs everything inside it, that includes mysql, redis, etc. does that mean this number is collective counting all the pods and not a single database?

What does distribute monolith mean? Is it just a monolith app deployment with distributed (master and replica) db servers in this case?

pmontra 3y ago
You might want to read this post from Shopify's engineering blog https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity https://shopify.engineering/deconstructing-monolith-designin...
deleted 3y ago
[deleted]
CharlieDigital 3y ago
I'm sure there are many variants and definitions, but the company I'm at runs one. The gist of it is that there's one codebase with multiple separate "modules". This codebase is packed and linked as a library and then we build different super slim hosts that load different parts of the monolith in production containers. Usually just different environment variables or config. But locally, we can run the whole thing in one process. We're.using .NET so `dotnet run` brings up the whole app. Whereas we might run parts of the app in different console hosts in prod, locally they are hosted background services in-process. From a debug perspective, this is super awesome since you can just launch and trace one codebase. If we broke it out into 3-4 separate services, we'd have to run 3 processes and 3 debuggers. 3 sets of configuration, 3 sets of CI/CD, 3 sets of testing. Terrible for productivity. We have parts of the system connected to SQS for processing events and if we need more throughout, we simply start more instances of the container all running the same monolith. I think GCP is probably one of the best platforms for building modular monoliths because of its tight orientation around HTTP push.
- figassis 3y ago
  I implemented this in one of my past startup jobs. Basically a core banking system that implemented multiple roles of the system, including open banking. Depending on config, it would act as a bank, as an OB service provider, as an OB registry, as a merchant, etc. In addition, it was built in a way that instances, if allowed could talk to each other in 2 ways: 1. As part of the same entity, so you could scale your operation. 2. As part of an ecosystem, so you could for example create an entire open banking network or just a regular network with bank transfers and card payments using proper protocols such as ACH, ISO8583 and ISO2022 for example.
- pshirshov 3y ago
  We call it "multi-tenant applications" or "role-based applications": https://www.youtube.com/live/Zk0Il6I5MQI?si=BFVL3JkHaj1hcGrZ https://www.youtube.com/live/Zk0Il6I5MQI?si=BFVL3JkHaj1hcGrZ And there is a separate concept of a configurable application which can completely reshape its component graph according to some high level configuration flags (like database=prod|dummy), we call it "multi-modal applications". And we created the perfect tool for wiring them: https://izumi.7mind.io/distage/index.html https://izumi.7mind.io/distage/index.html
- alserio 3y ago
  I'm looking to break an old java monolith application in something like that: modular code base, single deployment artifact, multiple configurable use cases. I've yet to find a good existing tool to compose the application at build time, other than using a godawful lot of maven profiles combinations
  - CharlieDigital 3y ago
    Hmm; at least in the more recent versions of .NET, Microsoft has really cleaned up the runtime host paradigm so that it's consistent across console (think background services, timer jobs, pull-oriented processing) and web (classic HTTP). For us, it just becomes a matter of configuring the correct construction of the dependency injection container at host startup using some flag (usually environment variable) to pick the right bits and pieces to load into the container and which services to run from the monolith. Then each of the host "partitions" gets its own Dockerfile.
konart 3y ago
Often this is view as counter microservices pattern. Microservice architecture assumes each service has its own data storage and is logically independable from other services. Monolith on the other hand is a single application with all of the logic in one place. Distibuted monolith is a set of applications\services like in microservices pattern but they can share common data storage and depend on each other.
- mlrtime 3y ago
  Data Storage and Compute should be separate orthogonal issues, it's not needed in this comparison. Stateful vs Stateless. Your monolith is a binary that gets distributed to hosts to perform some function. The binary has multiple entry points that can be envoked. Most calls are via internal library call. Microserverices (also stateless) have a different artifacts for each component, services call other services via a private API (often grpc/httprc).
  - konart 3y ago
    Good point. I was just trying to (awkwardly) say that monolith can also be split into a separate parts communicating via a private API.

aunwick 3y ago

My Master's thesis was on high performance distributed computing. And my conclusion was you likely don't have a problem that is hard enough to justify it. Thank you for promoting reality!

ajmurmann 3y ago
How is that the case? This example uses a distributed MySQL cluster which was of course tuned for high performance. Similarly the Rails app is distributed as well. Arguably the Rails app likely wouldn't qualify as "high performance", but it's distributed.
- ericb 3y ago
  Aren't the full set of these numbers definitionally "high performance"?
  - deleted 3y ago
    [deleted]
  - ajmurmann 3y ago
    Oh, I read the parent comment to thank them for confirming that "you likely don't have a problem that is hard enough to justify it". But reading it again, it could be read both ways. Edit: To be clear, I agree that this is an example of distributed, high-performance which is why the comment made little sense to me.
- gls2ro 3y ago
  It amazes me that even when we have numbers people still dismiss Rails. Nothing will run at that scale on a single VPS. All companies will have a wide range of languages used. If this is not Rails supporting high traffic then what do we need more?
  - chris12321 3y ago
    For some reason, some people with discount any example of Rails scalability as not counting.

jamil7 3y ago

I guess the more interesting data point is not how many queries it served but a what the cost was for serving them.

pmontra 3y ago
Ultimately the interesting thing to know is why they decided to stay on Rails instead of rewriting everything in a more performant language.
- aaronbrethorst 3y ago
  Joel Spolsky wrote about it, actually: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
- ericb 3y ago
  CPU cost is the obvious driver of who will win in this space, ergo, the correct optimization is to rewrite in assembly.
- iimblack 3y ago
  Don’t they also have a ton of custom ruby stuff like their own jit?
  - campbel 3y ago
    They employee the creator of yjit https://shopify.engineering/yjit-just-in-time-compiler-cruby https://shopify.engineering/yjit-just-in-time-compiler-cruby
  - riffraff 3y ago
    Shopify did develop a jit, but it's part of the main ruby runtime (they did other performance improvement too, all available in the standard ruby).
- abhibeckert 3y ago
  Modern CPUs are fast, and it's trivial to scale compute instances if you need that. The database would be the bottleneck and that's got nothing to do with rails - it's all mysql.

whalesalad 3y ago

Would love to see the infrastructure bill. Anything is possible if you throw enough $$$ at it. I’m not a ruby hater, but the average joe can’t accomplish this. If you consider each store an individual instance with an isolated/shard of a db it makes sense. But the underlying foundation is immense. Partitions are usually the key to scale. I’d imagine parts of their infrastructure would be better served by different runtimes. They’d save a lot of money. But if your entire team is hyper focused on ruby there is something to be said for a huge monolith.

MuffinFlavored 3y ago
> If you consider each store an individual instance with an isolated/shard of a db it makes sense. But the underlying foundation is immense. Each instance is probably just a k8s pod /s
- deleted 3y ago
  [deleted]
SkyPuncher 3y ago
Most software organizations (remember most are not FAANG) will be better served by paying the $$$ to have their engineers focused on building more/better features for customers.
- whalesalad 3y ago
  True up until a point. Once you plateau and your infra spend is $25-50k+ per month it’s wise to start re architecting.
  - aledalgrande 3y ago
    Rearchitecting doesn't necessarily entail changing technology/platform
davidw 3y ago
> the average joe can’t accomplish this The average joe does not need this. If you have decent margins on the pages you're serving, Rails is fine. Where you might want to investigate other things is if you're, say, an ad driven business with really thin margins and you want to minimize costs. Or if you've got things dialed and you're just not changing things much any more and you want to eke out some savings. And in any event, Rails is a good choice to figure out the problem space you're working in. Even Twitter started with it, and objectively, Twitter is very much not the sweet spot for Rails.

oopsthrowpass 3y ago

blasphemy, this should immediately be moved to microservices and AWS and cost optimized and each component should be scaled separately (maybe some components only need 0.5 CPU, think of the cost saving t3.micro could bring and you have a highly available 3AZ potato to make sure it never goes down!!) Then to handle any load we need to build autoscaling and spin up the toaster to medium potato and 20 instances (this costs 60k a month, but no worries we only pay for what we use so we will only run this for 27 minutes during our big sale). Oh what wonderful world we live in and the pain we inflict on ourselves. GJ Shopify for running a sane (tm) tech stack :) (btw of course Rails scales it's shared nothing setup, spin up infinite app servers as long as the db can handle it. It's pretty expensive for compute though)

stevebmark 3y ago
Maybe you haven't worked on a large production Rails app? Rails takes 3-4x the infrastructure hosting costs as other languages/frameworks. Not to mention the insane testing costs from Ruby test bloat.
- briantakita 3y ago
  I got burnt out on Rails after the third app in a row that I was responsible for upgrading. I appreciate Rails' contribution to web development. It took about a decade for the front end framework-library ecosystem to figure out that MPAs are more effective than SPAs for most apps. Fortunately, Astrojs is part of the ecosystem. It's sortof like the fullstack JS version of Rails/Sinatra...without the ORM which IMO adds incidental complexity. If you have a thriving business already using Rails, it's difficult to justify moving off of Rails...now that painful upgrades seem to be mostly in the past. However, I do find isomorphic JS components & state management to be a pleasure to develop & maintain compared developing an app in two languages.
  - throwaway2990 3y ago
    I had to upgrade a project from rails 4.2 to 5.0 and it took like 3 weeks because things just broke randomly. Upgraded a .NET 4.7 project to .NET 6 in 3 days and haven’t touched it since. And the .NET project was larger. I haven’t done rails since that upgrade and I hope i never have to touch it again.

Reptur 3y ago

They may be bragging about that, but they still make people use Liquid and tell their customers that features were released when they're in beta for another 3-6 months.

stanislavb 3y ago
What's wrong with Liquid?
- Reptur 3y ago
  It can't loop arrays with indices from a variable for one. Its basically a 90s scripting language that has had no growth.
softwaredoug 3y ago
Liquid is pretty phenomenal piece of engineering IMO

8f2ab37a-ed6c 3y ago

My impression is that it's simply harder to get promoted as an engineer in the industry by using boring, sustainable, unexciting solutions that have been used by everybody else and their dog. How do you even stand out that way? Looks bad on the resume, like you didn't even try. Great for the business, but maybe terrible for one's career? You could turn that one Rails app into a complex microservices architecture and do a conference talk about it, and get a promotion. Then you can undo the microservices architecture, write a blog post about returning to the majestic monolith, and do another conference talk about it, maybe get another promotion. Abstract, de-abstract, bundle, unbundle, rinse and repeat. It feels like a tragic situation that's nobody's fault, just the reality of human psychology being wired to reward the wrong things.

ankitrgadiya 3y ago
I absolutely agree with this. I sometimes use the phrase optimising for the resume.
- 000ooo000 3y ago
  I like to think I coined 'Resume Driven Development' but chances are I'm dev #3747 to say that phrase
  - arp242 3y ago
    #628 https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%22Resume%20Driven%20Development%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
    - 000ooo000 3y ago
      Hey #628 is way better! Thanks!
- stuxnet79 3y ago
  > optimising for the resume. Otherwise known as RDD (Resume Driven Development) - https://rdd.io https://rdd.io We value: - Specific technologies over working solutions - Hiring buzzwords over proven track records - Creative job titles over technical experience - Reacting to trends over more pragmatic options

ipsum2 3y ago

Neat, it seems like Shopify's product design choices make it easy to shard across database servers: https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale https://shopify.engineering/mysql-database-shard-balancing-t...

zombiwoof 3y ago

imagine what they could do with a real stack and programming language ;-)

AHTERIX5000 3y ago
Such as?

darth_avocado 3y ago

This is great, but it would also be great if they accept other solutions than monolith Ruby on Rails on MySQL for their systems design interview rounds. I’ve interviewed twice and there seems to be a resistance from engineers both the times I argued that MySQL wasn’t the right approach. Sure, MySQL could possibly run any use case possible in the world if you throw enough engineering resources at it and highly optimize for that use case, but why do that if there is another engine specifically designed for your use case. The argument was “Shopify runs on MySQl and we can handle millions of queries… blah… blah…”.

pas 3y ago
what do they use to shard MySQL? are they using upstream? fb's fork? what storage engine? innodb? myrocks? at millions of q/s "mySQL" is a bit meaningless :D
vidarh 3y ago
I dislike MySQL as much as the next guy, but if you go into an interview-situation and argue that their technology choices are wrong then that is a big red flag not so much because you're wrong, because you might be 100% right, but because it is likely to indicate to interviewers that you will be difficult to work with, possibly pushing for changes they don't want to happen, and that you won't read social and political situations within the team well - for starters by arguing with an interviewer. It's fine to argue with interviewers, but the threshold where it indicates to both sides that you're probably wrong for that job is pretty low.
- intelVISA 3y ago
  As much as producing 'technically sound' decisions is paramount so is being able to work with other people, namely your boss whose decisions are always technically sound if asked. ;)
  - vidarh 3y ago
    I think if asked it's fine to probe how much disagreement your boss has the stomach for. Sometimes you're being brought on because they know they need to fix things. But, yeah, don't start to rip apart technical decisions they appear wedded to from the first moment. Not least because a lot of the time as you say 'technically sound' decisions aren't everything. A whole lot of technical decisions we - me included - have very strong opinions about don't really have that much of a measurable effect on technical outcomes. I once, many years ago, had a conflict with someone reporting to me because I refused to entertain rewriting our entire frontend in Rails. This was just after Rails was released, and we had a lot of PHP code. We had PHP code because PHP frontend devs were "cheap" and plentiful, not because any of us liked PHP. I agreed with him about the preference for Ruby, and we used Ruby for other things (ironically all our Ruby use was on the backend), but he kept pushing on the basis of no insight into the relative market conditions for hiring at that time and on the basis of making assumptions about how "trivial" a rewrite was because he had no insight into why our then-current frontend had all of the capabilities it had which he'd chosen to ignore because he didn't know the roadmap. He went to my boss - the CEO and co-founder - and tried to get me fired. The CEO went to me and asked if I wanted to fire him instead. I didn't, but I did have a rather serious chat with said developer about our respective roles, and how it was about more than technical preferences, and how he might get a lot further if he actually tried to work more constructively with me instead of thinking he had the clout to get me fired. Said CEO hired me to run a development department again in my previous job, while he to my knowledge has never again worked with that developer - looking like a troublemaker to the wrong person can have long term effects.

imheretolearn 3y ago

> But Rails doesn't scale so what are we even doing This without context is meaningless. What is the cost in $$ and engineering time to scale to that level? Would a native image be able to scale to the same level at half the total cost?

gls2ro 3y ago
You seem to imply that you know better than the CEO if Rails is a good fit both from product and cost perspective. Unless you are the CFO or have inside information I will be skeptical of the idea that I - an outsider of Shopify - would know better than them what would works best for them in terms of tech stack and costs savings. Even if I would be a consultant for them I will first try to understand the current situation and then imply a native image will have half the total cost is a good solution. What if reducing the hosting costs will actually damage their speed of pivoting and adapting to changes?
- charcircuit 3y ago
  The ability to scale is independent from the cost to build / change that thing.

wg0 3y ago

Oh no I want my pizza size teams with five thousand Microservices written in Clojure, Haskel, Erlang and Scala. The cost argument about this monolith is just a straw to clutch at. Microservices are not cheaper than a monolith. Operationally or infrastructure wise. Logs, monitoring, tracing and what not for each Microservice.

pas 3y ago
microservices are a solution to certain problems. if the stack is already pretty diverse, needs a lot of separate teams, hiring is hard, coordinating deployments is hard, etc, then it makes sense. Shopify probably looks more like what you ridiculed than not. we can guess that it's not one big team, and it's not hundreds of identical copies of this big monolith (but configured during deployment to run in different roles).
- gls2ro 3y ago
  > if the stack is already pretty diverse, needs a lot of separate teams, hiring is hard, coordinating deployments is hard, etc, then it makes sense. So what you are saying is that microservices are not a solution to a technical problem but a solution to organizational problems?
  - pas 3y ago
    I'm saying that microservices as a tech-cultural phenomena (or even era) was a response to a very specific set of constraints. The whole cloud thing was new, hiring was hard, expertise was scarce, business was booming so loud people got deaf from simply thinking about it, the church of true scalability was omnipotent, tools were crude, marvelously maintained monorepos were only artifacts of seriously wet [FAAN]G-fueled dreams, etc. It made sense for Netflix, because they had a big Cassandra cluster, too much money, and a very picky organizational/hiring culture, and so on. https://www.infoq.com/presentations/microservices-netflix-industry/ https://www.infoq.com/presentations/microservices-netflix-in... (https://www.youtube.com/watch?v=TOM6UhCetQ0 https://www.youtube.com/watch?v=TOM6UhCetQ0)

firefoxd 3y ago

I worked at a fortune 10, backing up our mysql db daily took 8 hours. We had a job that processed data that also took 8 hours. We had a small window of time during work hours to fetch the data, if you happened to query when either service was running, you'd probably not get any data back. Few months in, both services reached 12 hours to run. We had to change the start time just so they'd happen back to back. I rolled up my sleeves and got to work on the db. A week later, the job's runtime was down to 2 hours. I was nearly promoted out of the job. I kept making improvements to the database, reorganizing the application, and optimizing. One day I manually ran the job, then went to get coffee. When i came back, prompt was awaiting my next command. I thought it had silently failed. I ran it several times throughout the day and checked the response. It ran in ~17 minutes. The backup was also reduced to less than 20 minutes. This was all on mysql 5.7 We literally gained 23 hours of availability. We had no idea what to do with it. I was fired shortly after.

appplication 3y ago
Ha, that was quite a twist at the end there. Was this due to your self-created redundancy?
- soks86 3y ago
  or embarrassment and concern on the part of the people who created this system before him. I've made similar improvements (40x speedup, 8x reduction in RAM footprint) and had been shown the door. This was for software that was a bit over 30 years old too so it was a bit involved. The team winning does not help a narcissist feel better. You'd be better off getting nothing done while stroking their ego daily.
  - alserio 3y ago
    it seems like they did you a solid showing you that you were wasted working in that environment
- thinkingemote 3y ago
  "job security through database complexity"

OfficeChad 3y ago

Gigachad Monolith vs virgin micro service.

herbst 3y ago

Was running a rails monolith with up to 300 concurrent users and a huge growing database on a $20 VPS and free cloudflare. Sure it only had highly optimized queries (something rails makes kinda easy too), but that's what you are supposed to do anyway. There wasn't ever a need to recode to run it on a $10 instance. Nether do I think 99% of all database driven projects, including mine, are ever going to reach as many concurrent users. IMO: talking about performance for ANY major coding stack is just premature optimization

TheRealDunkirk 3y ago

Why do so many people on this forum have kittens whenever someone points to a success story about Rails and Ruby? It's like the hamburger people can't stand it when someone enjoys a plain hotdog with ketchup and mustard. It's tiring. Get a life, and just enjoy your bland, overpriced, gourmet, too-big-to-fit-in-your-mouth, Instagrammed burgers.

ek750 3y ago
heh, I know right? I see all those comments shitting on ruby and rails, and it's like, yeah, I get it dude. They hate rails, but luckily nobody is forcing them to use it. The more choices the better. Some people hate dynamic types, some people hate static types. There's room for both, imo. It's not like it's a secret there are faster or more efficient stacks. I guess they can't help themselves and need to make their position known. Personally, I love ruby, and rails is pretty good too.

mortallywounded 3y ago

Cool, MySQL scales. What does RoR have to do with it?

quazar987 3y ago

A while ago I read shopify hosts stores in pods. Pods are self contained instances that runs everything inside it, that includes mysql, redis, etc. does that mean this number is collective counting all the pods and not a single database?