4 ms·

> 12GB to 5MB of memory consumption, then you have a design issue and not a language one. That's not Ruby's fault. I thought the same thing, I find that figure

by neikos 12y ago

> 12GB to 5MB of memory consumption, then you have a design issue and not a language one. That's not Ruby's fault. I thought the same thing, I find that figure also very hard to believe. Perhaps I just lack experience, but if you get 12G of memory usage that drops to 5M after rewriting it (whether another language or not) seems to me as if they did something pretty wrong in their previous iteration.

patio11 12y ago
This is a function of how Rails does concurrency, which is by using multiple processes. Each additional process consumes X MB. If you have something which needs a throughput of 100 requests per second, and each request takes 0.2 seconds, you require a minimum of 20 Rails processes and you probably want to have more. The X is largely dependent on how many dependencies your app pulls in, on your garbage collection settings, and on how much state you carry around with you. I don't know what an off-the-shelf Rails app requires as a minimum, but as a comparable, BCC requires ~120 MB per process and AR requires approximately ~200 MB. Neither are particularly mammoth code bases. Thus, if AR develops a business requirement for 100 requests per second on something which takes 200 ms per request, my server budget just increased by 4 GB of RAM. That is absolutely unavoidable in the Rails concurrency model. It doesn't matter how well you code the small thing at issue: you pay a memory tax based on desired maximum volume and app complexity, always. (Neither here nor there, but AR has features quite similar to the notification feature that the app described in this article. We handle this by using queue workers -- which each require 200 MB, as described above -- and the decision that queues do not require immediate servicing. Instead, if thousands of calls are dumped into the system, we add them to a queue quickly and then consume the queue leisurely. Different apps are different apps, but I'd wager a guess that we do substantially more volume in terms of calls on ~400 MB of RAM.) Also neither here nor there, but engineers are expensive and gigabytes are cheap. There exist many circumstances in which the architecture which requires 12 GB of RAM is a defensible engineering tradeoff versus the one which takes 5 MB. As Thomas told me once when I was asking him how to cut 100 MB off of Redis so that I could avoid upgrading a VPS by 1 GB: "How much does the extra gig cost? $10? Why are we even having this discussion?" And he was right.
- nexneo 12y ago
  Thanks, you explained it very well. Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
  - boundlessdreamz 12y ago
    Did you try sidekiq? Btw how are you generating the PDF from HTML and are able split a single HTML into multiple PDFs?
    - nexneo 12y ago
      wkhtmltopdf and phantomjs both worked similarly, currently I'm using phantomjs. And I'm not splitting pdf but splitting html generation work load, and then create individual pdfs from those html chunks. Then they will be joined together (using pdfunite). I found this much faster then joining html and generating large pdf.
      - boundlessdreamz 12y ago
        Ok. Are you using phantomjs 1 or 2 ? Any reason to choose phantomjs or wkhtmltopdf? We are using wkhtmltopdf because it creates Table of Contents for PDFs and also clickable links
        nexneo 12y ago
        PhantomJS 1. But nothing is tide to phantomjs, wkhtmltopdf should work as well. (I'm planning to test with PhantomJS 2)
  - legedemon 12y ago
    Yeah, everyone wants to process their queue as fast as possible but "as fast as possible" practically means a cap on the maximum allowed delay. Otherwise, why stop at 30 workers? Go for 300. 3000? Also, if the workers shared all the code, you could have used unicorn to fork the processes after the code loading was complete. The 400MB per process would then instantly come down to something ~10MB per process at which point rewriting would have been delayed for another year or so.

by neikos 12y ago

patio11 12y ago

This is a function of how Rails does concurrency, which is by using multiple processes. Each additional process consumes X MB. If you have something which needs a throughput of 100 requests per second, and each request takes 0.2 seconds, you require a minimum of 20 Rails processes and you probably want to have more. The X is largely dependent on how many dependencies your app pulls in, on your garbage collection settings, and on how much state you carry around with you. I don't know what an off-the-shelf Rails app requires as a minimum, but as a comparable, BCC requires ~120 MB per process and AR requires approximately ~200 MB. Neither are particularly mammoth code bases. Thus, if AR develops a business requirement for 100 requests per second on something which takes 200 ms per request, my server budget just increased by 4 GB of RAM. That is absolutely unavoidable in the Rails concurrency model. It doesn't matter how well you code the small thing at issue: you pay a memory tax based on desired maximum volume and app complexity, always. (Neither here nor there, but AR has features quite similar to the notification feature that the app described in this article. We handle this by using queue workers -- which each require 200 MB, as described above -- and the decision that queues do not require immediate servicing. Instead, if thousands of calls are dumped into the system, we add them to a queue quickly and then consume the queue leisurely. Different apps are different apps, but I'd wager a guess that we do substantially more volume in terms of calls on ~400 MB of RAM.) Also neither here nor there, but engineers are expensive and gigabytes are cheap. There exist many circumstances in which the architecture which requires 12 GB of RAM is a defensible engineering tradeoff versus the one which takes 5 MB. As Thomas told me once when I was asking him how to cut 100 MB off of Redis so that I could avoid upgrading a VPS by 1 GB: "How much does the extra gig cost? $10? Why are we even having this discussion?" And he was right.

nexneo 12y ago
Thanks, you explained it very well. Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
- boundlessdreamz 12y ago
  Did you try sidekiq? Btw how are you generating the PDF from HTML and are able split a single HTML into multiple PDFs?
  - nexneo 12y ago
    wkhtmltopdf and phantomjs both worked similarly, currently I'm using phantomjs. And I'm not splitting pdf but splitting html generation work load, and then create individual pdfs from those html chunks. Then they will be joined together (using pdfunite). I found this much faster then joining html and generating large pdf.
    - boundlessdreamz 12y ago
      Ok. Are you using phantomjs 1 or 2 ? Any reason to choose phantomjs or wkhtmltopdf? We are using wkhtmltopdf because it creates Table of Contents for PDFs and also clickable links
      - nexneo 12y ago
        PhantomJS 1. But nothing is tide to phantomjs, wkhtmltopdf should work as well. (I'm planning to test with PhantomJS 2)
- legedemon 12y ago
  Yeah, everyone wants to process their queue as fast as possible but "as fast as possible" practically means a cap on the maximum allowed delay. Otherwise, why stop at 30 workers? Go for 300. 3000? Also, if the workers shared all the code, you could have used unicorn to fork the processes after the code loading was complete. The 400MB per process would then instantly come down to something ~10MB per process at which point rewriting would have been delayed for another year or so.