3 ms·

Trust me. You won't hit 424,712 requests per seconds per each node (that is 1,528,963,200 per hour). You need to have many nodes and a a well distributed and s

by stunt 8y ago

Trust me. You won't hit 424,712 requests per seconds per each node (that is 1,528,963,200 per hour). You need to have many nodes and a a well distributed and scalable infrastructure to handle 1B req/hr. There is no point to benchmark your programming-language/framework when doing web development. When it comes to web, you will hit N different capacity limits before that even with slowest framework, starting from your database, network connections, bandwidth, third-party API rate limits, IO performance etc... Consider only a few options base on project requirements, ecosystem, productivity, and available human resources. then compare the top two on performance. You can't compare H2O and Rails on the same list. they are two different animals. Moreover, I'm sure all of these frameworks are not well optimized for a raw performance benchmark, and also these results are heavily affected by filters you are choosing. Don't spend time on pointless benchmarks.

3pt14159 8y ago
Yep. It took me about six months to write a replacement of ActiveRecord with an identical API (for what was needed) that could scale to arbitrarily large number of writes. Shit, pre-A16Z raise 500px ran on Rails and they were a social network around photography. You think Twitter is bad? Try timelines where you have half a million photographers liking hundreds of photos an hour and every like gets pushed into every feed of the people following them. Guys syncing their entire photo library. It was 5 application servers, 4 MongoDB servers for the timeline with some crazy data structures, one or two MySQL DBs. I think most web projects these days should be in either Pheonix, Rails, or similar. If you need something really fast here or there just fork off the request in Nginx or compile something in Rust or C and extend it into Ruby (or whatever). Or have a compiled worker that communicates through the DB or Redis. There is this long tail of UI you need to make for every web project and nobody uses it 99.999% of the time. It should be in whatever secure language / framework that brings it to market fastest.
- ksec 8y ago
  >Yep. It took me about six months to write a replacement of ActiveRecord with an identical API (for what was needed) that could scale to arbitrarily large number of writes Sorry to ask, would that be open sourced? >It was 5 application servers, 4 MongoDB servers for the timeline with some crazy data structures, one or two MySQL DBs. That is a little hard to warp my head around, because pre-A16z 500px was a long time ago? Hardware should be much slower and Ruby was way slower as well. We have 2-3x faster hardware and faster ruby today, and the same hardware surely won't do much with Discourse and Gitlab scale.
  - 3pt14159 8y ago
    > Sorry to ask, would that be open sourced? No. I would have if I owned the code personally, but it wasn't in the companies interests to open source it. > That is a little hard to warp my head around... I was there between was 2012—2013. The hardware wasn't bad, I don't remember the exact specs but it was colocated at a place in downtown TO on some decent but not crazy gear. It wasn't perfect, but it worked pretty well. For the actual image processing we just shelled out to imagemagick and I vaguely recall some fancy batch write code for keeping things like photo view counts from completely destroying the DB. We had a hardware load balancer that worked pretty good and we compiled ruby ourselves for a performance boost, though that ended up killing us this one time when there was a bug that only showed up in the compiled version of Ruby when all the dev boxes ran the universal binary. People care so much about performance, and in some cases like computer games it's totally warranted, but if at the end of the day you have a DB the DB is going to be the problem long, long, long before the application code.
    - ksec 8y ago
      > No. I would have if I owned the code personally, but it wasn't in the companies interests to open source it. Thanks, I just thought that might solve the ActiveRecord is slow problem.
      - 3pt14159 8y ago
        It didn't solve the ActiveRecord is slow problem, it was marginally faster, but not a whole lot faster. What it solved was handling models over an arbitrary number of databases. Imagine billions of writes of a single model type across multiple databases. Imagine needing to join across databases and not knowing which DB a joined model would be in.

by stunt 8y ago

3pt14159 8y ago

Yep. It took me about six months to write a replacement of ActiveRecord with an identical API (for what was needed) that could scale to arbitrarily large number of writes. Shit, pre-A16Z raise 500px ran on Rails and they were a social network around photography. You think Twitter is bad? Try timelines where you have half a million photographers liking hundreds of photos an hour and every like gets pushed into every feed of the people following them. Guys syncing their entire photo library. It was 5 application servers, 4 MongoDB servers for the timeline with some crazy data structures, one or two MySQL DBs. I think most web projects these days should be in either Pheonix, Rails, or similar. If you need something really fast here or there just fork off the request in Nginx or compile something in Rust or C and extend it into Ruby (or whatever). Or have a compiled worker that communicates through the DB or Redis. There is this long tail of UI you need to make for every web project and nobody uses it 99.999% of the time. It should be in whatever secure language / framework that brings it to market fastest.

ksec 8y ago
>Yep. It took me about six months to write a replacement of ActiveRecord with an identical API (for what was needed) that could scale to arbitrarily large number of writes Sorry to ask, would that be open sourced? >It was 5 application servers, 4 MongoDB servers for the timeline with some crazy data structures, one or two MySQL DBs. That is a little hard to warp my head around, because pre-A16z 500px was a long time ago? Hardware should be much slower and Ruby was way slower as well. We have 2-3x faster hardware and faster ruby today, and the same hardware surely won't do much with Discourse and Gitlab scale.
- 3pt14159 8y ago
  > Sorry to ask, would that be open sourced? No. I would have if I owned the code personally, but it wasn't in the companies interests to open source it. > That is a little hard to warp my head around... I was there between was 2012—2013. The hardware wasn't bad, I don't remember the exact specs but it was colocated at a place in downtown TO on some decent but not crazy gear. It wasn't perfect, but it worked pretty well. For the actual image processing we just shelled out to imagemagick and I vaguely recall some fancy batch write code for keeping things like photo view counts from completely destroying the DB. We had a hardware load balancer that worked pretty good and we compiled ruby ourselves for a performance boost, though that ended up killing us this one time when there was a bug that only showed up in the compiled version of Ruby when all the dev boxes ran the universal binary. People care so much about performance, and in some cases like computer games it's totally warranted, but if at the end of the day you have a DB the DB is going to be the problem long, long, long before the application code.
  - ksec 8y ago
    > No. I would have if I owned the code personally, but it wasn't in the companies interests to open source it. Thanks, I just thought that might solve the ActiveRecord is slow problem.
    - 3pt14159 8y ago
      It didn't solve the ActiveRecord is slow problem, it was marginally faster, but not a whole lot faster. What it solved was handling models over an arbitrary number of databases. Imagine billions of writes of a single model type across multiple databases. Imagine needing to join across databases and not knowing which DB a joined model would be in.