4 ms·

> it's stateless by design (much easier to scale) I noticed being stateless made it difficult to scale for any complex app. It’s been 7-10 years since I used i

> it's stateless by design (much easier to scale) I noticed being stateless made it difficult to scale for any complex app. It’s been 7-10 years since I used it but Drupal used 60-128MB+ plus per request. As it was stateless it had to do a full bootstrap for every request coming in, set up a new database connection, module discovery etc, all which was heavy to do for every request. Most people scaled Drupal by putting Varnish in front preventing the origin from being hit so not really scaling Drupal as such, varnish is taking off some load where it can. Drupal is probably the extreme though. If you have a simple app with a few scripts, light use of a DB it’s not to bad. I’ve heard it’s come on a lot in the past 10 years and Laravel. I certainly had less issues using PHP than I do with JavaScript.

dmitryminkovsky 4y ago
Had same experience, had same thoughts. I bet I could make my way around the Drupal 7 bootstrap even these days, I looked at it so much... and so often thought... all this... every request?
larve 4y ago
Yes, nothing comes for free. opcache, object cache + db pooling is something I always turn on. Then, for app performance: My approach is to usually design the app / refactor the app so static content can be fully cached. In ecommerce sites this can make an absolutely impressive difference. Then, leverage opcache and object cache and further separate cacheable path from dynamic path in dynamic SSR queries. Always drive using APM performance metrics as a guide. No need to optimize the DELETE /user endpoint called twice a week. Then, identify the heavy hitters in the dynamic DB queries, and basically write those in hand rolled SQL. It usually boils down to 4 or 5 queries doing 99% of the work (GET /posts, GET /categories, GET /products, etc...). I do this fairly late in the game, but think about it very hard up front. I like to use some kind of ORM for everything writing to the DB, and sql builders / pure SQL for reading. The split between cacheable / non-cacheable can be subtle depending on the application, and might require some non-obvious structures in the codebase. For real tricky performance issues (not something I had to do in PHP projects), I will either ducttape some AWS infrastructure or handroll a microservice in C++ or golang. This is something I had to do for realtime / transient loads.
- jamesfinlayson 4y ago
  Yes a properly tuned opcache (not hard - I think there are only a handful of settings) really helps.
mikeytown2 4y ago
Drupal misused the database. For Drupal 7 I made the apdqc module for async prefetch MySQL. no deadlocks makes it easy to scale, load balance more Apache servers in front. DB size was 100gb; well into millions of rows of nodes; all logged in users. Site I made this for has been up for years; Fix the database layer and Drupal 7 will scale like crazy.
lmm 4y ago
> I noticed being stateless made it difficult to scale for any complex app. It’s been 7-10 years since I used it but Drupal used 60-128MB+ plus per request. As it was stateless it had to do a full bootstrap for every request coming in, set up a new database connection, module discovery etc, all which was heavy to do for every request. It doesn't make scaling hard, it makes scaling very easy because you can just horizontally scale out as many instances as you need. The constant factor is pretty awful, but that's the tradeoff you make for statelessness.
- BFLpL0QNek 4y ago
  Ok, it makes scaling technically easy with horizontal scaling for the web tier, ignoring the DB. In practice it makes scaling hard for financial reasons if you require 128MB+ per request. A 32GB server will struggle with more than 256 concurrent requests at 128MB+ per request. A 32GB server is an extremely large server for such low request numbers. If you can bring the memory usage down stateless works well. I use a bit of rust on aws lambda with the lowest memory settings so pay fractions of a cent and can handle more traffic than I ever need. I do set up persistent connection pools for the life of the lambda and only bootstrap once to bring down latency per-request but treat it as an ephemeral horizontal scaled service.
  - lmm 4y ago
    The problem there isn't scaling - it scales perfectly - the problem is low performance.
WatchDog 4y ago
It might be much less efficient per request than other popular languages, but it's stateless design, does at least prevent developers from using local memory caches, which can prevent a number of problems from cropping up when scaling horizontally. With that said, I think the stateless nature of php is talked up too much. While not enforced by the nature of the language like php, writing stateless apps in other languages comes quite naturally. Scaling difficulties are almost always to do with the database solution rather than the app, and overcoming those problems has little to do with the choice of language.
- larve 4y ago
  It always depends on the situation, but I do value the stateless approach "by default" of PHP because it makes shoddy code / legacy code that has grown through generations of uninterested developers much easier to deal with. I of course value at least partial "stateless" (functional core / hexagonal architecture, event driven architecture) in any language, because they reduce complexity so much.
conradfr 4y ago
That's not that true anymore with https://www.php.net/manual/fr/opcache.preloading.php https://www.php.net/manual/fr/opcache.preloading.php (and PHP-FPM).
dbbk 4y ago
That might be mainly a Drupal problem, I use Laravel and it can do some significant caching of all the bootstrap stuff for production. Plus as mentioned PHP itself is pretty strong now in terms of opcache and such.
stef25 4y ago
With both Drupal and Laravel it's very easy to overload the database. It's shocking when you start using some tool that gives you insights in to this and it tells you you're running 6000+ queries every time your homepage loads. In Laravel it's not too hard to avoid this problem but a lot of noobs don't get the lazy / eager loading thing and then you end up like this. Drupal seems to add hundreds of queries for every module you add.
- larve 4y ago
  I feel this is more a "classic" ORM problem. Wordpress is absolutely egregious here as well. I wrote up my strategy in another post, but I usually develop using the ORM (often, ORM for writing, SQL builder + ORM for querying, because I hate ORM filter/join abstractions). I do think very hard up front what the product will require in terms of: - static content - dynamic content that is cacheable and which keys can be used to cache it Static content can use ORM as much as it wants, cloudfront will take care of it. dynamic content that is rarely hit doesn't really need much help either if there are no UX latency requirements. dynamic content that is hit very often (usually only a couple of endpoints) can be rewritten into manual SQL. If you think about it, most page hits with dynamic content can be rendered basically by the DB + templating engine, and no other logic should be involved.

dmitryminkovsky 4y ago

Had same experience, had same thoughts. I bet I could make my way around the Drupal 7 bootstrap even these days, I looked at it so much... and so often thought... all this... every request?

larve 4y ago

Yes, nothing comes for free. opcache, object cache + db pooling is something I always turn on. Then, for app performance: My approach is to usually design the app / refactor the app so static content can be fully cached. In ecommerce sites this can make an absolutely impressive difference. Then, leverage opcache and object cache and further separate cacheable path from dynamic path in dynamic SSR queries. Always drive using APM performance metrics as a guide. No need to optimize the DELETE /user endpoint called twice a week. Then, identify the heavy hitters in the dynamic DB queries, and basically write those in hand rolled SQL. It usually boils down to 4 or 5 queries doing 99% of the work (GET /posts, GET /categories, GET /products, etc...). I do this fairly late in the game, but think about it very hard up front. I like to use some kind of ORM for everything writing to the DB, and sql builders / pure SQL for reading. The split between cacheable / non-cacheable can be subtle depending on the application, and might require some non-obvious structures in the codebase. For real tricky performance issues (not something I had to do in PHP projects), I will either ducttape some AWS infrastructure or handroll a microservice in C++ or golang. This is something I had to do for realtime / transient loads.

jamesfinlayson 4y ago
Yes a properly tuned opcache (not hard - I think there are only a handful of settings) really helps.

mikeytown2 4y ago

Drupal misused the database. For Drupal 7 I made the apdqc module for async prefetch MySQL. no deadlocks makes it easy to scale, load balance more Apache servers in front. DB size was 100gb; well into millions of rows of nodes; all logged in users. Site I made this for has been up for years; Fix the database layer and Drupal 7 will scale like crazy.

lmm 4y ago

> I noticed being stateless made it difficult to scale for any complex app. It’s been 7-10 years since I used it but Drupal used 60-128MB+ plus per request. As it was stateless it had to do a full bootstrap for every request coming in, set up a new database connection, module discovery etc, all which was heavy to do for every request. It doesn't make scaling hard, it makes scaling very easy because you can just horizontally scale out as many instances as you need. The constant factor is pretty awful, but that's the tradeoff you make for statelessness.

BFLpL0QNek 4y ago
Ok, it makes scaling technically easy with horizontal scaling for the web tier, ignoring the DB. In practice it makes scaling hard for financial reasons if you require 128MB+ per request. A 32GB server will struggle with more than 256 concurrent requests at 128MB+ per request. A 32GB server is an extremely large server for such low request numbers. If you can bring the memory usage down stateless works well. I use a bit of rust on aws lambda with the lowest memory settings so pay fractions of a cent and can handle more traffic than I ever need. I do set up persistent connection pools for the life of the lambda and only bootstrap once to bring down latency per-request but treat it as an ephemeral horizontal scaled service.
- lmm 4y ago
  The problem there isn't scaling - it scales perfectly - the problem is low performance.

WatchDog 4y ago

It might be much less efficient per request than other popular languages, but it's stateless design, does at least prevent developers from using local memory caches, which can prevent a number of problems from cropping up when scaling horizontally. With that said, I think the stateless nature of php is talked up too much. While not enforced by the nature of the language like php, writing stateless apps in other languages comes quite naturally. Scaling difficulties are almost always to do with the database solution rather than the app, and overcoming those problems has little to do with the choice of language.

larve 4y ago
It always depends on the situation, but I do value the stateless approach "by default" of PHP because it makes shoddy code / legacy code that has grown through generations of uninterested developers much easier to deal with. I of course value at least partial "stateless" (functional core / hexagonal architecture, event driven architecture) in any language, because they reduce complexity so much.

conradfr 4y ago

That's not that true anymore with https://www.php.net/manual/fr/opcache.preloading.php https://www.php.net/manual/fr/opcache.preloading.php (and PHP-FPM).

dbbk 4y ago

That might be mainly a Drupal problem, I use Laravel and it can do some significant caching of all the bootstrap stuff for production. Plus as mentioned PHP itself is pretty strong now in terms of opcache and such.

stef25 4y ago

With both Drupal and Laravel it's very easy to overload the database. It's shocking when you start using some tool that gives you insights in to this and it tells you you're running 6000+ queries every time your homepage loads. In Laravel it's not too hard to avoid this problem but a lot of noobs don't get the lazy / eager loading thing and then you end up like this. Drupal seems to add hundreds of queries for every module you add.

larve 4y ago
I feel this is more a "classic" ORM problem. Wordpress is absolutely egregious here as well. I wrote up my strategy in another post, but I usually develop using the ORM (often, ORM for writing, SQL builder + ORM for querying, because I hate ORM filter/join abstractions). I do think very hard up front what the product will require in terms of: - static content - dynamic content that is cacheable and which keys can be used to cache it Static content can use ORM as much as it wants, cloudfront will take care of it. dynamic content that is rarely hit doesn't really need much help either if there are no UX latency requirements. dynamic content that is hit very often (usually only a couple of endpoints) can be rewritten into manual SQL. If you think about it, most page hits with dynamic content can be rendered basically by the DB + templating engine, and no other logic should be involved.