4 ms·

Is it really that hard? Tons of companies do it. LAMP stack has been scalable for couple decades or so. There are platforms and services that you can rent/buy

Is it really that hard? Tons of companies do it. LAMP stack has been scalable for couple decades or so. There are platforms and services that you can rent/buy if you don't wanna role your own. I can get a free databricks account and go to town with pyspark there's even free courses on it. Threading is hard and I'm sure will always be hard. But threading is the wrong way to scale or parallelise.

pcwalton 10y ago
> But threading is the wrong way to scale or parallelise. Depends on your domain. Multicore shared-memory parallelism is still king for many use cases, with no realistic alternative in sight. You heavily depended on threading to post this comment (because your GPU used it, along with vector instructions, to render the page).
hbogert 10y ago
LAMP scalable for a couple of decades? PHP is hardly usuable for two decades, let alone scalable since that very beginning. Same for Mysql. Did you actually ever try to scale Mysql? Not trivial. Did you every try to scale webserver with PHP where sessions are being used? Not trivial.
- Thaxll 10y ago
  Yes it's called sharding for MySQL.
  - prodigal_erik 10y ago
    The query engine doesn't support sharding. You're starting with a straightforward relational database and ending up with a bunch of fragments of relations that must only be used with great caution because most joins quietly give incorrect results.
  - duaneb 10y ago
    If you dislike joins and love data inconsistencies, go for it.
    - kod 10y ago
      Joins are still possible in sharded relational models. I wouldn't want to roll my own on mysql, but there are plenty of data stores that use this model (vertica, citus, etc)
      - deleted 10y ago
        [deleted]
      - duaneb 10y ago
        > Joins are still possible in sharded relational models. For some definition of joins, this is true. For a SQL-oriented version, I don't think it's possible, let alone easy, to run arbitrary joins across sharded data. Can you write subqueries that cross shard boundaries? How about aggregate functions? I would be (very pleasantly) surprised if this were possible outside of directly calling the aggregation/join. > I wouldn't want to roll my own on mysql, but there are plenty of data stores that use this model (vertica, citus, etc) Yes, and none of them have succeed at scaling mysql without sacrificing heavily in functionality. I don't believe foreign keys are supported cross-shard anywhere. I'm also pretty sure the shard schemes heavily dictate which operations are efficient and even possible, and which are not, so you really need to design around scaling to begin with anyway. So—yes, you can scale MySQL, but avoid using any of the features that make MySQL worth using, except intrashard.
        kod 10y ago
        None of the stores I mentioned use mysql. I'm not talking about scaling mysql, I'm talking about scaling relational databases, specifically the comment about joins. Joins arent a big deal, just join on (at least) the shard key, or for smaller hash joins ship the small side of the join. If you have multiple kinds of joins keep multiple copies with different shard keys. Yes, (monoidal) aggregations are possible with this, yes subqueries are possible with this.
- njharman 10y ago
  P is Perl PHP Python. The other letters have various incarnations. Point being horizontal scale is known, and not difficult solution for many, many domains. Starting a long time ago.
Jtsummers 10y ago
This depends on your definition of scaling. You're creating (relatively) heavyweight servers to handle multiple instances and hiding them behind a reverse proxy. And this is for a very specific use-case. Not everything is HTTP, and not everything HTTP is handled (well) by a LAMP stack. Certainly, as pcwalton points out, a LAMP stack will not handle scalability of your graphics (while hitting desired performance characteristics, at least). A LAMP stack is unlikely to be appropriate to the sort of scaling that map-reduce can perform. A LAMP stack won't handle, in the case of my office, handling an arbitrary number of connecting radios. ===EDIT=== > Threading is hard and I'm sure will always be hard. But threading is the wrong way to scale or parallelise. Threading is an implementation of concurrency/parallelism. Spawning multiple servers behind a reverse proxy, like you seem to be suggesting, is one method for handling scaling issues. But the models used to make effective use of threaded concurrency (actor model, CSP, even shared memory done well), also work in designing larger systems that need to scale beyond a single computer/CPU.
Jweb_Guru 10y ago
On the contrary, scaling and parallelism are much easier on a multicore machine (possibly with a NIC queue per socket) because most nontrivial workloads are ultimately communication bound. Single load balancers can handle most real sites' traffic and direct them to different cores, and for those concerned about (for example) DDOS attempts, services like Cloudflare will allow you to combine IP anycast with a reverse proxy to avoid this problem far from your machine. For a large percentage of use cases, the only reason to have multiple machines at all is for fault tolerance, and for most of the others, it's just not being able to fit all your data in memory (increasingly rare). That said, modern networking has gotten good enough that in a really high-end datacenter you can sometimes still get increased performance out of the network (using RDMA, etc.), but you have to really work hard for it (usually bypassing the OS networking stack).

pcwalton 10y ago

> But threading is the wrong way to scale or parallelise. Depends on your domain. Multicore shared-memory parallelism is still king for many use cases, with no realistic alternative in sight. You heavily depended on threading to post this comment (because your GPU used it, along with vector instructions, to render the page).

hbogert 10y ago

LAMP scalable for a couple of decades? PHP is hardly usuable for two decades, let alone scalable since that very beginning. Same for Mysql. Did you actually ever try to scale Mysql? Not trivial. Did you every try to scale webserver with PHP where sessions are being used? Not trivial.

Thaxll 10y ago
Yes it's called sharding for MySQL.
- prodigal_erik 10y ago
  The query engine doesn't support sharding. You're starting with a straightforward relational database and ending up with a bunch of fragments of relations that must only be used with great caution because most joins quietly give incorrect results.
- duaneb 10y ago
  If you dislike joins and love data inconsistencies, go for it.
  - kod 10y ago
    Joins are still possible in sharded relational models. I wouldn't want to roll my own on mysql, but there are plenty of data stores that use this model (vertica, citus, etc)
    - deleted 10y ago
      [deleted]
    - duaneb 10y ago
      > Joins are still possible in sharded relational models. For some definition of joins, this is true. For a SQL-oriented version, I don't think it's possible, let alone easy, to run arbitrary joins across sharded data. Can you write subqueries that cross shard boundaries? How about aggregate functions? I would be (very pleasantly) surprised if this were possible outside of directly calling the aggregation/join. > I wouldn't want to roll my own on mysql, but there are plenty of data stores that use this model (vertica, citus, etc) Yes, and none of them have succeed at scaling mysql without sacrificing heavily in functionality. I don't believe foreign keys are supported cross-shard anywhere. I'm also pretty sure the shard schemes heavily dictate which operations are efficient and even possible, and which are not, so you really need to design around scaling to begin with anyway. So—yes, you can scale MySQL, but avoid using any of the features that make MySQL worth using, except intrashard.
      - kod 10y ago
        None of the stores I mentioned use mysql. I'm not talking about scaling mysql, I'm talking about scaling relational databases, specifically the comment about joins. Joins arent a big deal, just join on (at least) the shard key, or for smaller hash joins ship the small side of the join. If you have multiple kinds of joins keep multiple copies with different shard keys. Yes, (monoidal) aggregations are possible with this, yes subqueries are possible with this.
njharman 10y ago
P is Perl PHP Python. The other letters have various incarnations. Point being horizontal scale is known, and not difficult solution for many, many domains. Starting a long time ago.

Jtsummers 10y ago

This depends on your definition of scaling. You're creating (relatively) heavyweight servers to handle multiple instances and hiding them behind a reverse proxy. And this is for a very specific use-case. Not everything is HTTP, and not everything HTTP is handled (well) by a LAMP stack. Certainly, as pcwalton points out, a LAMP stack will not handle scalability of your graphics (while hitting desired performance characteristics, at least). A LAMP stack is unlikely to be appropriate to the sort of scaling that map-reduce can perform. A LAMP stack won't handle, in the case of my office, handling an arbitrary number of connecting radios. ===EDIT=== > Threading is hard and I'm sure will always be hard. But threading is the wrong way to scale or parallelise. Threading is an implementation of concurrency/parallelism. Spawning multiple servers behind a reverse proxy, like you seem to be suggesting, is one method for handling scaling issues. But the models used to make effective use of threaded concurrency (actor model, CSP, even shared memory done well), also work in designing larger systems that need to scale beyond a single computer/CPU.

Jweb_Guru 10y ago

On the contrary, scaling and parallelism are much easier on a multicore machine (possibly with a NIC queue per socket) because most nontrivial workloads are ultimately communication bound. Single load balancers can handle most real sites' traffic and direct them to different cores, and for those concerned about (for example) DDOS attempts, services like Cloudflare will allow you to combine IP anycast with a reverse proxy to avoid this problem far from your machine. For a large percentage of use cases, the only reason to have multiple machines at all is for fault tolerance, and for most of the others, it's just not being able to fit all your data in memory (increasingly rare). That said, modern networking has gotten good enough that in a really high-end datacenter you can sometimes still get increased performance out of the network (using RDMA, etc.), but you have to really work hard for it (usually bypassing the OS networking stack).