3 ms·

> the central issue of scalability is "canonical state" Exactly. I've written a reasonable amount of Erlang, and yeah, from a purely technical perspective you'

by patrec 5y ago

> the central issue of scalability is "canonical state" Exactly. I've written a reasonable amount of Erlang, and yeah, from a purely technical perspective you'd be much better off handling concurrent requests with something actor based than async-du-jour dumpster fire (python is particularly bad). But the real difficulty ultimately is managing canonical persistent state, and actors won't magically solve that for you. Having said this, empirically speaking, almost none of the companies who believe they have scalability problems actually do and a relational DB combined with a not super inefficient architecture are all that's needed.

samsquire 5y ago
Problem with python is that async doesn't confer any performance benefit because of the GIL. You cannot actually run async functions in parallel. Each function makes forward progress independently but in serial. One problem I have is that the performance of computation is tied to performance of the storage, hence bringing computation to the storage for performance. If you have to communicate to get storage, then your solution will be slow.
- eloff 5y ago
  Async is not really about parallel. Async in nodejs works the same way. The point is for IO to happen concurrently. To scale to multiple cores in Python or node you run multiple processes. If you're not sharing mutable state between them, that scales great. If you are, you really shouldn't be using those languages, they're not the right tool for the job.
  - samsquire 5y ago
    I take advantage of I/O being parallel in Python in my mazzle continuous integration pipeline tool. I'm not sharing mutable state. I spin up a graph of python Threads and each joins others in a graph. This way we can run graphs in parallel. See this graph - the parts that look like this: dependency -> {parallel1; parallel2; parallel3} -> postparallel parallel1, parallel2, parallel3 can run in parallel in a separate python thread because the IO is parallel. postparallel joins parallel1, parallel2, parallel3 and waits for them all to complete. Where parallel1-3 is things like ansible, packer (slow), AMI builds, chef runs etc. https://github.com/samsquire/mazzle-starter/blob/master/architecture.png https://github.com/samsquire/mazzle-starter/blob/master/arch...

by patrec 5y ago

samsquire 5y ago

Problem with python is that async doesn't confer any performance benefit because of the GIL. You cannot actually run async functions in parallel. Each function makes forward progress independently but in serial. One problem I have is that the performance of computation is tied to performance of the storage, hence bringing computation to the storage for performance. If you have to communicate to get storage, then your solution will be slow.

eloff 5y ago
Async is not really about parallel. Async in nodejs works the same way. The point is for IO to happen concurrently. To scale to multiple cores in Python or node you run multiple processes. If you're not sharing mutable state between them, that scales great. If you are, you really shouldn't be using those languages, they're not the right tool for the job.
- samsquire 5y ago
  I take advantage of I/O being parallel in Python in my mazzle continuous integration pipeline tool. I'm not sharing mutable state. I spin up a graph of python Threads and each joins others in a graph. This way we can run graphs in parallel. See this graph - the parts that look like this: dependency -> {parallel1; parallel2; parallel3} -> postparallel parallel1, parallel2, parallel3 can run in parallel in a separate python thread because the IO is parallel. postparallel joins parallel1, parallel2, parallel3 and waits for them all to complete. Where parallel1-3 is things like ansible, packer (slow), AMI builds, chef runs etc. https://github.com/samsquire/mazzle-starter/blob/master/architecture.png https://github.com/samsquire/mazzle-starter/blob/master/arch...