3 ms·

I wish I was smart enough to get what you were saying here but I got a bit lost on the analogy. Could you ELI5 that for me? It sounded interesting

by mhoad 5y ago

rapsey 5y ago
It always comes down to your data that must be persistent and consistent. This data must be stored somewhere. The real problem is always storage. Message passing is all fine and good, but the real problem is your data. How scalable is your data layer in terms of size and operations per second.
slver 5y ago
When you scale, your basic solution is "I take this job done by one person/machine, and I split it to multiple people/machines". Sometimes you can't do that. Either because you're "outsourcing" part of the solution to someone beyond your control, or because the data relationships of your system require a "synchronizer", someone who works with this data in a serial, specific way. For example we have some government agency. Long queues, lots of waiting. One office for registration. What do we do? We add more registration offices, so people's forms can be processed in parallel. But every person needs to get a unique ticket number let's say. Now they still need to go through a single office to get this number, because the independent offices can't guarantee global uniqueness as they don't synchronize between each other. You need a "synchronizer" whose only purpose is to provide unique numbers. And this works, you have 100 offices and 1 "number" office. You scaled 100x over having one office for everything. But when you get to 200 offices, suddenly the queue in front of the "number" office starts to grow. You can't scale this office, because there's a "data relationship" in there that you can't split up and scale. You wanna identify and avoid such bottlenecks, and when you can't avoid them you want those bottlenecked systems to do as little as possible (so they can run as fast as possible) and to be hit as rarely as possible (by having someone in front take care of common cases: for example you can have a front-desk before a "number" office that checks you have all needed documents to get a number, before you get sent to get a number, now the "number" office has less to do). Sorry that's very abstract, but without specific examples it's hard to explain anything.
- devoutsalsa 5y ago
  The concept was nice of the one office that gives out numbers is similar to auto incrementing integer IDs in DB. To get around one single thing having to generate all the numbers, you can have every office generate UUIDs, and toss in a timestamp if you need ordering. I’ve heard people are playing with UUID v6, too, but I’m not sold on it yet.
- mhoad 5y ago
  No need for the apology, I appreciate you taking the time to event attempt explaining it :) Let me point out where I am falling over specifically because it might be more helpful. My understanding of Actors is that because they are essentially operating as in-memory objects with their own state, that with the exception of ad-hoc queries on my data (where SQL shines for example) I am doing the majority of my operations on various resources which handle persistence in a eventually consistent fashion and therefore aren't really obliged to immediately write to a database with each individual change in state thus freeing up the bottlenecks. This is all a pretty new concept to me so I feel like I might have made some bad assumptions in there for example, I am just not 100% sure where yet :)
  - deleted 5y ago
    [deleted]
mamcx 5y ago
In other words: You can't add many workers to a linear job and expect the task is done in SizeOfJob/Workers. Or: If you put many car lanes that go into one, you will get contention. This will manifest in many ways. But you "can" if along ALL the pipeline you have a way to coordinate the task!. However, if componentA is paradigmA and componentB is paradigmB you will get trouble. --- In practical term: Wanna kill your database? Open N connections launched by thousand of actors/async/threads. You need to instead use a pool, coordinate which jobs are fast to done or slow to complete. AND You need to architect the code to exploit both what this paradigm is, + what that make happy the DB (typical: NOT do sql + n, think in batches, group operations, etc) This is the part you don't get much help not matter what you use.

I wish I was smart enough to get what you were saying here but I got a bit lost on the analogy. Could you ELI5 that for me? It sounded interesting

by mhoad 5y ago

I wish I was smart enough to get what you were saying here but I got a bit lost on the analogy. Could you ELI5 that for me? It sounded interesting

rapsey 5y ago

It always comes down to your data that must be persistent and consistent. This data must be stored somewhere. The real problem is always storage. Message passing is all fine and good, but the real problem is your data. How scalable is your data layer in terms of size and operations per second.

slver 5y ago

When you scale, your basic solution is "I take this job done by one person/machine, and I split it to multiple people/machines". Sometimes you can't do that. Either because you're "outsourcing" part of the solution to someone beyond your control, or because the data relationships of your system require a "synchronizer", someone who works with this data in a serial, specific way. For example we have some government agency. Long queues, lots of waiting. One office for registration. What do we do? We add more registration offices, so people's forms can be processed in parallel. But every person needs to get a unique ticket number let's say. Now they still need to go through a single office to get this number, because the independent offices can't guarantee global uniqueness as they don't synchronize between each other. You need a "synchronizer" whose only purpose is to provide unique numbers. And this works, you have 100 offices and 1 "number" office. You scaled 100x over having one office for everything. But when you get to 200 offices, suddenly the queue in front of the "number" office starts to grow. You can't scale this office, because there's a "data relationship" in there that you can't split up and scale. You wanna identify and avoid such bottlenecks, and when you can't avoid them you want those bottlenecked systems to do as little as possible (so they can run as fast as possible) and to be hit as rarely as possible (by having someone in front take care of common cases: for example you can have a front-desk before a "number" office that checks you have all needed documents to get a number, before you get sent to get a number, now the "number" office has less to do). Sorry that's very abstract, but without specific examples it's hard to explain anything.

devoutsalsa 5y ago
The concept was nice of the one office that gives out numbers is similar to auto incrementing integer IDs in DB. To get around one single thing having to generate all the numbers, you can have every office generate UUIDs, and toss in a timestamp if you need ordering. I’ve heard people are playing with UUID v6, too, but I’m not sold on it yet.
mhoad 5y ago
No need for the apology, I appreciate you taking the time to event attempt explaining it :) Let me point out where I am falling over specifically because it might be more helpful. My understanding of Actors is that because they are essentially operating as in-memory objects with their own state, that with the exception of ad-hoc queries on my data (where SQL shines for example) I am doing the majority of my operations on various resources which handle persistence in a eventually consistent fashion and therefore aren't really obliged to immediately write to a database with each individual change in state thus freeing up the bottlenecks. This is all a pretty new concept to me so I feel like I might have made some bad assumptions in there for example, I am just not 100% sure where yet :)
- deleted 5y ago
  [deleted]

mamcx 5y ago

In other words: You can't add many workers to a linear job and expect the task is done in SizeOfJob/Workers. Or: If you put many car lanes that go into one, you will get contention. This will manifest in many ways. But you "can" if along ALL the pipeline you have a way to coordinate the task!. However, if componentA is paradigmA and componentB is paradigmB you will get trouble. --- In practical term: Wanna kill your database? Open N connections launched by thousand of actors/async/threads. You need to instead use a pool, coordinate which jobs are fast to done or slow to complete. AND You need to architect the code to exploit both what this paradigm is, + what that make happy the DB (typical: NOT do sql + n, think in batches, group operations, etc) This is the part you don't get much help not matter what you use.