5 ms·
Some cities are bigger than others. That's a heavily imbalanced sharding key.
by xapata 10y ago
Some cities are bigger than others. That's a heavily imbalanced sharding key.
- amelius 10y agoWhy is that a problem? You could count several smaller cities as a single large city, as far as sharding is concerned.
- kowdermeister 10y agoThat's like a post office way of thinking about the internet, we already past that point. Plus you haven't solved the problem of managing 800 microservices and 2000 programmers, but now you have a deployment problem also.
- jon-wood 10y agoWith 800 services and 2000 programmers you already have a deployment problem.
- eeZah7Ux 10y agoWith 800 services and 2000 programmers you already have a problem.
- new_hackers 10y agoWith programers you already have a problem.
- jrochkind1 10y agowith computers.
- kazinator 10y agoWith zeros, there are generally no issues, until you add ones.
- jrochkind1 10y agoNah, you're making a mistake with the zeroes, you should totally stick to all ones instead, everyone knows ones are better than zeroes (they're one more man! are you blind?!)
- kazinator 10y agoSee? As soon as we introduce 1, we give rise to the question: what is 0 - 1? And that, alas, is the root origin of downvoting.
- latch 10y agoWhich isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of concern is only needed when a car changes grid, and then only on the two grids in questions. This can be layered multiple levels, and your final car-holding structure could be an r-tree or something. The grid for a city can be sharded across multiple servers. And, if you told me that was necessary, fine..but as-is, I'm suspicious that a pretty basic server can't handle a tens of thousands of cars sending updates every second. Friends tell me the heavy processing is in routing / map stuff, but this is relatively stateless and can be sent off to a pool of workers to handle.
- andy_ppp 10y agoI always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free. The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.
- lgas 10y agoYou're saying you want to extend "just let it crash" to the individual car level? :)
- andy_ppp 10y agoDepends on how smart your supervisor gets doesn't it but sure, why not? Many times processes can be restarted in microseconds...
- ramchip 10y agoI've given that kind of model a lot of thought too, but I did not find very satisfying solutions to these problems: - How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy) - What happens if a node fails? What mechanism is going to respawn the car actors on a different node? How do you ensure minimal downtime for the cars involved? - What happens if there's a netsplit, e.g. how do you ensure no split brain where two nodes think they're responsible for a car? It feels to me like the traditional erlang-process-per-request architecture coupled with a distributed store (riak or w/e) makes it possible to avoid the very difficult problem of ensuring one and only one actor per car in a distributed environment.