3 ms·

Hi Simon, this is a good question. What I think is that this ops ultimately will be too slow to be useful, anyway you need a massive amount of data transfer to

by antirez 16y ago

Hi Simon, this is a good question. What I think is that this ops ultimately will be too slow to be useful, anyway you need a massive amount of data transfer to compare things in different instances. But Redis Cluster supports the MIGRATE command, so what we plan is to provide the client with a very cool way to do this kin do things. Basically MIGRATE is able to transfer a key from an instance to another (or to copy it). The performance is pretty impressive. So the idea is that you can migrate two keys from two different instances into a "temporary" instance that you use just for computation, and then perform what you want in this instance. This way we'll provide all the tools needed to people that understand what they are doing to do that. But will not provide direct support for something that in the general case can't be super fast and horizontally scalable.

simonw 16y ago
It sounds like I'll just need to make sure I keep sets I'm going to need to intersect with each other on the same node. They're usually sets of integers so that shouldn't be too much of a problem for the kinds of thing I use Redis for.
- antirez 16y ago
  Cool that it works for you, but for an use case where you have many distributed sets, and you need to intersect cross-node, the simplest thing you could do is to get the smallest transfered in the other node via MIGRATE against a temporary key, intersect, delete it. At least we know there is an exit strategy :)
  - jaksprats 16y ago
    that will work (as good as can be hoped for). the process can be thought of as a map-reduce. first pass would be figure out where all your SETs are located and how big they are. then intersect all co-located SETs. then (in parallel) take smaller SETs and MIGRATE them towards larger SETS, intersect those and repeat (respecting new sizes). A pyramid of SET intersection can be done until the result is the final two SETs being intersected, a process will need to coordinate all of this, but luckily intersections are idempotent, so if one parallel job finishes quicker, it doesn't need to wait. W/ redis speed, the overhead (for the coordinator) to get the size of the different SETs in all these steps should be minimal as compared to the time taken to MIGRATE the data (meaning for-free). Also w/ redis' speed the MIGRATE will be very fast and the cross-node-join (or cross-node-intersecton) bottlenecks on network I/O, so if a good framework for this redis map-reduce is created, it will be a pretty optimal setup, and it wont bloat the server w/ tasks that can not be done directly in the server (cross-node-intersections are done at something more like a proxy level, they require intermediate results). this problem is a hard one, data analytic stores optimise to this problem by storing data redundantly w/ a "pre-joined" colocation strategy, which works for star schemas and a limited number of tables, but doesnt make sense w/ 1000s of SETs, so this is a real good solution and classic redis anti-bloat.
- bretthoerner 16y ago
  We have the same need at Disqus, and I talked about our solution here: http://news.ycombinator.com/item?id=2247132 http://news.ycombinator.com/item?id=2247132

by antirez 16y ago

simonw 16y ago

It sounds like I'll just need to make sure I keep sets I'm going to need to intersect with each other on the same node. They're usually sets of integers so that shouldn't be too much of a problem for the kinds of thing I use Redis for.

antirez 16y ago
Cool that it works for you, but for an use case where you have many distributed sets, and you need to intersect cross-node, the simplest thing you could do is to get the smallest transfered in the other node via MIGRATE against a temporary key, intersect, delete it. At least we know there is an exit strategy :)
- jaksprats 16y ago
  that will work (as good as can be hoped for). the process can be thought of as a map-reduce. first pass would be figure out where all your SETs are located and how big they are. then intersect all co-located SETs. then (in parallel) take smaller SETs and MIGRATE them towards larger SETS, intersect those and repeat (respecting new sizes). A pyramid of SET intersection can be done until the result is the final two SETs being intersected, a process will need to coordinate all of this, but luckily intersections are idempotent, so if one parallel job finishes quicker, it doesn't need to wait. W/ redis speed, the overhead (for the coordinator) to get the size of the different SETs in all these steps should be minimal as compared to the time taken to MIGRATE the data (meaning for-free). Also w/ redis' speed the MIGRATE will be very fast and the cross-node-join (or cross-node-intersecton) bottlenecks on network I/O, so if a good framework for this redis map-reduce is created, it will be a pretty optimal setup, and it wont bloat the server w/ tasks that can not be done directly in the server (cross-node-intersections are done at something more like a proxy level, they require intermediate results). this problem is a hard one, data analytic stores optimise to this problem by storing data redundantly w/ a "pre-joined" colocation strategy, which works for star schemas and a limited number of tables, but doesnt make sense w/ 1000s of SETs, so this is a real good solution and classic redis anti-bloat.
bretthoerner 16y ago
We have the same need at Disqus, and I talked about our solution here: http://news.ycombinator.com/item?id=2247132 http://news.ycombinator.com/item?id=2247132