8 ms·

Plans for Redis 3.2

115 points by mostafah 11y ago

nefasti 11y ago
It's almost scary to see how productive, focused and well versed Salvatore Sanfilippo is. Most of the developers on my team struggle to build simple CRUD web apps, wasting long hours complaining about how wrong the Domain models is or how why this things is not in that place.
- zapt02 11y ago
  I get this as well. In my experience these are people who just don't like what they're doing (and are also often bad at it), so they try to undermine every decision. I haven't found a good way of dealing with this so I'd love to hear some pointers.
  - modarts 11y ago
    Quitting
- Klinky 11y ago
  The simplistic low-level nature of redis probably helps avoid fatigue. Many of the data structures implemented are well defined in their purpose, it's more a matter of implementation & optimization. The same is not true for higher-level projects, such as CRUD apps, where project structure and project goals can often be vague and are likely to change during the course of the project. This isn't to take away from Salvatore, who is obviously talented and seems to have a knack for researching and implementing the data structures in an efficient manner.
  - banachtarski 11y ago
    I don't agree with this comment at all. Low-level high-perf programs are extremely demanding and require a good deal of care and persistence. "The simplistic low-level nature of redis probably helps avoid fatigue." Yea I'm going to bet you never wrote anything where microseconds or L1 cache misses mattered.
    - Klinky 11y ago
      I didn't say it was easy. redis is full of reinventing the wheel in the name of improved performance. This is something developers often do, but are often told it's pointless, just use built-ins. Salvatore has found a niche where he can see these optimized implementations to their end. I would imagine there is a certain level of satisfaction that comes from thoroughly researching a topic, implementing it as best you can, and deploying it to the public you're popular with. It probably helps having firm control of the project with less red tape. It's like a comparison between a craft woodworker and someone who would like to play around with the concepts, but doesn't have the tools, time or know how, or they're told there is no point in building your own furniture from scratch, so they are instead building Ikea furniture.
    - bch 11y ago
      A humble idea: Maybe it's a case of "developer interest dissonance". Technical people digging technical problems (cache hit issues) versus business logic... which might be why they "got into computers ".
  - jkot 11y ago
    This. Redis has single-threaded compact code base.
- eknkc 11y ago
  To be fair, he seems to like doing these things.
- deleted 11y ago
  [deleted]
Freaky 11y ago
Native rotating bloom filters, fantastic! Sort of thing I've had to do myself more than once - great efficient way to be able to ask "Have I seen this recently?".
VieElm 11y ago
Aphyr has a great post about Redis: https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis Recommends using it on a single server or if distributed as a cache or for data that can tolerate write losses.
- Goopplesoft 11y ago
  That post was a bit before the new redis clustering in 3.0 (http://antirez.com/news/79 http://antirez.com/news/79 -- Redis cluster, no longer vaporware). Wonder if anything has changed.
  - lobster_johnson 11y ago
    Aside from the fact that Aphyr showed basic replication to be flawed, as far as I know Cluster isn't even designed to be consistent across partitions. The announcement page you linked to doesn't let me copy any text on iOS, but he's basically saying that if you get a partition and resolve it, the merge algorithm used is prone to data loss. I don't know if Redis Cluster has improved since.
- antirez 11y ago
  Hello, Redis uses asynchronous replication, in order to have a latency you expect for most use cases where Redis is used, however after the tests Aphyr performed we tried to improve what was possible to improve without transforming Redis into something completely different. Basically even with asynchronous replication and a drastic merge function (last failover wins), you can still do things to improve the real world behavior of the system, and this is what we did: 1) Improve Redis master-slave protocol in order to have asynchronous acknowledges, at least. So it is now possible, during partitions, to have a bound data loss: after some time the isolated master stops accepting writes. This is implemented both in Redis Cluster (natively) and in Redis vanilla replication (so can be used with Sentinel or other failover mechanisms). 2) Use a replication offset in order to try, during failovers, to pick the slave with the most updated state. Currently it is just an idea but in the future for certain data types and operations, Redis Cluster may take a log of the operations that were not acknowledged by all the slaves of a given master, and when the master is turned into a slave, it would re-play the log into the new master, at least for operations which are idempotent and commutative. This would allow for safer writes. The other (more used) approach of merging values directly is not very easy to perform with Redis for a number of reasons.
Axsuul 11y ago
Has Salvatore ever considered forking redis into a legal entity and providing official commercial support? I'm willing to bet alot of startups and enterprises don't use redis because of the lack of this.
- tvmalsv 11y ago
  According to the Redis support page, Pivotal sponsors Redis and they provide commercial support services for it. Although, to be honest, the page linked from the Redis site (http://www.pivotal.io/big-data/redis http://www.pivotal.io/big-data/redis) doesn't seem to have much to say on the matter. Looks like an ad for their cloud platform, and I wouldn't have guessed they offer support for Redis itself.
  - jacques_chester 11y ago
    Ah yes, our brochureware site is too vague for my tastes, but Pivotal-the-company is mostly pitched at F500s, not engineers per se. At the coalface, Pivotal also includes Pivotal Labs (for whom I work and from which the larger company took its name), which includes people who've done a lot of production Redis work. And we can always ask Salvatore for a tip if we get stuck. Similarly, I have access to Hadoop, Gemfire, Greenplum, Cloud Foundry, RabbitMQ or Spring devs for help. Being inside the same tent is nifty.
    - itamarhaber 11y ago
      > "brochureware site" <- don't be too hard on yourselves, there's plenty of goodness there too :) Besides Pivotal's commercial support for Redis (which, from what I've heard, is superb) there are also at least two hardcore Redis community members who're providing professional support for Redis deployments - for more details, contact Michel Martens and Damian Janowski. Redis Labs, my employer, is also heavily into Redis (duh) so naturally we give support for our product line that consists of a Redis on-prem solution and a cloud service.
      - jacques_chester 11y ago
tlarkworthy 11y ago
Scalable bloom filters! [1] Drop size configuration and just have a maximum false positive rate. FWIW I built one by wrapping Guava's, you can take an existing bloom filter implementation and wrap auto-scaling around them. [1] http://gsd.di.uminho.pt/members/cbm/ps/dbloom.pdf http://gsd.di.uminho.pt/members/cbm/ps/dbloom.pdf

115 points by mostafah 11y ago

It's almost scary to see how productive, focused and well versed Salvatore Sanfilippo is. Most of the developers on my team struggle to build simple CRUD web apps, wasting long hours complaining about how wrong the Domain models is or how why this things is not in that place.

zapt02 11y ago
I get this as well. In my experience these are people who just don't like what they're doing (and are also often bad at it), so they try to undermine every decision. I haven't found a good way of dealing with this so I'd love to hear some pointers.
- modarts 11y ago
  Quitting
Klinky 11y ago
The simplistic low-level nature of redis probably helps avoid fatigue. Many of the data structures implemented are well defined in their purpose, it's more a matter of implementation & optimization. The same is not true for higher-level projects, such as CRUD apps, where project structure and project goals can often be vague and are likely to change during the course of the project. This isn't to take away from Salvatore, who is obviously talented and seems to have a knack for researching and implementing the data structures in an efficient manner.
- banachtarski 11y ago
  I don't agree with this comment at all. Low-level high-perf programs are extremely demanding and require a good deal of care and persistence. "The simplistic low-level nature of redis probably helps avoid fatigue." Yea I'm going to bet you never wrote anything where microseconds or L1 cache misses mattered.
  - Klinky 11y ago
    I didn't say it was easy. redis is full of reinventing the wheel in the name of improved performance. This is something developers often do, but are often told it's pointless, just use built-ins. Salvatore has found a niche where he can see these optimized implementations to their end. I would imagine there is a certain level of satisfaction that comes from thoroughly researching a topic, implementing it as best you can, and deploying it to the public you're popular with. It probably helps having firm control of the project with less red tape. It's like a comparison between a craft woodworker and someone who would like to play around with the concepts, but doesn't have the tools, time or know how, or they're told there is no point in building your own furniture from scratch, so they are instead building Ikea furniture.
  - bch 11y ago
    A humble idea: Maybe it's a case of "developer interest dissonance". Technical people digging technical problems (cache hit issues) versus business logic... which might be why they "got into computers ".
- jkot 11y ago
  This. Redis has single-threaded compact code base.
eknkc 11y ago
To be fair, he seems to like doing these things.
deleted 11y ago
[deleted]

Freaky 11y ago

Native rotating bloom filters, fantastic! Sort of thing I've had to do myself more than once - great efficient way to be able to ask "Have I seen this recently?".

VieElm 11y ago

Aphyr has a great post about Redis: https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis Recommends using it on a single server or if distributed as a cache or for data that can tolerate write losses.

Goopplesoft 11y ago
That post was a bit before the new redis clustering in 3.0 (http://antirez.com/news/79 http://antirez.com/news/79 -- Redis cluster, no longer vaporware). Wonder if anything has changed.
- lobster_johnson 11y ago
  Aside from the fact that Aphyr showed basic replication to be flawed, as far as I know Cluster isn't even designed to be consistent across partitions. The announcement page you linked to doesn't let me copy any text on iOS, but he's basically saying that if you get a partition and resolve it, the merge algorithm used is prone to data loss. I don't know if Redis Cluster has improved since.
antirez 11y ago
Hello, Redis uses asynchronous replication, in order to have a latency you expect for most use cases where Redis is used, however after the tests Aphyr performed we tried to improve what was possible to improve without transforming Redis into something completely different. Basically even with asynchronous replication and a drastic merge function (last failover wins), you can still do things to improve the real world behavior of the system, and this is what we did: 1) Improve Redis master-slave protocol in order to have asynchronous acknowledges, at least. So it is now possible, during partitions, to have a bound data loss: after some time the isolated master stops accepting writes. This is implemented both in Redis Cluster (natively) and in Redis vanilla replication (so can be used with Sentinel or other failover mechanisms). 2) Use a replication offset in order to try, during failovers, to pick the slave with the most updated state. Currently it is just an idea but in the future for certain data types and operations, Redis Cluster may take a log of the operations that were not acknowledged by all the slaves of a given master, and when the master is turned into a slave, it would re-play the log into the new master, at least for operations which are idempotent and commutative. This would allow for safer writes. The other (more used) approach of merging values directly is not very easy to perform with Redis for a number of reasons.

Axsuul 11y ago

Has Salvatore ever considered forking redis into a legal entity and providing official commercial support? I'm willing to bet alot of startups and enterprises don't use redis because of the lack of this.

tvmalsv 11y ago
According to the Redis support page, Pivotal sponsors Redis and they provide commercial support services for it. Although, to be honest, the page linked from the Redis site (http://www.pivotal.io/big-data/redis http://www.pivotal.io/big-data/redis) doesn't seem to have much to say on the matter. Looks like an ad for their cloud platform, and I wouldn't have guessed they offer support for Redis itself.
- jacques_chester 11y ago
  Ah yes, our brochureware site is too vague for my tastes, but Pivotal-the-company is mostly pitched at F500s, not engineers per se. At the coalface, Pivotal also includes Pivotal Labs (for whom I work and from which the larger company took its name), which includes people who've done a lot of production Redis work. And we can always ask Salvatore for a tip if we get stuck. Similarly, I have access to Hadoop, Gemfire, Greenplum, Cloud Foundry, RabbitMQ or Spring devs for help. Being inside the same tent is nifty.
  - itamarhaber 11y ago
    > "brochureware site" <- don't be too hard on yourselves, there's plenty of goodness there too :) Besides Pivotal's commercial support for Redis (which, from what I've heard, is superb) there are also at least two hardcore Redis community members who're providing professional support for Redis deployments - for more details, contact Michel Martens and Damian Janowski. Redis Labs, my employer, is also heavily into Redis (duh) so naturally we give support for our product line that consists of a Redis on-prem solution and a cloud service.
    - jacques_chester 11y ago

tlarkworthy 11y ago

Scalable bloom filters! [1] Drop size configuration and just have a maximum false positive rate. FWIW I built one by wrapping Guava's, you can take an existing bloom filter implementation and wrap auto-scaling around them. [1] http://gsd.di.uminho.pt/members/cbm/ps/dbloom.pdf http://gsd.di.uminho.pt/members/cbm/ps/dbloom.pdf