5 ms·

> In addition we wanted open source implementations of features currently only available in proprietary modules. We feel a fork is the best way to accelerate d

> In addition we wanted open source implementations of features currently only available in proprietary modules. We feel a fork is the best way to accelerate development in the areas of most interest to us. This is probably the most important part of the fork.

jdsully 8y ago
In that vein we have FLASH support baked in. It works a bit differently than Redis Enterprise (uses File System snapshotting), but it’s also a lot less complicated.
- otterley 8y ago
  What tradeoffs have you made by doing so?
  - jdsully 8y ago
    The trade off is the kernel decides what is paged into RAM and what stays on disk. Less control, but also less overhead. This is a similar approach to Squid.
    - the_duke 8y ago
      Kafka also leaves the caching to the OS.
      - otterley 8y ago
        ... which is great until a consumer or replica decides to replay a topic/partition from the beginning, and forces all the recently-written data to be evicted, adversely impacting performance for all the well-behaved consumers :(
        cafxx 8y ago
        > forces all the recently-written data to be evicted That's unlikely unless 1) your hosts are seriously underpowered and 2) you have no per-consumer quotas set
        otterley 8y ago
        Can you clarify what you mean by underpowered? Not every site wants to replace terabytes of disk with NVMe storage to make paging faster - especially sites that have a lot of throughput and long retention demands. Second, even if quotas could be used - and there are plenty of sites where they cannot - there’s still a broker-replacement scenario that needs to be accounted for. Most folks don’t want to intentionally impede the recovery rate of a failed broker.
    - otterley 8y ago
      I don’t think users are going to be happy with that approach in the long term. It’s well advised to control eviction behavior yourself. (Varnish learned this lesson ages ago, notwithstanding phk’s original paper.) InnoDB’s buffer pool documentation is instructive: https://dev.mysql.com/doc/refman/5.7/en/innodb-buffer-pool.html https://dev.mysql.com/doc/refman/5.7/en/innodb-buffer-pool.h...
      - derefr 8y ago
        If you're running a Redis instance as the only thing on the VM instance it's on (very likely in my experience), is there a reason to reimplement all this policy logic in Redis, rather than simply tuning the caching policy of the existing OS kernel to make it perform well for Redis (maybe at the expense of performing well for any other process you might run under that same kernel)?
        otterley 8y ago
        In theory that makes sense, but I'm not aware of any mechanism that allows you to tune a kernel's VMM to suit an arbitrary workload/access pattern. Are you?
        batbomb 8y ago
        this is kind of a pain the container world
      - jdsully 8y ago
        Certainly we can do smarter things if we know an access is going to result in a page. But we're not in a position to take advantage of that until green threads are in place.
kristoff_it 8y ago
Considering that modules are (like the name suggests), detached components, how's that relevant with the fork? If anybody wants to build an alternative to Redis Labs' (or anybody else's) modules they can already do so.
antirez 8y ago
I beg to differ: a much better route would be to send PRs, or even add in the fork itself, modules APIs support to do what those modules do (there are Redis Labs modules that cannot be modeled via the current module interface). I did this error with Disque and now I'm implementing Disque back in terms of Redis modules, because keeping a fork in sync is a nightmare.

jdsully 8y ago

In that vein we have FLASH support baked in. It works a bit differently than Redis Enterprise (uses File System snapshotting), but it’s also a lot less complicated.

otterley 8y ago
What tradeoffs have you made by doing so?
- jdsully 8y ago
  The trade off is the kernel decides what is paged into RAM and what stays on disk. Less control, but also less overhead. This is a similar approach to Squid.
  - the_duke 8y ago
    Kafka also leaves the caching to the OS.
    - otterley 8y ago
      ... which is great until a consumer or replica decides to replay a topic/partition from the beginning, and forces all the recently-written data to be evicted, adversely impacting performance for all the well-behaved consumers :(
      - cafxx 8y ago
        > forces all the recently-written data to be evicted That's unlikely unless 1) your hosts are seriously underpowered and 2) you have no per-consumer quotas set
        otterley 8y ago
        Can you clarify what you mean by underpowered? Not every site wants to replace terabytes of disk with NVMe storage to make paging faster - especially sites that have a lot of throughput and long retention demands. Second, even if quotas could be used - and there are plenty of sites where they cannot - there’s still a broker-replacement scenario that needs to be accounted for. Most folks don’t want to intentionally impede the recovery rate of a failed broker.
  - otterley 8y ago
    I don’t think users are going to be happy with that approach in the long term. It’s well advised to control eviction behavior yourself. (Varnish learned this lesson ages ago, notwithstanding phk’s original paper.) InnoDB’s buffer pool documentation is instructive: https://dev.mysql.com/doc/refman/5.7/en/innodb-buffer-pool.html https://dev.mysql.com/doc/refman/5.7/en/innodb-buffer-pool.h...
    - derefr 8y ago
      If you're running a Redis instance as the only thing on the VM instance it's on (very likely in my experience), is there a reason to reimplement all this policy logic in Redis, rather than simply tuning the caching policy of the existing OS kernel to make it perform well for Redis (maybe at the expense of performing well for any other process you might run under that same kernel)?
      - otterley 8y ago
        In theory that makes sense, but I'm not aware of any mechanism that allows you to tune a kernel's VMM to suit an arbitrary workload/access pattern. Are you?
      - batbomb 8y ago
        this is kind of a pain the container world
    - jdsully 8y ago
      Certainly we can do smarter things if we know an access is going to result in a page. But we're not in a position to take advantage of that until green threads are in place.

kristoff_it 8y ago

Considering that modules are (like the name suggests), detached components, how's that relevant with the fork? If anybody wants to build an alternative to Redis Labs' (or anybody else's) modules they can already do so.

antirez 8y ago

I beg to differ: a much better route would be to send PRs, or even add in the fork itself, modules APIs support to do what those modules do (there are Redis Labs modules that cannot be modeled via the current module interface). I did this error with Disque and now I'm implementing Disque back in terms of Redis modules, because keeping a fork in sync is a nightmare.