And to add to point (4): as Marek says KT's replication is great... to a point. When Cloudflare was small and we only had to replicate to a relatively small num

And to add to point (4): as Marek says KT's replication is great... to a point. When Cloudflare was small and we only had to replicate to a relatively small number of machines it coped fine. As we scaled out each data centre we quickly hit limitations in KT's ability to replicate to many clients. We ended up having to install many more replication points in each data centre or region just to allow KT to replicate to all downstream clients, otherwise it would regularly stall while trying to service all replication requests, massively slowing down replication, or sometimes even missing updates. There's also two more points to be made: (6) As our KC (Kyoto Cabinet) databases grew, KT had trouble keeping consistency of the file on disk. Sometimes a restart of the service would leave you with a corrupted KC database, causing us to have to resync the node from scratch. This was a huge operational burden - especially if corruption somehow replicated downstream. (7) KT allows you to write to replicas. This one we always found crazy. You can write (or overwrite) a key on a downstream replica and that only ever stays on the replica - it can't propagate upwards (and nor should it). Until the top-level writes to that key again it will remain with this erroneous value. This made monitoring problematic because it would cause a keycount or replication diff if it ever happened.