9 ms·
Cloud Memorystore: A fully managed in-memory data store service for Redis
- caleblloyd 9y agoWhy is it that GCP can provide an internal IP address for Cloud Memorystore but not for Google Cloud SQL? It would be beneficial for these teams to work together, no internal IP in cloud SQL makes it much less appealing then AWS RDS.
- laixer 9y agoPrivate IP support for Cloud SQL is on the roadmap. The reality of development is that some things are easier to do on a new product. It's not a matter of communication.
- peter_retief 9y agoGreat concept, I am having a serious look
- jchw 9y agoThis is exciting, but I really hope that Terraform will add support for it in reasonable time. These days I'd prefer to not manually manage cloud resources if I don't have to.
- poooogles 9y agoAFAIK the policy is will be added once it's GA?
- jkaplowitz 9y agoSooner would be helpful to people who want to test the service while it's in beta in anticipation of relying on it in production after GA. Do you know if they document their Terraform timeline policy for GCP anywhere?
- dimitropoulos 9y agoIf postgres on cloud sql is any indication, then it will only be another year and a half! :) They seem to take the Beta/GA process very seriously and don't rush things to GA.
- tecleandor 9y agoHashicorp doesn't seem to like Google Cloud too much. I've been moving to Google Cloud Builder and it's Jinja and Python templates, which I happen to be a little familiar with...
- jkaplowitz 9y agoGoogle Cloud Deployment Manager, you mean? Yeah, that is linked tightly with Google's API system which means it has the fastest support for new things. Google does (last I checked) put some paid time into systems like Terraform so that they support GCP well, but there's always going to be more of a lag, especially before GA. I don't know Terraform well enough to know how big the lag is for new AWS features, but some lag probably exists even there despite AWS's dominant market share.
- samstave 9y agoPlz do a write-up. EOM
- dantiberian 9y agoI’m not sure why you’d say that Hashicorp doesn’t like Google Cloud. I can think of recent partnerships between Hashicorp and Google on Vault and Terraform that are both substantial integrations. Google Cloud and Hashicorp both have people that work on the GCP Terraform Provider, and they do an excellent job keeping it up to date. By all external measures this looks like a great relationship.
- _asummers 9y agoThere's a bunch of undocumented GCB features by the way. Pop into their Slack and they'll often give you tips that haven't made their way into the docs yet. We've been mostly happy with it, but still use Jenkins as the kickoff point.
- outworlder 9y agohttps://github.com/terraform-providers/terraform-provider-google https://github.com/terraform-providers/terraform-provider-go... "This provider plugin is maintained by: The Google Cloud Graphite Team at Google The Terraform team at HashiCorp " So Google can also help in this regard.
- danawillow 9y agoHey jchw, Dana from GCP here. No promises on the timeline, but I can tell you that adding a request in the issue tracker (https://github.com/terraform-providers/terraform-provider-google/issues https://github.com/terraform-providers/terraform-provider-go...) will at least put it on our radar so we start working on it sooner.
- dimitropoulos 9y agodone: https://github.com/terraform-providers/terraform-provider-google/issues/1467 https://github.com/terraform-providers/terraform-provider-go...
- neuland 9y ago> authorized networks ensure that the Redis instance is accessible only when connected to the authorized VPC network That's good that this is handled smoothly as it's the way Redis wants auth to work. I've always seen the security story to be one of the big weaknesses of Redis. No TLS, opt-in password with no user and only one shared password, no per-database privileges, etc. Does anyone know the status of TLS in Redis? I heard somewhere that Amazon has a patch to add that directly, rather than having to use stunnel.
- kozziollek 9y agoIt's being worked in in pull request: https://github.com/antirez/redis/pull/4855 https://github.com/antirez/redis/pull/4855 Also, antirez recently met with patch author: https://news.ycombinator.com/item?id=16943289 https://news.ycombinator.com/item?id=16943289
- neuland 9y agoCool! I saw that thread last week, but didn't check back in to see that antirez responded.
- dudus 9y agoThere's a fine print that says: > ¹Basic Tier instances experience a downtime and a full cache flush during scaling. Standard Tier instance experience very minimal downtime and loss of some unreplicated data during scaling operation. ²Applicable for GA release only. Does that mean that right now there are no persistance options with this Managed Redis cluster? I have never tried to use Redis but I've been eyeing it for a while and the most confusing part is how it handles persistence. Apparently there are ways to have Redis configured to be pretty durable as outlined here: https://redis.io/topics/persistence https://redis.io/topics/persistence But it doesn't look like Google is supporting either RDB nor AOF with this managed service, they say RDB is coming though. Doesn't it heavily limits the use cases for this product and relegate it to a cooler memcache? This is not criticism, it's just curiosity on good use cases for this product.
- ihsw2 9y agoOn the surface it looks like this isn't a true-blue Redis server (or even a fork of Redis) but support for the Redis protocol bolted onto an in-house memory-based caching product.
- panarky 9y agoTheir post isn't very clear on persistence, here's how I read it. 1) Yes, they're using Redis persistence (RDB, AOF, or a Google-written persistence layer, it's not specified). 2) Basic Tier scales by tearing down the smaller Redis instance and migrating the data (???) to a larger instance. This causes downtime, but not lost data. 3) Standard Tier migrates using replication during scaling, so there's little downtime but if data is in flight at cutover, it could be lost. 4) And "coming soon" you can migrate data in and out of Cloud Memorystore by importing and exporting RDB files.
- dudus 9y agoWhy do you think they are using persistence at all, and why do you think Basic Tier doesn't have data loss on scaling? I read it differently. When they say full cache flush doesn't that imply full database flush, I was thinking all data gets wiped on scaling operation.
- anonfunction 9y agoIs it managed redis or something google made themselves that has the same API? I wonder if they will keep up with all the new stuff in redis like streams.
- itcmcgrath 9y agoIt's the open source version of Redis v3.2.11
- anonfunction 9y agoThanks!
- f311a 9y agoCan someone explain use cases of using remote redis? Doesn't network delay is a big downside?
- ssambros 9y agoIt is intended to be accessed by resources running in the same Google Cloud zone/region, so the network latency is minimal.
- monocasa 9y agoIn a datacenter, a network request to a server that has the information in DRAM can be faster than pulling it locally off of an SSD. It goes back and forth with perf gains on either side, but they're on the same order with different tradeoffs.
- ryanworl 9y agoEspecially if that "local SSD" is really a virtualized block device over a network anyway, which is how most people are deploying their cloud stuff for convenience.
- qeternity 9y agoIs this still the case? We don’t use much cloud from the big boys but I thought even they were moving towards local storage. All of our OVH instances are local SSD backed.
- ryanworl 9y agoBoth are available and AWS and GCP are pretty up-front about what you should use depending on the use case. The virtualized devices do have advantages in terms of being able to be moved around and not tied to the life of a single instance.
- deleted 9y ago[deleted]
- 9y ago
- nodesocket 9y agoIf Redis Cluster support is coming soon, how are they currently handing cross region replication and failover? Is Redis Sentinel (painful and requires client support) deprecated now?
- ssambros 9y ago> how are they currently handing cross region replication and failover? We do not, unfortunately. Redis service is regional and only failover from one zone to another within a region is provided. -- I am an engineer on Memorystore Redis team.
- pulkitsh1234 9y agoIt seems like they might be running Redis Sentinels (for handling failover and replication), but exposing it outside as a single IP, instead of giving a list of sentinels to connect to. Applications might use that endpoint like a standalone Redis instance. Requests to that endpoint will be routed to the current master. Effectively abstracting the details about the Sentinels and their configuration from the clients. Not too sure, just a guess.
- meteor333 9y agohow does this compare with Redis Labs?
- nodesocket 9y agoIt is in your GCP account, and thus VPC. Has a private ip address.
- manigandham 9y agoRedis Labs (the official sponsor of Redis now) uses a custom proxy layer on top of Redis, giving you the latest 4.0 version with support for modules, no downtime and features like storing the values of keys on SSDs. They also support cross-region multi-master using CRDTs. If you need VPC/internal access only, or are fine with the lower level of features, or just want to consolidate on GCP services, then Cloud Memorystore can work, but otherwise I'd recommend Redis Labs.
- gingerlime 9y agoUnfortunately redis labs isn’t available in all regions of GCloud. I was effectively begging them to add support in Europe a couple of months ago.
- sandGorgon 9y agosame here for India datacenter of AWS or GCloud
- panarky 9y agoThe three big reasons I use Redis are: 1) Data structures like lists, hashmaps and sets 2) Persistence 3) Blindingly fast I wanted to see if Google's implementation sacrifices any speed. snip EDIT: I deleted the benchmark results from this post because they're meaningless. I used the "intrinsic latency" tool [0] that Redis provides, but it must be run on the server. So my results only reflected the intrinsic latency of the client VM, not the new Cloud Memorystore. My apologies. [0] https://redis.io/topics/latency https://redis.io/topics/latency
- manigandham 9y agoMicro instances are shared CPU cores, not very useful for measuring latency in a single-threaded application like Redis.
- sabareesh 9y agoThis is not fair to compare with micro instance
- douglasfshearer 9y agoIt would be fairer to compare with an AWS ElastiCache Redis instance [0]. [0] https://aws.amazon.com/elasticache/redis/ https://aws.amazon.com/elasticache/redis/
- detaro 9y agoA more interesting comparison might be an n1-standard-1, since that's roughly the same price as the 1 GB Memorystore (but has more memory)
- fusl 9y agoYour benchmarks: Google's new Cloud Memorystore, 1.0 GB: redis-cli -h 10.0.0.3 --intrinsic-latency 100 677411416 total runs in 100 seconds avg latency: 0.1476 microseconds / 147.62 nanoseconds per run Open-source Redis on a Google Cloud Platform micro instance, 0.6 GB: redis-cli -h localhost --intrinsic-latency 100 353427208 total runs in 100 seconds avg latency: 0.2829 microseconds / 282.94 nanoseconds per run Open-source Redis on an AWS EC2 micro instance, 1.0 GB: redis-cli -h localhost --intrinsic-latency 100 21681751 total runs in 100 seconds avg latency: 4.6122 microseconds / 4612.17 nanoseconds per run My benchmark: Open-source Redis on a Hetzner Cloud VPS, CX11 (92% cheaper than Google's Cloud Memorystore): redisbench-client:~# redis-cli -h 88.99.124.195 --intrinsic-latency 100 Max latency so far: 1 microseconds. Max latency so far: 77 microseconds. Max latency so far: 113 microseconds. Max latency so far: 130 microseconds. Max latency so far: 2562 microseconds. Max latency so far: 2835 microseconds. Max latency so far: 4165 microseconds. Max latency so far: 5497 microseconds. 757281326 total runs (avg latency: 0.1321 microseconds / 132.05 nanoseconds per run). Worst run took 41628x longer than the average latency.
- merb 9y agohow can a instance with 1gb cost 35 USD per month... also I would've hoped that google won't add tiers, more like "limits" and that if I only host 1mb on my instance that I would only pay for 1mb... besides that I could scale to XGb (depending on my limit)...
- GordonS 9y agoI totally get the benefits of a managed service, but...$35/month for 1GB is just silly.
- dsymonds 9y agoAre you sure you fully understand the benefit? $35 is, what, 30 minutes of an engineer's time from a company's perspective? If the management of the service would take any more than that every month (and it easily could!), you're ahead.
- GordonS 9y agoNot everyone works in the Bay making boat-loads of cash. My comment was from the perspective of a small business in Europe.
- dsymonds 9y agoEven then, I think it'd be unlikely that the time-cost wouldn't make it worthwhile. People generally underestimate how much time gets soaked up by managing a computer service.
- Can_Not 9y ago$35 seems to be 3x more cost than AWS's smallest tier (no further comparison of features/value beyond price), so it does seem expensive for an introductory project, considering a single $35~ Digital Ocean VPS might all some companies need their first year. On the other hand, the number of times I've seen an engineer suggest spending $500-$2000 in labor/hours (when already behind on mission critical work) as alternative to spending $5/month in some SaaS is just staggering.
- Elect2 9y agoWhat are the benefits of using this compare to self-hosted Redis on a Compute instance?
- jpatokal 9y agoIt's fully managed, so you don't need to worry about OS security patches, rotating logs and all the other administration that comes with running your own server. Basically the same benefits as using Cloud SQL/Aurora RDS vs maintaining your own MySQL server. Disclaimer: I work at GCP, but not on Memorystore.
- kraftman 9y agoNot having to deal with failover and scaling is also handy.
- Arqu 9y agoI was hoping to see something of a more flexible pricing like per 100 MB provisioned based on the far fetched assumption that it is not just a managed redis(like?) offering but a fully abstracted managed redis service where you are just charged by GB/s used as with lots of other services. This just boils down to the same cost as self hosting minus the self hosting hassle. So not too bad as its just easier to use + no vendor lock in since it is redis compatible at a similar price point.
- edf13 9y agoBiggest concern I have about all these hosted Redis replacements is that of latency. I use Redis on a highly controlled internal network and also local for its speed... This goes when moving to a hosted solution.
- jkaplowitz 9y agoIf you're already accessing it from within GCP, their network is excellent, not much lower latency than a high-quality internal physical network (under an order of magnitude) and highly flexible and configurable. In that use case it should be comparable to any solution you can host within GCP except hosting on the same instance as each client and using loopback access. For accessing from outside GCP, yeah the broader internet latency would figure into this as any other external solution.
- edf13 9y agoIt depends on what you mean by not much lower latency though... At Redis speeds the network latency can quite often be the bottleneck.
- jkaplowitz 9y agoDouble digits of microseconds of difference as of some time last year, if I remember correctly - it might have gotten even better since then since I know Google wanted to narrow that gap. (Disclaimer: While I have in the past worked for the GCP team, nothing in this comment relates to my time at Google or to info I learned then.)
- PRJ2999 9y agoI was pretty excited when Memorystore came out, as there is very little I enjoy less than managing redis servers. However, I've had nothing but trouble since switching. Applications than ran smoothly on redis instances deployed on kubernetes engine are giving me all sorts of problems now. "NOREPLICAS Not enough good slaves to write" errors have been extremely common, sometimes followed by problems connecting. Then operations on the memorystore instance start pending (I mean "repairing"), taking 20+ minutes. During that time I obviously get connection refused errors. This usually comes with huge spikes in network in/out that are completely unexplained by the app pointing to the service. I thought the pitch for MemoryStore was that it was a managed service; y'know, less time on devops and all of that. I've found the exact opposite to be the case. Pay more to Google, spend more time on devops, get a redis service that doesn't work.