13 ms·
Redis 6.0 GA
- alufers 7y agoI have a bit of a tangent question for more experienced back-end developers: where do you fit Redis (or other caching mechanisms) in the traditional MVC model? I haven't had a use-case for Redis yet, but I'd like to know how should I approach the architecture of an app using Redis.
- hrdwdmrbl 7y agoIF you want to use it for caching, THEN you would use it to cache stuff for your controller. But, Redis is much more than caching. It supports all kinda of fun data-structures like sorted lists, timeouts, sets, pub-sub and more! You can almost think of it like memory that is held by another process. In that way, there are SO many uses cases.
- penagwin 7y agoI think a good example of this is session storage - just store the session with a ttl and now redis will automatically "expire the session" when the time is over.
- jrochkind1 7y agoWell, the use for a cache is caching expensive operations. Sorry if this is just stating the obvious, but I'm not sure how else to answer how it fits into traditional MVC operations. It could be a front-end HTTP cache (although you'd probably use a CDN for that instead). It could be caching something expensive to look up or calculate, for which it's fine to not have up-to-the-second-current value. Many people at least in Rails also use Redis to hold a queue for background processing, where it ends up working well, although hypothetically many other things could too. You can also use redis for all sorts of other stuff that isn't caching or a queue.
- pletnes 7y agoYou can use it to keep track of session cookies across multiple web workers.
- csunbird 7y agoOur main use case in $JOB for Redis is distributed locking. We usually do not need key-value storage and even if we do, we just go with DynamoDB instead.
- ryanisnan 7y agoThere's lots of functionality that Redis provides beyond a key-value store. For example the data structures that it supports. These are very powerful on their own. Also I understand after considerable investments vendor freedom is a bit of an illusion, but you know, if you can choose an OS technology under the hood, it's effectively like not nailing shut a door, but still leaving it closed.
- Axsuul 7y agoAre you using the method[0] suggested by antirez for distributed locking? 0. https://redis.io/topics/distlock https://redis.io/topics/distlock
- csunbird 7y agoWe use Redisson, yes.
- chrstphrhrt 7y agoSTREAMS, best pubsub solution imho. I used it as a backing store for an MQTT frontend once, and also for generally coordinating worker processes to handle background tasks.
- chrstphrhrt 7y agoJust to add a bit on that, if you dynamically generate channel names according to a validatable naming convention that any consumer can predict (ideally with a client lib for generating them), you can do pretty complex message passing that doesn't blow up code complexity. Combine that with the locking and consumer groups built in, it's pretty much distributed computing "for free" even if stuck with multiprocessing for runtime scaling (e.g. Python/JS without the builtin concurrency or multithreading of VMs/hosted languages).
- nurettin 7y agoI made a poor man's rpc using hashsets for persistence and pubsub to notify for new calls Works great!
- DethNinja 7y agoI actually use it as primary database on some parts of the mmorpg I’m developing, Redis actually has ACID capabilities so it is actually very suitable to use as primary database in gaming platforms. That being said main game server requires Mongodb due to sheer size of the data.
- takeda 7y agoWell, if you have only a single instance that doesn't support threading it is trivial to get those properties, but what about durability? Do you realize you store all of the data in the RAM?
- latortuga 7y agoThis is your second comment in this thread about Redis being in-memory. Redis can function in-memory only but it also has at least 2 persistence stories (rdb snapshots, aof logs), and out of the box is configured use rdb.
- Scarbutt 7y agoYour data still has to fit in memory though.
- devit 7y agoRedis can dump and fsync data to disk WAL every query, and can run multiple commands in a transaction, so it can provide the highest guarantees possible for a database. The catch is that the data must fit in RAM, you can only use 1 core for query processing, there is no support for long-lived transactions and no built-in SQL support. Obviously in an ideal world something like Redis would be useless since proper databases would cover all use cases, but unfortunately the state of database software is disastrous.
- skrebbel 7y agoRAM is cheap and lots of realistic data sets never exceed some tens of GBs, even in extreme scenarios. What's wrong with keeping (a copy of) all data in RAM? It's what makes Redis fast yet simple.
- olavgg 7y agoWe use Redis to store user sessions. Spring Session Data Redis is an awesome project for this.
- misiti3780 7y agoI always use it as my default session store. I build out large python backends and I also have been using it as a celery backend, and prefer for it to memcache for those type of tasks also.
- penagwin 7y agoWe use redis for session storage, jobs with celery/redis-rq, view fragment caching, and certain query caching. It can really give you a good boost in performance especially on frequently accessed pages were the content rarely changes - database queries are often "expensive" - so for frequently accessed data that doesn't change frequently - such as product descriptions - it can be a huge help.
- neuland 7y agoI use it for caching, temporary content, pubsub, and distributed locking. It's been particularly useful in load balancers / proxies for authorization, managing csrf, and tracking session activity to auto-log out users. I do this with OpenResty. In async job or internal rpc systems, I use pubsub and streams for distributing work, ensuring exactly once processing, and getting results back to the caller.
- gautamdivgi 7y agoWe use it as the main database for low latency (less than 150ms) response time machine learning services. Store a pretty massive amount of data as well - close to 750GB.
- takeda 7y agoThat's really dangerous. What will you do if data is gone?
- xfalcox 7y agoRedis has persistence.
- gautamdivgi 7y agoyes - persisted and geo distributed
- enitihas 7y agoIsn't that just writing to disk every n seconds? Even if n is 1, data will be lost.
- manigandham 7y agoRedis has RDB (snapshots) and AOF (append log) with configurable fsync policies. You can also use replication to avoid a single server failure. https://redis.io/topics/persistence https://redis.io/topics/persistence
- yesbabyyes 7y agoYou can configure Redis for different persistence strategies. The default is to sync every second, but you can sync on every write. The chapter on Redis persistence is, like Redis documentation in general, quite readable and informative. https://redis.io/topics/persistence https://redis.io/topics/persistence
- deleted 7y ago[deleted]
- aclelland 7y agoAs well as caching, we make really heavy use of Redis for Leaderboards for our games. The sorted sets are perfect for storing score along with the userid. Scanning the leaderboard, finding neighboring scores, etc are all really fast operations. This could probably be done with a number of other storage system but we already used Redis and we've never had a problem.
- slifin 7y agoWe use it to crash our servers on larger customers, by using it to cache all our user entities, pull all of them out at runtime filter them in PHP then stampede ourselves when we clear the cache
- polynomial 7y ago8D
- cs02rm0 7y agoIsn't MVC a pattern for UI layers? I'm not sure redis, a database, is relevant.
- e12e 7y agoIt is many things, but started as more of an architecture for user-data interaction, where the data is in a computer system - and the user wants to interact with it (the user in user-model-view-controller unfortunately fell off at some point). See: http://heim.ifi.uio.no/~trygver/themes/mvc/mvc-index.html http://heim.ifi.uio.no/~trygver/themes/mvc/mvc-index.html As for where something like redis fits - I don't think it'd show up on the design that concerns mvc, no (it could be a cache inside model, inside controller - or even inside view.. Caching ui fragments for example?).
- rukugu 7y agoI use it as a general cache, especially for data that I know hitting the database will be expensive. I also use it for pubsub.
- helaoban 7y agoUse it to cache certain request payloads that are guaranteed not to change for a certain amount of time (e.g. 1-minute stock market aggregates that only change once a minute).
- sidm83 7y agoRedis can do far more than just caching. However for your specific use case, considering a typical MVC web app with RDBMS data storage, you would add a check at the beginning of your Model method to return cached data if it exists, else go through the method execution and write the data to the cache just before returning it back to the controller. This way the cache would be 'warmed' on first call and data will be served directly from the cache (memory) next time till it is cleared, saving expensive disk I/O.
- sergiotapia 7y ago1. Use it for caching 2. Use it for quick lookups for user accounts 3. To queue up jobs that need to run whenever the job runner has slots available. 4. Use it to crash your entire web stack when you accidentally clear your redis instance
- resca79 7y agoReally strange that how back-end developer you never had a use-case for Redis yet. First you could check at https://redis.io/topics/data-types-intro https://redis.io/topics/data-types-intro If you look at single data type you can see how redis takes care about complexity(indicating the Big O notation) of each single data structure and operations. Many devs use it for caching but in my opinion is super nice for evil-write applications.
- alufers 7y agoI mostly work on solutions used in-house by the client. The most used app that I had created was used by maybe 50 people at the same time, and it was mostly manipulating spreadsheet-like data, so querying the database directly was fast enough. I know broadly what Redis can be used for, I was just asking for some practical tips.
- danmaz74 7y agoYou need to be careful with caching inside models, because you want your models to reflect the current, completely up-to-date state of the application. Conceptually, the best place to do it is inside controllers, where you know when you can serve data which isn't completely up to date. With skinny controllers, very often you have some specific places (eg service objects or similar layers) where the controller logic lives, and that is where you can do your caching.
- vorpalhex 7y agoRedis can be a very powerful tool, it can also be a sign that something has gone wrong. Redis is a high performance key/value store with adjustable consistency. It can be used as a caching layer, and it can also do a solid job of representing a basic message queue. It typically fits in on the data layer to store computed results for fast retrieval, but it can also behave like a read replica (storing data from outside your domain). That being said, when Redis becomes a critical part of your stack for performance reasons it usually means something has gone wrong in your data layer. I often times see teams use an antipattern of taking data from their database, changing it's shape to what they want, and then storing that new model in redis. If you're the only consumer of your data, then your data model is incorrect and you're using Redis to paper over that fact.
- takeda 7y agoAgree with you, that's bad, there are also places which use redis as a primary data store. That's actually messed up.
- A-Train 7y agoIs this that bad? At our company we are using redis for real time recommendations. And this is our primary database for storing accumulated statistics about items and users. It has been important to us to have a quick way to share the data between the workers and we chose redis.
- jakearmitage 7y agoYou're fine. The simplistic interface that Redis provides to sets, and therefore empowering pragmatic recommendation engines, is the perfect tool for this job.
- vorpalhex 7y agoThere's nothing a priori wrong with using redis as a primary data store if your data is temporal and your durability needs are aligned with it's model.
- 7y ago
- motogpjimbo 7y agoWe use it for storing user sessions, for caching responses from a third-party API we access, and for imposing per-IP address rate limits on the use of that API. We've also previously used it for lightweight worker queues.
- Deimorz 7y agoFor my site, I use Redis in addition to PostgreSQL for these purposes: - User session storage - Asynchronous "queues" for background processing using Streams (was able to eliminate RabbitMQ from my stack when I switched to this) - Rate-limiting with a GCRA (via https://github.com/brandur/redis-cell https://github.com/brandur/redis-cell) - Bloom filter checks, specifically for making sure people aren't using passwords in data breaches (via https://github.com/RedisBloom/RedisBloom https://github.com/RedisBloom/RedisBloom with data from https://haveibeenpwned.com/passwords https://haveibeenpwned.com/passwords)
- ransom1538 7y agoJust restart your memcache server. Now you will see the point of redis.
- Beefin 7y agomostly as an intermediary for pubsub architectures. ZMQ is what I've used.
- pachico 7y agoFor PHP applications Redis has been a must for storing sessions, especially when distributing load between multiple servers. It is also used to collect data for Prometheus exporters, for example, for languages that don't share memory between requests.
- NightlyDev 7y agoIt's definitely not a must when using multiple PHP servers, but it sure is common and useful.
- Lazare 7y agoRedis is a flexible tool. Some things you could potentially do: - This DB query is pretty slow. Cache the result in Redis. - I'm using server-side sessions, but I have multiple servers. Where should I store session data where it'll be super fast, but all my servers can access it? You could use Redis. - I need to do distributed locking. Use Redis - Simple job queue? Redis. - I'm processing a lot of inbound webhooks, and I want to avoid trying to process the same one twice. I'd like to keep a list of all webhooks I've seen in the last week, and quickly check if a new webhook is in the list, but it needs to be really fast. Redis. Basically, Redis has support for some nifty data structures, and it's very fast. There's a lot of places in most apps where being able share a very fast list, set, map, stream, whatever between servers can be useful. Of course, all the above uses cases can be solved by other more specialised tools too, and in some cases better. (That being said, it's so useful and generally applicable that you should be careful not to ignore fundamental issues. For example, if you have an unavoidably slow query, then by all means cache it in Redis. But if all your queries are slow because you forgot to add indexes, maybe you should fix that instead of using Redis to cache every query!)
- Thaxll 7y agoIt's actually a big release with a lot of changes!
- DethNinja 7y agoRESP3 looks very interesting, unfortunately I made the mistake of writing my own RESP2 protocol and now I don’t have time to upgrade to RESP3. I guess I’ll move to some third party C++ client library.
- antirez 7y agoRESP3 is a strict superset of RESP2! You can just add to your implementation and end with RESP3 finally :-) It means also that a RESP3 parser can parse any RESP2 reply without issues, in case you use it against a RESP2 server.
- DethNinja 7y agoOh that is great to know, because there was a very specific reason why I implemented my own RESP2 client that was not possible with other libraries.
- skrebbel 7y agoout of curiosity, what is that reason?
- DethNinja 7y agoI need to pipeline around 100000 commands every 30 seconds as a transaction. So if any of the 100000 commands fail due to database failure, then I need to crash the server and restart. That is the easy part though, problem is these commands are coming from 12 different cores with possibly that cores will increase to 24 on future. One core might issue 5000 commands, other core might issue 20000 commands, there is no possibility to predict how many commands will come from a specific thread. It is extremely important that this is done with highest performance possible otherwise my system will have bunch of other issues with catching up on other non-db related client-side commands. So I devised a special Redis pipeline structure where each of the cores can issue commands without any kind of mutex during the course of a tick (server is tick based), mutexes for the threads are locked only at the end of the tick to compile different pipeline commands coming from different threads into one giant pipeline command along with their asynchronous callbacks. So practically I'm utilizing a giant redis pipeline with 12 threads on Redis client side where I'm locking only 12 mutexes at the end of the tick, this is extremely high performance as far as letting server process other non-db related client requests while all that data is getting written to server. Truth to be told it is an over-engineered crap and I guess it might be possible to implement something similar with other redis client libraries but this is something I already build so I would like to keep it as it is.
- kawsper 7y agoCongratulations to the Redis team, it's always nice to see improvements to the cluster story and tooling.
- Dowwie 7y ago"Redis 6 is the biggest release of Redis ever, so even if it is stable, handle it with care, test it for your workload before putting it in production. We never saw big issues so far, but make sure to be careful. " Congratulations, team!
- jrochkind1 7y ago> Threaded I/O Woah, I missed that redis changed their mind about this. Clearly haven't been paying attention.
- antirez 7y agoIt didn't, but we identified something that could be done without throwing away our vision. Basically Redis remains completely single threaded, but once is about to re-enter the event loop, when we have to write replies to all the clients, we use threads just for a moment in that code path, and return single threaded immediately after. It is not as performant as full threading, but we believe the right thing to do is sharding and share-nothing. Yet this 2x improvement for the single instance was almost for free so... btw it's disabled by default.
- jrochkind1 7y agoMakes sense, thanks for clarifying!
- ksec 7y ago>btw it's disabled by default Why is that? Thanks for the explanation on Threaded I/O, I was about to ask the difference between Redis 6 and KeyDB.
- jdsully 7y agoKeyDB multithreads the entire event loop from end to end. This provides better performance [1], and combined with MVCC will put us in a place to support more expensive commands like searches without blocking other clients. 1. https://docs.keydb.dev/blog/2020/04/15/blog-post/ https://docs.keydb.dev/blog/2020/04/15/blog-post/
- unrequited 7y agoCongratulations @antirez and the team! Can't wait to play with it.
- pachico 7y agoWe use it also for queueing messages to be consumed by logstash. Not as pub/sub but regular list and has never failed.
- philjackson 7y agoRedis is a workhorse.
- noitpmeder 7y agoOur redis guys do that as well, but it has ran out of memory on big days. Couldn't pop out of it fast enough I guess.
- elpatoisthebest 7y agoThis is great timing. I was just spinning up a new service using redis this morning. Thanks for the constant work you put into this project.
- _pgmf 7y agoContrarian view here. I really liked Redis for a long time. Simple, fast data-structures in-memory. That's it. Along the way there have been some nice enhancements like Lua, which solves a lot of the atomicity issues. But somewhere after 4.0 I feel they have lost their way. First they saw all the attention Kafka/event-stuff was receiving, so they baked-in this monstrous, complicated, stateful streams feature. Now we have SSL (do people really expose Redis on the internet??), ACLs, cluster stuff, and most relevant to me a new wire protocol. To my thinking, Redis fit very well in the "lightweight linux thing" category. It seems they aspire to be enterprise software, and this may be a good move for Redis-the-Business, but it's not good for users like me who just want simple in-memory data-structures and as little state as possible. Forcing a new protocol that adds very little value (in my opinion) also seems like a great way to alienate your users.
- brodouevencode 7y agoAgree on the Kafka, disagree on the SSL. There are compliance factors in place for certain use cases (PII).
- sgt 7y agoAs a comparison, we run (and I think this is pretty common) nginx proxy servers that point to app servers. The proxy servers handle SSL to the outside, whilst the connection to the app servers is simply http. Pretty sure that is an acceptable solution in most cases. So then this would apply to the SSL argument here as well.
- jrockway 7y agoYou can stick a proxy in front of apps that don't have features you need like mTLS, tracing, metrics, etc. to get those. Google "service mesh" to explore that space. But to some extent, I think it's all a bit easier if your apps just do the right thing out of the box. Less moving parts. Better integration testing. Like medicine, every piece of software you use has effects and side-effects. If the advantage of the effects outweigh the disadvantages of the side-effects, then something is a good deal. But if you can avoid the side-effects entirely, that's best.
- dirtydroog 7y agoWhy does redis recommend to not use huge tables? I'd like to experiment with using DPDK with it, but that requires huge tables
- antirez 7y agoBecause when Redis forks to persist, every write will copy a 2MB huge page, and that's not good. All the memory will end to be copy-on-write-ed: latency spikes, big memory usage, ...
- uhoh-itsmaciek 7y ago>So what changed between RC1 and today, other than stability? >... That's quite the list of changes after a release candidate...
- antirez 7y agoYep I use a non confirming release cycle and even actively backport new features form development branches to RCs or patchlevel releases if they are safe and don't have interactions with other code. This is done to make things available faster. Other times new features are almost like fixes because the old alternative resulted in user pain.
- jamsb 7y agoI'm waiting for HTTPS support on antirez.com. If a site doesn't support HTTPS nowadays I mostly ignore it. Otherwise, I enable a VPN connection overseas and use an isolated browser for this traffic.
- therealdrag0 7y agoWhat do you care about TLS for a blog?
- Can_Not 7y agoI've had ISPs insert ads/garbage into my pages.
- kaoussi 7y agoTo read an article you have to do all this?
- Tim25659 7y agoCongrats antirez and redis labs!!
- Tim25659 7y agoWhat is ACL? Can anybody please explain
- dudus 7y agoThanks antirez for all the work you've done on redis. Personally I can say that no one had such an impact on my work as you did. Reading through your source code is educating and inspiring. You are a hero.
- antirez 7y agoThanks, to me it means a lot that somebody out there perceives my work as something of positive :-). Live long and prosper.
- gsempe 7y agoI have to join the OC. Redis impact on my work has been huge. Thank you Antirez
- enitihas 7y agoI don't use redis, but the code quality is so high and it helped me understand how to write good looking C APIs which are also performant. Congratulations on the release.
- rochak 7y agoYour way of writing code is an inspiration to new developers like me. Thanks!
- adventured 7y agoI'll also take the opportunity to say thank you, for creating Redis. I tried Redis when it first came out, mostly just experimenting with it. I had not used it in production for many years and have recently made aggressive use of Redis 5 as a diverse caching layer, replacing Memcached among other things. It has simplified my service infrastructure and it has been super reliable. It's delightfully easy to use. Probably my favorite software to work with right now.
- whalesalad 7y agoDude... I do not think you know truly how many people love Redis. For every contrarian armchair quarterback here on HN there are 1000 people who are out using/coding/hacking w/ Redis and not caring about the noise. Myself included. Thank you and all the contributors for the hard work on an absolutely killer tool.
- sandGorgon 7y agoNoob question - is redis streams completely able to replace the Kafka usecase?
- itamarhaber 7y agoNope - similar concept, different stories.
- powerslacker 7y ago> 6. RDB files are now faster to load. You can expect a 20/30% improvement, depending on the file actual composition (larger or smaller values). INFO is also faster now when there are many clients connected, this was a long time problem that now is finally gone. Talk about humble bragging.
- stephenr 7y agoGreat work. I’ve been quite critical of Redis for its lack of TLS before. This is good to see. Well done!
- tiffanyh 7y ago@antirez Have you considered hosting your projects on Source Hut? https://news.ycombinator.com/item?id=23030489 https://news.ycombinator.com/item?id=23030489
- benatkin 7y agoThis seems wildly off topic. Redis has 342 contributors on GitHub. The SourceHut repo, which I assume is one of the most popular repos on SourceHut, only has six contributors with more than one commit in the last year: https://git.sr.ht/~sircmpwn/git.sr.ht/contributors https://git.sr.ht/~sircmpwn/git.sr.ht/contributors Github is an order of magnitude more popular than GitLab and Bitbucket, which in turn are an order of magnitude more popular than SourceHut.
- tiffanyh 7y ago>>” SourceHut, only has six contributors with more than one commit” And wasn’t antirez practically the only developer of Redis for years as well. Why does the number of developers of a piece of software indicate it’s better? Having a small development teams has worked for numerous successful projects. Take SQLite as another example. So why the downvote?
- mmm_grayons 7y agoNot the guy you asked, but I didn't downmod your comments. However, I don't think moving would make sense right now for big projects. Here's why: 1. Sourcehut is still pretty new; people aren't as familiar with its interface and workflow (which is intentionally different from the github PR-based one). Better or worse aside, this would create an additional barrier to contributions. 3. Sourcehut is still in alpha as of writing this comment; though it's stable, stuff might change. This is no ding on the site, which is actually really good (I greatly prefer the lightweight UI to github), but projects like redis have different considerations than a personal hobby project. 2. It takes time and effort to move a project to different hosting, and from my experience, this grows in a greater-than-linear fashion as the size of the project increases. Any project must therefore ask itself if there is a compelling reason to expend that time on that goal, some benefit to developers or to users.
- janpot 7y agoOne caching pattern I find myself doing a lot on Redis is where multiple clients try to access the same cached value, but if it doesn't exist only one is allowed to revalidate it, all other clients wait until it's revalidated and get notified with the new value. Currently for the clients it involves: 1. subscribe for updates on the key you're trying to retrieve 2. retrieve cached value or set a lock 3. if lock acquired, unsubscribe, then fetch value 4. set the result in a key and publish event with the value I wish there was a command that does: retrieve value of a key, if it doesn't exist lock the key and notify me with an id, if it's locked subscribe for the next update on the key. With a second command that acknowledges the lock with a new value Now I know this can be implemented and I've done so multiple times. It's just that it's tricky to get right and consistent.
- maxpert 7y agoThreaded IO, Server assisted client caching, and Redis proxy in my opinion is an important milestones. Threaded IO will fully utilize multiple cores; and improvements are amazing. Server assisted caching will reduce client round-trips. Redis proxy will remove the debate of which client library to use, Jedis for example has not allowed reading from replicas or doing multiplexing distributed commands; proxy is gonna solve all those gripes. I can't wait for it to be stable and production ready.
- munawwar 7y agoHaving client cache a subset of data is very much needed for the most frequently accessed data.. but it seems the various clients will yet have to implement this? (it could have been the best feature in the list) Previously I had to implement this myself -> https://github.com/Munawwar/redicache https://github.com/Munawwar/redicache