25 ms·

Do you really need Redis? How to get away with just PostgreSQL

• chmod775 5y ago
  Funny. My approach is usually the other way around: Can I get away with just Redis?
  • grenoire 5y ago
    My arsenal is Redis to SQLite to pg.
  • wokwokwok 5y ago
    Why would you choose to use a system that doesn't scale by default? Single user local applications? Fair. Web applications? Very strange choice imo. Reddis is great, but it is *not* a database, and it's thoroughly rubbish at high load concurrency without clustering, which is (still) a massive pain in the ass to setup manually. Of course, you can just use a hosted version off a cloud provider... but, it's generally about 10x more expensive than just a plain old database. /shrug I mean, sure, it's (arguably...) step up from just using sqlite, but... really, it's easy, and that's good... but it isn't good enough as a general replacement for having a real database. (To be fair, sqlite has got some pretty sophisticated functionality too, even some support for concurrency; it's probably a step up from redis in many circumstances).
    • tehbeard 5y ago
      > Why would you choose to use a system that doesn't scale by default? By all accounts Postgres seems to be a pain to scale off a single machine, much more so than redis.
      • cbsmith 5y ago
        Which PostreSQL scaling pain point would you be referring to? Citus?
      • hardwaresofton 5y ago
        Postgres is not as automatic as other tools but is mostly an artifact of it being around so long, and focus being on other things. Few projects have been around and stayed as relevant as postgres. Most of the time, you really don't need to scale postgres more than vertically (outside of the usual read replicas), and if you have tons of reads (that aren't hitting cache, I guess), then you can scale reads relatively easily. The problem is that the guarantees that postgres gives you around your data are research-level hard -- you either quorum or you 2pc. Once you start looking into solutions that scale easily, if they don't ding you on performance, things get murky really quick and all of a sudden you hear a lot of "read-your-writes" or "eventual consistency" -- they're weakening the problem so it can be solved easily. All that said -- Citus and PostgresXL do exist. They're not perfect by any means, but you also have solutions that scale at the table-level like TimescaleDB and others. You can literally use Postgres for something it was never designed for and still be in a manageable situation -- try that with other tools. All that said, KeyDB[0] looks pretty awesome. Multithreaded, easy clustering, and flash-as-memory in a pinch, I'm way more excited to roll that out than I am Redis these days. [0]: https://github.com/EQ-Alpha/KeyDB https://github.com/EQ-Alpha/KeyDB
        • victor106 5y ago
          KeyDB is really good. We use it in production to achieve millisecond response times on millions of requests per second.
          • hardwaresofton 5y ago
            It really looks absolutely amazing, I feel guilty because I want to run a service on it, there's almost no downside to running it everywhere you'd normally run Redis. Also in the cool-redis-stuff category: https://github.com/twitter/pelikan https://github.com/twitter/pelikan Doesn't have the feature set that KeyDB has but both of these pieces of software feel like they could the basis of a cloud redis product that would be really efficient and fast. I've got some plans to do just that.
          • killingtime74 5y ago
            Are you Google search? How do you have millions of requests per second?
    • arpa 5y ago
      redis is not a database. It's a key-value based cache. If you're using it as a database, you're gonna have a bad time.
      • cube2222 5y ago
        Why so? It has persistence and I'm not aware of any reported data loss happening with it. It's also got loads of complex and useful instructions.
        • arpa 5y ago
          Data loss can occur between flushes to disk, for example (by default every 2 seconds / every I_FORGOT megabytes). Perhaps (most likely) it is possible to fine-tune the configuration to have redis as a very reliable data store, but it doesn't come with such settings by default, unlike most of RDBMSes.
          • miohtama 5y ago
            Not all use cases require reliable data storage and it is ok lose few seconds of data. Think simple discussion forums, internal chat applications. There are some scenarios where ease of use and a single server scalability pays off in the faster development and devops cost.
            • arpa 5y ago
              GP was asking why redis is not a reliable storage solution/database. Redis is great as an unreliable (not source-of-truth) storage.
              • kwdc 5y ago
                For that temporary use case, how does it compare to memcached?
                • jasonwatkinspdx 5y ago
    • zigzag312 5y ago
      I would say Redis with RediSearch is a database.
    • Hendrikto 5y ago
      > sure, it's (arguably...) step up from just using sqlite How so? What‘s wrong with SQLite?
      • wokwokwok 5y ago
        I suppose it's a bit more suitable to networked services than sqlite is, since it's natively a web api, and sqlite is natively a local-only solution. ...but, I started writing about clustering and the network API, but, I can't really articulate why those are actually superior in any meaningful way to simply using sqlite, and given the irritation I've had in maintaining them in production in the past... I guess you're probably right. If I had to pick, I'd probably use sqlite.
  • himinlomax 5y ago
    If your data has no value whatsoever, sure.
    • gizdan 5y ago
      Why? Redis has persistent data stores, and doesn't necessarily need to be memory only.
      • himinlomax 5y ago
        When you commit to Postgres and the server acknowledges it, you know for sure that it's been written to disk and that it will survive anything but a hardware disk loss (or, obviously, system/FS bug). When clustering is enabled with synchronous writes, you can also be confident that the data has been recorded to another node as well. With redis clustering, there's no guarantee the data has been replicated. I'm not even sure there's any guarantee the data you just asked to be recorded be stored even once if a power outage happens immediately after the request.
    • cheald 5y ago
      Redis can be made durable. The WAIT command allows you to guarantee writes to a quorum of nodes, and it can be configured for on-disk persistence rather easily. That said, due to it's single-threaded nature, blocking on quorum writes is likely to bottleneck your application under any kind of significant load. It really shines at volatile data, and while it can work for valuable data, there are better tools for the job.
      • himinlomax 5y ago
        > Redis can be made durable Postgres, SQLite and many others are durable by default. Almost all so-called databases are like that. When you need a database, 90% of the time, you want durable. People make mistakes, developers are people, developers make mistakes, and one such mistake is assuming that Redis is like other databases in being durable by default when it's not. It's not conjecture, I've seen it done in production.
  • hardwaresofton 5y ago
    Hey I assume this is like a joke and not too serious, and we'd all switch off when things got a bit hairy, but I sure hope other readers can tell. I am literally in the middle of digging a company out of this mistake (keeping Redis too long) right now. If your software/data is worth something, take a week or a month and figure out a reasonable schema, use an auto-generation tool, ORM, or hire a DB for a little bit to do something for you. Even MongoDB is better than redis if your're gonna do something like this.
    • tluyben2 5y ago
      > take a week or a month and figure out a reasonable schema, use an auto-generation tool, ORM, or hire a DB for a little bit to do something for you. Sorry but am I the only one who is very worried about the state of software? There are people who drank so much of the schemaless (which was not an actual issue for any dev worth her salt to begin with) that you have to dispense this kind of advice? I find that bordering on criminal if someone did that to you and carries the title programmer. Again, maybe that is just me. Edit: not an attack on the parent: good advice. Just didn't know it was that bad. And sad.
    • chrisdinn 5y ago
      If you store protos in your Redis keys (like most people using “NoSQL” for data storage), this comment doesn’t have much punch. Pretty sure we all can think of some pretty high profile examples of NoSQL + structured data working very very well at scale.
      • hardwaresofton 5y ago
        I'm not trying to get on people who are using redis as a cache (for photos, or any other ephemeral data). The idea I was trying to get at was using redis to store data traditionally reserved for OLTP workloads. > Pretty sure we all can think of some pretty high profile examples of NoSQL + structured data working very very well at scale. Well that's the thing, you very rarely hear of companies who cursed their decision early on to use NoSQL when they realized that their data was structured but in 20 different ways over the lifetime of the product. Some datasets only need light structure (key/value, a loosely defined document, schema-included documents), and other things should probably have a schema and be stored in a database with a tight grip on that schema and data consistency/correctness. Please don't use redis in that latter case.
  • siscia 5y ago
    It was the reason with https://zeesql.com https://zeesql.com formerly know as https://RediSQL.com https://RediSQL.com was written. My clients seems rather happy with the project.
  • mianos 5y ago
    If you want to use transactions for multi table updates you are probably best to use a proper rdbms. Not to mention read consistency. If you only have one table Redis may be fine. I usually find my work grows beyond one table. Redis does make an unbeatable cache.
  • deleted 5y ago
    [deleted]
  • dionian 5y ago
    well... how much uptime do you need and how much resources do you have to devote towards achieving your desired uptime
  • Jemaclus 5y ago
    I sorta do this, but my approach is more Redis-first than _just_ Redis. I try to see if I can use Redis for 99.999% of my operations and have a more durable store (like Postgres or something) as a "backup". The nature of Redis is such that even with some persistence features, we kinda have to assume that the data could go away at any minute, so I always build some way to rebuild Redis as fast as possible. But I've run billions of queries per day against a single Redis instance with zero failures serving up traffic to large, enterprise-level customers with no downtime and no major issues (knock on wood). The only minor issues we've run into were some high-concurrency writes that caused save events and primaries to failover to replicas, and resulted in a few minutes downtime at the beginning of our experiments with Redis-first approaches, but that was easily mitigated once we realized what was happening and we haven't had a problem since. Huge fan of a Redis-first approach, and while the haters have _some_ valid points, I think they're overstated and are missing out on a cool way to solve problems with this piece of tech.
• petepete 5y ago
  I've used PostgreSQL in the first two scenarios and would love an opportunity to in the third. It's worked really well and hasn't caused any problems under decent loads. The one feature of Redis I'd love to have supported in PostgreSQL is to be able to set a TTL on a record. On a recent project where we could only save personal data for so long, it was a must have feature so we had to use Redis for that purpose instead.
  • sologoub 5y ago
    Wouldn’t a simple scheduled batch job be enough to go through periodically and drop records older than N days? This would also give you audit logs, etc. As well as flexibility to adjust business logic without updating the TTL on all records.
    • radicalbyte 5y ago
      It would, and you'd also get nice features like proper transactions and referential integrity. BUT that comes at a cost. Doing that in Redis is just so easy, and the vast majority of the time no-one is going to notice a few concurrency errors, YOLO! ;-) To be serious: it's a nice tool to have in the box and is amazing for inexperience/start developers because you can do a lot with very little time investment or knowledge.
      • zigzag312 5y ago
        What's wrong with Redis' transactions?
    • tbarbugli 5y ago
      Deleting large amount of data on Postgresql is expensive. First you need an index on a column to select the expired data, then you actually need to delete the rows which creates a lot of garbage, is heavy and slow. Creating and deleting a lot of data on PG is a pain because of MVCC and vacuum. One useful trick is to partition data into tables and to truncate/drop entire tables, drop/truncate is instant and reclaims space immediately.
• arpa 5y ago
  As German engineers are known to say: "Why make things simple when complicated will do?"
  • Fannon 5y ago
    If you already have Postgres in your project and now you get the requirement for an additional cache store, I think it will be less complicated to reuse what you already have instead of adding another DB to your stack. Of course, at some point of scaling needs a dedicated cache store will make sense anyway. (Just some justifications from a german engineer :) )
    • arpa 5y ago
      True. However, since cache is usually transient, adding this key-value store/cache is as easy as "docker run redis". No need to provision block storage, and it's really lightweight in comparison. (that being said, I try really hard not to judge; after all, i'm not without fault: it's 2021 and i'm using bash over cgi-bin to serve web pages for my own hobby projects :))) )
      • rualca 5y ago
        > True. However, since cache is usually transient, adding this key-value store/cache is as easy as "docker run redis". No need to provision block storage, and it's really lightweight in comparison. If you're using postgres for caching only, as you do Redis, then you also do not need to provision block storage. If you happen to already have Postgres running for other uses, you also do not need to provision block storage. Finally, I would add that Redis clients such as Redisson are resource hogs that cause performance problems on apps, while pg clients are barely noticeable.
    • mosselman 5y ago
      Redis as a cache server requires different settings from redis as a job queue. So you can’t reuse the same server anyway.
    • himinlomax 5y ago
• chx 5y ago
  You really don't need anything fancy to implement a queue using SQL. You need a table with a primary id and a "status" field. An "expired" field can be used instead of the "status". We used the latter because it allows easy retries. 1. SELECT item_id WHERE expire = 0. If this is empty, no items are available. 2. UPDATE SET expire = some_future_time WHERE item_id = $selected_item_id AND expire = 0. Then check whether UPDATE affected any rows. If it did, item_id is yours. If not, loop. If the database has a sane optimizer it'll note at most one document needs locking as the primary id is given. All this needs is a very weak property: document level atomic UPDATE which can return whether it changed anything. (How weak? MongoDB could do that in 2009.) Source code at https://git.drupalcode.org/project/drupal/-/blob/9.2.x/core/lib/Drupal/Core/Queue/DatabaseQueue.php#L116 https://git.drupalcode.org/project/drupal/-/blob/9.2.x/core/... (We cooked this up for Drupal in 2009 but I am reasonably sure we didn't invent anything new.) Of course, this is not the fastest job queue there is but it is quite often good enough.
  • habibur 5y ago
    Right. And this article reminds me of people trying to use a SQL database like a filesystem. "Lock file before update otherwise it will get overwritten". Don't use record lock as like it's an attribute of a row. Add a column on the table called "status" and use it.
  • 0x0nyandesu 5y ago
    I recently switched from rolling my own table to SQS. The main issue is making a job queue dispatcher and runner which most people aren't familiar with.
  • edwinyzh 5y ago
    Great! So I learned this technique. But one thing that's not clear to me - how a "expired" datetime field can allow easy retries? Anyone give me an example? Thanks!
    • chx 5y ago
      Successful items are deleted from the queue before they expire. So an item with expired > current time needs to be retried because it failed processing. So, a cronjob resets these to expire 0 and the next worker can pick them up.
• axiosgunnar 5y ago
  The point the author is missing is that most people are not deliberately choosing Redis. They use packages such as celery which happen use Redis under the hood but the user doesn‘t have to deal with these details other than pasting some Redis connection URL somewhere.
  • Nicksil 5y ago
    >They use packages such as celery which happen use Redis under the hood but the user doesn‘t have to deal with these details other than pasting some Redis connection URL somewhere. Celery doesn't use Redis under the hood. However, you may use Redis as a message broker and/or result store. https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/index.html https://docs.celeryproject.org/en/stable/getting-started/bac...
• Aeolun 5y ago
  This seems a bit silly. We might be able to use only postgres, but this seems harder than using Postgres and Redis. Any time someone says ‘this works for small workloads’ I hear, ‘we’ll eventually have to migrate this’. That’s not a problem if you are talking about introducing a hugely complex piece of infrastructure, but Redis is hardly that.
  • anaganisk 5y ago
    For thousands of applications deployed, they're most probably never need scale.
  • ryanjshaw 5y ago
    Or don't use Postgres to begin with. The reason you need to SELECT FOR UPDATE is because Postgres doesn't implement READ UNCOMMITTED. If you're implementing an ordered queue, you have no way of reliably getting the next page of queue items without taking a lock using SELECT FOR UPDATE because there may be pending uncommitted transactions that you need to block on. On MySQL and SQL Server you can compare counts computed with READ UNCOMMITTED and READ COMMITTED (on MSSQL, with snapshot isolation) to quickly figure out whether there are any uncommitted transactions in the range without taking a lock. That being said, SELECT FOR UPDATE and UPDLOCK equivalents are the easiest to get right, and if you need higher throughput you probably need a message queue or Kafka-equivalent.
  • gizdan 5y ago
    > Any time someone says ‘this works for small workloads’ I hear, ‘we’ll eventually have to migrate this’. Personally I aim to develop with future migrations in mind. Certain abstractions are worth it. Put an interface around a cacher, db store etc. Day to day dev overheads is low, but your future self will thank you should you need to migratw away. Sure this isn't always possible, but do it as much as possible.
  • manigandham 5y ago
    Most workloads are small workloads. Modern hardware is incredibly powerful.
  • kristopolous 5y ago
• ram_rar 5y ago
  Why do you need postgres, if you can get away with Redis? For all the 3 points mentioned in the article, I would rather just use redis as apposed to postgres. Unless, you really need a rds solution for the project.
  • dkarras 5y ago
    I think the article is assuming you already have Postgres for data storage and arguing against the addition of redis when that is the case, for the given common requirements that typically calls for the addition of redis into the mix.
• mosselman 5y ago
  For rails I’ve used que in the past, which is a job queue adapter for Postgres. It supports activejob. https://github.com/que-rb/que https://github.com/que-rb/que
  • shaicoleman 5y ago
    Another alternative is GoodJob: https://island94.org/2020/07/introducing-goodjob-1-0 https://island94.org/2020/07/introducing-goodjob-1-0 https://github.com/bensheldon/good_job https://github.com/bensheldon/good_job
  • pablodavila 5y ago
    For Elixir, Oban is a great library that allows the same, in case anyone was looking for one. https://github.com/sorentwo/oban https://github.com/sorentwo/oban
• skunkworker 5y ago
  Hopefully one day projects like que[1][2] will become stable and battle-tested enough to use in a production environment. Until then I'll be using something like sidekiq (if you're going for a client-side job queue, eg: the clients don't really know about each other and only have rate-limiting, not true throttling). With Postgres you also need to worry about high churn, especially since you are creating/locking/deleting rows constantly. This can be alleviated through a variety of means, of which personally I would use per-day table partitioning and truncate older partitions on a cron, not to mention the sharp increase in database connections to the host now required. Ignoring the literal elephant in the room of synced writes to the store. Redis can be used quite effectively in a blocking manner with RPOPLPUSH/LMOVE(6.2+) for a reliable queue, allowing an item to not be lost because atomically the pop and push from two different lists are done together. [1] https://github.com/que-rb/que https://github.com/que-rb/que [2] https://gist.github.com/chanks/7585810 https://gist.github.com/chanks/7585810
• phendrenad2 5y ago
  Wordpress historically didn't need a cache and just used page visits and/or a cron job to kick off automated processes, backed by MySQL. Is it fast? Yes. Is it nearly as fast as Redis? No. Do you need it to be? Not for Wordpress lol
• avinassh 5y ago
  Most of the times Redis's distributed lock works fine, however one should know that its not fail proof and you might run into really weird bugs references: 0 - https://aphyr.com/posts/283-jepsen-redis https://aphyr.com/posts/283-jepsen-redis 1 - https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
  • antirez 5y ago
    0 - unrelated to RedLock algorithm, since RedLock has its own replication in the protocol. 1 - I replied in a blog post refusing most of the arguments, and then there was maybe another reply. You may want to read the full thing to form an idea about the safety of RedLock.
• yukinon 5y ago
  This is a great article because it outlines an alternative to using Redis for any given use cases. If we don't constantly re-evaluate our toolsets and their capabilities, it can lead to poor decision making. That being said, I've found Redis largely a pleasure to work with for these use cases and don't really see a real incentive to changing my current approach.
• tim44 5y ago
  ia this a bub het, found, hard srry gl hf hf
• tim44 5y ago
  men have never found me. at least
• tim44 5y ago
  found at at a deal
• tim44 5y ago
  any
• tim44 5y ago
  any never alt homely
  • deleted 5y ago
    [deleted]
• cpursley 5y ago
  With Elixir you can replace both with something like Quantum or backed by Postgres with Oban. Or even just a hand rolled genserver. You can also listen to Postgres WAL changes to build an Event like system with Elixir. Supabase is doing this with their Realtime app.
• pilif 5y ago
  For pub/sub, I would recommend against using PostgreSQL if you're doing it at any kind of scale because LISTEN ties up one connection completely and Postgres connections are very expensive compared to a redis connection.
  • dvfjsdhgfv 5y ago
    I was looking for your comment here, thank you. I hope they solve this problem somehow in a future release.
  • gunnarmorling 5y ago
    The other issue with LISTEN/NOTIFY is that a client will miss any notifications sent while it is not running (e.g. due to crash, update, etc.). An alternative would be logical replication and change data capture (CDC), as provided for Postgres by Debezium for instance. That way, any consumers won't miss events during downtimes, as they'll continue to read from the replication slot from the last offset they had processed. Disclaimer: I work on Debezium
    • jbverschoor 5y ago
      This is by design and very common with pub/sub.
    • supermatt 5y ago
      isnt this also the case for redis pubsub?
      • philjohn 5y ago
        I'd need to check, but I think so. Which is probably why if you don't want loss if consumers go down a proper queue system (RabbitMQ, ActiveMQ, Amazon SQS, Kafka if you don't care about ordering between partitions) is the way I'd go.
      • Nick-Craver 5y ago
        For pub/sub yes that's correct. For full info though: Redis later added streams (in 5.x) for the don't-wan't-to-miss case: https://redis.io/topics/streams-intro https://redis.io/topics/streams-intro
• dvfjsdhgfv 5y ago
  Has anyone tested how listen/notify in pg (for in-memory databases) compares to pub/sub in Redis?
• posharma 5y ago
  What happened to RabbitMQ? Is that not used as a queue any more?
  • lwn 5y ago
    Other than it's absence from this thread, Rabbit still seems to be going strong.
• sgt 5y ago
  As much as I love Postgres, I would rather use Redis for this. I haven't used Redis much though, and on our project we actually decided on using Kafka. Admittedly much heavier and maintenance intensive, it seems to do the job very well. Any opinions and experiences here with Kafka vs Redis as a queue?
  • arpa 5y ago
    It's apples versus oranges with rabbits in the middle.
    • des1nderlase 5y ago
      Ha ha. So true. In theory though you can make any Turing complete piece of software to do anything, so rabbits can be turned into oranges...
  • ickyforce 5y ago
    Don't use Kafka as a queue. (I haven't used Kafka for a while, if anything below is outdated let me know) The main issue I've experienced was balancing work across consumers - if one consumer is slower than others (e.g. running on a machine with some other heavy workload). In such case when a larger number of jobs is enqueued and expected to process quickly it's possible that everything will complete except jobs which landed on the partition belonging to the slow consumer. There can be only one consumer per partition so others can't help. One could consider using a single partition to avoid this imbalance but this means that there could be only a single consumer - not good. In other queue systems consumers fetch up to N jobs from a shared pool when they need more work. This means that in the case of a slow consumer the lag is limited to by the time required to process N jobs. This situation also arises if consumers are equally fast but some kinds of jobs are considerably slower. Random distribution across partitions mitigates it to a degree. My case was precomputing caches after some event, the goal was to do this within 10 seconds. First implementation with Kafka worked but very often some consumer couldn't use as much CPU time as others and we didn't want dedicated nodes for this because this even was happening at most few times a day. As a result jobs from a single partition were processed 2x slower - we had to overprovision. Replacing Kafka with a queue based on Redis solved this right away, all consumers were finishing at the same moment. Although it wasn't a large scale operation. The second complaint about using Kafka as a job queue is not being able to modify the queue, e.g. to reorder jobs, postpone jobs or cancel jobs
• antihero 5y ago
  I think one of the draws of redis was, back when it came out, how simple it was to set up and maintain in comparison to an RDBMS.
• BiteCode_dev 5y ago
  Oh but redis is much more than that. It's so simple any little script can interact with it in seconds, instead of having to craft complex SQL or import your wrapper. You can call redis from the weirdest places, like from the inside of an nginx config file. You needn't even a compiled driver if that matters. It's easier to get it to perform. It's possible to get great perfs with PostgreSQL, but you just got them for free with redis. Very hard to screw up. Read, write ? N+1 ? Batching ? Who cares, it will be fast no matter the concurrency or the load. Sure you can expire with Postgres, but having done so in practice, it's way harder than it looks to get right. With redis, you don't have to care. Set a cache, set a lock with a timeout, store data your are not sure you need. It will all disapear. Redis is not just key/value. You have sets, so ensuring unicity is a piece of cake, no constraints to define, then insertion to check against. And of course, you have sorted sets, which you are kept ordered at insertion by any number you pair the value with, such as a timestamp/score/index, and truncate by range for extra magic. And then you have bonuses like hyperloglog which you need an extension for in Posgres. Super handy for stats. Finally, you have streams, which for most apps will fill the gap for a timeserie database or a way to store your app logs. All that with a 2ms latency at 10k/s requests. None of my projects ever needed more than 1K/s though, even one with 1M users/month. You have all of that with a dead simple install and basically no maintenance. In fact, redis by itself consume almost no resource, it's not even worth it to not have it in your toolset. I just install it by default on all my projects: I never regretted it, there is always something it can do for you. It not now, just wait a bit, it's costing you nothing, and something will come up. So no, let use Postgres for what it's great at, which is being a robust all purpose database. Redis is a fantastic complement to it, not a competitor, just use both. Unless you are google size, there are little chances you will reach a stage where you need to migrate from any of them. It's part of those tech that are just too good to be true, like SQLite or Python. My only regret is that it doesn't exist on windows. P.S: if you need caching and can't afford redis on a small python script, use Disk Cache, it's awesome: http://www.grantjenks.com/docs/diskcache/index.html http://www.grantjenks.com/docs/diskcache/index.html
  • junon 5y ago
    Not to mention I've never personally experienced redis go down by itself - it's always some stupid user error that could have happened to any part of the infrastructure. Redis is, far and away, _the_ most robust piece of software I've ever had the pleasure of deploying to production.
• fasteo 5y ago
  For use case 1 (job queue) I can only recommend beanstalkd[1]. Simple tcp plain text protocol. Install and forget. Plus you get some really useful features like deferred jobs, burying, kicking, job priorities, etc. We have literally processed tens of billions of jobs without a single failure. Old and simple technology that just works [1] https://beanstalkd.github.io/ https://beanstalkd.github.io/
  • mianos 5y ago
    It is funny, every time I have used beanstalk over the years I have put it in thinking I'll replace it later. I never have. Multiple startups I have left years ago are still running it, untouched years later. No one mentions it probably because it didn't do anything but run. I wish the same could be said for RabbitMQ.
    • GordonS 5y ago
      I've used RabbitMQ on several projects over the last decade or more, and have never once known it to fail. It can be tricky to configure it you want clustering, or have fancy requirements (e.g. client certificate authentication), but once running, it's solid.
• deckard1 5y ago
  I imagine most people using Redis as a queue were already using it as a cache and just needed some limited queuing ability. Much like how places end up using a DB as a queue. Using a DB as a queue has been a thing for a very long time. Every billing system I've seen is a form of a queue: at a certain point in the month a process kicks off that scans the DB and bills customers, marking their record as "current". The challenge is always going to be: what if the worker dies. What if the worker dies, the job is re-ran, and the customer is billed twice. Thank god it's been many years since I've had to touch cron batch jobs or queue workers. The thought of leaving the office knowing some batch job is going to run at 3am and the next morning might be total chaos... shudder.
  • cerved 5y ago
    How would double billing occur if the worker dies. The way I would design this, the billing request and bill would be committed atomically such that a request can only be completed with exactly one associated bill. If the worker dies, no bill is created. Also I'd detect a worker has died by recording the start-time and using a timeout. Furthermore I'd requeue requests as distinct new entities. A requeued entity would have a self-referencing nullable FK to reference its parent request.
    • jffry 5y ago
      For many billing activities, "days" is a small amount of time delay. One very viable option to "the worker died" is to just look for stuck jobs once the run is complete and kick out an alert to let a human decide what to do.
    • deckard1 5y ago
      Murphy's law says that you're going to screw this up any number of ways. Maybe not you, specifically, but perhaps your coworker. > committed atomically Complex billing systems don't work that way. Worker processes are not always in these neat boxes of "done" or "not done". Much like rollbacks are a developer myth. If a process was that trivial then you wouldn't need a queue and workers in the first place! > Also I'd detect a worker has died by recording the start-time and using a timeout. There are many ways to solve this and many ways to get it wrong. Not working in UTC? Oops, better hope nothing runs during daylight savings changeover. Parent process died but worker finished job? Let's hope the parent isn't responsible for updating the job completion status. Large job is borderline on the timeout? Better hope parent process doesn't restart the job while the worker is still working on it. Network partition? Ut oh. CAP theorem says you're out of luck there (and typically there is at least one network hop between the DB server/controlling process and the system running the workers). Probably the more straightforward solution is to give each worker an ID and let them update the database with the job they pick up. Then, something robust like systemd, would monitor and restart workers if they fail. When a worker starts, they find any jobs where table.worker_id = myId and then start back on those. But you still have network partitions to worry about. Again, not at all trivial.
• deleted 5y ago
  [deleted]
• hardwaresofton 5y ago
  A blog post series I've been meaning to write for over 3 years now: * Every database a Postgres 1: Key/Value store * Every database a Postgres 2: Document stores * Every database a Postgres 3: Logs (Kafka-esque) * Every database a Postgres 4: Timeseries * Every database a Postgres 5: Full Text Search * Every database a Postgres 6: Message Queues Low key, you could make almost every single type of database a modern startup needs out of Postgres, and get the benefits (and drawbacks) of Postgres everywhere. Should you do it? Probably not. Is it good enough for a theoretical ~70% of the startups out there who really don't shuffle around too much data or need to pretend to do any hyper scaling? Maybe. If anyone from 2ndQuadrant/Citus/EDB see this, please do a series like this, make the solutions open source, and I bet we'd get some pretty decent performance out of Postgres compared to the purpose built solutions (remember, TimescaleDB did amazing compared to InfluxDB, a purpose built tool, not too long ago). New features like custom table access methods and stuff also shift the capabilities of Postgres a ton. I'm fairly certain I could write a table access method that "just" allocated some memory and gave it to a redis subprocess (or even a compiled-in version) to use. [EDIT] - It's not clear but the listing is in emacs org mode, those bullet points are expandable and I have tons of notes in each one of these (ex. time series has lots of activity in postgres -- TimescaleDB, native partitioning, Citus, etc). Unfortunately the first bullet point is 43 (!) bullet points down. If someone wants to fund my yak shaving reach out, otherwise someone signal boost this to 2Q/Citus/EDB so professionals can take a stab at it. [EDIT2] - I forgot some, Postgres actually has: - Graph support, w/ AgensGraph now known as AGE[0] - OLAP workloads with Citus Columnar[1] (and zedstore[2]). [0]: https://age.apache.org https://age.apache.org [1]: https://www.citusdata.com/blog/2021/03/05/citus-10-release-open-source-rebalancer-and-columnar-for-postgres/ https://www.citusdata.com/blog/2021/03/05/citus-10-release-o... [2]: https://github.com/greenplum-db/postgres/tree/zedstore https://github.com/greenplum-db/postgres/tree/zedstore
  • rkwz 5y ago
    > Should you do it? Probably not. Is it good enough for a theoretical ~70% of the startups out there who really don't shuffle around too much data or need to pretend to do any hyper scaling? Maybe. It's also useful when you want to quickly build a "good enough" version of a feature like search so you can get it in front of your users fast and iterate on their feedback. Most of the time, they'd be quite happy with the results and you don't have to spend time on something like managing Elasticsearch. I wrote a post on how you can use postgres to add search capabilities with support for queries like jaguar speed -car ipad OR iphone "chocolate chip" recipe http://www.sheshbabu.com/posts/minimal-viable-search-using-postgres/ http://www.sheshbabu.com/posts/minimal-viable-search-using-p...
• Rygu 5y ago
  Don't forget to use Partial Indexes on jobs/tasks tables with queries like WHERE status = 'pending' More: https://use-the-index-luke.com/sql/where-clause/partial-and-filtered-indexes https://use-the-index-luke.com/sql/where-clause/partial-and-...
• iamgopal 5y ago
  Does groupcache for golang works with cloud run ? As a redis replacement ?
• nickjj 5y ago
  I think one of the biggest advantages of using Redis for job queing vs Postgres comes down to library support. For example Python has Celery and Ruby has Sidekiq. As far as I know there's no libraries in either language that has something as battle hardened with comparable features for background tasks using Postgres as a backend. There's a big difference between getting something to work in a demo (achievable by skimming PG's docs and rolling your own job queue) vs using something that has tens of thousands of hours of dev time and tons of real world usage. I'm all for using PG for things like full text search when I can because it drastically reduces operation complexity if you can avoid needing to run Elasticsearch, but Redis on the other hand is a swiss army knife of awesome. It's often used for caching or as a session back-end so you probably have it as part of your stack already. It's also really easy to run, uses almost no resources and is in the same tier as nginx in terms of how crazy efficient it is and how reliable it is. I don't see not using Redis for a job queue as that big of a win.
  • RedShift1 5y ago
    But in Postgres you could write functions in your schema to handle job queueing/dequeueing with the additional benefit of being able to use it in any language that can connect to Postgres and being able to reuse the same SQL/interface across all languages.
    • nickjj 5y ago
      I'm totally with you in this regard and I'd like to see that too but the reality of the situation is a job queue is more than slapping together a few SQL queries. A good job queue will have all or most of these features: Prioritize jobs (queues, weighs to certain jobs, etc.) Scheduled jobs (running them once but X time in the future) Periodic jobs (running them every 2nd Tuesday at 3:33am) Static and dynamic configuration (CRUD'ing jobs at runtime, like adding new scheduled tasks) Re-try jobs (customizable strategy, such as exponential back off) Rate limit jobs Expire jobs Cancel jobs Unique jobs Batch executing Handling graceful shutdown (integration with your app server) Get metrics (status, health, progress, etc.) Browse job history Web UI (nice to have) And I'm sure I'm missing things too. These are only off the top of my head based on features I tend to use in most applications. In other words, this isn't a laundry list of "nice to haves in theory", most of these are core or essential features IMO. I use them in nearly every web app. Rolling all of these things on your own would be a massive undertaking. Tools like Celery and Sidekiq have been actively developed for ~10 years now and have likely processed hundreds of billions of jobs through them to iron out the kinks. Even if you managed to do all of that and created it as a Postgres extension (which I think is doable on paper), that's only half the story. Now you'd have to write language specific clients to interface with that so you can create jobs in your application using a nice API. This would be a very welcome project but I think we're talking a year+ of full time development time to release something useful that supports a few languages, assuming you're already an expert with writing pg extensions, have extension knowledge about job queues and know a few popular programming languages to release the initial clients.
• larodi 5y ago
  yes, do you really need PDF convertors when you can have them as PostgreSQL extensions. the point (ok, one point of many) of REDIS is that it is not the main DB so you can have a sense of security and decoupling in the architecture. Besides - there is no silver bullet for all things. While you can have your app do everything with PostgreSQL (and much more with Oracle, something people dislike it about), the fact itself does not mean is a good design decision or is a more stable decision. Because when you have redis for sessions, kafka for event streams, postgre (or else) for data storage - you have components that can fail separately and thus the system degrades gracefully.
  • xupybd 5y ago
    Yes but have you built a tool for the job or a Rube Goldberg machine? Complexity comes at a huge cost. Only add it when the benefits out weigh the costs. You could start out building a product that could scale to millions of uses overnight. But if you do that you've spent months building something with no users. You could have been in the market already, building revenue already. Your requirements will change as you find your market fit and you'll need to change things. The less you have built the easier it is to change. Why not leave the million user capabilities until you actually need it?
    • berkes 5y ago
      > Yes but have you built a tool for the job or a Rube Goldberg machine? > Complexity comes at a huge cost. Only add it when the benefits out weigh the costs. Im honestly unsure if you mean this as opposing "doing everything in Postgres" or as opposing "throw more services on the stack". Because both are true for the statements. You have that complexity, regardless of where you implement it. Either you are building the rube-goldberg machine inside of postgres out of modules, config and SQL or outside of postgres with additional services. The only way to really solve this is to avoid building that RG machine in the first place. In this case: don't have queues. In practice that probably means introducting complexity elsewhere, though.
      • xupybd 5y ago
• dandarie 5y ago
  How about tags/sets?
• flefto 5y ago
  MySQL and Microsoft SQL server also support SKIP LOCKED.
• dutchbrit 5y ago
  Why use Redis when you have more suitable solutions like RabbitMQ or Kafka? Obviously depends on the scale and needs of a project, Postgres etc is fine for simple queues. I often see people waste unnecessary time by writing their own complex solutions, resulting in increasing technical debt, when you already have perfectly suitable open source options available that do a better job..
  • brentjanderson 5y ago
    I have had two separate benefits from using just Postgres in apps: 1. It’s one tool to learn. No need to upskill on topics, partitions, consumer groups, routers, brokers, etc. 2. It’s one transaction boundary. You get a lot of “magic” for free when you know that a rollback will not only revert application state but also NOTIFY events published and queue jobs in flight. This alone makes Postgres a powerful option.
• qatanah 5y ago
  For simple task queues, yes pg is ok. For high loads, redis is still better as PG generates a lot of WAL & connection overhead.
• leetrout 5y ago
  Watch out for transaction ID and sequence exhaustion if you have a lot of things rolling through a table as a queue. Postgres is awesome but logging and queuing in a table can cause gotchas you won’t have with redis.
• manishsharan 5y ago
  Sometimes we need network based data structures that are other than relational tables. Redis delivers quite a few of those .. You can always implement then in a SQL database but Redis is just a better solution for those cases.
• beforeolives 5y ago
  Can anyone explain the need for redis in a clear way for someone who knows how databases work but isn't a backend developer? What alternatives are there? What did people do to solve the same problem before redis existed?
  • snarkypixel 5y ago
    ELI5: Redis is really a mix-bag of many useful commands, see [https://redis.io/commands https://redis.io/commands]. It has been referred to as the "swiss army knife" because you can build many custom solutions from these building blocks. Still, its most common use-case is for in-memory key/value caching for performance reasons. For instance, if you have one database query that takes a particularly long time, you can execute it once and store the result in redis, and then retrieve that value extremely fast.
  • mixmastamyk 5y ago
    Generally used for caching expensive lookups, including job queues, and other nifty patterns. Before that memcached was most popular, but it didn’t save to disk, where redis eventually does. Before that, it was roll your own or ram disk.
• MichaelMoser123 5y ago
  it's cool that they added listen notify to postgres; I wonder when they will add many-to-many relationships; my problems with postgress start when i have to do a many to many relationship. For these you need to do an extra table for the link entries; now performance tanks, and you need to denormalise this for read requests, that's where the fun starts...
  • mixmastamyk 5y ago
    M2M tables shouldn’t be a problem except at the very high end, in which case you’ll have a number of mitigations at your disposal. ?
• dave_sid 5y ago
  I’d think you don’t really need redis or elastic search in about 80% of the places you see it. You certainly don’t need the baggage.
• nameless912 5y ago
  It's always wild when an article on HN has an answer to the exact problem i wanted to solve this weekend. Kudos!
• truth_seeker 5y ago
  While you are at it, don't forget to use UNLOGGED tables. UNLOGGED == In Memory. But if you must use disk based table for Job queueing, set fillfactor = 50. This takes care of heavy updates. Indexes are helpful but costs memory and CPU, so always make sure you partition the table based on job_type for performant pending job query. I wouldn't recommend using LISTEN/NOTIFY unless you are okay with "at most once" semantics. I have used disk table based approach for PUB/SUB to replace Kafka. More fine tuned approach will also allow (job_type, consumer_group, publisher) as a partition key. Ref - https://www.postgresql.org/docs/current/sql-createtable.html https://www.postgresql.org/docs/current/sql-createtable.html
  • kureikain 5y ago
    Thanks so much for this. I have been looking for this in Postgres. MySQL has memory back table which helped us a lot in the past.
  • osigurdson 5y ago
    My understanding is, UNLOGGED means that changes are not written to the WAL and data can be lost in the event of an unscheduled shutdown. It doesn't mean that the table only exists in memory however - the data is still eventually persisted to disk.
    • truth_seeker 5y ago
      From the official doc link i shared already: UNLOGGED If specified, the table is created as an unlogged table. Data written to unlogged tables is not written to the write-ahead log (see Chapter 29), which makes them considerably faster than ordinary tables. However, they are not crash-safe: an unlogged table is automatically truncated after a crash or unclean shutdown. The contents of an unlogged table are also not replicated to standby servers. Any indexes created on an unlogged table are automatically unlogged as well.
      • andrewflnr 5y ago
        Uh, yeah, that confirms what osigurdson said, not that they're in-memory. For that IIRC you need to mount a ram-disk in your OS and put the table on that. Definitely also make it UNLOGGED, though.
• Xophmeister 5y ago
  SKIP LOCKED looks interesting; I'll have to try that. I've used advisory locks in the past, but I kept running into deadlocks when I tried to acquire them intelligently (just for popping off the queue). It was unclear why, at the time, so I just put an advisory lock on every transaction. Obviously that causes serious contention problems as the number of jobs and workers increase.
• ibraheemdev 5y ago
  Okay, but why? Redis is perfect for job queues and pub/sub. Use the right tool for the job.
• chrisallick 5y ago
  interesting... im in opposite camp. do you really need anything other than key/value store and a data structure? ive used redis exclusively for close to a decade.
• taf2 5y ago
  Redis to me is the magic solution to solve so many complex multiple process syncing issues, a global lock, rate limiter, uniqueness with a set, stasher of all the temporary things… no migration, quick solution to complex problems- streams of events to organize pending tasks, etc… to me it’s all about the temporal / temporary state my application needs to keep track in order to work in a multiuser - multiprocess and multi service environment… just a different tool for a different job then what I would use a database for… but really it’s computer science, to each is their own
  • 19h 5y ago
    This is most definitely correct .. until you want to scale beyond a few gigabytes of memory and then with a distributed cluster many synchronization issues don’t seem that trivial anymore.
• osigurdson 5y ago
  This seems to come up on HN at least once a year. Sure it can work but LISTEN ties up a connection which limits scalability as connections are limited and expensive. Also, mitigation strategies like PgBouncer cannot be used with this approach (nor can scale out solutions like CitusDB I don't think). Of course, if scalability is not a concern (or the connection limitations are eventually fixed in postgres), this would be a very viable approach.
• simonw 5y ago
  The best reason I know of to use a relational database as a queue is that it lets you trigger queue operations from within a transaction - so a queue item is guaranteed to be created if the transaction succeeds, and guaranteed not to be created if the transaction fails. Brandur wrote a great article about that here: https://brandur.org/postgres-queues https://brandur.org/postgres-queues
• mxyzptlk 5y ago
  We tried it with just PostgreSQL and struggled with missed jobs. We tried it with Redis + PostgreSQL and haven't looked back. I'll take the blame for not engineering the first version adequately (this was before upsert) but Redis has been useful in so other ways that I'm glad it ended up in our architecture.
• papito 5y ago
  The world would be a simpler place if people actually knew their storage engine like Postgres. I mean, knew it. All the features and how to leverage them in a performant way. Would eat less cloud energy too.
• deleted 5y ago
  [deleted]
• u678u 5y ago
  Surely redis is trivial to set up. Postgres gives you lots more maintenance and admin headaches.
• gchamonlive 5y ago
  Get your requirements in check. How sensitive is your app to latency? How much data and request volume you need to handle? Do proof of concepts, write thorough load tests and go for what makes sense. Either way, no matter which tech you choose, make sure monitoring and alarms are in place and that you do regular maintenance exercises.
• bob1029 5y ago
  I feel I could write a similar article titled: Do you really need PostgreSQL? How to Get Away with Just SQLite How many apps are running pg on a single node and don't really have plans to scale beyond this? Exclusive ownership of a SQLite database by the application can deliver better performance than exclusive ownership over a single node postgres database. This also extends to SQL Server, DB2, Oracle, or any other hosted solution.
  • aeoleonn 5y ago
    Yep. Or even "Do you really need SQLite? How to get away with a dozen secretaries, pencil, paper, and a fax machine" I'm with ya. Why not use the right tool for the right job, given the right tool exists?
• x7777 5y ago
  Sometimes I wonder what kind of apps you are all developing where using postgres instead of redis makes sense? do you have 3 people using your app? Is 800ms latency ok for your users? Or is it that you don't care. Honest question.
  • jayd16 5y ago
    Postgres is plenty fast. If you think it can only do 3 users or 800ms, you're doing something wrong.
    • x7777 5y ago
      It is the blind claim of it being "plenty fast" that is resonated in this thread that I find problematic. The article describes how to implement queueing/app locks/pub sub, without providing any benchmarks on how the same implementations compare to redis under different loads. If you are using redis and you don't care about perf you are probably doing it wrong. If you are using postgres and you care about perf you are probably doing it wrong. "Probably" because context is key.
      • mixmastamyk 5y ago
        Pg is very fast. If it is slower it is because it does more and does it more safely. When not needed those things can be disabled. There are a number of tips about that in the discussion above, such as turning off logging of and partitioning a job table.
• encoderer 5y ago
  Because one can, does not mean one should.
• jdlyga 5y ago
  It's kind of like buying all sorts of running outfits and a camelbak to go on a 3 mile jog once a week. It's about overeager optimism of doing races and marathons in the future. Where in reality, you can get away with just running in the track pants and tshirt you already have on.
• sometimesshit 5y ago
  I need an article how to get away with just MangoDB
• fiznool 5y ago
  We’ve seen success in our current node.js application using pgboss [1] for job queueing. Since the primary database is Postgres, it’s nice to not introduce another dependency into the system, and take advantage of the infrastructure that we already have. The library supports most of what you’d expect from a job queue, and is crucially well maintained! That being said, I agree with other comments that this is somewhat swimming against the tide. Redis is much more commonly used, and so if you don’t mind adding another service into the mix, I’d probably recommend going with Redis instead. [1] https://github.com/timgit/pg-boss https://github.com/timgit/pg-boss
  • queicherius 5y ago
    Out of curiosity, how many jobs are you queuing on average and how did it perform for you at that level? I have been thinking about using the library, mainly because it's less operational hassle than running another service that has to be sticky to one host with persistent storage and backups.
    • fiznool 5y ago
      We are using it quite lightly at the moment, only a few jobs per second, and with the worker running in the same thread as the publisher. The scheduled jobs feature is nice too, replacing the need for a recurring crib job. So far our experiences have been positive!
      • queicherius 5y ago
        Thanks!
• jugg1es 5y ago
  Why would you use a DB or Redis for job queuing when there are extremely inexpensive and highly optimized queuing systems in every major cloud provider? I've had so many horrible experiences with DB-based queuing over my career once you get to a certain scale. Just use a message bus, bro!
  • mixmastamyk 5y ago
    Some folks are not using the cloud for everything. Portability may be an issue too.
    • jugg1es 5y ago
      Some kind of message bus implementation (RabbitMQ etc...) should be a must-have even in an on-premise environment. Queues are a very useful tool in almost every conceivable non-trivial software application.
• markus_zhang 5y ago
  I think it more or less depends on the mindset of the developer. For example from my observation developrrs from telecom or other big companies tend to turn every program into a "solution" i. e. it must have a full weaponry and other bells and whistles. Me? I'm the opposite amd won't add anything or fix anything unless it is really useful.
• gsvclass 5y ago
  I just did a thread on the topic myself https://twitter.com/dosco/status/1402909104012623873 https://twitter.com/dosco/status/1402909104012623873
• gsvclass 5y ago
  And others like this thread on building a MongoDB like JSON database in Postgres in one line. https://twitter.com/dosco/status/1401413712842346501 https://twitter.com/dosco/status/1401413712842346501
• gsvclass 5y ago
  And a search engine in Posgres instead of Elastic Search https://twitter.com/dosco/status/1400643969030127620 https://twitter.com/dosco/status/1400643969030127620
• void_mint 5y ago
  "Do you really need < insert well-tested, verified technology > ? Why not just hand-roll your own in < different technology > ?" Hopefully the emphasis is clear why this is silly.
  • meitham 5y ago
    Indeed this is silly, to me this sounded like "Do you really need a drill? Just hammer this screw down!".
    • void_mint 5y ago
      As with all things, if the goal is exploration, hell yes try this out. If you're actually trying to solve the problems Redis is meant to solve, just use Redis. Otherwise you become the sole maintainer of a glued together, suboptimal version of Redis based in Postgres. For most people, they'd rather build whatever caused them to need Redis in the first place, rather than building their own internal worse-version-of Redis.
• didip 5y ago
  The issue with Redis is that it's distributed story is not great. I wonder if their Raft experiment is finally GA or not. https://github.com/RedisLabs/redisraft https://github.com/RedisLabs/redisraft
• cs02rm0 5y ago
  I tend to ask the opposite question more often. Do you really need Postgres?
• deleted 5y ago
  [deleted]
• munro 5y ago
  I feel like sanity is being restored. Maybe I’m lazy, but yeah I use PostgreSQL for everything I can. Zookeeper is great, but I’ve used PostgreSQL for distributed locking & queueing. Sometimes it’s quicker for me to write a 10–20 line algorithm than install a new piece of infrastructure.
• zomgwat 5y ago
  Security is another factor in choosing not to use a database for things like queues. I like to keep access to the database as tight as possible. I don’t want to start dishing out network and account access to services just because they need a queue or a distributed lock. I could run a separate database instance but that’s worse than just running Redis.
• gregors 5y ago
  Remember when Circleci kept going down due to weirdness with their queued jobs in their database? Read here https://status.circleci.com/incidents/8rklh3qqckp1 https://status.circleci.com/incidents/8rklh3qqckp1
• busymom0 5y ago
  How does this compare in terms of performance since Redis keeps everything in memory while Postgres doesn't as far as I know?
• xdanger 5y ago
  A great job queue for PostgreSQL running on Node.js https://github.com/graphile/worker https://github.com/graphile/worker I've been very happy with it.