6 ms·

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts

• oulipo2 1y ago
  I want to rewrite some of my setup, we're doing IoT, and I was planning on MQTT -> Redpanda (for message logs and replay, etc) -> Postgres/Timescaledb (for data) + S3 (for archive) (and possibly Flink/RisingWave/Arroyo somewhere in order to do some alerting/incrementally updated materialized views/ etc) this seems "simple enough" (but I don't have any experience with Redpanda) but is indeed one more moving part compared to MQTT -> Postgres (as a queue) -> Postgres/Timescaledb + S3 Questions: 1. my "fear" would be that if I use the same Postgres for the queue and for my business database, the "message ingestion" part could block the "business" part sometimes (locks, etc)? Also perhaps when I want to update the schema of my database and not "stop" the inflow of messages, not sure if this would be easy? 2. also that since it would write messages in the queue and then delete them, there would be a lot of GC/Vacuuming to do, compared to my business database which is mostly append-only? 3. and if I split the "Postgres queue" from "Postgres database" as two different processes, of course I have "one less tech to learn", but I still have to get used to pgmq, integrate it, etc, is that really much easier than adding Redpanda? 4. I guess most Postgres queues are also "simple" and don't provide "fanout" for multiple things (eg I want to take one of my IoT message, clean it up, store it in my timescaledb, and also archive it to S3, and also run an alert detector on it, etc) What would be the recommendation?
  • zozbot234 1y ago
    > Also perhaps when I want to update the schema of my database and not "stop" the inflow of messages, not sure if this would be easy? Doesn't PostgreSQL have transactional schema updates as a key feature? AIUI, you shouldn't be having any data loss as a result of such changes. It's also common to use views in order to simplify the management of such updates.
  • singron 1y ago
    Re (2) there is a lot of vacuuming, but the table is small, and it's usually very fast and productive. You can run into issues with scheduled queues (e.g. run this job in 5 minutes) since the tables will be bigger, you need an index, and you will create the garbage in the index at the point you are querying (jobs to run now). This is a spectacularly bad pattern for postgres at high volume.
  • singron 1y ago
    Re 1. Look up non-blocking migrations for postgres. You can generally do large schema migrations while only briefly taking exclusive locks. It's a common mistake to perform a blocking migration and lock up your database (e.g. using CREATE INDEX on an existing table instead of CREATE INDEX CONCURRENTLY). There are globally shared resources, but for the most part, locks are held on specific rows or tables. Unrelated transactions generally won't block on each other. Also running a Very High Availability cluster is non-trivial. It can take a minute to fail over to a replica, and a busy database can take a while to replay the WAL after a reboot before it's functional again. Most people are OK with a couple minutes of downtime for the occasional reboot though. I think this really depends on your scale. Are you doing <100 messages/second? Definitely stick with postgres. Are you doing >100k messages/second? Think about Kafka/redpanda. If you were comfortable with postgres (or you will be since you are building the rest of your project with it), then you want to stick with postgres longer, but if you are barely using it and would struggle to diagnose an issue, then you won't benefit from consolidating. Postgres will also be more flexible. Kafka can only do partitions and consumer groups, so if your workload doesn't look like that (e.g. out of order processing), you might be fighting Kafka.
  • notepad0x90 1y ago
• Capricorn2481 1y ago
  > If it can't fit on one node, do you really need a distributed queue? (Alternative: good ol' load balancing and REST API's, maybe with async semantics and retry semantics) That sounds distributed to me, even if it wires different tech together to make it happen. Is there something about load balancing REST requests to different DB nodes that is less complicated than Kafka?
  • munchbunny 1y ago
    > Is there something about load balancing REST requests to different DB nodes that is less complicated than Kafka? To be clear I wasn't talking about DB nodes, I was talking about skipping an explicit queue altogether. But let's say you were asking about load balancing REST requests to different backend servers: Yes, in the sense that "load balanced REST microservice with retry logic" is such a common pattern that is better understood by SWE's and SRE's everywhere. No, in the sense that if you really did just need a distributed queue then your life would be simpler reusing a battle-tested implementation instead of reinventing that wheel.
• lumost 1y ago
  I suspect the common issue with small scale projects is that it's not atypical for the engineers involved to perform a joint optimization of "what will work well for this project", and "what will work well at my next project/job." Particularly in startups where the turnover/employer stability is poor - this is the optimal action for the engineers involved. Unless employees expect that their best rewards are from making their current project as simple and effective as possible - it is highly unlikely that the current project will be as simple as it could be.
  • procaryote 1y ago
    This is something to catch in hiring and performance evaluation. Hire people who don't build things to pad their own CVs, tell them to stop if you failed, fire them if that failed
    • lumost 1y ago
      Hiring irrational players, or forcing rational people to act outside of their own self-interest is not a winning strategy either. There is nothing wrong with building stuff, or career development. There is also nothing wrong with experimentation. You certainly would not want to incentivize the opposite behavior of never building anything unless it had 10 guarantors of revenue and technical soundness. If you need people to focus, then you need them to be incentivized to focus. Do they see growth potential? Are they compensated such that other employers are undesirable? Do they see the risk of failure?
      • procaryote 1y ago
        Some people do things for other reasons than maximising their monetary gain at every step; there are all kinds of motivations out there including liking solving problems There's also a huge spectrum between "pick a job that's good for your career" and "at every step of the way I'll do whatever is best for me, the company and my coworkers be damned" If you can't see that, just be open with it in the interview process
    • 59nadir 1y ago
• javier2 1y ago
  I dont disagree, and I am trying to argue for it myself, and have used postgres as a "queue" or the backlog of events to be sent (like outbox pattern). But what if I have 4 services that needs to know X happened to customer Y? I feel like it quickly becomes cumbersome with a postgres event delivery to make sure everyone gets the events they need delivered. The posted link tries to address this at least.
  • ThreatSystems 1y ago
    Call me dumb - I'll take it! But if we really are trying to keep it simple simple... Then you just query from event_receiver_svcX side, for events published > datetime and event_receiver_svcX = FALSE. Once read set to TRUE. To mitigate too many active connections have a polling / backoff strategy and place a proxy infront of the actual database to proactively throttle where needed. But event table: | event_id | event_msg_src | event_msg | event_msg_published | event_receiver_svc1 | event_receiver_svc2 | event_receiver_svc3 | |----------|---------------|---------------------|---------------------|---------------------|---------------------|---------------------| | evt01 | svc1 | json_message_format | datetime | TRUE | TRUE | FALSE |
  • dagss 1y ago
    The standard approach, which Kafka also uses beneath all the libraries hiding it from you, is: The publisher has a set of tables (topics and partitions) of events, ordered and with each event having an assigned event sequence number. Publisher stores no state for consumers in any way. Instead, each consumer keeps a cursor (a variable holding an event sequence number) indicating how far it has read for each event log table it is reading. Consumer can then advance (or rewind) its own cursor in whatever way it wishes. The publisher is oblivious to any consumer side state. This is the fundamental piece of how event log publishing works (as opposed to queues which is something else entirely; and the article talks about both usecases).
• ozim 1y ago
  Periodic polling of a DB gets bad pretty quick, queues are much better even on small scale. But then distributed queue is most likely not needed until you hit really humongous scale.
  • TexanFeller 1y ago
    Maybe in the past this was true, or if you’re using an inferior DB. I know first hand that a Postgres table can work great as a queue for many millions of events per day processed by thousands of workers polling for work from it concurrently. With more than a few hundred concurrent pollers you might want a service, or at least a centralized connection pool in front of it though.
    • skunkworker 1y ago
      Millions of events per day is still in the small queue category in my book. Postgres LISTEN doesn't scale, and polling on hot databases can suddenly become more difficult, as you're having to throw away tuples regularly. 10 message/s is only 860k/day. But in my testing (with postgres 16) this doesn't scale that well when you are needing tens to hundreds of millions per day. Redis is much better than postgres for that (for a simple queue), and beyond that kafka is what I would choose in you're in the low few hundred million.
    • 59nadir 1y ago
      This "per hour" and "per day" business has to end. No one cares about "per day" and it makes it much harder to see the actual talked about load on a system. The thing that matters is "per second", so why not talk about exactly that? Load is something immediate, it's not a "per day" thing. If someone is talking about per day numbers or per month numbers they're likely doing it to have the numbers sound more impressive and to make it harder to see how few X per second they actually handled. 11 million events per day sounds a whole lot more impressive than 128 events per second, but they're the same thing and only the latter usually matters in these types of discussions.
• drdaeman 1y ago
  > Do you really need a queue? (Alternative: periodic polling of a DB) In my experience it’s not the reads, but the writes that are hard to scale up. Reading is cheap and can be sometimes done off a replica. Writing to a PostgreSQL at high sustained rate requires careful tuning and designs. A stream of UPDATEs can be very painful, INSERTs aren’t cheap, and even a batched COPY blocks can be tricky.
  • bostik 1y ago
    Plus of course you can take out the primary even with a read from a replica. It's not a trivial feat, but you can achieve it with the combination of streaming replication and an hours-long read from the replica for massive analytical workloads. For large reads Postgres will create temporary tables as needed, and when those in the replica end up far enough, the cascading effect through replication backpressure will cause primary to block further writes from getting through... The scars from that kind of outage will never truly heal.
    • baq 1y ago
      IME (...don't ask) it's easy enough if you forget to set idle in transaction timeout, though I haven't... tried... on replicas
  • sgarland 1y ago
    Postgres’ need (modulo running it on ZFS) for full-page writes [0], coupled with devs’ apparent need to use UUIDv4 everywhere - along with over-indexing - is a recipe to drag writes down to the floor, yes. 0: https://www.enterprisedb.com/blog/impact-full-page-writes https://www.enterprisedb.com/blog/impact-full-page-writes
    • mobilemidget 1y ago
      Did you try uuidv7 yet?
  • enether 11mo ago
    The key question here is what is "high sustained rate" in numbers?
• jsolson 1y ago
  I agree with nearly everything except your point (1). Periodic polling is awkward on both sides: you add arbitrary latency _and_ increase database load proportional to the number of interested clients. Events, and ideally coalesced events, serve the same purpose as interrupts in a uniprocess (versus distributed) system, even if you don't want a proper queue. This at least lets you know _when_ to poll and lets you set and adjust policy on when / how much your software should give a shit at any given time.
  • JohnBooty 1y ago
    From a database load perspective, Postgres can get you pretty far. The reads triggered by each poll should be trivial index-only scans served right out of RAM. Even a modest Postgres instance should be able to handle thousands per second. The limiting factor for most workloads will probably be the number of connections, and the read/write mix. When you get into hundreds or thousands of pollers and writing many things to the queue per second Postgres is going to lose its luster for sure. But in my experience with small/medium companies, a lot of workloads fit very very comfortably into what Postgres can handle easily.
    • enether 11mo ago
      +1 on connections being the limiting factor. Curious how further a connection pooler can get you
• EdwardDiego 1y ago
  Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.
  • codeflo 1y ago
    Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?
    • EdwardDiego 1y ago
      Exactly. There's no concept in Kafka (yet...) of "acking" or DLQs, Kafka is very good at what it does by being deliberately stupid, it knows nothing about your messages or who has consumed them and who hasn't. That was all deliberately pushed onto consumers to manage to achieve scale.
      • speed_spread 1y ago
        Kafka is the MongoDB of sequential storage. Built for webscale and then widely adopted based on impressive single-metric numbers without regard for actual fitness to purpose in smaller operations. Fortunately it always what reliable enough. I believe RabbitMQ is much more balanced and is closer to what people expect from a high level queueing/pubsub system.
      • LgWoodenBadger 1y ago
        What do you mean "no concept...of asking?" During consumption, you must either auto-commit offsets, or manually commit them. If you don't, you'll get the same events over and over again.
        • deleted 1y ago
          [deleted]
        • EdwardDiego 1y ago
• raducu 1y ago
  > 1. Do you really need a queue? I'm a java dev and maybe my projects are about big integrations, but I've always needed queue like constructs and polling from a db was almost always a headache, especially with multiple consumers and publishers. Sure it can be done, and in many projects we do have cron-jobs on different pods -- not a global k8s cron-job, but legacy cron jobs and it works fine. Kafka does not YET support real queue (but I'm sure there's a high profile KIP to have true queue like behavior, per consumer group, with individual commits), and does not support server side filtering. But consumer groups and partitions have been such a blessing for me, it's very hard to overstate how useful they are with managing stateful apps.
  • mwarkentin 1y ago
    https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=255070434#content/view/255070434 https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...
• fragmede 1y ago
  re 4) If you're there, at the risk of drawing the ire of the "cloud is always too expensive" club, be sure you really really really want to run something like Kafka yourself, and not use a hyperscaler's platform queue/queue-ish system, aka SQS or pubsub or whatever Azure/your platform has. Kafka has its own foibles and isn't a trivia set-it-and-forget it to run at scale.
• Buttons840 1y ago
  5. Organize your code so it can work with both a PostgreSQL-based queue or a Kafka based queue. There should be only one code file that actually knows which of the two you are using. Then, if you ever need to switch to something more performant, it will be relatively easy. It's a queue... how bad can you screw this up? My guess is, in most corporate environment, very very badly. Somehow something as complicated as consuming a queue (which isn't very complicated at all) will be done in such a way that it will require many months to change which queue is used in the future.
• Tycho 1y ago
  What were you working on that had trillions of events per hour?
  • munchbunny 1y ago
    An internal data lake at a FAANG company. I don't really want to go into more detail for my own privacy, but it was definitely the kind of problem you only have when you have a FAANG-sized footprint.
• BinaryIgor 1y ago
  100%; often batch integration - pulling data from REST endpoint/endpoints in the background, with reasonable frequency like every 5 -15 minutes - is good enough ;)