24 ms·

Just use Postgres for everything

• throwaway67743 4y ago
  Ok but postgres is the wrong choice for everything - never ever have I seen a database so out of touch with actual database requirements. Working replication - barely and still awful 2 decades after MySQL In place upgrades - nope Access control that doesn't require root access to filesystem - nope The list goes on... Disclosure: I do postgres in production for 10000s of machines in like 49 countries, I understand how databases work and have been doing this for 20 odd years, it's a bad choice because postgres is written by purists, not people who deal with it in the real world.
  • mikl 4y ago
    Not sure what you’re talking about wrt. access control, but PostgreSQL replication works just fine for most people, and the lack of in-place upgrades isn’t really a problem unless you’re working on a massive scale with no downtime tolerance. You run things at an unusual scale, you have unusual problems.
    • throwaway67743 4y ago
      Having to edit a flat file for access control is absurd, perhaps 2 decades ago when it was common sure but everyone figured out how to do it without, also it's not really at that much scale when every other database implementation makes this painless (even Oracle!) Postgres still lacks basically everything that other replication implementations do, if they have to mention log shipping by rsync etc they've already got it wrong. For clarity they only recently figured out "streaming" replication which again, is what everyone else has been doing for years.
      • throwaway67743 4y ago
        Annoyingly can't reply to your reply below but: Replication can't (unless it's changed recently) be granted via ddl, and isn't propagated either - but otherwise sure It's not just that though, the general behaviour is counter to how real world deployments work, like for example basically anyone at scale moving away to something else because the admin and operational overhead is way too much
        • deleted 4y ago
          [deleted]
        • pritambaral 4y ago
          > Replication can't (unless it's changed recently) be granted via ddl, and isn't propagated either - but otherwise sure It can, and is, at least as of the five year old (and now unsupported) release 10.
          • throwaway67743 4y ago
            "Client authentication for replication is controlled by a pg_hba.conf record specifying replication in the database field. For example, if the standby is running on host IP 192.168.1.100 and the account name for replication is foo, the administrator can add the following line to the pg_hba.conf file on the primary:" https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION https://www.postgresql.org/docs/current/warm-standby.html#ST...
      • mikl 4y ago
        Again, not sure what you’re talking about here. `pg_hba.conf`? Normally you don’t need to modify that regularly (if ever). Normal access control can be done with GRANTs, just like MySQL or Oracle. As for streaming replication, that was added in PostgreSQL 9.0, six major versions ago. Sure, it was late to the party, but it’s not exactly a recent addition.
        • throwaway67743 4y ago
          Ah for some reason I can reply now: Regardless being 2 decades late to the party is unacceptable - it's still not complete and makes real world management a chore, I still admire the purist attitude it has its place but you can't also then claim it's a useful competitor - which is what they do. Unfortunately it's a perl postgres vs the world thing - both are irrelevant.
          • mikl 4y ago
            That they were late to the party doesn’t really matter _now_. It might have mattered five years ago. As for the config, one might argue that reconfiguring replication isn’t a common task, usually something you do when setting up the cluster and never mess with again. I understand that there are pain points running PostgreSQL at massive scale. But 99.9% of projects needing a database never reach that scale. And once they do, they probably need to re-engineer big chunks of the application anyway.
            • throwaway67743 4y ago
              Well this is one aspect - application should not have to care about underlying db topology - if it does then the db is wrong, my main issue tbh is that other databases implement the inevitable like proxying or sharing while postgres leaves it to horrific perl scripts from the last century, it's really not admissable as a production solution
              • pritambaral 4y ago
                > while postgres leaves it to horrific perl scripts from the last century Your knowledge of what Postgres supports and how to administrate it seems to be a few years out of date. There's plenty of examples in this thread alone.
  • pritambaral 4y ago
    Funny you say that, because I got the same feeling from MySQL. I mean, dropping data, ignoring constraints, all done silently. A database that cannot keep my data safe is just not a database.
    • throwaway67743 4y ago
      Ah yes, the old MySQL sucks argument based entirely on a bug fixed a decade ago
      • pritambaral 4y ago
        I'm not trying to pick a fight with you. Nor am I trying to say "MySQL sucks". I'm commiserating with you, or at least trying to, at your level. You're the one bursting in here shouting "Postgres sucks!" "based entirely" on a lack of features added more than "a decade ago". You have your choices, we have ours. You'd rather have the convenience of features, even at the risk of losing your data; we'd rather have the peace of mind of knowing our data is safe, and build features that don't exist out of whatever else we have, however we can. It's not always about what bugs or features exist or don't exist now. It's also about the nature of a project and how it's developed, and how it's going to evolve or otherwise behave in the future. After all, we're talking about databases here; we trust our data to them.
• HeavyStorm 4y ago
  Not read it. But. I second the title.
• vbezhenar 4y ago
  If I know how to use Postgres as a relational DB, it does not mean that I know how to turn it into message queue. I guess I can build some kind of implementation but that would take time to research and implement it. And I can spend that time to implement RabbitMQ. Which is likely million times more popular than Postgres for this particular task, so I'll have more information. I don't agree to this post. One should use old, proven and stable solutions. If you need message queue, you need rabbit MQ. If you need cache, you need Redis. Don't implement non-standard non-conventional solutions if you can avoid it. Find out what's industrial standard and use it. Avoid dying solutions, avoid new solutions. But also remember that you should have something unique. You don't want your business to be easily repeated by someone, because you will compete to margin zero. It's not necessarily tech thing. But it might be tech thing. You don't build cloudflare using off-the-shelf nginx.
• Havoc 4y ago
  The truth lies somewhere in the middle most likely. Replacing redis with PG for example feels very "when you have a hammer everything looks like a nail" to me.
• jmartin2683 4y ago
  This seems like a collection o terrible advice. These things are all very conditional, and such blanket statements are nonsense. Postgres is great, but it’s not all the things for all the applications. I wouldn’t hire (or work for!) this guy, fwiw.
• pyuser583 4y ago
  If you’re going to “use for everything” why not be more creative? Use DynamoDB for everything? Use SQLite + S3 for everything. Heck use Redis for everything. I understand that Postgres is insanely flexible. It supports pretty much any type of data storage you want. But if you’re going to focus all your developer knowledge on on product, consider whether another DB might be better.
• pyuser583 4y ago
  I think a better way of handling this is creating layers of abstraction between data storage and code. Annotate whether this is a hot cache or cold storage, and be able to flip out the infrastructure. ETL frameworks support this. Probably should be more widely used (web devs, etc).
• mannyv 4y ago
  This is good advice, unless it doesn't fit your use case. As an example, using pg instead of redis is great until you need to handle hundreds or more requests per second. Using pg instead of kafka works great until you have to ingest a few gigs of data a second. Using pg instead of mq works until you have a few thousand messages a second. Etc etc.
• johnthescott 4y ago
  postgresql handles queues well. the tricks are skip lock, hiding the db behind protocols/micro services and judicious use of unlogged tables. here is a 2015 talk on critical messaging using pg. https://github.com/jmscott/talk/blob/master/ams-pgopen-20150917.pdf https://github.com/jmscott/talk/blob/master/ams-pgopen-20150...
• scrollaway 4y ago
  I'm just about there and agree in general. Currently integrating Procrastinate (https://procrastinate.readthedocs.io/en/stable/ https://procrastinate.readthedocs.io/en/stable/) to use Postgres as a job queue in a Django API backend. Dropping dependencies is very nice especially when dealing with an MVP / small apps.
  • whatever1 4y ago
    Why not the battle tested celery?
    • llIIllIIllIIl 4y ago
      Battle tested Celery has dropped the ball so many times during operations and updates, that I would be very uncomfortable giving that recommendation.
    • scrollaway 4y ago
      See article? But also, Celery has an awful DX. I think the question is more, why do I need celery when Postgres can do the job itself?
    • simonw 4y ago
      Here's a really good article about why you might want to involve your database in queues (short version: transactions): https://brandur.org/job-drain https://brandur.org/job-drain
  • phoebefactorial 4y ago
    I do Django freelancing and have had a fair amount of small clients with relatively few users. Adding workers and a message broker adds a lot of complexity over just a single Django server. I even had the thought to build something like this but looks like someone already did! Thanks for sharing this, looks like a great tool to know about.
• jb1991 4y ago
  Why does the article talk about client complexity, then say Postgres can be used for everything, but then not address client complexity. Does Postgres have a nifty server-side rendering pipeline that could replace front-end complexity?
  • Finnucane 4y ago
    "You probably don't need as many back-end services as you think you do" isn't quite as good a title.
  • glutamate 4y ago
    Yes: HTML. Seriously, just generate HTML on the backend and serve web pages like it's 1999. Easiest way to get 100% on all Lighthouse scores.
    • jshen 4y ago
      Don’t write any code, easiest way to not have bugs.
      • ThePadawan 4y ago
        You joke, but I have literally never had as great of a success at work as that one time my team didn't end up building the software. We had great ideas about scheduling and caching and task priorities... ...and then we asked the customer what they wanted, and they wanted none of it. So we built none of it, and produced a solution that just did the stupidest thing, and did it without any edge cases, without ever crashing, reliably, as a cronjob, once on Sunday night. Some people would be disappointed that they couldn't put this on their CV because it didn't involve FancyTech #413, but damn it, I am still proud of that stupid thing.
        • robin_reala 4y ago
          I interviewed a (junior) candidate once that had tried and failed at a hackathon to build an Rails system that connected restaurants and shops with excess food to charities that gave food away to those that needed it. I asked him what he’d done to work around the technical difficulties, and it turned out that he’d set up a Wordpress site with a phone number of the guy running the scheme, and a Google sheet to manage contact details. A better definition of MVP I’m yet to see.
• jimbokun 4y ago
  As long as you have good interfaces in your code to make it possible to swap out Postgres to more specialized components when needed, I think it’s a good idea. Saving ops resources by managing fewer services when getting started is a good idea, until scale necessitates dedicated technologies for certain components.
  • glutamate 4y ago
    I wouldn't even worry about coding to interfaces. When you need to replace Postgres because you have more than 1M users, you will probably have the revenue to refactor your code to swap out Postgres dependencies. Because you have good integration and E2E tests, right?
    • thomaslangston 4y ago
      Good tests are a lot easier to write with good interfaces.
      • isbvhodnvemrwvn 4y ago
        Database abstraction layers are leaky, if you don't test things like isolation levels or other tricky behaviors, those tests will have limited value.
        • jimbokun 4y ago
          Limited value is better than no value.
    • marcosdumay 4y ago
      At such time you will also have plenty of interesting new problems that require changing most of your code anyway. I agree that hedging against having to move from a FOSS database is pure waste.
  • cube2222 4y ago
    That will make you lose a lot of productivity boosts SQL can give you. Obviously, if you anticipate to have very high traffic and applicable usage patterns, do use that advice, but if your apps anticipated usage pattern is in fact not "very high rps per buck made", then I recommend the opposite. I've went the whole way from very-well-abstracted-away services/repositories to an almost complete lack of abstraction. Direct SQL or ORM in your functions, operating on many models at once, with a real postgres database available to all your unit tests, treating the SQL code as part of your application logic. Transaction per test so that unit tests are fast to run. I've been very happy since. SQL is very powerful if you use it as SQL and not as a glorified KV store.
• PaulsWallet 4y ago
  > Use Postgres with TimescaleDB as a data warehouse. How does this stack compare to Snowflake or Redshift?
  • tootie 4y ago
    Redshift is just a heavily customized Postgres
    • PaulsWallet 4y ago
      C++ is heavily customized C. The heavy customization make Redshift a columnar database and more ideal for querying large amounts of data quickly. How does Timescale help Postgres in this area?
      • chrisdalke 4y ago
        Timescale is built around a concept they call "hypertables", which automatically partition data into a set of smaller tables segmented by time range. Timescale exposes the time-series data as if it was a single table, but behind the scenes is managing queries against the individual table partitions and automatically creating new partitions as data is inserted. By tuning the chunk sizes so their data fits in memory, many common queries gain a lot of efficiency. It's built around some assumptions of time-series data: Most inserts and queries are for recent data and are generally ordered. I've had great experience with TimescaleDB for small-medium time-series loads such as sensor or analytics data; I've found it's pretty plug-and-play and have used it to store tables with ~1B time-series rows of geospatial data, sensor values, etc.
    • darkr 4y ago
      It uses Postgres wire protocol sure, and may even contain some Postgres code; otherwise this is like saying “a tank is just a heavily customised car”
• bioemerl 4y ago
  Man, I agree but I also really disagree. I think you should keep complexity down by only using postgres, but just use it as a sequel database until you scale enough that it's a problem. 90% of the time it's never going to scale that far. When it does, use the tools people have made to do those jobs correctly. Do not hack you way into half-made tools put into a database that isn't specialized in that job. It seems like you're making your life more simple by having only one system, but you're having that one system to so many things at the same time that it's going to be an absolute kludge in the long term. Does anyone have experience trying something like this? If you did, is this how it turned out or did it actually work all right?
  • sk55 4y ago
    Nailed it.
  • N_A_T_E 4y ago
    Its somewhat situation dependent, but I've developed healthy systems that scale to surprisingly high throughputs with just a couple well designed sql databases, read replicas and memcached.
    • spiffytech 4y ago
      Gall's Law: > A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system. A SQL database, maybe supplemented by a cache, can carry most projects as far as they'll ever go. And if you outgrow it, replace it with something that meets your new needs. Any given project's "right tool for the job" can change over the project's lifetime, and optimizing for problems you don't have is quite harmful.
  • jb3689 4y ago
    Yep, let’s not forget that your “simple” Postgres cluster gets complicated really fast when you start having to graft layers of tools on top of it. There’s a reason why Spanner and DynamoDB exist. God forbid someone try to use Postgres to solve the same problems and actually get the configs wrong leading to years of inconsistent data. Definitely never seen that happen…
• viach 4y ago
  > Use stored procedures For God's sake, please don't
  • jmull 4y ago
    Your app logic concerned with data executes somewhere, right? So… why exactly would you exclude compute running close to the storage? You can use that minimal latency. Of course, people can create an uncontrolled mess of spaghetti code out of sps/funcs… like they can with any kind of code.
  • daneel_w 4y ago
    I second that. SPs/funcs have this weird tendency to always stay hidden in the fringes and out of sight, easily forgotten when adding new functionality, easily overlooked when making changes elsewhere.
    • KingOfCoders 4y ago
      From my experience only if there are dedicated DBAs and you have too many systems running - then you forget one. If you only have server code and the stored procedures in the same repository, with migrations, this problem goes away.
    • maerF0x0 4y ago
      I believe that stems from people frequently not including them in version control, or not doing tests.
      • daneel_w 4y ago
        We version-control our SPs/funcs. We have unit tests, we have integration tests.
    • simonw 4y ago
      I think stored procedures can be perfectly safe provides you follow these rules: - they live in source control - they are covered by automated tests - they are applied using some form of automatic database migration system (not by someone manually executing SQL against a database somewhere) If you don't have the discipline to do these things then they are likely best avoided.
• reese_john 4y ago
  > Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). I wouldn't describe Kafka as a message queue. It's more of a distributed commit log.
  • pdntspa 4y ago
    I have done this, and the one thing I would note is that if you are requesting a maximum number of rows per call, and if something in the queue breaks for all returned rows (like because it crashed with an exception or something and its status field isnt updated), then you can run a situation where every call to get more rows goes to the same broken rows and the queue becomes blocked. I hadn't figured out a great solution for this but otherwise my SKIP LOCKED queue worked great!
  • KingOfCoders 4y ago
    Yes I do agree. But looking into many startups, many of them use it as a message queue with the ability for transformations (E.g. Spark).
    • jpgvm 4y ago
      It's fine for streaming, i.e Spark and friends. I generally define streaming as the success of each individual message is entirely uniform, i.e the system can either process messages or it cannot. If this holds none of Kafkas drawbacks should significantly affect you. When people talk messaging they normally mean something lower latency, potentially out of order and where the success of each message is likely independent of other messages. For these use-cases Kafka is not well suited. For these cases you will want something like Pulsar, SQS, PubSub, etc.
  • manicennui 4y ago
    Unfortunately many (most?) people use Kafka as a message queue with larger message size.
  • ARandomerDude 4y ago
    Messaging is the first use-case given by the Kafka docs [1] and is the first core capability they mention on the Kafka landing page [2]. 1. https://kafka.apache.org/documentation/#uses_messaging https://kafka.apache.org/documentation/#uses_messaging 2. https://kafka.apache.org/ https://kafka.apache.org/ I use (and think of) Kafka as a message bus, with distributed commit logging being the mechanism for how it accomplishes that task.
• radicalbyte 4y ago
  I spent a decade using SQL Server for everything (back when Postgres wasn't close) and it is an exceptionally effective strategy. Deployments are simple, debugging is simple, operations are simple. Do that until scale forces you to start specialisation. It's surprising how far you can go with database-as-MQ. Even database-as-IPC can work for smaller systems.
  • DenisM 4y ago
    How do you handle high availability? Ideally I would want to not lose many transactions and have automatic failover. SQL Server has that, but it not the default config, and it becomes expensive af.
    • popotamonga 4y ago
      Do you really need it early? Got like 2 failures in 20+ years, i just let it fail.
      • kkielhofner 4y ago
        Exactly. A lot of startups are convinced they'll need Google scale from day one. Then, of course, the overwhelming majority fail in the first year. Get a big, reliable, and cheap vhost/server somewhere, use as many "can't really go all that wrong" components like postgres, minio, etc and dockerize everything. If you want to get "fancy" use ZFS and setup some snapshots and backup. Most solutions don't even need 100% uptime. Communicate maintenance windows to customers and you'll be fine with a total of an hour of downtime (or whatever) in the first year. Most big, really complex, early over-engineered and unnecessarily "optimized" solutions have enough footguns you'll probably end up with more unscheduled downtime in the first year anyway. In the rare event the startup really succeeds and customer demand, load, uptime requirements, etc demand it you can throw revenue/funding/etc at a K8s control plane on your favorite hosting provider, use a managed postgres/db/whatever, and S3 compatible object store, etc. Or, if things get really big skip all of that and hire in house talent to manage a couple of racks of leased hardware (same opex as cloud but almost always SUBSTANTIALLY cheaper) in geo redundant/distributed colocation facilities. I've launched multiple startups with this strategy and it's gone very well. My current startups all run from the same big (but 10yr old) hardware that has loooooong since paid for itself even with lots of GPU, storage, etc upgrades over the years. People can be kind of scared of hardware but I've never had downtime or data loss caused by a hardware failure in almost 20 years of this approach. People are always amazed when I do things with ML, TBs of data, lots of bandwidth, etc and I tell them my total hosting costs are $150/mo.
• jawns 4y ago
  The link to full-text search uses a relatively small dataset. Can anyone provide advice about how large a dataset has to be before this breaks down?
  • buro9 4y ago
    I don't know when it breaks. But I'm on a single medium sized Linode and doing full-text search over 120M small (< 1KB) to medium (< 10KB) text documents and it's still so fast I'm not sure that I will ever need to consider an alternative.
• dpedu 4y ago
  What's the favored way to run highly available postgres these days?
  • bsagdiyev 4y ago
    I use Patroni behind HAProxy setup to automatically point to the new active host in the case of a failover. It's not a huge scale setup though so I'm sure there will be better ways or other input on this.
  • whitepoplar 4y ago
    pg_auto_failover is nice: https://github.com/citusdata/pg_auto_failover https://github.com/citusdata/pg_auto_failover
  • cube2222 4y ago
    AWS Aurora has been working well for me in general.
    • baq 4y ago
      How painful is the cost?
      • throwawaythekey 4y ago
        It's been a while now but we saved about 10-100x moving from aurora to dynamo for a pure/simplish OLTP use case. Lost a lot of flexibility but the aurora costs were going to send us under. We also had a few issues where aurora was giving us *incorrect query results* and aws support were not helpful. Most of the issues were denied, even with simple reproductions, and then got fixed years later (~2-3) after we had given up. We still use it today but stay away from cutting edge features, unusual query patterns, and high volume use cases. Having said all that, SQL still seems like the correct choice for getting an MVP out quickly.
      • phamilton 4y ago
        Not OP, but a big Aurora fan. Not that bad actually, but it varies by use case. Reasons aurora can be cheaper than you think: 1) Autoscaling. Adding a new reader takes 15 minutes. Instead of provisioning for peak traffic and paying for it 24/7, run an extra reader for a few hours each day. 2) Metered IO. Aurora IO can handle 400k+ iops when needed. Or it can hum along at 100 iops. You pay only for what you use, you don't have to provision for peak load 24/7. Switching to Aurora saved us money over vanilla RDS. Self hosting postgres may be cheaper, but not by as much as would first appear.
• haolez 4y ago
  I've gone this route before and debugging your system can become problematic, especially when using adapters to format tables as JSON. But it works and the dev speed is unparalleled.
• posharma 4y ago
  Just use the right tool for the job at hand. If it’s Postgres for now, sure. There’s really nothing more to this. Period. But saying this would not make for a blog post. I can’t help but categorize this post as juvenile at best. Generalizations like “use XXX for everything’ should be avoided at all costs. No software product serves well to such sweeping generalizations. I’m surprised that it’s coming from a CTO. They should know better if they’re worth their salt .
  • simonw 4y ago
    It's often better to use the tool that is capable but isn't the absolute best possible option to solve multiple problems, rather than taking on the burden of running five different "best" tools for five different problems.
    • cortesoft 4y ago
      If you are at a startup, sure. But many of us work at big companies that have the scale that requires specialized solutions. I just frustrated that everyone always seems to assume everyone is working on POCs at startups.
      • necovek 4y ago
        Somebody just jumping on a new tool thinking it's the right tool for the job might still end up with worse performance than someone proficient with another tool (like Postgres) might get in less time. No matter the scale or how big it is. If you are talking of specialized solutions, that means you know exactly where an existing, well known solution is failing you and why. Eg. if you drop all foreign key references and constraints in Postgres, you might get similar write performance to other databases which can't make those guarantees when you do need them.
      • simonw 4y ago
        I've worked at big companies, where I have championed introducing "the right" technology for scale reasons... and have in some cases later regretted it because with hindsight we would have been fine sticking with what we had already, at a greatly reduced cost in terms of time and complexity.
• mgkimsal 4y ago
  > Use Postgres to generate JSON in the database, write no server side code and directly give it to the API. Who does this? "give it to the API"... ? Still sounds like there's "server side code" there if there's an "API" involved. Surely they're not meaning let front-end code hit the database directly?
  • scrollaway 4y ago
    It's possible to have Postgres serve HTTP, actually. I've seen many experiments like this. I don't remember the name, but... actually... Yeah, thanks ChatGPT - It's called PostgREST: https://postgrest.org/en/stable/ https://postgrest.org/en/stable/ I've obviously not used it and to be honest, I probably wouldn't. But IMO Postgres is a radically different type of framework. People just think of it as a SQL database, it can be much more though. Tooling is just a bit lacking. Edit: Heh, Retool is sponsoring them. "PostgREST for the backend, Retool for the frontend". Cool idea. I might reconsider...
    • fnimick 4y ago
      You can go even farther and generate an entire GraphQL api with row level security, custom functions, etc. I've had a great experience developing with Postgraphile - https://www.graphile.org/postgraphile/ https://www.graphile.org/postgraphile/
      • xrayarx 4y ago
        Looks and sounds great, but nodejs
    • dragonwriter 4y ago
      PostgREST is an external web server; postgres isn’t serving HTTP, though its effectively doing everything else that the backend requires besides serving HTTP when you use PostgREST.
      • scrollaway 4y ago
• daneel_w 4y ago
  "Use Postgres for caching instead of Redis with UNLOGGED tables and TEXT as a JSON data type." What the heck?! No!
  • robin_reala 4y ago
    More info on why not?
    • qatanah 4y ago
      You offload some load like pg connections from postgres to redis. Your pg shared buffers would also be available for other stuff.
    • daneel_w 4y ago
      If the suggestion truly is to try squeeze as much utility as possible out of a single RDMBS setup, and if you already need and use an RDBMS, then I would consider it as an option. If not, it's an absolutely idiotic choice. And that's the problem with this article - it really comes of as a suggestion to supplant your Redis setup, a dedicated key-value store, with Postgres, a full-fledged and highly complex relational database.
    • kmbfjr 4y ago
      Because it is not a cache. High entropy tables require a lot of vacuuming, which adds overhead to actual database operations. While you can set autovacuum parameters per table, they still are not free . If you need something like Redis or memcache, use that for anything that is heavy on writes and deletes.
• jamal-kumar 4y ago
  I love postgres for most things, but these days (Especially while my product is in early development, embedded, or just not internet-facing) sqlite is amazingly workable. Killer postgres features however: Row-level security (Fantastic when you're using something like postgrest for rapid backend development [1]), and its built in fulltext search engine is 'good enough' for use cases like when you have an enormous users table and need to index something simple enough, like email addresses for quick login. [1] https://postgrest.org/ https://postgrest.org/
  • kristianp 4y ago
    One thing I miss in sqlite is numeric types that store binary-coded decimal. (Is there a plugin for that?) This is very useful for currency calculations.
  • giovannibonetti 4y ago
    It is not very useful to add full text search to an email field used for login. A regular unique index, perhaps case insensitive, is what you should be using.
    • VWWHFSfQ 4y ago
      is case insensitive fulltext faster or slower than case insensitive index search on a varchar
      • microsoftdoes 4y ago
        The index should not be slower unless something is seriously wrong.
    • lolinder 4y ago
      There's an extension for trigram similarity operators which is useful for a quick and easy fuzzy search for small things like email and name: https://www.postgresql.org/docs/current/pgtrgm.html#id-1.11.7.44.6 https://www.postgresql.org/docs/current/pgtrgm.html#id-1.11....
    • 4y ago
• klysm 4y ago
  Fully agree with this philosophy. With json support and timescale you can get really far. If you run into performance problems you’ve already won.
• samsquire 4y ago
  If only there was a infrastructure in a box that gave you all the features he mentioned in a well maintained distribution. Then rather than saying just use Postgres you say just use X. I like postgres but I've only used it in production once. My experiences were fine. We used alembic for database migrations. It was a Python Flask app. We also used it as a message queue and stored JSON form data. There's still hand crafted code for using Postgres as a message queue. A bit like Linux distributions which are aggregations of desktop or server software collections of well integrated software. I tried to build a stack that could be spun up with all goodies included. But I would not want to inherit something hand stuck together. But at the same time, the thought of setting up Kubernetes for everything from scratch is also a lot of work. If you need to use cloud, that's also a lot of work. https://devops-pipeline.com https://devops-pipeline.com
• nikeee 4y ago
  You can so use pgnotify for a simple pub/sub.
• johne20 4y ago
  NOTIFY / LISTEN is handy too. (pubsub)
• higeorge13 4y ago
  If we are talking about really small business, yeah fine but I really don’t see the point of such articles without any numbers. Do some benchmarks and compare postgres with kafka, postgres with elasticsearch, etc. Define the use cases or minimum number of table rows or number of requests that performance deteriorates.
  • themenomen 4y ago
    And not just performance, also costs related to scalability
• dathinab 4y ago
  Just one thing: 1. Teach people transaction isolation levels. 2. Have some standard written down rules about what SQL type to use in which situation and what approaches to use for table structure. 3. don't overuse triggers or stored plSQL procedures or similar they are hard to test and debug The two main problems I have seen with SQL databases is: 1. People not understanding transaction isolation, most inconsistency bugs or strange behaviour no seem to be able to explain I have seen where due to this. As far as I can tell this is a HUGE problem, even through SQL databases are used much less and at least theoretically it's normally through in any bachelor level course about SQL. 2. people wasting time on discussion about types and table structure. For example weather this or that int type should be used,for a lot of use-cases it's just fine to use bigint (64bit) initially for everything. With 32bit there is often some edge case in which it isn't enough. For example if you do 5_000_000 increments per second on a 31bit (i32>=0) counter it overflows in ~7 minutes but if you the same with a 63bit counter it takes ~58494 years. Sure you sometimes might have to go back and optimize storage and there are cases with clearly constrained numbers, but it's the best default for not clearly bound integer numbers. Similar "initial start with" default approaches can be written down for most situations in just a single DIN-A4 page of paper or so.
  • KingOfCoders 4y ago
    I think (1.) is especially tricky b/c it works if you're small (experienced this myself with Hibernate, not understanding transaction levels) and then breaks with the number of concurrent transactions. When the company rapidly grows you get mysterious bugs - in the worst case customers seing other customers data because of the wrong transaction levels.
    • dathinab 4y ago
      Yes, thats why I think it's a huge problem for the industry. How the heck is it possible that so many developers with bachelor and master degrees which otherwise do a reasonable job don't even know that they have a dangerous knowledge gap there??? through that: > in the worst case customers seing other customers data because of the wrong transaction levels. should not happen even with wrong transaction levels, that indicates some additional serious design problems IMHO, likely related to premature optimizations
• ericHosick 4y ago
  My current stack is PostgreSQL (on Supabase) + retool (or similar front end tools). PostgreSQL Extensions: http, pg_cron, timescaledb To help with development, I am using https://www.npmjs.com/package/sql-watch https://www.npmjs.com/package/sql-watch (written by myself) to do continuous development and testing (TDD/BDD). The stack "doesn't scale" but the turn around time for development is crazy.
  • sk55 4y ago
    What are some other frontend tools you've found to be effective?
    • ericHosick 4y ago
      I've quickly looked at a few but for admin applications, retool seems to be a good choice. I don't have a lot of insight into other tools.
  • lysecret 4y ago
    Big fan of retool too. I also like to throw FastApi in front of Postgres on lambdas or cloud runs.
  • ttfkam 4y ago
    I prefer the model of event triggers writing to a DDL audit table and calling NOTIFY. Then your script can just LISTEN for changes after you run your idempotent scripts. Then you're capturing all changes to the DB, not just the ones you're expecting. Because it's always that guy making manual DDL updates that screws everything up, isn't it Mr. Hosick. ;)
• jerry1979 4y ago
  I really like postgres, but I'm starting to wonder if some of these articles are GPT or not.
  • donatzsky 4y ago
    The "Use stored procedures" link points to ChatGPT, so I'd say that's a distinct possibility here.
  • ls-lah_33 4y ago
    Good to see I'm not the first one to notice. Some of the author's points are super vague: > Use Postgres for caching instead of Redis with UNLOGGED tables and TEXT as a JSON data type. > Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). I'm sure if we try hard enough we could find some sort of meaning in these points, but then the content is coming from the reader not the author. Also from the author's mastodon: > Using ChatGPT As a Co-Founder
    • jerry1979 4y ago
      I wonder if the author is trying out GPT as a kind of auto-complete for ideas. I have not had a lot exposure to GPT, but it does seem to have something like an "accent". I was listening to This American Life, and they had a mini game show about accents and ages. So, maybe that's just on my mind.
• smitty1e 4y ago
  Then there is http://postgis.net/ http://postgis.net/ for Geospatial requirements. Serious question, though: when the data span several instances, how well does PostGreSQL do compared with Elasticsearch?
• friedman23 4y ago
  Can someone explain why I shouldn't just use something like cockroachdb, yugabyte or spanner for my primary database nowadays? I've seen the pain and resources wasted from teams needing to find ways to shard mysql and postgres and make them scale. Sure, you might say I might never need that scale but why not just avoid this problem entirely? If I'm building something I intend to support hundreds of millions of users why would I choose a technology I know will fail to scale to that?
  • ReflectedImage 4y ago
    Because you are working for a company that like 99% of companies have less than a million users? You want data analytics, financial reports, the ability to query your data in new ways you didn't think of when you first built the product? NoSQL comes at a great cost. When you hit a million users, the entirely of your codebase will be on-fire with scalability issues anyway so it won't make any difference?
    • endisneigh 4y ago
      Using a sharded database is not just for scale.
      • necovek 4y ago
        So instead, add more complexity? Isolation? Please do elaborate.
    • friedman23 4y ago
      Those databases I mentioned are not NoSQL, they are relational databases.
• Joel_Mckay 4y ago
  The best part of Postgres was it wasn't MySQL. ...geospatial types were just a nice addition later. ;)
  • wistlo 4y ago
    Agree. In my case, I went from Excel stare'n'compare to MS Access and finally mySQL for several years. mySQL worked great for my lil piece of company business involving a few million records. But then I volunteered to do some analysis with files an order or two of magnitude larger (10 mlllion-100 million). mySQL started coughing up with memory errors. Uh oh. Embarrassing. I should mention that I'm doing all this on a Windows desktop, as that's what is installed and locked down on the machines they give me. Enter postGRES. No matter what I throw at it, it gets to end of the queries without errors—in Windows, no less. With an nVME and 32 GB of ram, I can run a complex report on 70M records in seconds. Loading and indexing takes about 15 min. mySQL was very very good to me for a long time, but for the really big stuff I'm sticking with PostGRES.
• DrBazza 4y ago
  I've just started at a company using MSSQL and it feels like a massive backwards step. This article indirectly shows what's missing with MSSQL - an 'ecosystem' of people developing products on top of it, and a much larger number of people working on what is, ironically, a free product.
  • bags43 4y ago
    MSSQL features that are missing in PostgreSQL are impossible to be made by community. ;) I am using both PostgreSQL and MSSQL in the production since 2003 and they are simply not in the same league. Especially tooling around MSSQL is miles ahead.
  • baq 4y ago
    MSSQL is a beast of a RDBMS. There’s little it can’t do better than Postgres, so you either have to read up on it or the niche you’re in doesn’t fit.
    • tormeh 4y ago
      The db that makes you use bit instead of boolean is a beast? I disagree. IMO it’s one of those technologies waiting to die.
      • baq 4y ago
        If your niche is storing booleans in the DB… then I could see it matters just a bit? Otherwise it’s a stupid legacy nuisance with zero prod impact. You could’ve pointed out the lackluster json support or lack of generalized indexes, but this? Come on.
  • albertopv 4y ago
    Having experience with both, with small-medium size DB, they have pros and cons. MSSQL wins on ootb tooling, easier monitoring, collations, better query planner, ootb encryption, ootb data compression etc. I have seen running more than 100 databases of hundreds of GBs on a single MSSQL instance without a problem. I heard stories about data corruption, never happened to me. Postgres wins on (lots of) datatypes, indexes types and plugin ecosystem, eg Postgis is great. Per process connection is meh, but can be mitigated. It loses a lot on all features based on operating system libraries, different OSes give different results, not great, not at all. Also if you are using postgres at enterprise level, better have some kind of support, which may cost a lot. So far I had less issues with MSSQL than with Postgres. YMMV, ofc.
• cortesoft 4y ago
  So I should replace our 200 node Kafka cluster with Postgres? Ok, I’ll get started on Monday.
  • KingOfCoders 4y ago
    Tell me how it went ;-)
  • freedomben 4y ago
    > Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). you have a 200 node Kafka cluster just as a message queue?
• 29athrowaway 4y ago
  You can use a spoon as a knife, it will just take you more effort. Probably you can use Magic the gathering cards as a computer since it is turing complete, but it will just be inconvenient as hell.
• faebi 4y ago
  Can I use a few postgres tables as redis style in memory database? Can I finally have more connections to postgres without using pgbouncer?
  • Ingaz 4y ago
    What's wrong using pgbouncer? Today it's not rare to run http proxy in sidecars for every service.
• guidoism 4y ago
  Ouch. That first image has really bad kerning. It hurts my eyes.
• yehia2amer 4y ago
  We actually use dapr which provides abstraction layer over all of those things including State management, Pub/sub, etc.. So it’s really easy to switch between them without changing the code at all. https://docs.dapr.io/developing-applications/building-blocks/ https://docs.dapr.io/developing-applications/building-blocks...
• endisneigh 4y ago
  Though Postgres is great, I personally disagree. If you're starting a brand new project now I think you should use something redundant and distributed like CockroachDB or Vitess.
  • theptip 4y ago
    This seems like overkill to me; a single Postgres master with replication/standby gets you very far. Why pay the ops burden of Cockroach when it buys you nothing in your first N years of business pre-PMF? Concretely speaking most startups should be using RDS or whatever hosted Postgres their cloud provider offers, not running their own. I think it’s sensible to use what one knows best, but for default advice to others I think ”simplest viable option” is generally better.
    • heywherelogingo 4y ago
      "Why pay the ops burden of Cockroach" - sounds like you haven't used it. Considerably easier than postgres.
    • jpdb 4y ago
      > a single Postgres master with replication/standby gets you very far So does a distributed database. A primary/replica has an ops burden too, and measuring that complexity is subjective. With primary/replica you still take downtime during failover, you need to make sure you're testing the failover, upgrades aren't zero downtime in a lot of cases, etc. You're essentially treating your database as a pet and a clustered option is much more cattle vs pet. This saves you time to develop other features (including working on reliability -- the most important feature)
• theptip 4y ago
  I’d go with this approach for my next startup, unless I had specific performance requirements. (Having built a company to Series B on the standard Django/Celery/Redis stack). Very hard to overstate the benefit of having all data in one DB that your developers can trivially run, mutate, and step-debug in your application.
  • edgyquant 4y ago
    very hard to undersell just how bad this is. Now every action on your application it running database queries.
    • theptip 4y ago
      Which is totally fine for most early-stage applications.
      • edgyquant 4y ago
        It is, until it isn’t I suppose. I’ve worked at places where the ORM (or equivalent) was nicely utilized and we were able to cache things via redis as we got bogged down then migrate to a multi-layer structure with caching based on date + popularity across a DB cluster, nodql store and browser store (this was really nice.) But: I’ve also had to deal with “SPAs” that evolved a few dozen independent routes a massive backend all coupled to the database with no ORM/class layer. This was not so nice so refactor when it came time to scale
• sammy2255 4y ago
  A blog called “amazingcto” that isn’t loaded over HTTPS..
  • xrayarx 4y ago
    It loads via https for me
  • jmartrican 4y ago
    Oh wow. It shouldn't even work on http... it should redirect to https. If you calling yourself amazing CTO.
• nix23 4y ago
  I correct it....just use the filesystem for everything.
• madjam002 4y ago
  I need to store ~50 million records with around 40 columns of strings, integers, decimals, various data. Only needs to be indexed by two string columns, but every day I want to "upsert" ~10 million records with data that has potentially changed, plus update a column that represents the date that the row was last updated on every row. With Postgres it seems to be inefficient at storing rows with large amounts of columns, and "upserting" data (using UPDATEs or INSERT ON CONFLICT) results in huge amounts of disk writes, I think because Postgres writes the entire row even if a single column has been updated. I could store some of the regularly updated columns in a separate table, but I'd thought I'd ask, is there a database out there that is more suited for this type of workload? Because I feel like "just use postgres for everything" is not the answer here.
  • Nican 4y ago
    I was going to answer that the answer is column families [1], but I just realized that is a CockroachDB-specific feature. [1] https://www.cockroachlabs.com/docs/stable/column-families.html https://www.cockroachlabs.com/docs/stable/column-families.ht...
  • deleted 4y ago
    [deleted]
  • inkyoto 4y ago
    It sounds like you are dealing with a denormalised data model. Relational databases do not handle the denormalised data models well, and 50 million records is a small dataset. Consider remodelling the data model and going up to 3NF [0], or to a higher normal form if that is more appropriate for your use case. Throwing different relational database engine types at the problem might help with bandaiding the problem for a while, but one day it will come back and bite you again. Wikipedia has a good starting point [1] for the 1NF ↝ 2NF ↝ 3NF ↝ … design steps. Depending on the nature of your data, you might have to consider the Boyce–Codd normal form (3NF++ so to speak). Maybe not. Once you have remodelled the data model, selects and upserts will be very efficient, quick, and they won't result in the constant index update/rebuild for the entire single main table. You will have to rewrite your queries, but if the data model is normalised, joins will be straightforward, fun to write and will only update what has actually has changed in each, separate table. If the data model normalisation is absolutely impossible (e.g. you don't own the data model but rather your customer does), consider replacing upserts with the MERGE statement. [0] https://en.wikipedia.org/wiki/Third_normal_form https://en.wikipedia.org/wiki/Third_normal_form [1] https://en.wikipedia.org/wiki/Database_normalization#Example_of_a_step_by_step_normalization https://en.wikipedia.org/wiki/Database_normalization#Example...
• fidrelity 4y ago
  I love postgres, keen to recommend it and did use a couple of the suggested tips in my last startup. However: - After just the first couple of months we had to replace full text search with Elastic because it just wasn't up to the task fully. - We did introduce redis for cashing which was maybe 2 hours of setup overhead (using Rails, deploying on Render for $5/month). YMMV.
• huntertwo 4y ago
  Hot take but I prefer dynamo for storing persistent state. Much easier to scale.
  • ttfkam 4y ago
    Apples and oranges. DynamoDB is wicked fast for what it does. If you know your use cases ahead of time, you can figure out a data storage strategy where queries and secondary indexes can work wonderfully. $deity help you when the use cases change though. Rearchitect->dump->transform->reload. I really like DynamoDB. Its functional overlap with a fully-featured relational database is very narrow though. No graph queries or joins or advanced data validation or...basically anything on this list: https://www.sql-workbench.eu/dbms_comparison.html https://www.sql-workbench.eu/dbms_comparison.html Postgres can be made to work like DynamoDB: simple queries with no joins, table partitioning and tablespaces for I/O parallelization, etc. DynamoDB barely scratches the surface the other way around. PartiQL is the limit and it's a very basic limit.
• CodeWriter23 4y ago
  I'll tell you what Postgres, Kafka, Cockroach and Mongo DON'T do for your startup. Make sales.
• Lio 4y ago
  He forgot one, you can also use Postgres for ML tasks too: https://postgresml.org/ https://postgresml.org/
• kolar 4y ago
  Be careful when using Postgres as queue with priorites, especially if your messages are large. Removing rows doesn't free up the memory in Postgres and vacuum could remove rows only from the end of a 'page'. As a result of that queue will require enormous amount of space to work. The only way to free up space is to use VACUUM FULL which rewrites whole database and will lock queue for long time. I've had a lot of headaches when we've tried 'use Postgres for everything' on production :)
  • fuy 4y ago
    there's also things like pg_repack which don't lock the tables (but will incur high IO while freeing up the space).
  • kolar 4y ago
    Also Postgres is too slow for large analytical databases. You need columnar database to make fast queries on >1Tb of data.
    • bfelbo 4y ago
      You could use TimescaleDB which is a Postgres extension that adds support for columnar tables and time-based chunking. Works brilliantly IMO.
    • ithkuil 4y ago
      Does anyone have experience with some postgres columnar store extension like https://github.com/citusdata/cstore_fdw https://github.com/citusdata/cstore_fdw ?
      • ttfkam 4y ago
        AWS Redshift works wonderfully in that capacity.
      • whoopdeepoo 4y ago
        My experience was not enough support for common postgres features
• zkirill 4y ago
  I think that the job of a CTO is to minimize the surface area of "built in-house" so that your team can focus on things for which your customers actually pay you. Sure, PostgreSQL can be used for sessions, but Redis has solved this problem on virtually every single platform a long time ago. Your customer probably doesn't even know what a session is, but they will definitely learn about them when your in-house implementation inevitably encounters an edge case. Of course, PostreSQL has wonderful search capabilities but it was never intended to be used as a search engine. Solr was created in 2004 and is still being improved and used in production every day. Do you know what will happen to your full-text search when your customer types in a mix of English and Chinese characters? Yes, PostgreSQL addition of SKIP LOCKED was neat but AFAIK the author of that feature himself recommended using a traditional job queue unless you had a very good reason against it. RabbitMQ was designed as a message queue and when you read the documentation you realize that they had encountered virtually every single problem and figured out a way to deal with it so that you don't have to. The documentation pretty much tells you what problems you are going to have later so that you can plan for them today. Choose boring technology (tm), follow industry best practices, and enable your team to get their work done using the right tools for the job, so they can deliver the product to the customer, and leave the office on time.
  • jmull 4y ago
    Redis doesn’t “solve” sessions.
  • danielvaughn 4y ago
    I’m not sure I follow. Postgres has been around forever, it’s basically what I would call boring technology. Sure they’re still innovating, but the core tech is mature. An example of a “not boring” database would be like Cockroach or Planetscale.
  • Delemono 4y ago
    Choose PostgreSQL as long as possible. This is probably good enough for 99% of things out there running stuff. For everything else you have architects which will do the right thing for you.
  • 4y ago
• pkstn 4y ago
  I just use MongoDB for everything. It's easy to scale both vertically but also horizontally. Also enhances availability and avoids data loss to have replica set with at least three MongoDB instances geographically distributed. MongoDB also keeps most frequent queries in memory which makes it super fast. ObjectID UUIDs are generated client-side making it fast writing. GridFS makes storing files easy, fast and robust etc...
  • edgyquant 4y ago
    I tend to use redis but in my experience Mongo works for a similar purpose (plus has querying which is nice.). Use Mongo/Redis as the working store (build data client side, sync to mongo/redis which then inserts into db when it gets the chance.)
• simonebrunozzi 4y ago
  Dear Stephan, in my experience (6 years as tech evangelist for AWS, met with thousands of startups and large companies in 6 continents - not to brag, just to give you context, AWS' success and relevance is only very slightly attributable to my tiny contribution), what you suggest might work if and only if the company is not really successful, and its IT needs do not change quickly over time. If instead we're talking about a fast growing startup, a single solution like Postgres (btw, I adore Postgres, to be clear), it's going to become an issue. You can dramatically improve your performance if you allow the use of Redis, as a start. And your title could have been "Just use Postgres + Redis for everything", and it would have been half as bad, or quite good. I think it's ok to try to keep things simple, but limiting yourself to JUST Postgres is not going to work.
  • je42 4y ago
    I have seen postgres queries being answerd in low ms range. Why would I want to use redis ? Only if my loadtest suggest that postgres is a bottleneck and redis is faster. Keep complexity down to min to make it functionally work. Loadtest and fix the bottlenecks. add complexity when you know the gain.
  • christophilus 4y ago
    > might work if and only if the company is not really successful What is your definition of success? I’ve seen Postgres scale to pretty large loads. I’ve also seen plenty of complex stacks where the scale never warranted the complexity. That said, Redis + Postgres is a pretty simple stack, so I’m not complaining about your suggestion. I’m more curious about specifics.
• bitwize 4y ago
  Be like a former boss of mine. Replace your entire backend strategy with stored procedures. Fire your backend developers. Save cash. Get a promotion for saving the company millions a year.
• whoopdeepoo 4y ago
  We tried using postgres for a rewrite of an existing system but the lack of support for columnar data made it not possible.
  • ttfkam 4y ago
    AWS Redshift. Same Postgres application drivers with a columnar/analytic focus.
• JacksonGariety 4y ago
  The graphic is misleading: - You can easily do prerendering and asset packing in the frontend build step - You don't need MVC on the backend - You can use a third party for authorization - Frontend can just be a React app - VirtualDOM doesn't add any complexity in addition to the frontend View - You don't need MVC on the frontend - Frontend can fetch data from the API and cache it with cache-control header The simplest setup is actually to host a React app from an S3 bucket and use Lambda / Cloud Functions to respond to network requests.
  • adamckay 4y ago
    > The simplest setup is actually to host a React app from an S3 bucket and use Lambda / Cloud Functions to respond to network requests. You'd have to pry my monolith-on-a-vps out of my cold dead hands before you can call that the simplest setup.
• rubenfiszel 4y ago
  Our open-source project [1] was built with nothing but Rust + Postgres on the backend side of things and it allows us to iterate quickly and have very simple deployments for self-hosting. The queue which is the core bottleneck of our stack will eventually get replaced by redis but for a scale of less than 10k/s jobs, it scale incredibly well. [1]: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill
• k__ 4y ago
  I don't know if I wanted to iterate with a datastore that required me to define a schema.
  • necovek 4y ago
    You might want to look into tools for "evolutionary databases", or "database migration" tools as they are better known today. Btw, your comment strikes me like saying "I don't want to iterate with a strongly typed language".
    • k__ 4y ago
      I think, strongly typed is okay. Same goes for statically typed. Just nominally typed isn't my thing.
      • necovek 4y ago
        Proper databases like Postgres are pretty strongly and statically typed. Eg. try inserting a string into an int column and watch it complain. It even supports custom types out of the box, including enums.
• iends 4y ago
  My company has a similar philosophy. “Just use DynamoDB for everything.” Seemingly they don’t care about costs, but otherwise it is mostly working.
• tuukkah 4y ago
  In my next backend, I want to use serverless functions (Cloudflare), serverless Postgres (Neon), and nothing more.
  • nikita 4y ago
    We will certainly make it possible. There is something to also consider: software development lifecycle and native support for previews, schema PRs, and deploys. Lots to build to give lots of confidence to a developer that a PR won't break and won't regress performance. We will make Vercel + Neon to support this extremely well.
• sjducb 4y ago
  I used to think this, but now that I know a bunch of these cloud tools; I think these frameworks standardise things between companies. Kafka is well documented and understood. Your custom postgres solution is not. Best case you write high quality maintainable code and have a single dependency. Worst case is you create a complete mess that takes months to onboard people. Right now I'm contracting on a project that uses AWS BATCH, docker, kubernetes, terraform, Nextflow, Django and postgres. It took me 2 days to on-board myself and start delivering features because everything was standard.
  • chucke 4y ago
    I've worked for a while now in the field and I never worked with kafka. Nor most of the peers I've worked with. And I wouldn't even call it the most used message queue or data stream solution, so I don't understand why people treat it as ubiquitous software. SQL, on the other hand, is. And while it's certainly not built for message queues, that's not the point of the article.
    • sjducb 4y ago
      That's not my argument. I'm saying it'll be easier to onboard developers onto a standard message queue (like Kafka) than to onboard developers onto some custom thing you wrote. The original article is rebranding "Not invented here" as "Minimise dependencies"
      • chucke 4y ago
        But kafka isn't a standard message queue, and therefore won't make onboarding easier except for the ones who already know it, which is not a significant majority, right? Or at least we'll have to agree on the definition of standard here. For the equivalents of SQL, I'd say we have AMQP or MQTT, and kafka does not seem to support any (my goggling tells me it has its own protocol); deployment-wise, it doesn't seem easier to set up than, let's say, SQS; while used and supported in most common languages, most queue/job frameworks do not seem to support it OOTB (looked into ruby, python, php, javascript), so you'd have to write your own custom kafka handling code from scratch. So what would make it easier than just riding postgres?
• projectileboy 4y ago
  Context matters. If you’re going to be scaling huge, this advice is silly. If you’re just starting, or planning to rewrite later, or for good reasons never expect huge scale, staying as simple as possible makes sense. Also, tons of devs build to scale huge for systems that will never, ever scale huge. I have two competing values in this space: someone else has solved problem x better than you will, and you will live to regret every dependency you introduce.
• ebfe1 4y ago
  Add ClickHouse for analytic, columnar use case and you have everything.... Oh wait, Cloudflare already pointed that out XD: https://about.gitlab.com/blog/2022/04/29/two-sizes-fit-most-postgresql-and-clickhouse/ https://about.gitlab.com/blog/2022/04/29/two-sizes-fit-most-...
• didip 4y ago
  Yup, I like the general idea of using PG for everything. The author forgot to mention LISTEN/NOTIFY feature in PG. Perfect for low volume pubsub.
• gregopet 4y ago
  Oh god I am just looking at a piece of software written by some really untalented folk (who are no longer with the client's company) and it's an unholy mess of caches, Kafkas, Elastics.. because they prematurely decided that Postgres just won't cut it as they have almost a 100k items to keep track of! And since that mess of random arhitecture started generating millions of Kafka events their last act of engineering was to add kubernetes to the mix, because hey, more instances will solve the problem, right? I'm begging them to let me rewrite the whole thing just in Postgres, I think I'd rather quit than fix the original.
  • breckenedge 4y ago
    Résumé driven development at its finest
• willcodeforfoo 4y ago
  I love Postgres as much as the next dev, but why are COUNTs so slow?
  • pritambaral 4y ago
    Because of how MVCC is implemented in Postgres. However, it's not terribly hard to add efficient count tracking that works with this implementation of MVCC, using row triggers. I use it often in production. This might change in the future with the zHeap storage backend.