6 ms·

Just Use Postgres for Everything

zer00eyz 2y ago
I mostly agree with this... > Use Postgres for Fulltext Search instead of Elastic. Yea, that's not quite the same. Im going to make the argument that a well tuned search platform along side your data storage is probably one of the most useful things you can have. It's one of the first places where you will be "let down" by the Postgres solution. As a document store, a queue, Postgres will do the job up to a point. When and where it breaks down in one of these ancillary roles you're likely to have the resource to support more systems, or a more "focused" tool.
- tracker1 2y ago
  I think that's true of most of the examples. There are times you will want/need to break out. When you do, it should be apparent and compelling to do so. Whether it's Redis, RabbitMQ(or another queue option), or Elastic you should have a clue as to what and why you need to break out. Many/most applications don't have those needs. Then again, I'm also a fan of using JSONB for details in fewer tables over overly normalized structures. It varies.
deleted 2y ago
[deleted]
wavemode 2y ago
All of these points are valid, when given a caveat of "if Postgres covers your actual needs". Like, I've personally worked on many systems that used Redis for absolutely no reason. As it was not actually measurably improving the latency of any part of the system. It was basically just another place to store data. Obviously, Postgres and Redis are not the same, and there are situations where keeping data in memory at all times is important for latency. Hence the caveat. But most companies are not in any such situation. Same story with the rest of these technologies - people often reach for them without really considering whether they are any sort of improvement over just storing the data in a relational database. This is commonly done by engineers who underestimate the capabilities of a relational database.
- tracker1 2y ago
  For me, I tend to reach for Redis when I need/want caching or otherwise data that can expire or otherwise disappear. Once I have Redis, I may use it for other types of things as well as caching. I know that you can use persistence and other features for more use cases, which can work well. Of course, I'm more inclined today to just use PostgreSQL unless and until something truly needs to break the mold.
lucianbr 2y ago
As always, "it depends". There are situations where you can use Postgres as a queue perfectly fine. But there is no way Postgres can do as well or better than RabbitMQ in all situations. Same for text search, and so on.
- ejdhshsuwisjsh 2y ago
  Those articles are for the avg developer. Every expert / expert team should know what to use when.
- tracker1 2y ago
  Of course... that said, there are too many times where the additional complexity isn't necessary or worth it. I say this as someone who is usually somewhat early to introduce RabbitMQ or Redis into a new environment. Most applications, in most environments are not ever going to see more than 10k simultaneous users. You can vertically scale an RDBMS like PostgreSQL a LOT. Not to mention, there are also a lot of ways to approach similar techniques with PostgreSQL that you might reach for a different DB platform for. JSONB in Postgres is actually easier to reason with IMO than MongoDB as a platform. Depending on your needs, it's a much better fit with hybrid tables with additional details in JSONB.
lucianbr 2y ago
> Use Postgres for caching instead of Redis with UNLOGGED tables This is not the same thing, is it? Redis is in-memory, while the Postgres tables will be on disk, even UNLOGGED.
- donor20 2y ago
  I thought unlogged was used so crash recovery would truncate / dump the table on a crash with 100% Dataloss but this let you put stuff on a tempfs / ramdisk?
  - zactato 2y ago
    Isn't dataloss the main selling feature of redis /s
arp242 2y ago
Just use Postgres for everything - https://news.ycombinator.com/item?id=33934139 https://news.ycombinator.com/item?id=33934139 - Dec 2022 (431 comments)
webprofusion 2y ago
Sure, but I need it embedded (so my app controls the installed version it needs).
- rad_gruchalski 2y ago
  There you go: https://github.com/snaplet/postgres-wasm https://github.com/snaplet/postgres-wasm. It should be possible to run it on the server side.
cultofmetatron 2y ago
A lot of points made here is also is why I believe Phoenix framework is the BEST stack for building mvps in 2024. 1. you get concurrency and pubsub built in with no external dependancies. 2. postgres suppport out of the box. our entire deployment is just phoenix + postgres 3. creating a microservice is easy. create a genserver, add it to your application.ex -> DONE. no need to create a seperate repo. its already able to send and receieve messages from other parts of your monolith using the built in pubsub system 4. Oban, the background worker library uses postgres. end result is that as a small team, we are able to knock out a lot of features and scale it without any of the complexity most startups have. Its been 4 years and only now we are starting to consider bringing in an external message queue.
- Antitoxic6185 2y ago
  I feel like it is significantly harder to learn elixir and phoenix as a newbie. The learning curve is very steep owing to the fact that good learning resources seems to be scarce, maybe not for elixir but definitely for phoenix. I have been trying to grasp it for a month now, but an up-to-date resource is nowhere to be found and oh, please don't tell me to go read the documentation. It is not meant to be read by beginners. Almost everyone in the community recommends the prag studio course but $200?? I can't afford it XD
  - arrowsmith 2y ago
    You could try my new LiveView course: http://phoenixliveview.com/ http://phoenixliveview.com/ If you have a Rails background then I have another course specifically for you: https://phoenixonrails.com/ https://phoenixonrails.com/ Or click here to buy both together: https://arrowsmithlabs.com/checkout?bundle_id=1 https://arrowsmithlabs.com/checkout?bundle_id=1 ;) </selfpromo>
  - cultofmetatron 2y ago
    >The learning curve is very steep owing to the fact that good learning resources seems to be scarce thats a fair assessment and I sympathize. I'm at the point where being able to hire devs is a concern of mine. we just hired a new elixir dev to join my team and it was difficult finding people with experience. We had to go though a recruiter who specializes in elixir. That said, I encourage you to not give up. I became an elixir dev after 9 years of working with javascript and nodejs. Personally I found it pretty easy to learn (was building apps in it as quickly as I could in nodejs after jsut 2 weeks). Its a magnitude order easier to learn than rust (though thats admittedly a low bar). > I have been trying to grasp it for a month now, but an up-to-date resource is nowhere to be found This is definitly an issue I admit. once you know elixir, you know the idioms and picking up the newest changes is pretty easy but I remember how it was to be a begginer. would love your input on what would be needed for a learning resource as I have been thinking about writing a book on phoenix for newbies. As far as getting into elixir, if you have experience with javascript, I reccomend checking out immutable-js. the data structures are basicly elixir's data structures but in javascript. my experience ins scheme and functional style js were a huge help for me when I was learning elixir.
dhbradshaw 2y ago
I've found this bias to be useful with two main exceptions: 1. It's nice if you can avoid using Postgres to store large files. 2. It's nice if you can avoid using Postgres for an event stream with massive volume (say billions of records per month that must be maintained and queried). Postgres will work for either of these, but there are real advantages that come if you can keep your database lighter than these allow for.
- TexanFeller 2y ago
  An event stream / message queue implemented in PG can actually be fine at some billions of events per month and has some major advantages over standard queues. I just wouldn't get too high into the billions.
anacrolix 2y ago
Can you use a ton of extensions with a managed Postgres, or should you just host it yourself?
up2isomorphism 2y ago
“AmazingCTO” and “just using xxx for everything” surely fit each other .
jdboyd 2y ago
My leading use for redis on a system where I am already using Postgres is to use Celery. I suspect this is just one of many cases where people add an additional datastore because their frameworks and/or libraries make it easier to to that than to use Postgres for that type of data.

I mostly agree with this... > Use Postgres for Fulltext Search instead of Elastic. Yea, that's not quite the same. Im going to make the argument that a well tuned search platform along side your data storage is probably one of the most useful things you can have. It's one of the first places where you will be "let down" by the Postgres solution. As a document store, a queue, Postgres will do the job up to a point. When and where it breaks down in one of these ancillary roles you're likely to have the resource to support more systems, or a more "focused" tool.

tracker1 2y ago
I think that's true of most of the examples. There are times you will want/need to break out. When you do, it should be apparent and compelling to do so. Whether it's Redis, RabbitMQ(or another queue option), or Elastic you should have a clue as to what and why you need to break out. Many/most applications don't have those needs. Then again, I'm also a fan of using JSONB for details in fewer tables over overly normalized structures. It varies.

deleted 2y ago

[deleted]

wavemode 2y ago

All of these points are valid, when given a caveat of "if Postgres covers your actual needs". Like, I've personally worked on many systems that used Redis for absolutely no reason. As it was not actually measurably improving the latency of any part of the system. It was basically just another place to store data. Obviously, Postgres and Redis are not the same, and there are situations where keeping data in memory at all times is important for latency. Hence the caveat. But most companies are not in any such situation. Same story with the rest of these technologies - people often reach for them without really considering whether they are any sort of improvement over just storing the data in a relational database. This is commonly done by engineers who underestimate the capabilities of a relational database.

tracker1 2y ago
For me, I tend to reach for Redis when I need/want caching or otherwise data that can expire or otherwise disappear. Once I have Redis, I may use it for other types of things as well as caching. I know that you can use persistence and other features for more use cases, which can work well. Of course, I'm more inclined today to just use PostgreSQL unless and until something truly needs to break the mold.

lucianbr 2y ago

As always, "it depends". There are situations where you can use Postgres as a queue perfectly fine. But there is no way Postgres can do as well or better than RabbitMQ in all situations. Same for text search, and so on.

ejdhshsuwisjsh 2y ago
Those articles are for the avg developer. Every expert / expert team should know what to use when.
tracker1 2y ago
Of course... that said, there are too many times where the additional complexity isn't necessary or worth it. I say this as someone who is usually somewhat early to introduce RabbitMQ or Redis into a new environment. Most applications, in most environments are not ever going to see more than 10k simultaneous users. You can vertically scale an RDBMS like PostgreSQL a LOT. Not to mention, there are also a lot of ways to approach similar techniques with PostgreSQL that you might reach for a different DB platform for. JSONB in Postgres is actually easier to reason with IMO than MongoDB as a platform. Depending on your needs, it's a much better fit with hybrid tables with additional details in JSONB.

lucianbr 2y ago

> Use Postgres for caching instead of Redis with UNLOGGED tables This is not the same thing, is it? Redis is in-memory, while the Postgres tables will be on disk, even UNLOGGED.

donor20 2y ago
I thought unlogged was used so crash recovery would truncate / dump the table on a crash with 100% Dataloss but this let you put stuff on a tempfs / ramdisk?
- zactato 2y ago
  Isn't dataloss the main selling feature of redis /s

arp242 2y ago

Just use Postgres for everything - https://news.ycombinator.com/item?id=33934139 https://news.ycombinator.com/item?id=33934139 - Dec 2022 (431 comments)

webprofusion 2y ago

Sure, but I need it embedded (so my app controls the installed version it needs).

rad_gruchalski 2y ago
There you go: https://github.com/snaplet/postgres-wasm https://github.com/snaplet/postgres-wasm. It should be possible to run it on the server side.

cultofmetatron 2y ago

A lot of points made here is also is why I believe Phoenix framework is the BEST stack for building mvps in 2024. 1. you get concurrency and pubsub built in with no external dependancies. 2. postgres suppport out of the box. our entire deployment is just phoenix + postgres 3. creating a microservice is easy. create a genserver, add it to your application.ex -> DONE. no need to create a seperate repo. its already able to send and receieve messages from other parts of your monolith using the built in pubsub system 4. Oban, the background worker library uses postgres. end result is that as a small team, we are able to knock out a lot of features and scale it without any of the complexity most startups have. Its been 4 years and only now we are starting to consider bringing in an external message queue.

Antitoxic6185 2y ago
I feel like it is significantly harder to learn elixir and phoenix as a newbie. The learning curve is very steep owing to the fact that good learning resources seems to be scarce, maybe not for elixir but definitely for phoenix. I have been trying to grasp it for a month now, but an up-to-date resource is nowhere to be found and oh, please don't tell me to go read the documentation. It is not meant to be read by beginners. Almost everyone in the community recommends the prag studio course but $200?? I can't afford it XD
- arrowsmith 2y ago
  You could try my new LiveView course: http://phoenixliveview.com/ http://phoenixliveview.com/ If you have a Rails background then I have another course specifically for you: https://phoenixonrails.com/ https://phoenixonrails.com/ Or click here to buy both together: https://arrowsmithlabs.com/checkout?bundle_id=1 https://arrowsmithlabs.com/checkout?bundle_id=1 ;) </selfpromo>
- cultofmetatron 2y ago
  >The learning curve is very steep owing to the fact that good learning resources seems to be scarce thats a fair assessment and I sympathize. I'm at the point where being able to hire devs is a concern of mine. we just hired a new elixir dev to join my team and it was difficult finding people with experience. We had to go though a recruiter who specializes in elixir. That said, I encourage you to not give up. I became an elixir dev after 9 years of working with javascript and nodejs. Personally I found it pretty easy to learn (was building apps in it as quickly as I could in nodejs after jsut 2 weeks). Its a magnitude order easier to learn than rust (though thats admittedly a low bar). > I have been trying to grasp it for a month now, but an up-to-date resource is nowhere to be found This is definitly an issue I admit. once you know elixir, you know the idioms and picking up the newest changes is pretty easy but I remember how it was to be a begginer. would love your input on what would be needed for a learning resource as I have been thinking about writing a book on phoenix for newbies. As far as getting into elixir, if you have experience with javascript, I reccomend checking out immutable-js. the data structures are basicly elixir's data structures but in javascript. my experience ins scheme and functional style js were a huge help for me when I was learning elixir.

dhbradshaw 2y ago

I've found this bias to be useful with two main exceptions: 1. It's nice if you can avoid using Postgres to store large files. 2. It's nice if you can avoid using Postgres for an event stream with massive volume (say billions of records per month that must be maintained and queried). Postgres will work for either of these, but there are real advantages that come if you can keep your database lighter than these allow for.

TexanFeller 2y ago
An event stream / message queue implemented in PG can actually be fine at some billions of events per month and has some major advantages over standard queues. I just wouldn't get too high into the billions.

anacrolix 2y ago

Can you use a ton of extensions with a managed Postgres, or should you just host it yourself?

up2isomorphism 2y ago

“AmazingCTO” and “just using xxx for everything” surely fit each other .

jdboyd 2y ago

My leading use for redis on a system where I am already using Postgres is to use Celery. I suspect this is just one of many cases where people add an additional datastore because their frameworks and/or libraries make it easier to to that than to use Postgres for that type of data.