3 ms·

So, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified? Have you ever thought of using a pos

by dxxvi 1y ago

So, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified? Have you ever thought of using a postgresql db (also on aws) to store those files and use CDC to publish messages about those files to a kafka topic? In your original way, we need 3 aws services: s3, lambda and sqs. With this way, we need 2: postgresql and kafka. I'm not sure how well this method works though :-)

jrochkind1 1y ago
Like put the video blobs themselves in postgres data columns? Does putting very large (relative to what you normally put in postgres) files in pg work well? Genuine question, i do not know, I've been considering it too and hesitant about it.
- dxxvi 1y ago
  I already checked with AI before putting the comment :-) 1GB with the bytea data type (https://www.postgresql.org/docs/current/datatype-binary.html https://www.postgresql.org/docs/current/datatype-binary.html) and 4TB with the BLOB data type (https://wiki.postgresql.org/wiki/BinaryFilesInDB https://wiki.postgresql.org/wiki/BinaryFilesInDB).
  - jrochkind1 1y ago
    Have you done this? I can google or AI for the max size that postgres will allow, sure. I have googled in the past for whether this actually works well, and have gotten answers leaning towards most advice against it in real world scenarios. So if you have experience with this and it did work well, I'm curious to hear about it! That's why i asked about if it worked well, not about the maximum size postgres allowed in various data types. If you have no experience with it, but are just posting advice based on what AI tells you about max sizes of data allowed by pg that I can get from the same source too, then okay, fair enough, and certainly no need to give me any more of that!
- dxxvi 1y ago
  > I've been considering it too and hesitant about it Why hesitant? Just ask AI. It'll tell you how to do it and then you can experiment it yourself.
ravedave5 1y ago
I've dealt with images in a database and it was a disaster, the transfer times are garbage.

by dxxvi 1y ago

jrochkind1 1y ago

Like put the video blobs themselves in postgres data columns? Does putting very large (relative to what you normally put in postgres) files in pg work well? Genuine question, i do not know, I've been considering it too and hesitant about it.

dxxvi 1y ago
I already checked with AI before putting the comment :-) 1GB with the bytea data type (https://www.postgresql.org/docs/current/datatype-binary.html https://www.postgresql.org/docs/current/datatype-binary.html) and 4TB with the BLOB data type (https://wiki.postgresql.org/wiki/BinaryFilesInDB https://wiki.postgresql.org/wiki/BinaryFilesInDB).
- jrochkind1 1y ago
  Have you done this? I can google or AI for the max size that postgres will allow, sure. I have googled in the past for whether this actually works well, and have gotten answers leaning towards most advice against it in real world scenarios. So if you have experience with this and it did work well, I'm curious to hear about it! That's why i asked about if it worked well, not about the maximum size postgres allowed in various data types. If you have no experience with it, but are just posting advice based on what AI tells you about max sizes of data allowed by pg that I can get from the same source too, then okay, fair enough, and certainly no need to give me any more of that!
dxxvi 1y ago
> I've been considering it too and hesitant about it Why hesitant? Just ask AI. It'll tell you how to do it and then you can experiment it yourself.

ravedave5 1y ago

I've dealt with images in a database and it was a disaster, the transfer times are garbage.