4 ms·

I think there is too much emphasis on 'the right way' sometimes, which leads people to skip over an analysis for the problem they are solving right now. For ex

I think there is too much emphasis on 'the right way' sometimes, which leads people to skip over an analysis for the problem they are solving right now. For example, if you are building something that will have thousands or millions of records, and storing small binaries in postgresql lets you avoid integrating with S3 at all, then you should seriously consider doing it. The simplification gain almost certainly pays for any 'badness' and then some. If you are working on something that you hope to scale to millions of users then you should just bite the bullet and integrate with S3 or something, because you will use too much db capacity to store binaries (assuming they aren't like 50 bytes each or something remarkably small) and that will force you to scale-out the database far before you would otherwise be forced to do so.

crooked-v 3y ago
> and that will force you to scale-out the database far before you would otherwise be forced to do so Disk space is cheap, even when it's on a DB server and not S3. Why worry that much about it?
- tomnipotent 3y ago
  > Disk space is cheap Disk I/O less so. An average RDBMS writing a 10MB BLOB is actually writing at minimum 20-30MB to disk - once to journal/WAL, once to table storage, and for updates/deletes a third copy in the redo/undo log. You also get a less efficient buffer manager with a higher eviction rate, which can further exasperate disk I/O.
  - crooked-v 3y ago
    But if the files are important enough that you need transactional updates or point-in-time recovery for them, you're stuck with the equivalents of those things anyway, wherever those files end up, plus the extra overhead of coordinating two systems.
    - ltbarcly3 3y ago
      But there is a hard limit to db scale up and you don't want to hit it. Also, who said you need transactional updates or pit recovery in all cases?
- deleted 3y ago
  [deleted]
sorintm 3y ago
Brilliant point! It always depends on the problem you're trying to solve. What Xata did, was integrate files as a database column, BUT stored the binary content in S3. Thus offering both transactional file management and point in time recovery as well as high performance direct (S3) file access, through a CDN. Basically Xata implements the orchestration and takes away the pain of managing 2 services.
- ltbarcly3 3y ago
  The last thing I would do is use a proprietary solution from a startup to store data. Sorry, but that is just absolutely insane.
harshitaneja 3y ago
Going on a tangent here but why s3? Why not avoid vendor lock in and just serve static files? My setup is to post blobs over a small rest api which balances storage load over all the file servers using nginx. It responds with the serverSubdomainId which in turn gets saved in the database. So the url for file access can be generated https://{serverSubdomainId}.domain.tld/{resourceId} https://{serverSubdomainId}.domain.tld/{resourceId} The only costly part is writing which with caching of available storage of all the servers isn’t bad. The whole system is horizontally and vertically scalable. What am I missing that s3 is still the defacto even with vendor lock in and high egress costs?
- aforwardslash 3y ago
  There are plenty of oss solutions that talk "s3", like swift, ceph and seaweedfs. Why object storage? Its easier to backup, version and to scale horizontally. Most solutions will also provide encryption capabilities that can be commanded either via external key management systems or from the client application. Also, custom access policies are great for private documents and file uploads. Using static files is a good solution in some cases, nothing against it. But in many scenarios, there are huge benefits on using object storage instead, even if it is quite slower.
- timando 3y ago
  "S3" is often used to refer to any S3 compatible object storage system. You can even self-host it with something like MinIO.
- connordoner 3y ago
  I can't speak for everyone but, in my case, it comes down to the ease of setup and having someone else manage it.

crooked-v 3y ago

> and that will force you to scale-out the database far before you would otherwise be forced to do so Disk space is cheap, even when it's on a DB server and not S3. Why worry that much about it?

tomnipotent 3y ago
> Disk space is cheap Disk I/O less so. An average RDBMS writing a 10MB BLOB is actually writing at minimum 20-30MB to disk - once to journal/WAL, once to table storage, and for updates/deletes a third copy in the redo/undo log. You also get a less efficient buffer manager with a higher eviction rate, which can further exasperate disk I/O.
- crooked-v 3y ago
  But if the files are important enough that you need transactional updates or point-in-time recovery for them, you're stuck with the equivalents of those things anyway, wherever those files end up, plus the extra overhead of coordinating two systems.
  - ltbarcly3 3y ago
    But there is a hard limit to db scale up and you don't want to hit it. Also, who said you need transactional updates or pit recovery in all cases?
deleted 3y ago
[deleted]

sorintm 3y ago

Brilliant point! It always depends on the problem you're trying to solve. What Xata did, was integrate files as a database column, BUT stored the binary content in S3. Thus offering both transactional file management and point in time recovery as well as high performance direct (S3) file access, through a CDN. Basically Xata implements the orchestration and takes away the pain of managing 2 services.

ltbarcly3 3y ago
The last thing I would do is use a proprietary solution from a startup to store data. Sorry, but that is just absolutely insane.

harshitaneja 3y ago

Going on a tangent here but why s3? Why not avoid vendor lock in and just serve static files? My setup is to post blobs over a small rest api which balances storage load over all the file servers using nginx. It responds with the serverSubdomainId which in turn gets saved in the database. So the url for file access can be generated https://{serverSubdomainId}.domain.tld/{resourceId} https://{serverSubdomainId}.domain.tld/{resourceId} The only costly part is writing which with caching of available storage of all the servers isn’t bad. The whole system is horizontally and vertically scalable. What am I missing that s3 is still the defacto even with vendor lock in and high egress costs?

aforwardslash 3y ago
There are plenty of oss solutions that talk "s3", like swift, ceph and seaweedfs. Why object storage? Its easier to backup, version and to scale horizontally. Most solutions will also provide encryption capabilities that can be commanded either via external key management systems or from the client application. Also, custom access policies are great for private documents and file uploads. Using static files is a good solution in some cases, nothing against it. But in many scenarios, there are huge benefits on using object storage instead, even if it is quite slower.
timando 3y ago
"S3" is often used to refer to any S3 compatible object storage system. You can even self-host it with something like MinIO.
connordoner 3y ago
I can't speak for everyone but, in my case, it comes down to the ease of setup and having someone else manage it.