3 ms·

I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distrib

by ansible 11y ago

I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...

LoSboccacc 11y ago
but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space. I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.
- manigandham 11y ago
  I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.
  - LoSboccacc 11y ago
    > I have a strong sense that you're vastly over-engineering something. and you're 100% right it wouldn't be interesting otherwise :D (I'm not gonna use this on real work no worries)

3 ms·

I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distrib

by ansible 11y ago

LoSboccacc 11y ago
but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space. I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.
- manigandham 11y ago
  I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.
  - LoSboccacc 11y ago
    > I have a strong sense that you're vastly over-engineering something. and you're 100% right it wouldn't be interesting otherwise :D (I'm not gonna use this on real work no worries)