3 ms·
Ask HN: Does anybody store files in S3 for their SaaS?
Is anybody storing non-text files in S3 for their software? I'm referring to pdf, video, audio, images, etc.
If so, it would mean the world if I could talk to you and just ask about some of your experience with uploading/retrieving these files within your software.
To give context: I have a free managed API that offers full-text-search for your S3 bucket to integrate into your software. The website is https://mixpeek.com
Let me know if this matches you and if you're willing to answer some quick Qs in exchange to be involved in the free pilot :)
- rshnotsecure 5y agoWhat you describe is part of the AWS "Well Architected Framework". Static files should, as much as possible, be hosted on S3. This takes the pressure off the web/app server, is cheaper, increases resiliency to DDOS attacks, and allows much faster loading, particularly of video files, when one integrates the S3 storage with a CDN (such as CloudFront). What I would find interesting is anyone running a SaaS project and NOT using S3/object storage for a significant amount of their files.
- Beefin 5y agoI think you misunderstand. The non-text files remain in S3, my service just parses them for text, then stores it in a Lucene index to become searchable.
- Jugurtha 5y agoI think there's a typo in the code example for the Python client that would result in a NameError if run from mixpeek import mixpeek Should be from mixpeek import Mixpeek Given that it is the class instantiated below. One point in the Healthcare and Insurance use case reads as follows >Ensure the are up to date Insurance Documents Do you mean "Keep Insurance Documents up to date"? We use S3 for https://iko.ai https://iko.ai and give a filesystem like experience for notebook users. They can 'cd', 'ls', etc on S3 buckets right from a Jupyter notebook. We both integrate with external buckets, private or public, but also use an S3 compatible object storage (MinIO), for which I wrote this library: https://pypi.org/project/bmc/ https://pypi.org/project/bmc/ One thing I find will hinder your product's adoption is the "/upload" route. If I were to use your tool, I'd want to use it to search through already existing objects in my bucket, not to upload to the bucket. That was one objection we faced, and then we added the ability to mount S3 buckets because people had datasets of 1TB or so in existing buckets.
- Beefin 5y agoOh good catch ! Thanks for the feedback, you're not actually uploading the file. you're just sending it to be parsed and then the metadata is stored in a Lucene index.
- motyar 5y agoYes, we store the generated PNG images for CDN links at Bruzu.com
- Beefin 5y agoThis seems like an interesting use case, since Bruzu pictures are all text. How are users searching these today?