4 ms·
I'm very excited. I want to use this to version ConceptNet's raw input and its built data, all of which is public. So I can assume this isn't going to be afrai
by rspeer 9y ago
I'm very excited. I want to use this to version ConceptNet's raw input and its built data, all of which is public.
So I can assume this isn't going to be afraid of gigabytes, right? I've seen services before that want to be a repository of data, and I try to upload a mere 20 GB of data and they're like "oh shit nevermind". Even S3 requires it to be broken into files of less than 5 GB for some inscrutable reason.
- akarve 9y agoGigabytes should be no problem. We have yet to implement multi-part uploads though, so if your network gets interrupted you will need to re-upload the package from scratch. By design when the upload is in progress your client is talking directly to scalable blob storage. If you run into any issues, please email me: aneesh at quiltdata dot io. We also want to add server-side filtering for giant packages (where `quilt install` doesn't make sense). I can add you to our Slack channel if you'd like to discuss further.
- IanCal 9y agoI don't think you have had to break files up yourself for a long time on S3. You can treat files up to 5TB as a single object. I think you have to do a multipart upload but that's probably not a bad idea anyway.
- rspeer 9y agoAh, right. I'm remembering from when I was trying to use git-annex to version the data, which was a problem for multiple reasons, including that their S3 driver didn't use multipart uploads.