6 ms·
Its weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging
by hoodoof 10y ago
Its weird how S3 seems to be the unwanted stepchild of AWS.
So many obvious innovations just aren't turning up.
For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind.
In fact it's strange that there is NO useful filtering at all apart from the very useful folder/hierarchy/prefix filtering. But apart from that you can't do wildcard searches or filters or date filters or tag filters.
I'm building an application right now that needs to get a list of all the jpg files - the only way to do that is get every single object in the bucket and manually filter out the unwanted ones - feels like its 1988 again.
It seems like it would also be valuable for there to be alternate interfaces to S3 such as the ability to send data via ftp or SMTP or sftp or whatever, but there are no such interfaces.
Hopefully Google will goad AWS into action on S3 innovation by implementing such features.
- kot-behemoth 10y agoWhile great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).
- hoodoof 10y agoPerhaps true. I'll never do that though because I'd have to use DynamoDB, which is a technology that is high on my list of "technologies that I am least enthused about". Also, I really shouldn't have to go to all the work of creating and maintaining a metadata database and implementing a query API just because I want to do searches more powerful than "list all objects" - that's Amazon's job.
- maxaf 10y agoMoreover, even if you had gone to the trouble of building such an API, S3 still doesn't offer bulk operations, so you'd have to operate on each matching object... one object at a time.
- hoodoof 10y agoThis isn't such an issue because you can update the DynamoDB index using an AWS Lambda function on every putobject or removeobject event. It's still not something I want to do, mainly because I'd have to touch DynamoDB but secondly because, well, why the heck doesn't AWS do it?
- LightskinKanye 10y agoWhat do you have against DynamoDB?
- xtracto 10y agoThis, we love S3. What we did is add a SQL tier for some of the data we are storing there in case we want to do some more structured operations.
- hoodoof 10y agoYes, but are you sure your database matches the underlying data store? The real problem with building a metadata index outside is that you then have the synchronization validation - yuk.
- illumin8 10y agoYou can always do a full scan of your S3 namespace every week or so and synchronize the index. This gives your consumers low latency access to the object store, as index lookups are extremely fast, it minimizes the cost of lookup events on S3.
- hoodoof 10y agoSo my database is up to a week wrong? Errr.....
- posterboy 10y agoin that it stores undeleted files, until weekly clean-up the DB is only incomplete for as long it takes to commit to the SQL layer after storing successfully.
- otterley 10y agoThis was just introduced last November: http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inven...
- edblarney 10y agoSurely some extra functionality would not obfuscate the inherent simplicity of S3. A an S3Query module would not, I think, make things harder for S3 users. And frankly - it would be awesome. I used s3 a lot, and loathe to switch to a DB if I can avoid it. Some querying and indexing features I think would be taken up by a large number of devs.
- wellsjohnston 10y agoI've had the same frustration with S3, and the reasons you went over are what drove me to create this. It seems like they made S3 (10 years ago now?) and just forgot about it. There is no way to sort/filter or even list all the objects in a bucket without writing a recursive algorithm using one of their SDKs.
- lobster_johnson 10y agoS3's API is so rudimentary that I prefer to think of it as a non-enumerable key/value store. I learned this the hard way: We had an application where made the mistake of storing about a billion files in a nearly flat structure — one level of nesting, probably 100m "folders" in the root. Then one day we needed to go through it to prune stuff that was no longer in use. Unfortunately, if you don't have a "shardable" prefix, list requests are impossible to parallelize efficiently (because you can't subdivide the work), and our scripts took weeks to run to completion. Hard-earned experience: If you're storing large quantities of stuff in S3, always pick a shardable prefix. The upload date is a good choice. A random string will also do. After this, my solution for any non-trivially-sized storage use case is to store an inventory of objects separately in a performant PostgreSQL database, and make sure all writes go through a service layer that shields the consumer from the details of S3. This has some benefits over a hypothetical centralized approach (but some downsides, like the possibility that things get out of sync if you sidestep the inventory). Overall, I wish S3 would store its metadata in something like BigQuery. Anyone know if Google Cloud Platform's S3 equivalent, Cloud Storage, improves on these issues?
- jessaustin 10y agoI wonder if "bucket notifications" are reliable enough that one could keep such an index DB populated automatically?
- lobster_johnson 10y agoI think SQS would be reliable enough here, yes.
- nolite 10y agoYes, just hook those up to a lambda function and write to dynamodb or something
- otterley 10y agoI tried this, but if you want to query by tags, using an RDS database works much better. DynamoDB is not well suited to this particular problem.
- illumin8 10y agoFirst, I wanted to say, you bring up some very good points. S3 wasn't really designed to be a searchable key/value store, as you have to pay for lookups, and pagination kills your ability to effectively search anything greater than a few thousand objects in a hierarchy, within a reasonable amount of time. There are, however, ways to solve this: you could fire a Lambda function whenever an object is put into your S3 bucket that simply adds a single row to a DynamoDB table with the object name, along with any additional metadata you might like to capture to assure data provenance. Then, to search, you can simply query the DynamoDB table. As always, there are many basic building blocks at AWS, but you have to connect them together (like legos) before they become useful for most applications.
- hoodoof 10y agoAs mentioned elsewhere in this thread, an external metadata database of S3 object immediately introduces synching and validity issues. DOS is smarter than S3.
- philliphaydon 10y agoI believe it works fine.
- noobiemcfoob 10y agoDOS is 32 bit...
- tjholowaychuk 10y agoI just index S3 with SQL, that combination is plenty powerful, not a huge worry, but it might be interesting to see more native support for that kind of thing.