3 ms·
Since we're talking object storage, a question for the collective brain: are there any object storage solutions, cloud or on-prem, which support any sort of "op
by twic 2y ago
Since we're talking object storage, a question for the collective brain: are there any object storage solutions, cloud or on-prem, which support any sort of "operator pushdown"?
By "operator pushdown", i mean any ability to filter or map over the contents of the object on the server side in some way, sending only the results over the network to the client.
For example, say you have a huge CSV file of customer orders in a bucket. You might want to find the timestamp of all the orders which included a particular product. If all you can do is stream the whole file, then you need to do that, just to pick out a few timestamps. But you could imagine a kind of request where you say "only give me lines where the product ID is P01234, and only send the timestamp column". Perhaps you would express that as a pair of regular expressions, or a sed program, or a Lua script, or maybe the server would understand CSV and let you write something a bit like SQL. There are all sorts of ways it could be done. Providing a fully general way might be tricky, but it wouldn't need to be fully general to be useful.
I appreciate that if you want to do this sort of access frequently, you should probably be using a database, not object storage. But it seems like a very useful feature to layer on top of object storage, and one that feels like it should be fairly cheap to execute - the server has to do a small extra amount of computation, but then needs a lto less network bandwidth.
- rapatel0 2y agoLook into duckdb. It's not server side, but It's pretty phenomenal for this use case.
- patrickthebold 2y agoIn the AWS world, Athena does more or less what you are describing: but is is targeted (AFAIU) for large datasets. I'm not sure how it would perform on a single (small) CSV to shave off a few bytes in transit.
- weberer 2y agoYeah, on AWS you can set up a Glue Data Catalog and query it with Athena.
- dmw_ng 2y agoThat's been a feature of S3 for quite a long time now, called S3 Select https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/select... Despite it being an awesome feature I've been itching to use, I've never actually found a use for it beyond messing around. Most places where S3 Select might make sense seems to be subsumed (for my uses) by Athena. Athena has a rather large amount of conceptual and actual boilerplate to get up and running with, though, S3 Select requires no upfront planning beyond building a fancy query string (or using their SDK wrappers) Where S3 Select is likely to become fiddly is anywhere multiple files are involved. Athena makes querying large collections of CSVs (etc) straightforward, and handles all the scheduling and results merging for you.
- twic 2y ago> Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. :( But you and patrickthebold are spot on in pointing out Athena. I've always thought of it as a database you load via S3, but of course it's equally a tool for querying data in S3.
- ju-st 2y agoS3 Select is not available anymore for new customers. Athena with columnar file format (eg parquet) in S3 and partitioning with Glue Data catalog is the solution for OP's problem. The cost of this kind of queries is very low because you only pay the actual data consumed/requested. And with the columnar file format Athena only accesses the necessary columns. And the data in the columns is usually compressed so the amount of data is even less.
- thinkingemote 2y agoYou could use Range HTTP Headers if you know which bit to request. Kind of simple and not that useful for dynamic queries but quite good for queries where you know the queries beforehand and can index accordingly. https://docs.aws.amazon.com/AmazonS3/latest/userguide/range-get-olap.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/range-...
- 392 2y agoWhat you're describing is targeted by Parquet/Iceberg + an engine like Dremio. But targeted at big data / analytics.