orf

38 ms·

601.
by orf 3y ago
If only there was some kind of long-lived proxy you could put in front of things that would handle connection pooling and would forward requests onto a short-lived backend, scaling up instances as needed to serve those requests.
602.
by orf 3y ago
I don’t get it. A lot of this post is describing how they translate HTTP requests into function calls, and deal with issues around the HTTP protocol. If you’re invoking a heavy-duty, long running function then you’re likely doing something
603.
by orf 3y ago
No it doesn’t, absolutely not. It’s ironic that you say this after the post you’re commenting on spells out quite explicitly why things “in base” are hard to change and adapt.
604.
by orf 3y ago
And I hope you were not around many catholic priests when you were.
605.
by orf 3y ago
But this would now make the reference count increment require a conditional? It’s a very hot path, and this would cause a slowdown for single-threaded Python code.
606.
by orf 3y ago
Right click, move to recycle bin, wait for the progress bar to finish. Except the progress bar takes a day or so. This is only needed if you have a huge (100 million+) bucket, at which point you should be experienced with s3, otherwise yo
607.
by orf 3y ago
Incorrect. One call to trigger a step function that sets up the lifecycle rule, sleeps for 24 hours and then deletes the bucket. Stop being silly, as if 1 vs 2 API calls matters. You should empty large buckets with lifecycle policies. It&#x
608.
by orf 3y ago
Literally 1 API call.
609.
by orf 3y ago
What differences in behaviour from the AWS docs did you find, out of interest?
610.
by orf 3y ago
2 seconds is a nuts response time, but I guess it depends entirely on your file size. TTFB is usually 50ms.
611.
by orf 3y ago
It was both actually, but more for the listing issue. Netflix built a lot of tooling around this. But yeah: things like filtering on tags or created at dates requires another approach.
612.
by orf 3y ago
Neither of those are true though? Not sure if this is sarcastic or not, if so make it more clear in the future
613.
by orf 3y ago
> It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. Because there is no POSIX api for this. Depending on your requirements and query patterns, you may
614.
by orf 3y ago
You’re assuming you’re paginating through all objects from start to finish. A request to list objects under “foo/“ is a request to list all objects starting with “foo/“, which is constant time irregardless of the number of keys be
615.
by orf 3y ago
It’s fundamental to how S3 works and its ability to scale, so it is a defining feature of S3. If you think wider, a bucket itself is just a prefix.
616.
by orf 3y ago
Constant time irregardless of the number of objects in the bucket and irregardless of the initial starting position of your list request.
617.
by orf 3y ago
My overall point can be summarised as this: - Listing things is a very common operation to do. - The POSIX api and the directory/file hierarchy it provides is a restrictive one. - S3 does not suffer from this, you can recursively list
618.
by orf 3y ago
It mistakenly mentions UNIX whilst referencing the POSIX filesystem API, and I literally quoted where it talks about listing in my original comment.
619.
by orf 3y ago
It depends on a few factors. The list objects call hides deleted and noncurrent versions, but it has to skip over them. Grouping prefixes also takes time, if they contain a lot of noncurrent or deleted keys. A pathological case would be a p
620.
by orf 3y ago
From the article we're commenting on, which is comparing the interface of S3 to the POSIX interface. Not any given filesystem + platform specific interface.
621.
by orf 3y ago
Sure. It's kind of an index - limited to prefix-only searching, but useful. Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`. If you need to
622.
by orf 3y ago
Yes. You could also just use a SQLite table with two columns (path, contents), then just query that. Or do any number of other things. The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well s
623.
by orf 3y ago
S3 doesn’t have directories, it could be thought of a flat + sorted list of keys. UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That ca
624.
by orf 3y ago
Yes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents. You can maybe use “d_off” with readdir i
625.
by orf 3y ago
> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something cri
626.
by orf 3y ago
This is so laughably incorrect/out of date for the UK travel section that I doubt the rest is of any value. The majority of the advice can be distilled down to “use Trainline”. Call national rail for travel times? What??
627.
by orf 3y ago
> When millions of people understand a term "wrong", then that becomes "correct" on virtue of it being so widely used
628.
by orf 3y ago
You’d need to build a site with multiple pages, with multiple developers and lots of concurrent changes. Then you’d realise why people don’t use raw CSS
629.
by orf 3y ago
Yes. Much more than doing this via a number of detached and separate CSS classes defined in different files, with colliding rules and all the fun that goes with this. Or better yet, inventing enough small composable utility functions to wor
630.
by orf 3y ago
Ahh, amazing! That’s not entirely clear from the data loading docs[1], which when I read it seemed only focused around CSV and JSON. 1. https://observablehq.com/framework/loaders

More ›