3 ms·

> To be clear, did your debugging problem come down to the fact that the metadata for the objects was on "the other side of" the S3 URL, being held as object me

> To be clear, did your debugging problem come down to the fact that the metadata for the objects was on "the other side of" the S3 URL, being held as object metadata headers in S3 that you would need to do a HEAD request to retrieve per object, rather than as e.g. columns in a local RDBMS—such that you couldn't do aggregate queries on it to figure out what a customer's files "looked like" in a statistical sense? No. In this case, the metadata included some of that data, but it also stored information about past application state or data that was otherwise stored in the blob itself. For instance, when a file was replaced, it created an orphaned blob in S3. Tracking that down meant using rollbacks in Heroku postgres on at least one occasion. Keeping track of which files replaced other files is new functionality that was not possible using any reasonable means, but would have come for free with the feature I designed. Reading data out of S3 is painfully slow and I don't consider it to be a reasonable solution for anything that involves looking at more than one file. Consider the question "how many M4A files have a Content-Type set that isn't M4A?" Iterating hundreds of thousands of files and checking their magic number (ignoring ID3) and Content-Type is not only slow but also expensive. Storing the content type in advance would at least allow that process to be shaved down to a small fraction of what it otherwise would be. (and before you ask, file extension is a worthless heuristic when dealing with user-supplied data). But of course, I didn't know that storing an object with metadata would solve these problems, even though I knew that I should do it. Saying things like "put Varnish in front of S3" is suggesting putting your finger in the dike. The cost of setting up and maintaining Varnish (and paying for it) is far higher than just doing the thing the first time. And in fact, would have been far more costly than me putting my foot down and doing the one-time migration, as I did. Besides, our applications are on Heroku, where something like this would have been non-trivial anyway. My point is that YAGNI purism optimizes for short term wins. If you never do anything until you can make a compelling argument that it's something you NEED to do NOW, you're going to end up with Dr Seuss-esque systems that do everything possible to actually avoid solving the problem itself, especially if there's always a (less than ideal) workaround. This is one of the easiest ways for tech debt to accumulate.

derefr 8y ago
> The cost of setting up and maintaining Varnish (and paying for it) is far higher than just doing the thing the first time. Are you sure? What is your time worth? I mean, there's certainly a comparative-advantage thing here. A software developer setting up a Varnish instance themselves is going to take more time than a software developer writing code, because software developers know code and not Varnish. But my point was that most software isn't run by single do-everything DevOps people. In bigcorp production-scale systems (where all this software engineering advice both comes from and is targeted toward), you've got a dev team creating a particular software component, and then either internal or external ops teams running it as just one component of their system/solution. Those ops people know Varnish better than they know code. Picture a ticket triage that goes through a support person, and then gets elevated to an ops person. (Because this is what ops people are for, whereas the dev-team's time is far too valuable to the company to spend on things that could be handled by the pre-provisioned capacity of the ops team.) The ops person isn't going to turn around and ask a dev to solve the problem by writing code if he can at-all help it. The ops person is going to try to solve the problem themselves, with an infrastructure-level solution. And, given 1. the overhead of having to go to the dev-team and get the ops-supporting feature added into their priority list, and then work with the support person to interact with the customer while the dev-team maybe eventually fixes the problem (and where then the ops-team will have to both deploy, and understand how to maintain, the software in its newly stretched state, where it probably now has extra ops-time needs like cache storage!); vs. 2. the ease with which existing deployments of infrastructure components like Varnish, which were stood up to solve previous scaling challenges, can simply have their cluster-configurations extended to support new scaling challenges with no marginal increased maintenance burden; the infrastructure-level solution will win every time. Yes, sure, if you're one dev and you do your own ops and you don't actually know much about ops (in the comparative-advantage sense), then solving all your problems in code might have the highest ROI. YAGNI—and pretty much any other software-engineering principle—isn't targeted at you. You're doing artisanal software development—making the hand-woven wicker chairs of the software world. Most such principles focus on decreasing the Total Cost of Ownership of software, taking into account inter-departmental collaboration overheads, maintenance of the codebase after the original developers leave, etc. You've got none of those concerns. If you're a dev-team of one and you deploy your own code, go wild: write your whole system in a macro-heavy DSL dialect of Common Lisp with your own custom logging; or write your system as a single 46kb x86-64 assembler unikernel. It doesn't matter, because you still understand it, and can modify it just fine. (Just don't expect to sell your company down the line!) --- > Consider the question "how many M4A files have a Content-Type set that isn't M4A?" Iterating hundreds of thousands of files and checking their magic number (ignoring ID3) and Content-Type is not only slow but also expensive. Storing the content type in advance would at least allow that process to be shaved down to a small fraction of what it otherwise would be. (and before you ask, file extension is a worthless heuristic when dealing with user-supplied data). Ah, yeah, that's a different problem than I had assumed. In my mental model, you already had a proxy for handling your object creation, where audio files users uploaded would first go to you; you'd extract the file's indexable metadata (magic, ID3); and then your server would add said metadata into the S3 PUT request as headers (Content-Type & co, and then x-amz-meta- headers for anything else.) Under that model, you get back everything you need from just doing a HEAD request to your object. S3 HEAD requests are cheap. They're just not fast/highly concurrent, so the point of Varnish here is to make them so. S3 is annoying, though, in that you can't mutate metadata on an object without re-uploading the object. So, if you were allowing people to do direct object uploads to S3 using signed URLs, and then doing the indexing from there, it makes sense that you wouldn't want to write the resulting metadata back to the object, and would instead want to keep the canonical copy of it local. In that case, though, there's still an infrastructure-level solution. Replace S3 with https://www.minio.io https://www.minio.io backed by S3, with Minio's metadata storage pointed at your existing RDBMS. It's essentially just the same thing you built, without writing code. Metadata lives in Minio (and therefore your DB, where you can query it); object bodies get uploaded asynchronously to S3. Though, with this solution, you can optionally have the metadata get proxied through Minio during the PUT, and thus do the metadata-generating content analysis step asynchronously against that stream (as a Minio plugin), rather than having to wait for the upload to complete and then GET the result back to your business layer. Though, again, for your devs=1 use-case, maybe this has higher TCO than just writing code to track metadata within your system.

derefr 8y ago

> The cost of setting up and maintaining Varnish (and paying for it) is far higher than just doing the thing the first time. Are you sure? What is your time worth? I mean, there's certainly a comparative-advantage thing here. A software developer setting up a Varnish instance themselves is going to take more time than a software developer writing code, because software developers know code and not Varnish. But my point was that most software isn't run by single do-everything DevOps people. In bigcorp production-scale systems (where all this software engineering advice both comes from and is targeted toward), you've got a dev team creating a particular software component, and then either internal or external ops teams running it as just one component of their system/solution. Those ops people know Varnish better than they know code. Picture a ticket triage that goes through a support person, and then gets elevated to an ops person. (Because this is what ops people are for, whereas the dev-team's time is far too valuable to the company to spend on things that could be handled by the pre-provisioned capacity of the ops team.) The ops person isn't going to turn around and ask a dev to solve the problem by writing code if he can at-all help it. The ops person is going to try to solve the problem themselves, with an infrastructure-level solution. And, given 1. the overhead of having to go to the dev-team and get the ops-supporting feature added into their priority list, and then work with the support person to interact with the customer while the dev-team maybe eventually fixes the problem (and where then the ops-team will have to both deploy, and understand how to maintain, the software in its newly stretched state, where it probably now has extra ops-time needs like cache storage!); vs. 2. the ease with which existing deployments of infrastructure components like Varnish, which were stood up to solve previous scaling challenges, can simply have their cluster-configurations extended to support new scaling challenges with no marginal increased maintenance burden; the infrastructure-level solution will win every time. Yes, sure, if you're one dev and you do your own ops and you don't actually know much about ops (in the comparative-advantage sense), then solving all your problems in code might have the highest ROI. YAGNI—and pretty much any other software-engineering principle—isn't targeted at you. You're doing artisanal software development—making the hand-woven wicker chairs of the software world. Most such principles focus on decreasing the Total Cost of Ownership of software, taking into account inter-departmental collaboration overheads, maintenance of the codebase after the original developers leave, etc. You've got none of those concerns. If you're a dev-team of one and you deploy your own code, go wild: write your whole system in a macro-heavy DSL dialect of Common Lisp with your own custom logging; or write your system as a single 46kb x86-64 assembler unikernel. It doesn't matter, because you still understand it, and can modify it just fine. (Just don't expect to sell your company down the line!) --- > Consider the question "how many M4A files have a Content-Type set that isn't M4A?" Iterating hundreds of thousands of files and checking their magic number (ignoring ID3) and Content-Type is not only slow but also expensive. Storing the content type in advance would at least allow that process to be shaved down to a small fraction of what it otherwise would be. (and before you ask, file extension is a worthless heuristic when dealing with user-supplied data). Ah, yeah, that's a different problem than I had assumed. In my mental model, you already had a proxy for handling your object creation, where audio files users uploaded would first go to you; you'd extract the file's indexable metadata (magic, ID3); and then your server would add said metadata into the S3 PUT request as headers (Content-Type & co, and then x-amz-meta- headers for anything else.) Under that model, you get back everything you need from just doing a HEAD request to your object. S3 HEAD requests are cheap. They're just not fast/highly concurrent, so the point of Varnish here is to make them so. S3 is annoying, though, in that you can't mutate metadata on an object without re-uploading the object. So, if you were allowing people to do direct object uploads to S3 using signed URLs, and then doing the indexing from there, it makes sense that you wouldn't want to write the resulting metadata back to the object, and would instead want to keep the canonical copy of it local. In that case, though, there's still an infrastructure-level solution. Replace S3 with https://www.minio.io https://www.minio.io backed by S3, with Minio's metadata storage pointed at your existing RDBMS. It's essentially just the same thing you built, without writing code. Metadata lives in Minio (and therefore your DB, where you can query it); object bodies get uploaded asynchronously to S3. Though, with this solution, you can optionally have the metadata get proxied through Minio during the PUT, and thus do the metadata-generating content analysis step asynchronously against that stream (as a Minio plugin), rather than having to wait for the upload to complete and then GET the result back to your business layer. Though, again, for your devs=1 use-case, maybe this has higher TCO than just writing code to track metadata within your system.