3 ms·

> The OS's file reading mechanisms Did you mean "mechanism" as in singular, not plural? Because there are indeed many different ones, and that's why I asked t

> The OS's file reading mechanisms Did you mean "mechanism" as in singular, not plural? Because there are indeed many different ones, and that's why I asked this question. > or a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3. Why is it so hard to understand... No, they should not have used S3 to begin with. S3 is for Web, it's not for storage. These people used a wrong tool because they don't even know what tools they have... > if they mean something more native to the OS, like read(2)/write(2), Again, they don't know what they are talking about. read()/write() are just gateways that expose them to different ways OS can read or write. > or a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3 Of course it can. HTTP goes through a socket, which is a file. C can perfectly fine read from a socket file using file interface. In case you ever had the "engineer" title, things like this should be the reason to lose it. > That the data is in Blob Storage seems to me to be a given It's not a given. It's never a given. It's data stored in the proprietary data-center "somehow". Even if it was placed in the blob storage by someone else, nobody prevents the authors from moving it away from it. Not to mention that moving things inside that proprietary data-center is going to be much faster, so, even if they are required to acquire data as blob storage, they can almost instantly convert it into something more usable.

acdha 3y ago
> In case you ever had the "engineer" title, things like this should be the reason to lose it. Even if you weren't technically incorrect, this would be uncalled for. I would suggest toning down the heat a lot in your replies and asking what substance you're contributing to the conversation. For example, the POSIX filesystem APIs don't have a concept of TLS or any of the HTTP protocol-level handling (transfer compression, chunked encoding, etc.) so it's only true that C code can read bytes from the socket. There's a reason why projects like s3fs exist to do all of that work in addition to things like ranged requests and caching which are essential for performance so you don't have to wait to stream a 2GB file just to read the header.
deathanatos 3y ago
> Why is it so hard to understand... No, they should not have used S3 to begin with. S3 is for Web, it's not for storage. These people used a wrong tool because they don't even know what tools they have... S3 is absolutely for storage. So much so that it is quite literally in the name. I'm morbidly curious what purpose you think S3 fills, if not storage? > [capabilities of read/write] Yes, while read/write can operate on a socket, that doesn't mean it's usable by a library that's going to be expecting image data. Let's say our library which is going to expect to read image data does accept just any old FD, (and not a FILE pointer); even then, I can't simply hand it a socket: the socket is going to be speaking HTTP/TLS, and the library expects image data. In the best case, you'd get a parse error. I mention FUSE drivers in the comment, and so too does the article. The article itself notes they're on macOS, where FUSE is a second-class citizen. There's pragmatic reasons to avoid it. (They also note that it read the entire file, which, while that shouldn't be true, if they errantly believed it to be so, then it would reasonably drive their judgement in the direction they went.) > In case you ever had the "engineer" title, things like this should be the reason to lose it. I don't know why you feel the need to include personal attacks like this. Guess I'm not an engineer anymore! > It's not a given. It's never a given. It's data stored in the proprietary data-center "somehow". But it may be out of this engineers scope, or out of their control; they may have been given something resembling "hey, we have this data, already stored in Blobstore, can you do this task with it?", and within that scope, it's a given. Yes, in the grand design of the universe, it need not be, but for the engineer right there in OP, that might be boiling an ocean. But also … the data likely has to be stored somewhere, and there are good reasons to choose such a service. Given the file sizes mentioned in the article, they need a fair bit of storage (something S3 et al. is good at) and they need access to individual files without a lot of special needs that might be better filled by a true FS like ext4. Within reason, it seems like one can assume that Blobstore isn't an unreasonable choice here. > Not to mention that moving things inside that proprietary data-center is going to be much faster, so, even if they are required to acquire data as blob storage, they can almost instantly convert it into something more usable. And format change is going to be something you can do with S3, too. The only thing you really get by moving into a proprietary data center is that the data might be slightly more local, and thus have better latency. Format conversions are sometimes hard: that format may be required for other use cases. But again, this all seems like questioning things in the OP that … don't really need to be questioned from our armchairs.

acdha 3y ago

> In case you ever had the "engineer" title, things like this should be the reason to lose it. Even if you weren't technically incorrect, this would be uncalled for. I would suggest toning down the heat a lot in your replies and asking what substance you're contributing to the conversation. For example, the POSIX filesystem APIs don't have a concept of TLS or any of the HTTP protocol-level handling (transfer compression, chunked encoding, etc.) so it's only true that C code can read bytes from the socket. There's a reason why projects like s3fs exist to do all of that work in addition to things like ranged requests and caching which are essential for performance so you don't have to wait to stream a 2GB file just to read the header.

deathanatos 3y ago

> Why is it so hard to understand... No, they should not have used S3 to begin with. S3 is for Web, it's not for storage. These people used a wrong tool because they don't even know what tools they have... S3 is absolutely for storage. So much so that it is quite literally in the name. I'm morbidly curious what purpose you think S3 fills, if not storage? > [capabilities of read/write] Yes, while read/write can operate on a socket, that doesn't mean it's usable by a library that's going to be expecting image data. Let's say our library which is going to expect to read image data does accept just any old FD, (and not a FILE pointer); even then, I can't simply hand it a socket: the socket is going to be speaking HTTP/TLS, and the library expects image data. In the best case, you'd get a parse error. I mention FUSE drivers in the comment, and so too does the article. The article itself notes they're on macOS, where FUSE is a second-class citizen. There's pragmatic reasons to avoid it. (They also note that it read the entire file, which, while that shouldn't be true, if they errantly believed it to be so, then it would reasonably drive their judgement in the direction they went.) > In case you ever had the "engineer" title, things like this should be the reason to lose it. I don't know why you feel the need to include personal attacks like this. Guess I'm not an engineer anymore! > It's not a given. It's never a given. It's data stored in the proprietary data-center "somehow". But it may be out of this engineers scope, or out of their control; they may have been given something resembling "hey, we have this data, already stored in Blobstore, can you do this task with it?", and within that scope, it's a given. Yes, in the grand design of the universe, it need not be, but for the engineer right there in OP, that might be boiling an ocean. But also … the data likely has to be stored somewhere, and there are good reasons to choose such a service. Given the file sizes mentioned in the article, they need a fair bit of storage (something S3 et al. is good at) and they need access to individual files without a lot of special needs that might be better filled by a true FS like ext4. Within reason, it seems like one can assume that Blobstore isn't an unreasonable choice here. > Not to mention that moving things inside that proprietary data-center is going to be much faster, so, even if they are required to acquire data as blob storage, they can almost instantly convert it into something more usable. And format change is going to be something you can do with S3, too. The only thing you really get by moving into a proprietary data center is that the data might be slightly more local, and thus have better latency. Format conversions are sometimes hard: that format may be required for other use cases. But again, this all seems like questioning things in the OP that … don't really need to be questioned from our armchairs.