4 ms·

> But as it turns out, we can’t use it. Although it has Python bindings, OpenSlide is implemented in C and reads files using standard OS file handlers, however

> But as it turns out, we can’t use it. Although it has Python bindings, OpenSlide is implemented in C and reads files using standard OS file handlers, however our data sits on cloud storage that is accessible via HTTP. This is a self-inflicted problem. Very typical of people who don't know how storage works / what functionality is available will often push themselves into an imaginary corner. Why of all things use HTTP for this? No, of course you don't need to download the whole file to read it. "standard OS file handlers" -- this is a strong indicator that the person writing this doesn't understand how their OS works. What standard are we talking about? Even if "standard" here is used to mean "some common way" -- then which one? How the files are opened? And so on. The author didn't research the subject at all, and came up with an awful solution (vendor lock-in) as a result.

dekhn 3y ago
The standard OS file handlers they mean are the UNIX and Windows APIs to open and read/write file content. The open(), read(), seek(), and write() library calls, which wrap the OS's system calls that use VFS. HTTP is now a de-facto transport standard. S3 uses, and many other data storage systems do as well. It's highly tuned for performance and there is an entire ecosystem around it. You could implement a system like NFS over XDR-over-TCP-IP (https://en.wikipedia.org/wiki/Sun_RPC https://en.wikipedia.org/wiki/Sun_RPC) or HTTP (https://en.wikipedia.org/wiki/WebDAV https://en.wikipedia.org/wiki/WebDAV).
- crabbone 3y ago
  Those "file handlers" you speak of are just numbers. There's nothing special about them. There was no reason for OP to complain about them. It's just an indication that they don't understand the problem they are trying to solve. > HTTP is now a de-facto transport standard. S3 uses, Was the period here intentional? Because if not, then HTTP is the standard for Web applications. It's virtually unheard of in storage world. The standards used in the storage world are typically called X-over-Y eg. iSCSI (which is SCSI over IP) or NVMe over IP, NFS (which is Sun RPC over IP) and so on. In other words, it's usually a protocol intended for storage needs implemented on top of some network protocol (since we are talking about storage on the network). S3 is almost unique in its use of application-level protocol for storage. It is a bad decision from storage perspective as it is imposing too many unnecessary restrictions and would be generally slow for implementing some network-attached storage. > It's highly tuned for performance and there is an entire ecosystem around it. Not in the storage world. It's quite worthless as a conduit for storage. In storage, you operate in blocks, you don't need streaming, you can handle out-of-order reads and writes, you may or may not benefit from sessions, but you absolutely have no use for any of HTTP headers, you have no use for content-types. While you could probably interpret GET as "read" and POST as "write", the PATCH, TRACE and whatever else there is would be completely worthless. On the other hand, whenever you read or write in storage you need to transmit a lot of important details which are not reflected in these HTTP methods. None of the HTTP status codes are useful for storage... In other words, all the features HTTP has to offer on top of TCP are worthless for storage. It's designed for Web applications, and that's why S3 chose HTTP (to be usable directly from Web applications). If Web supported straight up TCP or UDP, S3 wouldn't be using HTTP. > XDR-over-TCP-IP What does this have to do with HTTP? (PS. I wrote an NFS3 testing client in Go some 5-7 years ago, no HTTP involved).
deathanatos 3y ago
> No, of course you don't need to download the whole file to read it. HTTP doesn't require this, nor do any of AWS S3/Azure Blog Storage/GCP Cloud Storage. > "standard OS file handlers" -- this is a strong indicator that the person writing this doesn't understand how their OS works. What standard are we talking about? Even if "standard" here is used to mean "some common way" -- then which one? How the files are opened? The OS's file reading mechanisms, whatever those are. For a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3. (Excepting something like a FUSE driver, which the article does even mention, in passing¹.) Even if they mean something more native to the OS, like read(2)/write(2), or whatever Windows uses … it doesn't change the point? They then describe an FS abstracting layer — those also aren't uncommon, e.g., gvfs, and often exist to bridge exactly the gap above: I have things like S3 that sort of look like files, and I have something that wants a file. I want to directly connect them. FS abstractions like what the article describes bridge that gap. > The author didn't research the subject at all, and came up with an awful solution (vendor lock-in) as a result. That the data is in Blob Storage seems to me to be a given in the article. I.e., we're starting with that assumption, so how can we make use of the data without incurring huge network transfers? The data could be in some FOSS storage system running on your own servers, serving up its data using the S3 API … and you'd have the same article. ¹it's confusing that they say the FUSE driver needs to download the entire file. It shouldn't require that — certainly nothing in the APIs available require it. But … macOS; yeah. That's a rougher problem, and does push one in the direction of something like where the author went.
- crabbone 3y ago
  > The OS's file reading mechanisms Did you mean "mechanism" as in singular, not plural? Because there are indeed many different ones, and that's why I asked this question. > or a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3. Why is it so hard to understand... No, they should not have used S3 to begin with. S3 is for Web, it's not for storage. These people used a wrong tool because they don't even know what tools they have... > if they mean something more native to the OS, like read(2)/write(2), Again, they don't know what they are talking about. read()/write() are just gateways that expose them to different ways OS can read or write. > or a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3 Of course it can. HTTP goes through a socket, which is a file. C can perfectly fine read from a socket file using file interface. In case you ever had the "engineer" title, things like this should be the reason to lose it. > That the data is in Blob Storage seems to me to be a given It's not a given. It's never a given. It's data stored in the proprietary data-center "somehow". Even if it was placed in the blob storage by someone else, nobody prevents the authors from moving it away from it. Not to mention that moving things inside that proprietary data-center is going to be much faster, so, even if they are required to acquire data as blob storage, they can almost instantly convert it into something more usable.

dekhn 3y ago

The standard OS file handlers they mean are the UNIX and Windows APIs to open and read/write file content. The open(), read(), seek(), and write() library calls, which wrap the OS's system calls that use VFS. HTTP is now a de-facto transport standard. S3 uses, and many other data storage systems do as well. It's highly tuned for performance and there is an entire ecosystem around it. You could implement a system like NFS over XDR-over-TCP-IP (https://en.wikipedia.org/wiki/Sun_RPC https://en.wikipedia.org/wiki/Sun_RPC) or HTTP (https://en.wikipedia.org/wiki/WebDAV https://en.wikipedia.org/wiki/WebDAV).

crabbone 3y ago
Those "file handlers" you speak of are just numbers. There's nothing special about them. There was no reason for OP to complain about them. It's just an indication that they don't understand the problem they are trying to solve. > HTTP is now a de-facto transport standard. S3 uses, Was the period here intentional? Because if not, then HTTP is the standard for Web applications. It's virtually unheard of in storage world. The standards used in the storage world are typically called X-over-Y eg. iSCSI (which is SCSI over IP) or NVMe over IP, NFS (which is Sun RPC over IP) and so on. In other words, it's usually a protocol intended for storage needs implemented on top of some network protocol (since we are talking about storage on the network). S3 is almost unique in its use of application-level protocol for storage. It is a bad decision from storage perspective as it is imposing too many unnecessary restrictions and would be generally slow for implementing some network-attached storage. > It's highly tuned for performance and there is an entire ecosystem around it. Not in the storage world. It's quite worthless as a conduit for storage. In storage, you operate in blocks, you don't need streaming, you can handle out-of-order reads and writes, you may or may not benefit from sessions, but you absolutely have no use for any of HTTP headers, you have no use for content-types. While you could probably interpret GET as "read" and POST as "write", the PATCH, TRACE and whatever else there is would be completely worthless. On the other hand, whenever you read or write in storage you need to transmit a lot of important details which are not reflected in these HTTP methods. None of the HTTP status codes are useful for storage... In other words, all the features HTTP has to offer on top of TCP are worthless for storage. It's designed for Web applications, and that's why S3 chose HTTP (to be usable directly from Web applications). If Web supported straight up TCP or UDP, S3 wouldn't be using HTTP. > XDR-over-TCP-IP What does this have to do with HTTP? (PS. I wrote an NFS3 testing client in Go some 5-7 years ago, no HTTP involved).

deathanatos 3y ago

> No, of course you don't need to download the whole file to read it. HTTP doesn't require this, nor do any of AWS S3/Azure Blog Storage/GCP Cloud Storage. > "standard OS file handlers" -- this is a strong indicator that the person writing this doesn't understand how their OS works. What standard are we talking about? Even if "standard" here is used to mean "some common way" -- then which one? How the files are opened? The OS's file reading mechanisms, whatever those are. For a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3. (Excepting something like a FUSE driver, which the article does even mention, in passing¹.) Even if they mean something more native to the OS, like read(2)/write(2), or whatever Windows uses … it doesn't change the point? They then describe an FS abstracting layer — those also aren't uncommon, e.g., gvfs, and often exist to bridge exactly the gap above: I have things like S3 that sort of look like files, and I have something that wants a file. I want to directly connect them. FS abstractions like what the article describes bridge that gap. > The author didn't research the subject at all, and came up with an awful solution (vendor lock-in) as a result. That the data is in Blob Storage seems to me to be a given in the article. I.e., we're starting with that assumption, so how can we make use of the data without incurring huge network transfers? The data could be in some FOSS storage system running on your own servers, serving up its data using the S3 API … and you'd have the same article. ¹it's confusing that they say the FUSE driver needs to download the entire file. It shouldn't require that — certainly nothing in the APIs available require it. But … macOS; yeah. That's a rougher problem, and does push one in the direction of something like where the author went.

crabbone 3y ago
> The OS's file reading mechanisms Did you mean "mechanism" as in singular, not plural? Because there are indeed many different ones, and that's why I asked this question. > or a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3. Why is it so hard to understand... No, they should not have used S3 to begin with. S3 is for Web, it's not for storage. These people used a wrong tool because they don't even know what tools they have... > if they mean something more native to the OS, like read(2)/write(2), Again, they don't know what they are talking about. read()/write() are just gateways that expose them to different ways OS can read or write. > or a C library, probably that means "fopen()" et al. which of course cannot read from, e.g., S3 Of course it can. HTTP goes through a socket, which is a file. C can perfectly fine read from a socket file using file interface. In case you ever had the "engineer" title, things like this should be the reason to lose it. > That the data is in Blob Storage seems to me to be a given It's not a given. It's never a given. It's data stored in the proprietary data-center "somehow". Even if it was placed in the blob storage by someone else, nobody prevents the authors from moving it away from it. Not to mention that moving things inside that proprietary data-center is going to be much faster, so, even if they are required to acquire data as blob storage, they can almost instantly convert it into something more usable.