4 ms·

The hash is not for cryptographic purposes, it's to detect corrupt downloads. Ideally you should provide both the hash and you should pgp sign your releases.

by 4ad 12y ago

colanderman 12y ago
That is bizarre. Like "I thought XMODEM went out of style in 1992" bizarre. Shouldn't checksumming the payload be performed at the application layer? (A quick Google shows that indeed it can be: the Content-MD5 header <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>. Wonder how widely supported it is by HTTP software used by people who like to check hashes of things they download.)
- sp332 12y ago
  Well what if you don't get the binary from the same source as the hash, or it's been sitting on a disk for a while and you want to double-check its integrity?
  - colanderman 12y ago
    The post to which I replied specifically focused on the case where the binary came from the same source as the hash, and was being used to confirm the integrity of the download. I was not addressing other cases. (Although I will note that file systems, like application protocols, should maintain their own integrity; however most do not. Which also seems bizarre given that it's 2014.)
    - sp332 12y ago
      Other programs could, accidentally or maliciously, step on the contents of a file. Filesystem checksums are awesome (I use btrfs), but it doesn't protect your file from everything. Hm... maybe a versioned filesystem where you can go back to the "original" version of a file and verify its checksum? :)
      - colanderman 12y ago
        Yes, protecting from malicious / accidental modification is a separate problem than protecting from bit rot / gamma rays. The former is obviously much more difficult than the latter. My surprise stems from the fact that common tools are not robust against the latter by now. (My day job is working with enterprise-grade content-addressed block storage, so maybe I've just set my data-protection expectations too high.)
    - derefr 12y ago
      Indeed, I find it surprising that there's no filesystem flag giving the opposite behaviour to "compress contents to save space": something like "transparently store this file bloated by a fountain codec to ensure integrity on an bitwise-unreliable backing store" or maybe "transparently generate a Parity Volume Set from this file and stripe it across the disk to protect against lost sectors." Both of these options would protect your important files against failing single disks without having to do any RAIDing. The unimportant data (e.g. the OS itself, caches, etc.) could be reduced-redundancy, since it doesn't need to be captured in a disk rescue.
      - angry_octet 12y ago
        It would be a great feature, but obviously adds complexity. You can get half way there with single disk ZFS, which lets you store multiple copies. But the key feature is that files and directory info are checksummed, and regularly checked, so corruption cannot silently occur. But I think that usually the whole disk is lost, rather than getting bad blocks (so 90's). For any archival DVDs I burn, I compress, then run through PAR2 http://en.m.wikipedia.org/wiki/Parchive http://en.m.wikipedia.org/wiki/Parchive
- 4ad 12y ago
  I am not so sure why you think this is bizarre. In an ideal world TCP is reliable, HTTP assumes it runs on reliable transport so it doesn't do any more integrity checking. In the real world there are bugs in TCP stacks and in HTTP implementations that cause HTTP traffic to get corrupted. I see this every day. Some applications do implement extra checking, most applications do not. Browsers and wget and curl can't implement any extra checking because the way you check is application specific. There is no standard way to do it; what you mention there is an esoteric feature. Just for anecdotal fun, Logic Pro tries to download over 50GB of assets over HTTP (each individual file is many-GB). It has never worked for me over any of my networks (and in fact I wrote a tool to fix this).
  - colanderman 12y ago
    > In the real world there are bugs in TCP stacks and in HTTP implementations that cause HTTP traffic to get corrupted I'm very much aware of this. Hence why I expressed surprise that such checksumming was not commonly performed at the application layer. > There is no standard way to do it; what you mention there is an esoteric feature. The Content-MD5 header is defined in RFC 2616. It is, by definition, standard. If it's not widely supported, then I think that it would behoove the people who care about these things to switch to servers/clients which do support it. (I suspect the intersection between "people who know/care how to use md5sum" and "people who know how to set/read an arbitrary HTTP header" is fairly large. Hence my surprise at the common practice of ignoring this capability.)
- adricnet 12y ago
  Well, I understand your reaction, but it may help to point our that there are three different algorithms involved here and that to ensure content completeness and ensure security concerns (CIA, NR, etc) you want them all, and all done correctly: * Checksums, like in Xmodem or CRC * Cryptographic hashes (including MACs) * Cryptographic signatures (ie OpenPGP key or cert) As noted in the other comments these protect against different kinds of problems in transmission, but in used correctly in combination can protect from both glitches and active attacks. To say that it is difficult to implement all of these correctly and in concert is a grave understatement, but this is what modern crypto software and network protocols that use it, have to do. Now back to the thread on HTTP header checksums :)
  - deleted 12y ago
    [deleted]
- syntheticnature 12y ago
  I'll note that even today "ASCII mode" in FTP haunts people who aren't careful when downloading -- and that it is a frequent reason I find myself having to urge MD5 checks of embedded firmware I work on. At least these days the firmware has enough room to validate a download itself before installing it.
sysk 12y ago
Do corrupt downloads really happen these days and what causes the downloads to become corrupt? I can't remember having a single corrupt HTTP download in the past 10 years.
- 4ad 12y ago
  Yes, they do happen. Not as often as 20 years ago, but they still happen. Personally I can't download more than about 1GB without an error. If you use bittorrent you might want to look in the logs to see how often it re-downloads blocks because the block it just downloaded has been corrupted. You might be surprised. I don't know the root cause in general, but for me the most common cause has been bugs in the TCP/IP stacks of cheap home routers.
- juliangoldsmith 12y ago
  I've had some corrupt downloads from a bad wireless NIC.

The hash is not for cryptographic purposes, it's to detect corrupt downloads. Ideally you should provide both the hash and you should pgp sign your releases.

by 4ad 12y ago

The hash is not for cryptographic purposes, it's to detect corrupt downloads. Ideally you should provide both the hash and you should pgp sign your releases.

colanderman 12y ago

That is bizarre. Like "I thought XMODEM went out of style in 1992" bizarre. Shouldn't checksumming the payload be performed at the application layer? (A quick Google shows that indeed it can be: the Content-MD5 header <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>. Wonder how widely supported it is by HTTP software used by people who like to check hashes of things they download.)

sp332 12y ago
Well what if you don't get the binary from the same source as the hash, or it's been sitting on a disk for a while and you want to double-check its integrity?
- colanderman 12y ago
  The post to which I replied specifically focused on the case where the binary came from the same source as the hash, and was being used to confirm the integrity of the download. I was not addressing other cases. (Although I will note that file systems, like application protocols, should maintain their own integrity; however most do not. Which also seems bizarre given that it's 2014.)
  - sp332 12y ago
    Other programs could, accidentally or maliciously, step on the contents of a file. Filesystem checksums are awesome (I use btrfs), but it doesn't protect your file from everything. Hm... maybe a versioned filesystem where you can go back to the "original" version of a file and verify its checksum? :)
    - colanderman 12y ago
      Yes, protecting from malicious / accidental modification is a separate problem than protecting from bit rot / gamma rays. The former is obviously much more difficult than the latter. My surprise stems from the fact that common tools are not robust against the latter by now. (My day job is working with enterprise-grade content-addressed block storage, so maybe I've just set my data-protection expectations too high.)
  - derefr 12y ago
    Indeed, I find it surprising that there's no filesystem flag giving the opposite behaviour to "compress contents to save space": something like "transparently store this file bloated by a fountain codec to ensure integrity on an bitwise-unreliable backing store" or maybe "transparently generate a Parity Volume Set from this file and stripe it across the disk to protect against lost sectors." Both of these options would protect your important files against failing single disks without having to do any RAIDing. The unimportant data (e.g. the OS itself, caches, etc.) could be reduced-redundancy, since it doesn't need to be captured in a disk rescue.
    - angry_octet 12y ago
      It would be a great feature, but obviously adds complexity. You can get half way there with single disk ZFS, which lets you store multiple copies. But the key feature is that files and directory info are checksummed, and regularly checked, so corruption cannot silently occur. But I think that usually the whole disk is lost, rather than getting bad blocks (so 90's). For any archival DVDs I burn, I compress, then run through PAR2 http://en.m.wikipedia.org/wiki/Parchive http://en.m.wikipedia.org/wiki/Parchive
4ad 12y ago
I am not so sure why you think this is bizarre. In an ideal world TCP is reliable, HTTP assumes it runs on reliable transport so it doesn't do any more integrity checking. In the real world there are bugs in TCP stacks and in HTTP implementations that cause HTTP traffic to get corrupted. I see this every day. Some applications do implement extra checking, most applications do not. Browsers and wget and curl can't implement any extra checking because the way you check is application specific. There is no standard way to do it; what you mention there is an esoteric feature. Just for anecdotal fun, Logic Pro tries to download over 50GB of assets over HTTP (each individual file is many-GB). It has never worked for me over any of my networks (and in fact I wrote a tool to fix this).
- colanderman 12y ago
  > In the real world there are bugs in TCP stacks and in HTTP implementations that cause HTTP traffic to get corrupted I'm very much aware of this. Hence why I expressed surprise that such checksumming was not commonly performed at the application layer. > There is no standard way to do it; what you mention there is an esoteric feature. The Content-MD5 header is defined in RFC 2616. It is, by definition, standard. If it's not widely supported, then I think that it would behoove the people who care about these things to switch to servers/clients which do support it. (I suspect the intersection between "people who know/care how to use md5sum" and "people who know how to set/read an arbitrary HTTP header" is fairly large. Hence my surprise at the common practice of ignoring this capability.)
adricnet 12y ago
Well, I understand your reaction, but it may help to point our that there are three different algorithms involved here and that to ensure content completeness and ensure security concerns (CIA, NR, etc) you want them all, and all done correctly: * Checksums, like in Xmodem or CRC * Cryptographic hashes (including MACs) * Cryptographic signatures (ie OpenPGP key or cert) As noted in the other comments these protect against different kinds of problems in transmission, but in used correctly in combination can protect from both glitches and active attacks. To say that it is difficult to implement all of these correctly and in concert is a grave understatement, but this is what modern crypto software and network protocols that use it, have to do. Now back to the thread on HTTP header checksums :)
- deleted 12y ago
  [deleted]
syntheticnature 12y ago
I'll note that even today "ASCII mode" in FTP haunts people who aren't careful when downloading -- and that it is a frequent reason I find myself having to urge MD5 checks of embedded firmware I work on. At least these days the firmware has enough room to validate a download itself before installing it.

sysk 12y ago

Do corrupt downloads really happen these days and what causes the downloads to become corrupt? I can't remember having a single corrupt HTTP download in the past 10 years.

4ad 12y ago
Yes, they do happen. Not as often as 20 years ago, but they still happen. Personally I can't download more than about 1GB without an error. If you use bittorrent you might want to look in the logs to see how often it re-downloads blocks because the block it just downloaded has been corrupted. You might be surprised. I don't know the root cause in general, but for me the most common cause has been bugs in the TCP/IP stacks of cheap home routers.
juliangoldsmith 12y ago
I've had some corrupt downloads from a bad wireless NIC.