5 ms·

The author is missing an obvious point that computing was way too different at the time the format was designed. You couldn't do a "find XXX | grep YYY | zip"

The author is missing an obvious point that computing was way too different at the time the format was designed. You couldn't do a "find XXX | grep YYY | zip" pipeline; you would instead often add files to an archive one-by-one. So the format consists of appendable records sharing the same structure, and adding new files is simply a matter of appending records to the end. Now sometimes you would want to quickly locate and extract just one file out of the entire archive, and scanning the entire archive would be ridiculously slow on a system with <1MB/s throughput a some kilobytes of RAM. Hence, the central directory at the end of the archive. Some other time, the power would go down while you are appending to the archive, or a bad block would pop up on your HDD, hence redundancy between the central directory and the records. Also, writing archivers back then was very different from what we do now. There were no unit tests, no abstraction layers, and very limited debuggers. Software was hand-coded in assembly, and adding an extra field or a check here or there was much less trivial than it is now. You can always find something that was designed under completely different constraints and call it a bad design. But in reality it only shows that the author hasn't done enough research on the topic.

rbobby 5y ago
Zip files on floppy disks!
- mrlonglong 5y ago
  There were zip floppies with 120MiB which was glorious back in those days.
PaulDavisThe1st 5y ago
> You couldn't do a "find XXX | grep YYY | zip" pipeline; The format was developed in 1989, at which time find(1) and grep(1) and command lines pipes had been around for at least 10 years (more, depending on how you measure it). It wasn't that you couldn't run pipelines like this, but more that zip's creators didn't care about Unix. > Software was hand-coded in assembly I'm not sure what version of software history you've been reading, but I can assure you that I personally was not hand-coding assembly in 1989, even though I was writing device drivers, PostScript RIPs and other stuff at that time.
- sroussey 5y ago
  I did some then (video device driver), and much more of it 10 years earlier.
- throw0101a 5y ago
  > The format was developed in 1989, at which time find(1) and grep(1) and command lines pipes had been around for at least 10 years (more, depending on how you measure it). PKZIP, which originated the Zip format, was developed on MS-DOS, which did not have find(1) and grep(1), nor did it have pipes (IIRC): * https://en.wikipedia.org/wiki/PKZIP https://en.wikipedia.org/wiki/PKZIP It was a third-party who created utilities that ran under Unix-y systems: * https://en.wikipedia.org/wiki/Info-ZIP https://en.wikipedia.org/wiki/Info-ZIP Unix had its own file archive formats at the time, but it seems that folks on the PC side of things either didn't know about them, or didn't care for them: * https://en.wikipedia.org/wiki/Tar_(computing) https://en.wikipedia.org/wiki/Tar_(computing) * https://en.wikipedia.org/wiki/Ar_(Unix) https://en.wikipedia.org/wiki/Ar_(Unix) * https://en.wikipedia.org/wiki/Cpio https://en.wikipedia.org/wiki/Cpio
  - PaulDavisThe1st 5y ago
    GP said "computing was different back then", not "Zip was developed on a toy operating system that lacked these sorts of facilities". I did note that likely what mattered the most was that pkzip's creators didn't care about (or maybe even know about) Unix.
  - pcwalton 5y ago
    > PKZIP, which originated the Zip format, was developed on MS-DOS, which did not have find(1) and grep(1), nor did it have pipes (IIRC): COMMAND.COM supported find(1) in the form of DIR /S and grep(1) in the form of FIND. It also supported Unix-like pipe syntax, though because DOS wasn't multitasking it was implemented under the hood by redirecting to a temporary file and running the programs sequentially.
- TimMeade 5y ago
  I can assure you this was exactly the case. The original was done in C and then compiled. Phil then would optimize it by hand resulting in highly optimized asm code. All the DOS versions were done this way. That was the genius of Phil Katz. The first VAX version was done in C and that code was then ported to the first unix variants at which time it was moved to 32 bit and the extended attributes fields were added.
GoblinSlayer 5y ago
>scanning the entire archive would be ridiculously slow on a system with <1MB/s throughput a some kilobytes of RAM That's how tar works. Just the other day I was making a backup and streamed 5gb of data as a tar archive, opening it in the archiver took a lot of time.
- sdfjkl 5y ago
  > >scanning the entire archive would be ridiculously slow on a system with <1MB/s throughput a some kilobytes of RAM > That's how tar works. Just the other day I was making a backup and streamed 5gb of data as a tar archive, opening it in the archiver took a lot of time. That's because tar is the tape archiver, specifically written for a medium whose very nature is linear reading/writing and where skipping around was highly impractical or even impossible.
  - codedokode 5y ago
    But having a file directory would be useful even with a tape because you wouldn't need to read the whole tape to get the list of files inside.
- josefx 5y ago
  Could that just be the archiver? I often skip UIs since they seem to take longer to show the file contents than it takes me to unpack the entire archive on the command line. Worse I am convinced some UIs end up unpacking the entire archive repeatedly when you select only two or three files.
  - account42 5y ago
    No, tar does not have a central directory record so you need to skip over all data even just to list files. Of course you can have additionall slowdowns on top of that by doing stupid things.
edflsafoiewq 5y ago
None of the author's suggestions are incompatible with those constraints.
pjmlp 5y ago
Sure you could, even though MS-DOS wasn't UNIX, and using pipelines meant writing temporary data into the current directory. In fact that was one of the reasons ARJ got more love than zip, given its flexibility and ability to split archives across floppies. Unless you were coding COM files, demoscene or commercial games, there was enough software being written in Turbo and Quick Basic, Turbo/Quick/TMT Pascal, Clipper/FoxPro/DBase III, Turbo/Quick C/C++, Turbo/TMT Modula-2,....
- hnlmorg 5y ago
  > Sure you could, even though MS-DOS wasn't UNIX, and using pipelines meant writing temporary data into the current directory. Assuming you're commenting on the GP's point about UNIX pipelines, I think your comment here is a little disingenuous. Akin to that famous HN comment mocking the then start up Dropbox saying "you can just use FTP...." while completely missing the reason Dropbox would then go on to be successful for. > In fact that was one of the reasons ARJ got more love than zip, given its flexibility and ability to split archives across floppies. Did ARJ actually get more love than ZIP or is that just your anecdotal recall of the era. I seem to recall the opposite with ZIP being far more prevolent. Remember ZIP also supported splitting archives over multiple floppies. Interestingly ARJ's website is still live and not been updated in 10 years[1]. It might be ugly by today's standards but man is it better designed for getting the core information out. We've definitely lost something when sites became more about presentation and less about information. [1] http://www.arjsoftware.com/arj.htm http://www.arjsoftware.com/arj.htm
  - tored 5y ago
    What I can remember from the 90s in Sweden the most common archive formats for spanning multiple 3 1/2 inch floppies was ARJ or RAR. ZIP existed but was uncommon in comparison. I think it wasn’t so much about the qualities about the zip format itself rather that WinZip software wasn’t as good compared to WinRAR. ARJ was command line, at least that is how I used it, it was well documented and easy to use. It may have been that the zip format was more common in the DOS 5 1/4 inch floppy days because zip (1989) existed prior to both both ARJ (1992) and RAR (1993)
PinguTS 5y ago
It seems you didn't bother to read the article completely, because the author also goes into the reasoning of this problem: > You might think this is nonsense but you have to remember, pkzip comes from the era of floppy disks. Reading an entire zip file's contents and writing out a brand new zip file could be an extremely slow process. In both cases, the ability to delete a file just by updating the central directory, or to add a file by reading the existing central directory, appending the new data, then writing a new central directory, is a desirable feature. This would be especially true if you had a zip file that spanned multiple floppy disks; something that was common in 1989. You'd like to be able to update a README.TXT in your zip file without having to re-write multiple floppies.
inglor_cz 5y ago
I think the point of the article is that we are now stuck with a very technically obsolete format that no longer reflects the wants and needs of our environment - and that, in the future, anyone who designs a new format (which, in the HN community, some people actually might do), should think a bit more generously than the creator of ZIP. For example, the need to be able to skip unknown blocks is fairly important and imcompatibility of parsers was a known technical problem in the early 1980s already.
bluedino 5y ago
>> The author is missing an obvious point that computing was way too different at the time the format was designed. I think the author is aware, at the time that PkZIP was created he was writing NES games and using data compression to fit more content into the games. https://games.greggman.com/game/programming_m_c__kids/ https://games.greggman.com/game/programming_m_c__kids/

rbobby 5y ago

Zip files on floppy disks!

mrlonglong 5y ago
There were zip floppies with 120MiB which was glorious back in those days.

PaulDavisThe1st 5y ago

> You couldn't do a "find XXX | grep YYY | zip" pipeline; The format was developed in 1989, at which time find(1) and grep(1) and command lines pipes had been around for at least 10 years (more, depending on how you measure it). It wasn't that you couldn't run pipelines like this, but more that zip's creators didn't care about Unix. > Software was hand-coded in assembly I'm not sure what version of software history you've been reading, but I can assure you that I personally was not hand-coding assembly in 1989, even though I was writing device drivers, PostScript RIPs and other stuff at that time.

sroussey 5y ago
I did some then (video device driver), and much more of it 10 years earlier.
throw0101a 5y ago
> The format was developed in 1989, at which time find(1) and grep(1) and command lines pipes had been around for at least 10 years (more, depending on how you measure it). PKZIP, which originated the Zip format, was developed on MS-DOS, which did not have find(1) and grep(1), nor did it have pipes (IIRC): * https://en.wikipedia.org/wiki/PKZIP https://en.wikipedia.org/wiki/PKZIP It was a third-party who created utilities that ran under Unix-y systems: * https://en.wikipedia.org/wiki/Info-ZIP https://en.wikipedia.org/wiki/Info-ZIP Unix had its own file archive formats at the time, but it seems that folks on the PC side of things either didn't know about them, or didn't care for them: * https://en.wikipedia.org/wiki/Tar_(computing) https://en.wikipedia.org/wiki/Tar_(computing) * https://en.wikipedia.org/wiki/Ar_(Unix) https://en.wikipedia.org/wiki/Ar_(Unix) * https://en.wikipedia.org/wiki/Cpio https://en.wikipedia.org/wiki/Cpio
- PaulDavisThe1st 5y ago
  GP said "computing was different back then", not "Zip was developed on a toy operating system that lacked these sorts of facilities". I did note that likely what mattered the most was that pkzip's creators didn't care about (or maybe even know about) Unix.
- pcwalton 5y ago
  > PKZIP, which originated the Zip format, was developed on MS-DOS, which did not have find(1) and grep(1), nor did it have pipes (IIRC): COMMAND.COM supported find(1) in the form of DIR /S and grep(1) in the form of FIND. It also supported Unix-like pipe syntax, though because DOS wasn't multitasking it was implemented under the hood by redirecting to a temporary file and running the programs sequentially.
TimMeade 5y ago
I can assure you this was exactly the case. The original was done in C and then compiled. Phil then would optimize it by hand resulting in highly optimized asm code. All the DOS versions were done this way. That was the genius of Phil Katz. The first VAX version was done in C and that code was then ported to the first unix variants at which time it was moved to 32 bit and the extended attributes fields were added.

GoblinSlayer 5y ago

>scanning the entire archive would be ridiculously slow on a system with <1MB/s throughput a some kilobytes of RAM That's how tar works. Just the other day I was making a backup and streamed 5gb of data as a tar archive, opening it in the archiver took a lot of time.

sdfjkl 5y ago
> >scanning the entire archive would be ridiculously slow on a system with <1MB/s throughput a some kilobytes of RAM > That's how tar works. Just the other day I was making a backup and streamed 5gb of data as a tar archive, opening it in the archiver took a lot of time. That's because tar is the tape archiver, specifically written for a medium whose very nature is linear reading/writing and where skipping around was highly impractical or even impossible.
- codedokode 5y ago
  But having a file directory would be useful even with a tape because you wouldn't need to read the whole tape to get the list of files inside.
josefx 5y ago
Could that just be the archiver? I often skip UIs since they seem to take longer to show the file contents than it takes me to unpack the entire archive on the command line. Worse I am convinced some UIs end up unpacking the entire archive repeatedly when you select only two or three files.
- account42 5y ago
  No, tar does not have a central directory record so you need to skip over all data even just to list files. Of course you can have additionall slowdowns on top of that by doing stupid things.

edflsafoiewq 5y ago

None of the author's suggestions are incompatible with those constraints.

pjmlp 5y ago

Sure you could, even though MS-DOS wasn't UNIX, and using pipelines meant writing temporary data into the current directory. In fact that was one of the reasons ARJ got more love than zip, given its flexibility and ability to split archives across floppies. Unless you were coding COM files, demoscene or commercial games, there was enough software being written in Turbo and Quick Basic, Turbo/Quick/TMT Pascal, Clipper/FoxPro/DBase III, Turbo/Quick C/C++, Turbo/TMT Modula-2,....

hnlmorg 5y ago
> Sure you could, even though MS-DOS wasn't UNIX, and using pipelines meant writing temporary data into the current directory. Assuming you're commenting on the GP's point about UNIX pipelines, I think your comment here is a little disingenuous. Akin to that famous HN comment mocking the then start up Dropbox saying "you can just use FTP...." while completely missing the reason Dropbox would then go on to be successful for. > In fact that was one of the reasons ARJ got more love than zip, given its flexibility and ability to split archives across floppies. Did ARJ actually get more love than ZIP or is that just your anecdotal recall of the era. I seem to recall the opposite with ZIP being far more prevolent. Remember ZIP also supported splitting archives over multiple floppies. Interestingly ARJ's website is still live and not been updated in 10 years[1]. It might be ugly by today's standards but man is it better designed for getting the core information out. We've definitely lost something when sites became more about presentation and less about information. [1] http://www.arjsoftware.com/arj.htm http://www.arjsoftware.com/arj.htm
- tored 5y ago
  What I can remember from the 90s in Sweden the most common archive formats for spanning multiple 3 1/2 inch floppies was ARJ or RAR. ZIP existed but was uncommon in comparison. I think it wasn’t so much about the qualities about the zip format itself rather that WinZip software wasn’t as good compared to WinRAR. ARJ was command line, at least that is how I used it, it was well documented and easy to use. It may have been that the zip format was more common in the DOS 5 1/4 inch floppy days because zip (1989) existed prior to both both ARJ (1992) and RAR (1993)

PinguTS 5y ago

It seems you didn't bother to read the article completely, because the author also goes into the reasoning of this problem: > You might think this is nonsense but you have to remember, pkzip comes from the era of floppy disks. Reading an entire zip file's contents and writing out a brand new zip file could be an extremely slow process. In both cases, the ability to delete a file just by updating the central directory, or to add a file by reading the existing central directory, appending the new data, then writing a new central directory, is a desirable feature. This would be especially true if you had a zip file that spanned multiple floppy disks; something that was common in 1989. You'd like to be able to update a README.TXT in your zip file without having to re-write multiple floppies.

inglor_cz 5y ago

I think the point of the article is that we are now stuck with a very technically obsolete format that no longer reflects the wants and needs of our environment - and that, in the future, anyone who designs a new format (which, in the HN community, some people actually might do), should think a bit more generously than the creator of ZIP. For example, the need to be able to skip unknown blocks is fairly important and imcompatibility of parsers was a known technical problem in the early 1980s already.

bluedino 5y ago

>> The author is missing an obvious point that computing was way too different at the time the format was designed. I think the author is aware, at the time that PkZIP was created he was writing NES games and using data compression to fit more content into the games. https://games.greggman.com/game/programming_m_c__kids/ https://games.greggman.com/game/programming_m_c__kids/