3 ms·
I ran into a somewhat similar issue recently: Sometimes I work on a piece of software that classifies files based on their contents, and with so many file forma
by 13of40 8y ago
I ran into a somewhat similar issue recently: Sometimes I work on a piece of software that classifies files based on their contents, and with so many file formats implemented as "just stuff it in a zip file" it ends up identifying a lot of files by looking for well known filenames in zip archives. An annoying quirk about the zip file format is that the authoritative list of the offsets of the files in the archive is at the end of the zip file, but not at any particular offset from the end because there's an optional, arbitrarily sized comment field in it. That means the only way for a program to correctly parse it is to read backwards from the end of the file looking for a magic number. The folks who wrote the .Net API for enumerating zip entries knew about this and wrote their code accordingly.
For our file classification tool...we made it work from a System.IO.Stream, with the only requirement being that the stream was seekable, and we have it classifying hundreds of millions of filesystem-backed file streams per day, no problem.
So we recently shared it with a partner team, and all of a sudden in their environment, every once in a while they get a file that takes five minutes to identify. The difference turned out to be that their stream implementation was backed by a database, so even though it was seekable, every read turned into a database query, so if you passed in a big zip file with the directory information missing from the end, .Net would read through the entire stream backwards, looking for the magic number, one 32-byte read (database query) at a time...