3 ms·

That's funny, I did the same analysis with sci-hub. Back when there was an organized drive to back it up. I downloaded parts of it and wanted to figure out why

That's funny, I did the same analysis with sci-hub. Back when there was an organized drive to back it up. I downloaded parts of it and wanted to figure out why it was so heavy, seeing as you'd expect articles to be mostly text and very light. There was a similar distribution of file sizes. My immediate instinct was also to cut off the tail-end, but looking at the larger files I realized it was a whole range of good articles that included high quality graphics that were crucial to the research being presented, not poor compression or useless bloat.

liberalgeneral 4y ago
I think Sci-Hub is the opposite since 1 DOI = 1 PDF in its canonical form (straight from the publisher) so neither duplication nor low-quality is the case.
- dredmorbius 4y ago
  It does depend on when the work was published. Pre-digital works scanned in without OCR can be larger in size. That's typically works from the 1980s and before. Given the explosion of scientific publishing, that's likely a small fraction of the archive by work though it may be significant in terms of storage.
dredmorbius 4y ago
It can be illuminating to look at the size of ePub documents. This is in general an HTML container (and compressed), such that file sizes tend to be quite small. A book-length text (~250 pp or more) might be from 0.3 -- 5 MB, and often at the lower end of the scale. Books with a large number of images or graphics, however, can still bloat to 40-50 MB or even more. Otherwise, generally, text-based PDFs (as opposed to scans) are often in the 2--5 MB range, whilst scans can run 40--400 MB. The largest I'm aware of in my own collection is a copy of Lyell's Geography, sourced from Archive.org. It is of course scans of the original 19th century typography. Beautiful to read, but a bit on the weighty side.

3 ms·

That's funny, I did the same analysis with sci-hub. Back when there was an organized drive to back it up. I downloaded parts of it and wanted to figure out why

by ad404b8a372f2b9 4y ago

liberalgeneral 4y ago
I think Sci-Hub is the opposite since 1 DOI = 1 PDF in its canonical form (straight from the publisher) so neither duplication nor low-quality is the case.
- dredmorbius 4y ago
  It does depend on when the work was published. Pre-digital works scanned in without OCR can be larger in size. That's typically works from the 1980s and before. Given the explosion of scientific publishing, that's likely a small fraction of the archive by work though it may be significant in terms of storage.
dredmorbius 4y ago
It can be illuminating to look at the size of ePub documents. This is in general an HTML container (and compressed), such that file sizes tend to be quite small. A book-length text (~250 pp or more) might be from 0.3 -- 5 MB, and often at the lower end of the scale. Books with a large number of images or graphics, however, can still bloat to 40-50 MB or even more. Otherwise, generally, text-based PDFs (as opposed to scans) are often in the 2--5 MB range, whilst scans can run 40--400 MB. The largest I'm aware of in my own collection is a copy of Lyell's Geography, sourced from Archive.org. It is of course scans of the original 19th century typography. Beautiful to read, but a bit on the weighty side.