4 ms·

I also had the idea of zip bomb to confuse badly behaved scrapers (and I have mentioned it before to some other people, although I did not implemented it). Howe

I also had the idea of zip bomb to confuse badly behaved scrapers (and I have mentioned it before to some other people, although I did not implemented it). However, maybe instead of 0x00, you might use a different byte value. I had other ideas too, but I don't know how well some of them will work (they might depend on what bots they are).

ycombinatrix 1y ago
The different byte values likely won't compress as well as all 0s unless they are a repeating pattern of blocks. An alternative might be to use Brotli which has a static dictionary. Maybe that can be used to achieve a high compression ratio.
- zzo38computer 1y ago
  I meant that all of the byte values would be the same (so they would still be repeating), but a different value than zero. However, Brotli could be another idea if the client supports it.
- dspillett 1y ago
  Compressing a sequence of any single character should give almost identical results length-wise (perhaps not exactly identical, but the difference will be vanishingly small). For example, with gzip using default options: me@here:~$ pv /dev/zero -s 10M -S | gzip -c | wc -c 10.0MiB 0:00:00 [ 122MiB/s] [=============================>] 100% 10208 me@here:~$ pv /dev/zero -s 100M -S | gzip -c | wc -c 100MiB 0:00:00 [ 134MiB/s] [=============================>] 100% 101791 me@here:~$ pv /dev/zero -s 1G -S | gzip -c | wc -c 1.00GiB 0:00:07 [ 135MiB/s] [=============================>] 100% 1042069 me@here:~$ pv /dev/zero -s 10M -S | tr "\000" "\141" | gzip -c | wc -c 10.0MiB 0:00:00 [ 109MiB/s] [=============================>] 100% 10209 me@here:~$ pv /dev/zero -s 100M -S | tr "\000" "\141" | gzip -c | wc -c 100MiB 0:00:00 [ 118MiB/s] [=============================>] 100% 101792 me@here:~$ pv /dev/zero -s 1G -S | tr "\000" "\141" | gzip -c | wc -c 1.00GiB 0:00:07 [ 129MiB/s] [=============================>] 100% 1042071 Two bytes difference for a 1GiB sequence of “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa…” (\141) compared to a sequence of \000.

ycombinatrix 1y ago

The different byte values likely won't compress as well as all 0s unless they are a repeating pattern of blocks. An alternative might be to use Brotli which has a static dictionary. Maybe that can be used to achieve a high compression ratio.

zzo38computer 1y ago
I meant that all of the byte values would be the same (so they would still be repeating), but a different value than zero. However, Brotli could be another idea if the client supports it.
dspillett 1y ago
Compressing a sequence of any single character should give almost identical results length-wise (perhaps not exactly identical, but the difference will be vanishingly small). For example, with gzip using default options: me@here:~$ pv /dev/zero -s 10M -S | gzip -c | wc -c 10.0MiB 0:00:00 [ 122MiB/s] [=============================>] 100% 10208 me@here:~$ pv /dev/zero -s 100M -S | gzip -c | wc -c 100MiB 0:00:00 [ 134MiB/s] [=============================>] 100% 101791 me@here:~$ pv /dev/zero -s 1G -S | gzip -c | wc -c 1.00GiB 0:00:07 [ 135MiB/s] [=============================>] 100% 1042069 me@here:~$ pv /dev/zero -s 10M -S | tr "\000" "\141" | gzip -c | wc -c 10.0MiB 0:00:00 [ 109MiB/s] [=============================>] 100% 10209 me@here:~$ pv /dev/zero -s 100M -S | tr "\000" "\141" | gzip -c | wc -c 100MiB 0:00:00 [ 118MiB/s] [=============================>] 100% 101792 me@here:~$ pv /dev/zero -s 1G -S | tr "\000" "\141" | gzip -c | wc -c 1.00GiB 0:00:07 [ 129MiB/s] [=============================>] 100% 1042071 Two bytes difference for a 1GiB sequence of “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa…” (\141) compared to a sequence of \000.