6 ms·
If you sort the numbers, then represent them as the amount you need to add to the prior number[1], it produces a very small file once compressed...about 4k[2].
by tyingq 5y ago
If you sort the numbers, then represent them as the amount you need to add to the prior number[1], it produces a very small file once compressed...about 4k[2].
So, even with the code to turn it back into a list, handle the leading zeros on many numbers, etc, it would be under 10k total, much smaller than the crazy regex.
Not general purpose, and a bit silly, but you can get the list pretty damn small.
[1] Like: "1234, 1235, 1435: becomes "1234, 1, 200"
[2] The < 4k gz file that you can reproduce most of the list from: https://github.com/tyingq/barclays/blob/main/out.c.gz?raw=true https://github.com/tyingq/barclays/blob/main/out.c.gz?raw=tr... Gunzip, then read each line, then add the prior read line to it, prepend a leading zero and print. "Prior read line" is 0 for the first line. There's a small amount of the list you would have to hardcode or handle another way...the numbers without leading zeros, total of 63 numbers.
- nerdponx 5y agoThe point is that they didn't think to do this or anything else to try to keep the bandwidth down, while still managing to deliver a lazy sloppy product that doesn't work very well for its sole purpose.
- tyingq 5y agoSure. It just made me curious what you could do. Also, there are a fair number of duplicates in Barclay's original list, so it's even more lazy an effort than that :)
- snet0 5y agoI don't even know if this qualifies as a lazy effort, as that implies some effort was put in. This was an effort-free implementation. The user, as usual, pays the price.
- AnotherGoodName 5y agoThe 1.3MB JSON file being discussed is literally sent gzipped. https://www.barclays.co.uk/content/dam/json-files/TelephoneNumberChecker_26_03_2021.json https://www.barclays.co.uk/content/dam/json-files/TelephoneN... content-encoding: gzip content-length: 186136 Don't be misled by the fact that your browser transparently hides the compression from you.
- coin 5y agoAnd possibly even smaller if their server were to support br compression
- gruez 5y ago> If you sort the numbers, then represent them as the amount you need to add to the prior number[1], it produces a very small file once compressed...about 4k[2]. That seems like a nightmare to do searches on. I'd probably compress it with a prefix tree or similar. In terms of off the shelf solutions, sorting the list, saving as a newline delimited text file, and then compressing with LZMA2 gets you down to 22KB.
- baobabKoodaa 5y agoYou don't need to do searches on the compressed JSON, you can uncompress it in memory in client-side to do searches.
- pradn 5y agoIt's easy to do searches on 74,000 numbers. Unpack them into an array and do binary search. No need for prefix trees when binary search will do just fine. Much simpler, too. The unpacking process is trivial if the list is packed using the method described above, where diffs are stored, not whole numbers.
- kccqzy 5y agoGoogle uses a similar scheme of compression in their Safe Browsing v4 protocol: https://developers.google.com/safe-browsing/v4/compression https://developers.google.com/safe-browsing/v4/compression It calculates deltas (adjacent difference) between a sorted set of numbers and then use Golomb encoding on that.