3 ms·
I wondered the same thing. I wrote a basic crawler to measure JS footprint on various sites on the web. Most of the JS bloat I found was due to misconfigured we
by andrethegiant 8y ago
I wondered the same thing. I wrote a basic crawler to measure JS footprint on various sites on the web. Most of the JS bloat I found was due to misconfigured webpack configs: leaving development mode on, inline source maps, etc. Tons of whitespace and comments. I tweeted at some of the offenders (https://twitter.com/bundlebloat https://twitter.com/bundlebloat for those interested), but only a few actually fixed the problem.
- azemetre 8y agoAre you willing to post your code for your crawler some where? Sounds like a neat project, shame most people aren't fixing the problem.
- JimDabell 8y agoYou may not need a crawler for this. The HTTP Archive has already done all the crawling for a million URLs from the most popular websites. If one of their existing reports doesn't cover what you need, you can write custom queries using BigQuery: https://httparchive.org/faq#how-do-i-use-bigquery-to-write-custom-queries-over-the-data https://httparchive.org/faq#how-do-i-use-bigquery-to-write-c...
- andrethegiant 8y agoMaybe one day. Right now it's really messy. It uses crawler[0] and rethinkdb[1] FYI. [0] https://www.npmjs.com/package/crawler https://www.npmjs.com/package/crawler [1] https://rethinkdb.com https://rethinkdb.com