3 ms·
The Readme says "Warning that it can take a long time: between 20 to 30 days." How in the world can it take so long time? The csv file seems to be 24mb in size
by capableweb 7y ago
The Readme says "Warning that it can take a long time: between 20 to 30 days."
How in the world can it take so long time? The csv file seems to be 24mb in size and the computation performed can't be that advanced. Did the author do something seriously wrong?
- detaro 7y agoIt just processes everything sequentially.
- capableweb 7y agoI saw that it's actually going through all the website urls and makes a request for each. To be optimisitic, each request takes 1 second, that's 1m seconds. That's ~10 days, so doesn't seem unrealistic it would actually take ton of time.
- buboard 7y ago> To run it you'll need either 2Gb of RAM What? I guess this is a toy program used to learn clojure or sth - it even uses sed for line parsing. A 10-line php script could do the same with a few MB of RAM
- Something1234 7y agoWhat is sth? I do have to agree this sounds relatively inefficient, especially considering you could just immediately flush the result out to disk.
- tanrax 7y agoI agree, even a Bash script would be more efficient. Everything was born as an exercise to learn Clojure. I can tell you that it actually uses 1.1Gb, but for safety I recommend twice as much. Why does it use so much RAM? Ask Java :) (I am the author of the script)
- eugenekolo2 7y agoPerforming a web request to 1M targets takes a long time. However, this is an [embarassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel) problem, and renting out some machines would speed it up.