3 ms·
Very cool tool. Useful too. =] Framework isn't the issue in this case, it just looks like you're loading all 100 pages into memory at one time? Or keeping the
by mitchellh 17y ago
Very cool tool. Useful too. =]
Framework isn't the issue in this case, it just looks like you're loading all 100 pages into memory at one time? Or keeping the previous pages in memory as you go "deeper" into the hierarchy.
I recommend you just keep a few pages in memory at any given time, keeping a simple list of the URLs scraped out of that HTML. If you're using PHP remember to unset() the variables storing the HTML and so on.
I haven't used PHP in a long time and of course I'm just speculating based on the error message, so slight disclaimer there.
- aarongough 17y agoYeah I'm pretty sure that I was storing all of the raw HTML for possible further analysis down the line. I haven't even looked at the code in months but it definitely needs over-hauling! Initially it started out as a 'what if I do this' kind of app. I would like to re-write the backend with more attention to detail...
- aarongough 17y agoI also meant to say: re: "Framework isn't the issue in this case" I agree. It is possible to turn out nice code in PHP and write fast, reliable apps. That being said Rails has me addicted to the idea of unit tests within easy reach. I'm not exactly a massive fan of any of the unit testing systems that I have seen for PHP and that is my main reason for wanting to switch away in this case... I use PHP all the time, I am just a little dissatisfied with it in light of the alternatives that are out there.