3 ms·
So, having done some recent scraping of HN (historical front-page views only, completed in a couple of days, with only incremental updates now) ... ... and loo
by dredmorbius 3y ago
So, having done some recent scraping of HN (historical front-page views only, completed in a couple of days, with only incremental updates now) ...
... and looked at the API and finding myself somewhat underwhelmed ...
... the JSON option would be interesting (much easier to parse than raw HTML, particularly HN raw HTML). Given that there are performance concerns with the primary server already, I'd suggest:
- Put JSON on a secondary server. (Automatic redirects for, say, id=?<ID>&json=1 requests being an option to transparently serve via requests made to the main site.)
- Use some sort of account-based mechanism to regulate access. I'm thinking more in terms of "how much" and "how fast" than "who" and "what", though that might also come into play.
I also suspect that a JSON delivery would also make alternate app / front-end design even more viable, so you might want to consider that in design and provisioning considerations.
- dang 3y agoOh believe me it's never going to happen until the performance problem is fixed. That's the other thing we've been waiting on for years. It's coming though.
- klyrs 3y agoRewrite in APL?
- bombcar 3y agoCOBOL
- seanhunter 3y agoFor something like this, you need a COBOL framework you can trust. COBOL on cogs http://www.coboloncogs.org/INDEX.HTM http://www.coboloncogs.org/INDEX.HTM It's robust, ... performant... well robust.... Fully conformant to COBOL 85...