6 ms·
Show HN: Page.REST – An API to fetch details from a web page as JSON
- ganessh 9y agoIf I have to know the elements' selectors, why should I prefer this service over using a HTML parser?
- AznHisoka 9y agoYeah, It kind of defeats the point/value prop of an API.
- laktek 9y agoIt depends on the use case. Say you want to get all images on a page you'd just use `img`. If you are going to find JSON-LD blocks defined in a set of pages, you can use - `script[type='application/ld+json']`. For MVP, I wanted to give the option of free-form selectors and then offer more targeted queries based on what people use.
- icebraining 9y agoConsider offering responses other formats, possibly protobuf. Then you can sell it as a solution for limited IoT devices where a full HTML parser is too much. If your service avoids having to host a custom server, it could be very useful.
- squiggy22 9y agoNot unlike YQL - https://developer.yahoo.com/yql/ https://developer.yahoo.com/yql/
- geetfun 9y agoLooks interesting. I wonder what kind of market this app might serve. For larger apps, I would worry about support. 5 dollars per year tells me that the developer is doing this as a hobby. For small side projects, I can see tinkerers building this themselves.
- laktek 9y agoIt's actually a utility I built for my own use (I bootstrap Pragma full-time). I thought there might be an interest among other developers for a similar utility. That's why I thought to put it out (wanted to charge $5 so I can reduce abuse). If you are interested in using it for a larger app and concerned about support, please drop me an email (see profile). I can set you up an instance with better rate limits and service gurantees.
- geetfun 9y agoGood to know! Thanks!
- cinooo 9y agoSomeone could really abuse this service, I don't see any mention of API limits.
- laktek 9y agoThe idea of charging $5 was a stop gap measure to prevent abuse. There are limits set for access token too. I actually rushed to ship before putting out a TOS which mentions limits.
- kowdermeister 9y agoI would make the 5$ price and token validity much larger, like "4rem" or something, I was looking at the CC input field and thinking "seriously? how much will you charge?"
- laktek 9y agoI bolded it for now. Need to rethink the presentation.
- xytop 9y agoFor absolutely 0-cost you can use http://proc.link http://proc.link which will return oembed info for ANY url. General example: http://api.proc.link/oembed?url=http%3A%2F%2Fpage.rest http://api.proc.link/oembed?url=http%3A%2F%2Fpage.rest Youtube: http://api.proc.link/oembed?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdA5qYrboTUE http://api.proc.link/oembed?url=https%3A%2F%2Fwww.youtube.co... Facebook: http://api.proc.link/oembed?url=https%3A%2F%2Fwww.facebook.com%2Fbritneyspears%2Fposts%2F10155665586428234 http://api.proc.link/oembed?url=https%3A%2F%2Fwww.facebook.c...
- afshinmeh 9y agoVery neat. But the question is, how they pay the cost of the server? I mean, do you think a web service without any API keys, subscriptions, etc. is reliable?
- xytop 9y agoIt's my own server and I'm the developer of the service ;) I pay $3/mo for hosting and $10/y for domain so it doesnt cost me much. Regarding reliability: all the code is opensourced and if you want to use the service in production - you can run this on your own server.
- have_faith 9y agoWhat levels of traffic does your server see on a normal day? has it ever gone down from a spike?
- xytop 9y agoUsual traffic is 0-1 req/sec. Even now it is about 2-3 req/sec. It is running for more than 2 years now, never failed :) Golang is pretty stable. Also the server is behind Cloudflare which caches responses and protects from DDoS, this can be another reason for low RPS to the server.
- 9y ago
- keyle 9y agoI did something similar a loooong time ago. Granted not as sexy. https://github.com/keyle/json-anything https://github.com/keyle/json-anything
- Luyanda 9y agoUhm this is interesting. Reminds me of: https://wrapapi.com https://wrapapi.com
- chenster 9y agoZapier it.
- janober 9y agoIf you are interested the same is possible with https://link.fish/hn-early-access/ https://link.fish/hn-early-access/ which has a Zapier integration. It allows to send an URL and it then returns its data. However, you would have to send me an email to info@link.fish because the integration is still in beta mode and so only available after invitation.
- andrevv_6 9y agoIs there a await for JS Frontends to initialise before the scrape via a selector (i.e. Angular/Ember etc)?
- laktek 9y agoIt runs page scripts and wait after DOMContentLoaded. But may not be accurate in some cases. Do you have any examples to try?
- throwaway12s34 9y agoHow do you handle sites that have scraper prevention? Such as captcha, IP throttling, etc. I'd pay $100-$500 per-month for a service that could reliably scrape some particularly difficult sites. With that being said, I'd need the service to be able to handle ~100 req/s in bursts and 2-4 req/s on average.
- laktek 9y agoAdded support for OpenGraph extraction https://www.page.rest/#open-graph https://www.page.rest/#open-graph
- laktek 9y agoI wrote a follow up blog post about what I learnt from shipping Page.REST http://www.laktek.com/what-i-learned-from-building-pagerest/ http://www.laktek.com/what-i-learned-from-building-pagerest/