7 ms·
Show HN: Jam API, turn any site into a JSON api using CSS selectors
- dimino 10y agoHow will someone like CloudFlare stop a tool like this from scraping their customer's sites? Just blocking the tool's IP?
- brbsix 10y agoCloudFlare will make sure the browser can run JS, which in the case of this service I assume it won't. There are ways around this of course, using headless browsers (e.g. PhantomJS), tools like cloudflare-scrape[0] (which uses PyExecJS[1]). I've even used PyQt5 to render webpages for similar purposes. Unfortunately the aforementioned tools are generally pretty slow (especially headless browsers). Also I can't imagine it's particularly safe running such a service. [0]: https://github.com/Anorov/cloudflare-scrape/ https://github.com/Anorov/cloudflare-scrape/ [1]: https://github.com/doloopwhile/PyExecJS https://github.com/doloopwhile/PyExecJS
- amelius 10y agoIsn't this exactly what XML (or for that matter XHTML) was supposed to do?
- smadge 10y agoOr I feel like anything surrounding Linked Data, Semantic Web, RDF, RDFa, microformats, etc.
- OJFord 10y agoYes, yes, yes! I'm using Apifier at the moment, which I really like, but my biggest gripe is the awkwardness of source (and VCS) integration. The best I've come up with is to export the JSON config (which contains the scraper source code as a value - yuck) and try to remember to keep re-exporting and checking it in. Having also had to hack around the inability to parameterise the scrape url (e.g. 'profile/$username') - which they've since added support for - I started to wonder if I mightn't as well just use BeautifulSoup (Python HTML parser lib) and check it in properly. This is probably my ideal. I can keep it all in source control because it's just an HTTP request body, and I can parameterise it because, well, it's just an HTTP request body! It's also open source because you're an amazing person; so if I had one little concern left about the availability of your site I can dismiss it right away since I could run my own on Heroku should jamapi.xyz prove unsustainable. It's possibly a better idea to do that anyway, but I often wonder if Heroku doesn't consider that a problem - multiple instances of the same app running on free dynos under different accounts...
- paulmd 10y agoI've been thinking about writing some website-to-JSON scrapers myself and this basically solves that problem (since I would have been going after CSS selectors or xpath anyway myself). Nice job.
- joelbondurant 10y agoThis!... is why we can't have nice things.
- jstanley 10y agowith curl: $ curl -d url=https://news.ycombinator.com/ -d json_data='{"title":"title"}' http://www.jamapi.xyz/ => { "title": "Hacker News" } Also, the Ruby example appears to post to the wrong URL?
- gavino 10y agoAh, yep, you're right, forgot to change the URL. Updated now. Thanks for letting me know.
- jstanley 10y agoAnd to get the HN post titles: curl -d url=https://news.ycombinator.com/ https://news.ycombinator.com/ -d json_data='{"title":[{"elem":".title > a","value":"text"}]}' http://www.jamapi.xyz/ http://www.jamapi.xyz/ This is cool :) EDIT: Incidentally, you don't really need to have that "index" key inside the values of an array, because in an array the order is preserved anyway. Unless I've misunderstood what it means?
- gavino 10y agoRegarding the "index" key, there are some JSON parsers for languages like Swift that will rearrange your JSON. By adding the index key, you'll still be able to sort after parsing. Also, thanks, it's really cool to see people liking this :)
- JelteF 10y agoThey might rearrange keys in a JSON object, but in an array they should be preserved in order as according to the spec[1]. If Swift does this (which I can't really check) than this would be a bug. [1] http://www.json.org/ http://www.json.org/: An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
- chriswarbo 10y ago
- deleted 10y ago[deleted]
- fryiee 10y agoGreat! I've been trying to get my head around Scrapy, and I have little Python experience. This seems to fit in a lot better with my skillset for the project I'm working on.
- WA 10y agoHTTPS results in 500 Internal Server Error. Edit: Well no, it's only some sites. E. g. https://medium.com https://medium.com
- danielsamuels 10y agoA CORS restriction perhaps?
- learned 10y agoI'm also getting 500s. Looks to be a CORS issue. I've tried about a dozen big name sites so far without any luck. Nonetheless, a great idea! Edit: Just tried it in Node and it works brilliantly. Cool project.
- diggan 10y agoIf you're running the example on the website/in a browser, it's probably CORS stopping you. Try using a backend language or just curl and it should be fine.
- WA 10y agoWell no, because my browser isn't doing the request. The underlying Node app (the Jam API) does it. I found it: The API responds with an HTTP 500 error if you use CSS selectors that don't select anything or are simply invalid. Probably makes sense to add some Exception handling right there.
- gavino 10y agoI had been trying to figure out what would be causing this issue, thanks for pointing it out, I've pushed a fix real quick that will respond whether JSON is invalid or a CSS selector wasn't found on the provided URL.
- deleted 10y ago[deleted]
- adriancooney 10y agoThis is a fantastic idea and I'm really surprised nothing like this has existed before, it seems like such a no-brainer. Great work.
- xaduha 10y agoUsing a 3rd party site to query HTML (which you should be able to do yourself, plenty of tools for that) isn't a fantastic idea. This one, for example http://www.videlibri.de/xidel.html#examples http://www.videlibri.de/xidel.html#examples
- martinvol 10y agoThe code is on Github, you can use this as a library, not as a 3rd party saas.
- deleted 10y ago[deleted]
- fizx 10y agohttps://github.com/fizx/parsley/wiki https://github.com/fizx/parsley/wiki looks pretty similar. Running this sort of thing as a service/api never panned out for us because you are almost universally robots.txt denied and/or blocked. We briefly tried, and supported a wiki of json extraction scripts at parselets.org, but it went nowhere after a few months.
- mappy 10y ago> I'm really surprised nothing like this has existed before But how would you monetize it? Unlike an RSS feed, you really don't know how the JSON response would be used, so you can't inject ads into it. And if you charge for it, wouldn't people assume it would continue to work, but site "scrapers", regardless of how they are configured, are likely to break, so it would be tougher having customers pay for something that could break at any time leaving them having to figure out if its the service that's changed/broken or the page that's changed. Don't get me wrong- some great businesses have been/are based on "scraping" in one way or another. However, as cool as this is, it's just another way to "scrape". If the person hosting the page would provide an API or JSON view instead, you'd be loads better off.
- NicoJuicy 10y agoI'm actually wondering if it would be possible to add forms authentication to this? Eg. Post with some sort of css selecters and then a "cookie memory".
- OJFord 10y agoIt would be possible to, of course. But you'd surely want to host it yourself.
- thomasahle 10y agoWhat do you think would be a good syntax for enabling following links? Say I wanted the Hacker News links + first comment?
- fizx 10y agoI wrote a language that's basically a superset of this (https://github.com/fizx/parsley/wiki https://github.com/fizx/parsley/wiki) back in 2008 and used it to crawl a variety of insane job posting sites. As crawling complexity increases, pretty soon you want an actual programming language to specify things like crawl order and cache behavior. Multi-page behavior was very hard to describe declaratively for misbehaving sites. Also, it's a terrible default (for security reasons) to let the web pages you're parsing automagically initiate new requests to arbitrary urls. Such as it is, I believe that the following works in some version of parsley, though I doubt its an official release. { "articles": [ { "title": ".title a", "comment_link": "follow(.subtext a:nth-child(3) @href) .athing:nth-child(1) .default" } ] } At some point, these json things might as well be as readable as regex :/
- jerf 10y ago"At some point, these json things might as well be as readable as regex :/" Don't feel :/ . The complexity is essential, and located in the remote website, not your code or your ideas. You still win isolating all the nasty stuff to one and precisely one location. :/ is on them, not you!
- thomasahle 10y ago> Also, it's a terrible default (for security reasons) to let the web pages you're parsing automagically initiate new requests to arbitrary urls. Right. We'd have to only grab the article-id, validate that it is in fact an interger in the right range, and only then piece the url back together and request it. On the other hand, maybe just checking that we stay within the domain is enough. If the website wants to screw with us, they can send us any reply they want to any url anyway.
- daw___ 10y agoWonderful idea. What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?
- gavino 10y agoIt doesn't currently do that, I think it'd be an interesting challenge to try and do that though. It's definitely possible to do.
- daw___ 10y agoYep. Have a look at phantomjs [1], or other phantomjs wrappers like casperjs [2]. [1] https://www.npmjs.com/package/phantomjs https://www.npmjs.com/package/phantomjs [2] https://www.npmjs.com/package/casperjs https://www.npmjs.com/package/casperjs
- etatoby 10y ago> interesting challenge Understatement of the year. You'd need to either re-implement an entire browser stack or run a headless version of gecko of webkit server-side. The former entails millions of man-hours of work. The latter opens up your server to all sorts of exploits. Overall a really bad idea. Besides, single page applications are the worst junk in the entire Web 2.0 cesspool. If you really need to scrape them, they usually come with their own JSON API which you can just piggyback.
- OJFord 10y ago> entails millions of man-hours of work Overstatement of the year. Why on Earth would the OP start from scratch? Besides, though not a solo and OSS effort, Apifier does this; certainly without "millions" of hours having been spent on it.
- staticelf 10y agoI just get "invalid json" when I try to use the form on the page.
- bartkappenburg 10y agoOT perhaps: I'm still looking for a solution that has a graphical UI that allows users to point and click an element on their page and returns the corresponding CSS-selector. SelectorGadget does this as a chrome-extension but I'm looking for something that works without an extension.
- dkopi 10y agoChrome Developer tools. Inspect an element to get it in the elements tab. Right click the element's HTML, copy -> copy selector. #hnmain > tbody > tr:nth-child(3) > td > table > tbody > tr.athing > td.default
- bartkappenburg 10y agoExplain that to a small business owner (our customers) using IE of Safari. ;-)
- toupeira 10y agoWhy not make a screencast?
- fizx 10y agoAFAIK, Selectorgadget's chrome extension is just a wrapper around the bookmarklet. It's pure JS, doesn't use any sort of elevated privileges, and is MIT licensed so you can include the core engine in your own projects.
- hegivor 10y agoFlutter Selector(https://flutter.social/bookmarklet https://flutter.social/bookmarklet) seems to do exactly what you want.
- lost_my_pwd 10y agoA little word of warning/encouragement. I did something similar a long time ago (JSONDuit), which got posted to HN by someone else. You will probably run into a healthy mix of "that's cool" / "I did that before you!" / "but how will it make money?". Ignore it and do your thing. If you figure out how to monetize it, great! Even if you don't or if you have no desire to, you will have learned and grown during the course of the project. That is invaluable. Have fun and screw the haters...
- xaduha 10y agoI find this attitude "shoot first (write code), ask questions later" as something to be admired and a bit worrisome at the same time. Nothing against people learning stuff, but why does it have to be promoted this way? Lack of humility is what gets me. Maybe I'm just jealous or something, but it rubs me the wrong way.
- iamben 10y agoI think it's less about promotion and more about feedback. "Here's what I've built, what do you think?" We're meant to be an inclusive community of smart people. The idea is we'll encourage the poster and offer constructive criticism (or praise). If the post is useful to no one, it simply won't get discussed or upvoted. When something does, it's validated as an idea, or as something of interest.
- xaduha 10y ago"Just because you could, doesn't mean you should" - that phrase should have been applied to both writing the software and posting about it here. Plenty of mediocre stuff gets to the frontpage and plenty of gems fall through the cracks.
- barranger 10y agoPerhaps your view of what's "mediocre" and what's a "gem" is not consistent with the views of Hacker News readers at large
- chriswarbo 10y agoVery nice idea. Although scraping should always be a last resort, I could imagine using this for semi-serious purposes, i.e. when I care enough about the output, will be doing many requests, don't mind relaying data via a third-party, etc. I currently do quite a bit of scraping for my own use (generating RSS feeds for sites, making simple commandline interfaces to automate common tasks, etc.). I've found xidel to be pretty good for this: it starts off pretty simple (e.g. with CSS selectors or XPath), but gets pretty gnarly for semi-complicated things. For example, it allows templating the output, using a language I struggle to grasp. This service seems to address that middle ground, e.g. restricting its output to JSON, and hence making the specification of the output much simpler (a nice JSON structure, rather than messing around with splicing text together).
- nsgi 10y agoIf it's going to be used for serious purposes it really needs HTTPS support, as most APIs do these days.
- splatcollision 10y agoNice work, thanks for adding the Github link. I can think of lots of immediate use for this. Consider publishing on NPM?
- ptwt 10y agoI put this similar project[0] together a while ago. Almost the same concept, but I skipped the json layer altogether as I just wanted a quick way of getting nuggets of content from webpages into my terminal. For example: curl https://news.ycombinator.com/news | tq -tj ".title a" 0. https://github.com/plainas/tq https://github.com/plainas/tq
- finnn 10y agoThat's awesome. Like jq but for html.
- karlcoelho1 10y agoIf anyone remembers, they was a YC company that did exactly this. It was called Kimono Labs. I think it failed and just got acquired a year ago. "Jam API" will probably do way better because, well, open source.
- uberneo 10y agohttp://blog.webkid.io/nodejs-scraping-libraries/ http://blog.webkid.io/nodejs-scraping-libraries/ -- Good scraping options in NodeJS .. my personal favourite is https://github.com/rc0x03/node-osmosis https://github.com/rc0x03/node-osmosis
- deleted 10y ago[deleted]
- MetaMetaApplyHN 10y agoDoes anyone have any information on anyone that's used HTTP as an API to share/create metadata for any transactions, content, etc. publicly online? I would very curious to know about it! Welcome feedback on my "Apply HN" on doing exactly this: https://news.ycombinator.com/item?id=11583348 https://news.ycombinator.com/item?id=11583348
- Mahn 10y agoJust a heads up, "Apply HN" is for built products/services, not ideas.
- joachimm 10y agoFantastic, reminds me of Yahoo Pipes
- loisaidasam 10y agoMight be helpful to have the example execute inline so you can see what's going on/experiment without having to leave the page.
- denishaskin 10y agoApplication Error An error occurred in the application and your page could not be served. Please try again in a few moments. If you are the application owner, check your logs for details.
- soheil 10y agoI think with advent of tools like this developers more and more will be thinking of ways to make it hard to have someone scrape their website into data structures. I wonder if we are going to see the same thing that happened to minimized js happening to html more and more. I know there are sites that dynamically change css class names and ids. But I think soon we will also see div hierarchies to dynamically change form without presentationally looking different to the end user.
- smadge 10y agoThat would be bad for the web. DRM and the web are incompatible concepts.
- smadge 10y agoI wish site publishers annotated their markup with RDFa tags so every web page was already an "api"