8 ms·

Show HN: ScreenSlicer – Automatic, zero-config web scraping

26 points by logn 12y ago

ivan_ah 12y ago
Is there part of this that I could use for "main text content" detection on generic web pages? (i.e. not the search results listings)
- logn 12y ago
  I recommend Boilerpipe, which happens to be embedded in ScreenSlicer too: https://code.google.com/p/boilerpipe/ https://code.google.com/p/boilerpipe/
Rodeoclash 12y ago
Looks like it's getting crushed at the moment.
aurelius 12y ago
What an offensive, restrictive, and anti-free software license! Too bad - this might have otherwise been a useful piece of software. The current license renders it toxic.
- logn 12y ago
  The GNU licenses are arguably the definition of free software. More permissive licensing just gives you the freedom to make software non-free. That's also offered for this project, but as a paid privilege.
- sciurus 12y ago
  To those curious, the license is the GNU Affero General Public License version 3.
  - rkowalick 12y ago
    It is a very open-source unfriendly modification of the AGPLv3 license.
    - logn 12y ago
      The modifications (available to paying licensees) are designed to let this project be useful to those running scraping SaaS's. It lets them hook this into their proprietary infrastructures without GPL'ing their whole stack. Revisions/additions to the project itself need to be licensed as AGPL but the rest can remain proprietary. The AGPLv3 license stands on its own without the modifications (they're optional). It's essentially a dual licensing option. There are a limited number of ways to do open source and make a living at it. And given that I am in business independently with no investors, I have not many other options. The one path I've ruled out, at least for now, is BSD-style licensing, as that just allows SaaS operators to leverage my work, deny users freedom, and also not pay me for my time to help their commercial projects.
      - sireat 12y ago
        Ahh, the Peanut Butter Hula Hoops crazy licence: http://www.billthelizard.com/2012/05/which-open-source-license.html http://www.billthelizard.com/2012/05/which-open-source-licen... The more I think about it the more I like this dual licencing setup which seems a realistic way to ensure those working on OS get paid. It lets everyone experimenting enjoy the code, whether they are a destitute student or an aspiring startup. Meanwhile, if you are someone who wants to actually make money from this AND want to hide your own code, you've got to pay the piper. Otherwise you end up in BSD land (or is it MIT land) where everyone takes from your project and give nothing back.
- Flimm 12y ago
  It is not unfree or anti-free at all. You can use and copy it and modify it and share it, unlike proprietary software.
kazinator 12y ago
Using neural nets and tuned heuristics, ScreenSlicer is able to intelligently find a search box, enter a query, extract the results, and page forward in the results. This sounds like it might be usable for making a universal program to go through the idiotic, annoying, hated-by-everyone web-based sign on screens required by many free Wi-Fi hot spots. As in, something that doesn't have a rigid template database of what UI it can deal with so the users have to beg "please support such and such hot spot in the next release".
- deleted 12y ago
  [deleted]
  - logn 12y ago
    There's nothing proprietary in the entire stack and there's no SaaS being offered here. The API is separated from the core logic to allow for clustering. You can put both API and core servers on the same machine.
- logn 12y ago
  Specifically, the neural nets are used in finding the parent node of the result set (or one of the parents). Heuristics (basically just generic strategies) are used for finding search boxes. There's also a similar function for finding authentication forms. Most of the functions in this project are static so it should be easy to use isolated components as needed.
- deleted 12y ago
  [deleted]
siegecraft 12y ago
You're supposed to build an audience first before you lock them in.. not do it right at the beginning.
CGamesPlay 12y ago
The first thing I tried it on was http://hcpdirectory.cigna.com/web/public/providers http://hcpdirectory.cigna.com/web/public/providers, which is a typical instance of a site I might like to scrape, but still one of the simpler ones. It was not able to extract meaningful information.
- logn 12y ago
  That works in my test locally. The cloud servers are rather overloaded at the moment and also sometimes Tor is just flaky. Here's the request/response: http://pastebin.com/raw.php?i=KncMgbfB http://pastebin.com/raw.php?i=KncMgbfB edit: note that there's a couple different search forms on that page. If you run this on your own machines you can use the Form Query API which lets you specify a form ID and then set specific values for each form field (e.g., for a provider search with City/State and Name). And you can use any non-Tor proxies or a direct connection.

26 points by logn 12y ago

Is there part of this that I could use for "main text content" detection on generic web pages? (i.e. not the search results listings)

logn 12y ago
I recommend Boilerpipe, which happens to be embedded in ScreenSlicer too: https://code.google.com/p/boilerpipe/ https://code.google.com/p/boilerpipe/

Rodeoclash 12y ago

Looks like it's getting crushed at the moment.

aurelius 12y ago

What an offensive, restrictive, and anti-free software license! Too bad - this might have otherwise been a useful piece of software. The current license renders it toxic.

logn 12y ago
The GNU licenses are arguably the definition of free software. More permissive licensing just gives you the freedom to make software non-free. That's also offered for this project, but as a paid privilege.
sciurus 12y ago
To those curious, the license is the GNU Affero General Public License version 3.
- rkowalick 12y ago
  It is a very open-source unfriendly modification of the AGPLv3 license.
  - logn 12y ago
    The modifications (available to paying licensees) are designed to let this project be useful to those running scraping SaaS's. It lets them hook this into their proprietary infrastructures without GPL'ing their whole stack. Revisions/additions to the project itself need to be licensed as AGPL but the rest can remain proprietary. The AGPLv3 license stands on its own without the modifications (they're optional). It's essentially a dual licensing option. There are a limited number of ways to do open source and make a living at it. And given that I am in business independently with no investors, I have not many other options. The one path I've ruled out, at least for now, is BSD-style licensing, as that just allows SaaS operators to leverage my work, deny users freedom, and also not pay me for my time to help their commercial projects.
    - sireat 12y ago
      Ahh, the Peanut Butter Hula Hoops crazy licence: http://www.billthelizard.com/2012/05/which-open-source-license.html http://www.billthelizard.com/2012/05/which-open-source-licen... The more I think about it the more I like this dual licencing setup which seems a realistic way to ensure those working on OS get paid. It lets everyone experimenting enjoy the code, whether they are a destitute student or an aspiring startup. Meanwhile, if you are someone who wants to actually make money from this AND want to hide your own code, you've got to pay the piper. Otherwise you end up in BSD land (or is it MIT land) where everyone takes from your project and give nothing back.
Flimm 12y ago
It is not unfree or anti-free at all. You can use and copy it and modify it and share it, unlike proprietary software.

kazinator 12y ago

Using neural nets and tuned heuristics, ScreenSlicer is able to intelligently find a search box, enter a query, extract the results, and page forward in the results. This sounds like it might be usable for making a universal program to go through the idiotic, annoying, hated-by-everyone web-based sign on screens required by many free Wi-Fi hot spots. As in, something that doesn't have a rigid template database of what UI it can deal with so the users have to beg "please support such and such hot spot in the next release".

deleted 12y ago
[deleted]
- logn 12y ago
  There's nothing proprietary in the entire stack and there's no SaaS being offered here. The API is separated from the core logic to allow for clustering. You can put both API and core servers on the same machine.
logn 12y ago
Specifically, the neural nets are used in finding the parent node of the result set (or one of the parents). Heuristics (basically just generic strategies) are used for finding search boxes. There's also a similar function for finding authentication forms. Most of the functions in this project are static so it should be easy to use isolated components as needed.
deleted 12y ago
[deleted]

siegecraft 12y ago

You're supposed to build an audience first before you lock them in.. not do it right at the beginning.

CGamesPlay 12y ago

The first thing I tried it on was http://hcpdirectory.cigna.com/web/public/providers http://hcpdirectory.cigna.com/web/public/providers, which is a typical instance of a site I might like to scrape, but still one of the simpler ones. It was not able to extract meaningful information.

logn 12y ago
That works in my test locally. The cloud servers are rather overloaded at the moment and also sometimes Tor is just flaky. Here's the request/response: http://pastebin.com/raw.php?i=KncMgbfB http://pastebin.com/raw.php?i=KncMgbfB edit: note that there's a couple different search forms on that page. If you run this on your own machines you can use the Form Query API which lets you specify a form ID and then set specific values for each form field (e.g., for a provider search with City/State and Name). And you can use any non-Tor proxies or a direct connection.