3 ms·

I'm sure there'd be a ton of people that would love to pay to use your platform (who cares if the source is available, I don't want to run my own because once t

I'm sure there'd be a ton of people that would love to pay to use your platform (who cares if the source is available, I don't want to run my own because once the code is written, it's ops thats hard). But then I suppose it would be hard to stay unnoticed.

atmosx 10y ago
Similar, paid solution: https://scrapinghub.com/crawlera/ https://scrapinghub.com/crawlera/
fake-name 10y ago
Yeah, running this thing publicly would be a huge mess from a copyright perspective, since it literally re-hosts everything as a core part of how it works. As it is, I think I'm OK, since it's basically just a "website DVR" type thing, for my own use. Really, if nothing else, the project has been enormously educational for me. I've learnt a boatload about distributed systems, learned a bit of SQL, dicked about with databases a bunch, and actually experienced deploying a complex multi-component application across multiple disparate data centers.
- nickysielicki 10y ago
  This project is really cool. Last year I was looking into open source projects that implement something like Readability so that I could scrape articles from my RSS feeds and turn them into plaintext. But I didn't find anything that blew me away. The best I got was stealing the implementation from Firefox, and I lost interest before I could make it worthwhile. (Now revisiting the idea, I wonder why I never thought of passing a user-agent from a mobile browser... Probably would have helped a lot.) I see you don't have a license listed on GitHub. Do you have a license in mind for these?
  - fake-name 10y ago
    It's probably GPL, I'll have to figure out my dependencies and see what it's infected with. I tend to err BSD on my own cruft. This isn't quite as fancy as readability, though I integrated a port of readability for a while. Now I just write a ruleset for a site that has stuff that interests me.
  - fake-name 10y ago
    Note: I stuck it under BSD license.

atmosx 10y ago

Similar, paid solution: https://scrapinghub.com/crawlera/ https://scrapinghub.com/crawlera/

fake-name 10y ago

Yeah, running this thing publicly would be a huge mess from a copyright perspective, since it literally re-hosts everything as a core part of how it works. As it is, I think I'm OK, since it's basically just a "website DVR" type thing, for my own use. Really, if nothing else, the project has been enormously educational for me. I've learnt a boatload about distributed systems, learned a bit of SQL, dicked about with databases a bunch, and actually experienced deploying a complex multi-component application across multiple disparate data centers.

nickysielicki 10y ago
This project is really cool. Last year I was looking into open source projects that implement something like Readability so that I could scrape articles from my RSS feeds and turn them into plaintext. But I didn't find anything that blew me away. The best I got was stealing the implementation from Firefox, and I lost interest before I could make it worthwhile. (Now revisiting the idea, I wonder why I never thought of passing a user-agent from a mobile browser... Probably would have helped a lot.) I see you don't have a license listed on GitHub. Do you have a license in mind for these?
- fake-name 10y ago
  It's probably GPL, I'll have to figure out my dependencies and see what it's infected with. I tend to err BSD on my own cruft. This isn't quite as fancy as readability, though I integrated a port of readability for a while. Now I just write a ruleset for a site that has stuff that interests me.
- fake-name 10y ago
  Note: I stuck it under BSD license.