6 ms·

Web scraping with Factor

79 points by otoburb 13y ago

techaddict009 13y ago
I do regular web scrapping using php and curl. This seems more interesting time to learn factor programming language. @otoburb can you recommend any better guide for the same?
- otoburb 13y ago
  The Factor documentation is hosted at http://docs.factorcode.org http://docs.factorcode.org. Feel free to browse! It's the same set of docs that you also have access to locally when you download the Factor binaries[1] or pull/compile from github[2]. The concatenative.org[3] wiki also has similar starting material and pointers. Factor is a fun language. The blog that I linked to is written by one of the Factor contributors. [1] http://factorcode.org/ http://factorcode.org/ [2] https://github.com/slavapestov/factor https://github.com/slavapestov/factor [3] http://concatenative.org/wiki/view/Factor/Learning http://concatenative.org/wiki/view/Factor/Learning
  - klibertp 13y ago
    Do you know about any "Factor for Forth programmers" tutorial? Factor is similar enough to irritate me with it's "for beginners" materials, but different (and larger) enough to make normal manuals useless by themselves. I'd especially appreciate a bottom-up write up, starting with the stack and cells (which feel familiar), and then introducing higher-level abstractions of Factor.
    - bjourne 13y ago
      Better wash your brain about that implied misconception. Factor is much more like Lisp and Haskell (esp point-free style) than Forth. I guess "learn Lisp and/or Haskell and Factor won't seem so foreign" isn't terrific advice. But right now there doesn't exist many newbie guides at all.
      - klibertp 13y ago
        I dunno, syntactically it resembles Forth quite a bit, what with : ; for defining words and () for comments and all that. Anyway, I have no problem whatsoever with high-level abstractions in Factor, nor with its concatenative nature, nor with its macros and so on. I know all these features from other languages. What I want is a just description of how these high-level things map to assembly, I guess. For example I just learned that: "Internally, a quotation is a pair, consisting of an array and a machine code entry point. The array stores the quotation's elements" - this is a kind of definition I want for all the abstractions in Factor. It's probably best to go through Slava Pestov blog and pick up such scattered descriptions, but I'd really appreciate if someone prepared a single article with all these definitions.
        csandreasen 13y ago
        Note that the stack comments aren't actually comments in Factor - they're part of the function definition and are mandatory. The compiler will do a simple check to ensure that all of your stack inputs and outputs match up for each function call.
        mrjbq7 13y ago
        We made stack effects mandatory for most definitions as it appeared an area of frustration with new Factor programmers. However, we have a stack checker that still supports optional stack effects if you yearn for the good ol' days: https://github.com/slavapestov/factor/issues/887 https://github.com/slavapestov/factor/issues/887
danso 13y ago
This didn't seem interesting to me, because there's so many ways to scrape a site, in every high level language, that learning one more way to do it is going to be a diminished return...and I also didn't know that "Factor" was a language. The OP's template could have some more info on what Factor is, but there's a few links, including this wiki for it: http://concatenative.org/wiki/view/Factor/Learning http://concatenative.org/wiki/view/Factor/Learning I honestly didn't know what a "concatenative" language was until I saw that "Six programming paradigms that will change how you think about coding" post that fronted HN last week: http://brikis98.blogspot.in/2014/04/six-programming-paradigms-that-will.html http://brikis98.blogspot.in/2014/04/six-programming-paradigm... So this is just a long way of saying...after just learning about concatenative languages, I'm really interested in what that paradigm brings to a common task...maybe there aren't productivity gains, but I love learning different philosophies of coding, and thanks to the OP for showing one practical example.
- mrjbq7 13y ago
  I added some information about the Factor language to the blog template - great idea, thanks. Concatenative languages are quite interesting, and I'd encourage you to try it out. You might find it helps your thinking about certain problems.
- xrange 13y ago
  In exploring concatenative languages, be sure to investigate the Joy language. https://web.archive.org/web/20111006171100/http://www.latrobe.edu.au/phimvt/joy.html https://web.archive.org/web/20111006171100/http://www.latrob... ...unfortunately, it appears that the webpage of Joy's author has been lost to the web, so therefore the link to the Internet Archive. And don't forget that there was a very mainstream concatenative language at one point in time: PostScript.
notastartup 13y ago
you can write an entire web scraper with just a url using http://scrape.ly http://scrape.ly With scrape.ly I can just do this to crawl the entire HackerNews site across pages and grab the urls and extract any data from the page it lands on without defining any fields (it discovers them on it's own) and so doesn't require you to 'relabel' fields when the site changes layouts. It also generates new IP addresses on the fly so you don't get stuck and launches multiple threads for you to speed up the process. It works fully with ajax sites and single page apps. Flash support is coming too. http://scrape.ly/s/{https://news.ycombinator.com/} {next:More}{Space Monkey dumps Python for Go}*{fields:'Auto'} Honest question (I don't mind downvoting if you disagree), but why would you want to waste time writing web scrapers, maintaining it to run and fixing the code? Multiply it by 100 or 1000 different websites and it becomes a full-time job. For me, I want to get the data I need with the least possible of overhead and as soon as possible and I don't really want to be bothered with setting up environments and hosting for it to run and fixing bugs when sites change layout.
- sogen 13y ago
  Don't forget https://scraperwiki.com/ https://scraperwiki.com/
- Jake232 13y ago
  Or if you're a python enthusiast, then shameless self link: http://jakeaustwick.me/python-web-scraping-resource/ http://jakeaustwick.me/python-web-scraping-resource/
  - notastartup 13y ago
    pretty good but that's an awful lot of reading and lot of work just to grab some data from a simple website.
- Avshalom 13y ago
  This is not a post about web scraping. It's a post about doing something in Factor.
naveen99 13y ago
factor has opengl built into its repl, so it kind of gives you the mythical modern terminal in one of gary bernhardt's talks.
- agumonkey 13y ago
  I forgot about that ... Factor impresses me a lot for there's so much packed in it. one man effort it seems ... even more impressive.
  - mrjbq7 13y ago
    Although it feels likes a one programmer effort, there have been a few of us contributing consistently over the years: https://github.com/slavapestov/factor/graphs/contributors https://github.com/slavapestov/factor/graphs/contributors
    - agumonkey 13y ago
      Thanks for showing me this, I was greatly misguided.
riffraff 13y ago
somewhat relative to factor: whatever happened to slava pestov? I used to follow his blog about factor with interest some years ago and then all of a sudden... he was gone.
- mrjbq7 13y ago
  He is working for Google. From the outside, it looks a lot like he disappeared from open source...

79 points by otoburb 13y ago

I do regular web scrapping using php and curl. This seems more interesting time to learn factor programming language. @otoburb can you recommend any better guide for the same?

otoburb 13y ago
The Factor documentation is hosted at http://docs.factorcode.org http://docs.factorcode.org. Feel free to browse! It's the same set of docs that you also have access to locally when you download the Factor binaries[1] or pull/compile from github[2]. The concatenative.org[3] wiki also has similar starting material and pointers. Factor is a fun language. The blog that I linked to is written by one of the Factor contributors. [1] http://factorcode.org/ http://factorcode.org/ [2] https://github.com/slavapestov/factor https://github.com/slavapestov/factor [3] http://concatenative.org/wiki/view/Factor/Learning http://concatenative.org/wiki/view/Factor/Learning
- klibertp 13y ago
  Do you know about any "Factor for Forth programmers" tutorial? Factor is similar enough to irritate me with it's "for beginners" materials, but different (and larger) enough to make normal manuals useless by themselves. I'd especially appreciate a bottom-up write up, starting with the stack and cells (which feel familiar), and then introducing higher-level abstractions of Factor.
  - bjourne 13y ago
    Better wash your brain about that implied misconception. Factor is much more like Lisp and Haskell (esp point-free style) than Forth. I guess "learn Lisp and/or Haskell and Factor won't seem so foreign" isn't terrific advice. But right now there doesn't exist many newbie guides at all.
    - klibertp 13y ago
      I dunno, syntactically it resembles Forth quite a bit, what with : ; for defining words and () for comments and all that. Anyway, I have no problem whatsoever with high-level abstractions in Factor, nor with its concatenative nature, nor with its macros and so on. I know all these features from other languages. What I want is a just description of how these high-level things map to assembly, I guess. For example I just learned that: "Internally, a quotation is a pair, consisting of an array and a machine code entry point. The array stores the quotation's elements" - this is a kind of definition I want for all the abstractions in Factor. It's probably best to go through Slava Pestov blog and pick up such scattered descriptions, but I'd really appreciate if someone prepared a single article with all these definitions.
      - csandreasen 13y ago
        Note that the stack comments aren't actually comments in Factor - they're part of the function definition and are mandatory. The compiler will do a simple check to ensure that all of your stack inputs and outputs match up for each function call.
        mrjbq7 13y ago
        We made stack effects mandatory for most definitions as it appeared an area of frustration with new Factor programmers. However, we have a stack checker that still supports optional stack effects if you yearn for the good ol' days: https://github.com/slavapestov/factor/issues/887 https://github.com/slavapestov/factor/issues/887

danso 13y ago

This didn't seem interesting to me, because there's so many ways to scrape a site, in every high level language, that learning one more way to do it is going to be a diminished return...and I also didn't know that "Factor" was a language. The OP's template could have some more info on what Factor is, but there's a few links, including this wiki for it: http://concatenative.org/wiki/view/Factor/Learning http://concatenative.org/wiki/view/Factor/Learning I honestly didn't know what a "concatenative" language was until I saw that "Six programming paradigms that will change how you think about coding" post that fronted HN last week: http://brikis98.blogspot.in/2014/04/six-programming-paradigms-that-will.html http://brikis98.blogspot.in/2014/04/six-programming-paradigm... So this is just a long way of saying...after just learning about concatenative languages, I'm really interested in what that paradigm brings to a common task...maybe there aren't productivity gains, but I love learning different philosophies of coding, and thanks to the OP for showing one practical example.

mrjbq7 13y ago
I added some information about the Factor language to the blog template - great idea, thanks. Concatenative languages are quite interesting, and I'd encourage you to try it out. You might find it helps your thinking about certain problems.
xrange 13y ago
In exploring concatenative languages, be sure to investigate the Joy language. https://web.archive.org/web/20111006171100/http://www.latrobe.edu.au/phimvt/joy.html https://web.archive.org/web/20111006171100/http://www.latrob... ...unfortunately, it appears that the webpage of Joy's author has been lost to the web, so therefore the link to the Internet Archive. And don't forget that there was a very mainstream concatenative language at one point in time: PostScript.

notastartup 13y ago

you can write an entire web scraper with just a url using http://scrape.ly http://scrape.ly With scrape.ly I can just do this to crawl the entire HackerNews site across pages and grab the urls and extract any data from the page it lands on without defining any fields (it discovers them on it's own) and so doesn't require you to 'relabel' fields when the site changes layouts. It also generates new IP addresses on the fly so you don't get stuck and launches multiple threads for you to speed up the process. It works fully with ajax sites and single page apps. Flash support is coming too. http://scrape.ly/s/{https://news.ycombinator.com/} {next:More}{Space Monkey dumps Python for Go}*{fields:'Auto'} Honest question (I don't mind downvoting if you disagree), but why would you want to waste time writing web scrapers, maintaining it to run and fixing the code? Multiply it by 100 or 1000 different websites and it becomes a full-time job. For me, I want to get the data I need with the least possible of overhead and as soon as possible and I don't really want to be bothered with setting up environments and hosting for it to run and fixing bugs when sites change layout.

sogen 13y ago
Don't forget https://scraperwiki.com/ https://scraperwiki.com/
Jake232 13y ago
Or if you're a python enthusiast, then shameless self link: http://jakeaustwick.me/python-web-scraping-resource/ http://jakeaustwick.me/python-web-scraping-resource/
- notastartup 13y ago
  pretty good but that's an awful lot of reading and lot of work just to grab some data from a simple website.
Avshalom 13y ago
This is not a post about web scraping. It's a post about doing something in Factor.

naveen99 13y ago

factor has opengl built into its repl, so it kind of gives you the mythical modern terminal in one of gary bernhardt's talks.

agumonkey 13y ago
I forgot about that ... Factor impresses me a lot for there's so much packed in it. one man effort it seems ... even more impressive.
- mrjbq7 13y ago
  Although it feels likes a one programmer effort, there have been a few of us contributing consistently over the years: https://github.com/slavapestov/factor/graphs/contributors https://github.com/slavapestov/factor/graphs/contributors
  - agumonkey 13y ago
    Thanks for showing me this, I was greatly misguided.

riffraff 13y ago

somewhat relative to factor: whatever happened to slava pestov? I used to follow his blog about factor with interest some years ago and then all of a sudden... he was gone.

mrjbq7 13y ago
He is working for Google. From the outside, it looks a lot like he disappeared from open source...