10 ms·

A bit offtopic, but how did you do the webscraping part? I'm a bit like the parent, in that I don't commit myself to projects in my "spare time" (ok, after wor

A bit offtopic, but how did you do the webscraping part? I'm a bit like the parent, in that I don't commit myself to projects in my "spare time" (ok, after working 10 hours for someone else). I've been meaning to build a website for real estate searches (the current local giant has several usability issues), but I've never gotten to the webscraping part. I have these 3 in my "to-research" list: Scrapy http://scrapy.org/ http://scrapy.org/ Beautiful Soup http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/ Nokogiri http://nokogiri.org/ http://nokogiri.org/ did you use any of those?

jlengrand 13y ago
BS is awesome, I used it in a former project. Good thing is that it tries to repair the parsed stuff too. And you'll see that most of what you retrieve will be broken :). I currently use Jsoup, simply because I am working in java. It gets the job done, I got only good things to say about it. Concerning the real estate thing, I coulnd't agree more. Being a french guy, I know that all real estate website suck, and it is nearly impossible to have a way to know all houses to sell in the neighborhood. Some kind of location based real estate aggregator with links to real estate websites could be a huge hit IMHO. Hope this helps
- GFischer 13y ago
  Thank you :) . That's exactly what I want to do, a location-based real estate aggregator. I want to add Airbnb-style sliders, and map-based search.
  - jlengrand 13y ago
    in which country?
    - GFischer 13y ago
      Uruguay. The main website for real estate here is El Gallito by the local newspaper El País. There's also BuscandoCasas (http://www.buscandocasa.com/ http://www.buscandocasa.com/) , which is pretty decent but doesn't have map-based search. I'm a bit worried about legal implications, linking and scraping are not well seen here.
      - jlengrand 13y ago
        Ok. I got only knowledge of the french and dutch market. The website you gave seems to be a good candidate for scrapping (simple layout, really straightforward). Web scraping is usually not well seen, but hey, if you can help them sell more / more efficiently you can make some friends pretty quickly! AFAIK, what is hugely missing in real estate websites currently is location based search. Some time ago, I had written some to play around with google maps and say "I want to live within 30 minutes of this place". If you can link that to a real estate listing that could really become awesome! Let me know if you actually start the project :).
        GFischer 13y ago
        Yeah, I probably won't, I have a job opportunity that, if I end up getting it, will take most of my time and focus for the next 2 years :) Thank you very much for your advice !! I now feel a bit bad if I don't do this :P
        jlengrand 13y ago
        Congratulations! Come on. . .You´ll still have all your nights free! :D

jlengrand 13y ago

BS is awesome, I used it in a former project. Good thing is that it tries to repair the parsed stuff too. And you'll see that most of what you retrieve will be broken :). I currently use Jsoup, simply because I am working in java. It gets the job done, I got only good things to say about it. Concerning the real estate thing, I coulnd't agree more. Being a french guy, I know that all real estate website suck, and it is nearly impossible to have a way to know all houses to sell in the neighborhood. Some kind of location based real estate aggregator with links to real estate websites could be a huge hit IMHO. Hope this helps

GFischer 13y ago
Thank you :) . That's exactly what I want to do, a location-based real estate aggregator. I want to add Airbnb-style sliders, and map-based search.
- jlengrand 13y ago
  in which country?
  - GFischer 13y ago
    Uruguay. The main website for real estate here is El Gallito by the local newspaper El País. There's also BuscandoCasas (http://www.buscandocasa.com/ http://www.buscandocasa.com/) , which is pretty decent but doesn't have map-based search. I'm a bit worried about legal implications, linking and scraping are not well seen here.
    - jlengrand 13y ago
      Ok. I got only knowledge of the french and dutch market. The website you gave seems to be a good candidate for scrapping (simple layout, really straightforward). Web scraping is usually not well seen, but hey, if you can help them sell more / more efficiently you can make some friends pretty quickly! AFAIK, what is hugely missing in real estate websites currently is location based search. Some time ago, I had written some to play around with google maps and say "I want to live within 30 minutes of this place". If you can link that to a real estate listing that could really become awesome! Let me know if you actually start the project :).
      - GFischer 13y ago
        Yeah, I probably won't, I have a job opportunity that, if I end up getting it, will take most of my time and focus for the next 2 years :) Thank you very much for your advice !! I now feel a bit bad if I don't do this :P
        jlengrand 13y ago
        Congratulations! Come on. . .You´ll still have all your nights free! :D