6 ms·

It's probably not what you meant, but I recently created a weekend project bot that scans facebook for apartments rental ads. Thanks to this bot, I landed an op

by snird 6y ago

It's probably not what you meant, but I recently created a weekend project bot that scans facebook for apartments rental ads. Thanks to this bot, I landed an opportunity I wouldn't have catch otherwise, saving me 6,000$ a year. Assuming I will hold my new apartment for 3 years, this is a net savings of 18,000$. I haven't sold anything, but yet I created something that "generated money" for me. My code is open source by the way, and I wrote about it here: https://snir.dev/blog/apartments-bot/ https://snir.dev/blog/apartments-bot/

HaloZero 6y ago
Interesting. I've never heard of apartment posts in facebook groups but I'm in the bay area. Do other people here in the US have this experience as well? I was thinking it'd be Facebook marketplace, craigslist, and apartment sites (like apartments.com)
- cedricium 6y ago
  FB buying/selling groups (particularly for housing) are quite common I'd say. In my experience, you will generally find them categorized as part of a college or university (e.g. "San Jose State University (SJSU) Housing"), but a quick search for "Bay Area housing" also results in numerous such groups.
- mbrameld 6y ago
  A Facebook group is the primary source for advertising rental properties in Juneau, AK. Just a normal group, not the marketplace.
  - stevekemp 6y ago
    Ditto Helsinki, Finland: https://www.facebook.com/groups/235368102105 https://www.facebook.com/groups/235368102105 I listed my rental flat a bunch of places before using the group, since I started using it I'd not look elsewhere.
- snird 6y ago
  I can say that people in Israel prefer facebook groups for many reasons: 1. It allows for "team search". My friends knew I'm looking, so they tagged me in relevant posts they saw. Can't do it anywhere else (including marketplace by facebook themselves) 2. It allows for group communication with the publisher of the post. One asks - everybody sees. 3. It gives a sense of power to those who seek to rent. Many posts with high prices have comments like "You are crazy", "You are a pig", "whoever pays this much is as sucker". It almost warms the heart, we fight the greedy landlords. It almost make it fun, like a witch hunt.
  - bradlys 6y ago
    As a landlord though, what’s the reason to use Facebook then? Is there no alternative like craigslist in Israel? I’ve only seen posts of apartments for rent in Facebook groups that are aimed at college students. (In the US) Otherwise, Craigslist is basically king.
    - snird 6y ago
      Young people use facebook to find apartments. So if your apartments target young couples or room mates, facebook is your place. For families we have craiglist-like solutions.
- asdff 6y ago
  typical at college campuses in the us
radihuq 6y ago
This is great, congrats on the savings. Do you have any opinions on the ethics of web scraping? I have some projects I want to pursue that involve web scraping but I'm worried about 1) any illegal technicalities 2) is it a dick move to the website owner
- ta17711771 6y ago
  > is it a dick move to the website owner FB gets more user engagement out of this, which is their goal. No harm, no foul.
- mmcwilliams 6y ago
  Not OP, but I find that the ethical questions tend to not be about web scraping itself and more what you are using it to accomplish. If you want to scrape and rehost some data then your ethical violation is in the copying and rehosting, not that you automated the downloading of data. It would be the same if you copied all the data by hand and then rehosted it. Then, of course, the way you go about it is important, too. Sapping the resources and bandwidth of a small site by sending it thousands of requests a minute is very different from automatically checking a public index once an hour.
- snird 6y ago
  I was a CTO of a company that did scraping to gather data and sell the analysis. We did this for sellers on Amazon. In short about legality - it depends on the country. But most will define user content as owned by the user, not the platform, therefore legal to scrape and use, as the user intents for the data to be public. But sites will fight you on this. Amazon specifically give up a really good fight. Ofcourse, this talks about user content in an "open" platform. If the content is owned by a business that has no intentions for it to be open for everyone in every medium, its probably illegal. In regard to being a dick move, it depends. If someone posted an ad in facebook, he intended for it to be seen. He doesn't care how. I just tunnel it to myself in easier way for me to consume.
  - yowlingcat 6y ago
    I think there was also case law about this recently[1] regarding LinkedIn, where if the content isn't behind a login wall, it can't legally be considered anything but public. The UGC aspect is an interesting angle I haven't thought of, but that's a fantastic point too. Would I be correct in saying that the same mechanism companies set up to evade responsibility for illegal UGC also sets it up to be legal to scrape? And if the user owns the material, what happens if you've already made money selling that data and they ask for their cut of the proceeds?
jack95 6y ago
Thanks for sharing this, I tried to write a bot to crawl a few music related Facebook groups a while ago and it was really difficult. I'm going to give it another shot following a similar method to this. Cheers!

by snird 6y ago

HaloZero 6y ago

Interesting. I've never heard of apartment posts in facebook groups but I'm in the bay area. Do other people here in the US have this experience as well? I was thinking it'd be Facebook marketplace, craigslist, and apartment sites (like apartments.com)

cedricium 6y ago
FB buying/selling groups (particularly for housing) are quite common I'd say. In my experience, you will generally find them categorized as part of a college or university (e.g. "San Jose State University (SJSU) Housing"), but a quick search for "Bay Area housing" also results in numerous such groups.
mbrameld 6y ago
A Facebook group is the primary source for advertising rental properties in Juneau, AK. Just a normal group, not the marketplace.
- stevekemp 6y ago
  Ditto Helsinki, Finland: https://www.facebook.com/groups/235368102105 https://www.facebook.com/groups/235368102105 I listed my rental flat a bunch of places before using the group, since I started using it I'd not look elsewhere.
snird 6y ago
I can say that people in Israel prefer facebook groups for many reasons: 1. It allows for "team search". My friends knew I'm looking, so they tagged me in relevant posts they saw. Can't do it anywhere else (including marketplace by facebook themselves) 2. It allows for group communication with the publisher of the post. One asks - everybody sees. 3. It gives a sense of power to those who seek to rent. Many posts with high prices have comments like "You are crazy", "You are a pig", "whoever pays this much is as sucker". It almost warms the heart, we fight the greedy landlords. It almost make it fun, like a witch hunt.
- bradlys 6y ago
  As a landlord though, what’s the reason to use Facebook then? Is there no alternative like craigslist in Israel? I’ve only seen posts of apartments for rent in Facebook groups that are aimed at college students. (In the US) Otherwise, Craigslist is basically king.
  - snird 6y ago
    Young people use facebook to find apartments. So if your apartments target young couples or room mates, facebook is your place. For families we have craiglist-like solutions.
asdff 6y ago
typical at college campuses in the us

radihuq 6y ago

This is great, congrats on the savings. Do you have any opinions on the ethics of web scraping? I have some projects I want to pursue that involve web scraping but I'm worried about 1) any illegal technicalities 2) is it a dick move to the website owner

ta17711771 6y ago
> is it a dick move to the website owner FB gets more user engagement out of this, which is their goal. No harm, no foul.
mmcwilliams 6y ago
Not OP, but I find that the ethical questions tend to not be about web scraping itself and more what you are using it to accomplish. If you want to scrape and rehost some data then your ethical violation is in the copying and rehosting, not that you automated the downloading of data. It would be the same if you copied all the data by hand and then rehosted it. Then, of course, the way you go about it is important, too. Sapping the resources and bandwidth of a small site by sending it thousands of requests a minute is very different from automatically checking a public index once an hour.
snird 6y ago
I was a CTO of a company that did scraping to gather data and sell the analysis. We did this for sellers on Amazon. In short about legality - it depends on the country. But most will define user content as owned by the user, not the platform, therefore legal to scrape and use, as the user intents for the data to be public. But sites will fight you on this. Amazon specifically give up a really good fight. Ofcourse, this talks about user content in an "open" platform. If the content is owned by a business that has no intentions for it to be open for everyone in every medium, its probably illegal. In regard to being a dick move, it depends. If someone posted an ad in facebook, he intended for it to be seen. He doesn't care how. I just tunnel it to myself in easier way for me to consume.
- yowlingcat 6y ago
  I think there was also case law about this recently[1] regarding LinkedIn, where if the content isn't behind a login wall, it can't legally be considered anything but public. The UGC aspect is an interesting angle I haven't thought of, but that's a fantastic point too. Would I be correct in saying that the same mechanism companies set up to evade responsibility for illegal UGC also sets it up to be legal to scrape? And if the user owns the material, what happens if you've already made money selling that data and they ask for their cut of the proceeds?

jack95 6y ago

Thanks for sharing this, I tried to write a bot to crawl a few music related Facebook groups a while ago and it was really difficult. I'm going to give it another shot following a similar method to this. Cheers!