20 ms·

I need to find an apartment

350 points by mattrighetti 5y ago

ab_testing 5y ago
May be I am too old fashioned but I have never been able to select apartments like this mathematically . Sure I have used methods like this to shortlist the apartments, but I still need to visit them and feel for the place before commiting to one place.
- mattrighetti 5y ago
  Nah you’re doing it right! I’m just using the tool to select the best ones to eventually visit and maybe rent
ChuckNorris89 5y ago
At least OP had the full address of those apartment on that website, so he could plot them on the map. Here in Austria, most leeches, sorry, realtors, avoid putting the street in the ad to prevent you from trying to contact the property directly and bypassing their fees, so apartment hunting is even more of a headache. Still, amazing work from OP.
- dazbradbury 5y ago
  There are genuine privacy concerns at play here, for example there may be existing tenants/owners who don't want people randomly knocking on the door because someone saw an ad online. Or getting a bunch of junk mail because there is an extra data point about the property. I guess the variety of concerns will vary from country to country depending on local laws / existing open data though.
  - bluefirebrand 5y ago
    I don't think it is reasonable to expect to be able to list real estate for rent or sale, and also expect privacy with regards to the address.
    - giraffe_lady 5y ago
      Yeah but this is a "the customer isn't the user" situation. The tenant wants and deserves privacy, and it's their privacy that can be compromised by the listing. The owner doesn't really have a reason to care, I don't think, and may even be incentivized in some situations to reveal information the tenants would rather they not.
      - fennecfoxen 5y ago
        What information? “This apartment exists at LOCATION and will be available for rent at DATE?” And you think the tenant is entitled to stop the world from knowing that because the tenant is residing at that location before DATE? Goodness. I should give the First Amendment a hug.
        giraffe_lady 5y ago
        I didn't make any of those claims at all. I just pointed out that if there is a privacy issue, it's not the lister's privacy at stake.
        bluefirebrand 5y ago
        I'm still not sure renters really have a reasonable expectation of privacy when the place they are living in is listed for rent or sale by someone else. In most places renters are only granted 24 hours notice to have complete strangers come walk through their living spaces while the space is shown off. They have no real right to refuse to allow that. If people start knocking on their door asking to come look at the place they are free to refuse, or ignore the doorbell or anything else. Of course they have a freedom against being harassed, so if people persist they can get police involved. That seems like a separate issue to me from having their address posted online in a rental listing.
        ChuckNorris89 5y ago
        >I'm still not sure renters really have a reasonable expectation of privacy when the place they are living in is listed for rent or sale by someone else In some EU countries they do. The landlord need to ask permission to have visitors at your place and the tenant has the right to refuse.
- est31 5y ago
  When I searched for my apartment, I was able to reverse engineer the home's address sometimes when the general vicinity was specified, then looking at the pictures, and comparing them with Google street view. This was a manual process however and only worked if enough outside views were available. Despite the time effort, you still spent less time on it per house than making an appointment for a viewing, getting there, and looking at it. If Google allows you to download the street views for an entire city, one might build an ML solution to bring up candidates at least. Similarly, in the city where I studied for university, I immediately knew where the house was located just from outside pictures of it in the ad, at least for the districts of the city I was familiar with.
  - fennecfoxen 5y ago
    I use a similar process to look at interesting job listings sent to me unsolicited by recruiters. You can anonymize the company but it only takes a few unique phrases to locate the posting with Google.
  - alex-nt 5y ago
    I am looking for an apartment in eastern Europe. I was shocked to see listings with the windows covered in Microsoft Paint. I was able to find the location in some instances by finding the same property in other listings (the images were still slightly altered). This is the amateur move, what I see more often is image alteration so that the light coming through the windows makes it impossible to see what is outside. The same apartment is usually posted by 5-10 agencies, pictures and description almost identical, with different location. You get different responses depending who you call. I hope to end my suffering this year and buy something.
    - est31 5y ago
      Oh it is definitely possible to avoid this. It's kind of an arms race. The overexposed window effect might be reversed with computational photography for example. Maybe one can just lower brightness in an image editor and discover that there is a residual of the outside left. You might use "looking around a corner" like tricks on white walls, etc etc.
- tecleandor 5y ago
  Oh, around here (Spain) usually they don't put the address in the ad so you can't cross check the information with the national registry and find that the apartment is 20% smaller than claimed.
- sswezey 5y ago
  Most of the apartments on that site don't have the actual addresses, instead just have the city/town. It's the same issue you've experienced.
- micheloosterhof 5y ago
  I've searched for houses in multiple places around the world and the unwillingness to provide the full addresses is often due to agent competition. More precise whether the seller has one or more selling agents. In places like The Netherlands, a seller generally works with one seller's agent only, and there's no incentive to hide full addresses. Funda.nl has a huge amount of detail, from high quality photos, to floor plans. On the contrary, in Singapore, a seller often has multiple agents. The agents compete with each other and would rather not share any information at all publicly to avoid other agents using their information/photos, or buyer's bypassing them. It leads to propertyguru.com.sg with no addresses, bad photos, all photos watermarked with the agent pictures. As little information as possible, just enough to draw you into an in-person visit.
- andbberger 5y ago
  can you register as a broker on the austrian version of MLS to get the listings
denysvitali 5y ago
Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible). I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here. I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust. Whilst the application is not available to the public (yet), all of the libraries I've created are now available. I should have both a Rust and a Golang version for most / all of them. On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites. [0]: https://github.com/denysvitali/sbb-api-rs https://github.com/denysvitali/sbb-api-rs [1]: https://github.com/denysvitali/homegate-rs https://github.com/denysvitali/homegate-rs [2]: https://github.com/denysvitali/flatfox-rs https://github.com/denysvitali/flatfox-rs [3]: https://github.com/denysvitali/go-sbb-api https://github.com/denysvitali/go-sbb-api
- vageli 5y ago
  > I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust. How did you convert an address to long/lat for postgis?
  - denysvitali 5y ago
    I was lucky, the data also included lat/lng and the address for most (if not all) the entries. If I would have been confronted with such a situation though, I would have probably used some Geocoding API like Google Maps's geocoding [0] or OSM's Nominatim [1]. [0]: https://developers.google.com/maps/documentation/geocoding/overview https://developers.google.com/maps/documentation/geocoding/o... [1]: https://wiki.openstreetmap.org/wiki/Nominatim https://wiki.openstreetmap.org/wiki/Nominatim
  - mattrighetti 5y ago
haswell 5y ago
I'm in the middle of an apartment search. I was reflecting the other day on how much searching for an apartment has changed over the last 10 years. Conclusion: it hasn't changed much at all. It's still necessary to search on a myriad of sites. Most of the time, by the time you inquire, a place is gone. Or the ad was misleading and that "$2000 2BR" is actually "$2000 for the cheapest Studio we have, but we also have 2BR for $3.5K". Finding places that match my criteria still requires careful inspection of the description. Checking a filter box for "Parking" excludes places that only mention parking in the description. With all of the modernization and streamlining that has happened in other parts of our lives, apartment hunting remains unstructured, messy, frustrating, stressful. Aggregators like Padmapper helped for awhile, but these days, it seems like most listings are posted by scammy middlemen just trying to make a commission on the listing, and now the site actively makes the process even harder. Is hunting for an apartment doomed to be this messy forever? It seems like an area that is ripe for disruption or a better option, but what would it take to actually solve this? --- With all of that said, this post is exactly what I needed right now. I've been toying with a very similar idea to improve the efficiency of my search process and this is good inspiration.
- deleted 5y ago
  [deleted]
- Etheryte 5y ago
  As someone who just jumped through all the hoops to find a new apartment, the pain is real. I think the main issue is that the incentives are misaligned from the perspective of the seeker. Where I'm currently at (the Netherlands), the market is dominated by agents. Using an agent makes sense from the perspective of the apartment owner — they will take care of managing the ads, initial screening, arranging viewings etc. Given how many people contact a single viewing, you definitely don't want them hitting your personal inbox or phone. From the agent's perspective, it's good to drive people to their company's website rather than one of the big aggregate ones. Having more people visit it means they're more likely to do business with that company specifically. Getting people to visit your company's site is pretty straightforward: they put up ads there earlier, often so much earlier that by the time you see an ad on an aggregate site, it's already fully booked for viewings. And that's how we end up with dozens of different company sites, each crappy in their own way, and in the middle of it all, people trying to find a home pulling their hair out, refreshing a bunch of different websites every few hours.
vageli 5y ago
This is a great project and reminds me of a small project I put together when searching for an off-grid property. Price, property size, and distance to home were key constraints and using something like follium would have made it much nicer than the tabular format I used. Thanks for sharing!
dangus 5y ago
I appreciate the talent and ingenuity behind this apartment hunting project. I just wonder if OP ever thought of contacting a broker? I would have done that before I wrote a single line of code. Scraping all this data is really great, but a human broker is going to know the local area and the other human nuances of apartment searching in the local area. Plus, brokers already have access to their own software that’s different than what’s on the public Internet, along with human connections. Why reinvent the wheel? My last apartment came up as an option before it even hit any sort of public property advertisement platform, and that was thanks to our broker working directly with the landlord.
- mattrighetti 5y ago
  I didn't actually, I'm trying to save as much money as possible at the moment and while a broker would be a good solution it would be a bit pricey for me. Maybe when I'm older and willing to spend money on relocating that would be a better solution for sure!
  - dangus 5y ago
    A lot of brokers work on commission that they get from the landlord. At least where I live, you don’t really pay for a broker as a buyer/renter, but I admittedly don’t know how that works in Luxembourg. A lot of brokers specialize on ex-pats, too! To expand on my point about knowing the local area, brokers know things like “what kind of person likes to live here?” Is it a young and trendy area or a quiet family neighborhood? (Admittedly, that information can be based on biases rather than data.) It’s something to think about. I know you might be looking to save money, but where you live is going to be a major purchase decision. Plus, spending time on your own apartment hunting platform isn’t technically free, your time is valuable. In any event, I’m sure with the kind of skills and dedication you have you’ll find a place in no time.
    - jrockway 5y ago
      > A lot of brokers work on commission that they get from the landlord. It depends on the city. In NYC, the broker is pretty much always paid for by the person looking for an apartment, and it's quite a bit of money. I haven't looked for an apartment for 10 years, but I think it was 5-10% of the annual rent. (I imagine this gravy train is coming to its end, though. 10% was fine when your employer was paying for relocation, but in the All Remote future, this free money goes away.)
bilekas 5y ago
Ciao, I had the same problem moving to Italy, specifically Bologna.. Super student rich town so apartments are gone fast, I setup a crawler of the usual sites but one or two had live updates for locations that would be published through their site's websocket, after a few hours I had the rasPi subscribed and could get notifications to me as soon as they were available, bypassing the need to sit on the website. It was a pain after for bureaucracy but that was after landing an apartment.
wildmanx 5y ago
Looks to me like procrastination from doing what he actually needs to do. Which is to click on those links to find an apartment. Tech won't solve that. Obligatory xkcd: https://xkcd.com/974/ https://xkcd.com/974/
- mattrighetti 5y ago
  I partially agree! But atm I’m using the tool a lot, it’s working well and as expected. Let’s say that for the time it took me to develop it was definitely worth it
  - wildmanx 5y ago
    Good for you! Glad you had some fun. And you are in the beginning of your career, can just as well post it to make yourself attractive for future employers. Better solve an actual problem that you had than to post some made-up nonsense-projects on Github just because people think you gotta have "a Github" when applying somewhere.
- actually_a_dog 5y ago
  I’d call it more of a variant of the XY problem: https://xyproblem.info/ https://xyproblem.info/
binarymax 5y ago
I love blogs like this. Showing the dirty side of data acquisition with an end result of useable software, that hacks around provider limitations to give something way better. We’ll done.
pimlottc 5y ago
It frustrates me that there are so many "information programs" like these that are computationally easy to solve but rely on the free availability of information. Sadly, companies like athome.lu have no interest in sharing this data and so here we are in 2022, still having to manually search multiple sites. In Internet olden times, we dreamed about automated agents that would run searches on your behalf, scour the web to compare prices and hunt down things you were interested in. But user-centric fantasies like this don't make fortunes, so walled gardens is what we ended up with.
- barnabee 5y ago
  This is why companies must be forced to make their data available. The value that could be unleashed is insane.
  - judge2020 5y ago
    athome.lu is specifically selling the 'search all listings' functionality, though. If they had to make this data available they wouldn't be interested in making the data in the first place, since it'd basically be charity work for whoever else is interested in throwing that data into elasticsearch and creating a frontend for it whilst undercutting athome.
  - golergka 5y ago
    How would bankrupting these companies would unleash value, exactly?
    - Gigachad 5y ago
      How is mandatory data reporting linked to bankruptcy? The more information available to the market, the better and more efficient they run.
      - deleted 5y ago
        [deleted]
      - 5y ago
abhaynayar 5y ago
I'm looking for an apartment in Bangalore right now, and hope to be done with it soon.
sofixa 5y ago
This reminds me of the small Go utility i wrote to help when i was in a similar situation. My focus was entirely on checking the location though, and see how much time it'd take to get to my various points of interest ( work addresses, places where things happen, favourite restaurants and bars, etc.). (It was just one of the things we were looking at with my SO, but it was a factor). It's all Google Maps with arguments so i should probably open source it, someone might find it useful.
aqme28 5y ago
I'm currently searching in Berlin and the market has become insane. Here's a response I got yesterday: "With more than 1200 prospective tenants, it was really difficult for the landlord to make a decision. Unfortunately, we have to inform you that the landlord has decided on other prospective tenants. Thank you for your understanding! "
- baobabKoodaa 5y ago
  This sounds like rent caps.
  - ghaff 5y ago
    My initial reaction was that it sounds as if they're not charging nearly enough. You may want to be picky but that's ridiculous. But you may be right.
    - wildmanx 5y ago
      These 1:1200 are exceptional anecdotes that happen if you have some idealistic landlord that doesn't max out the allowed rent but rents it out at some price point that they consider reasonable, which is perhaps the rent from 20 years ago. Of course they get flooded with applications. If they'd max out, they'd still get multiple applications, but maybe 10-30 of those, half of them from people with bad credit rating. Still bad, but not as extreme as it sounds like.
    - aqme28 5y ago
      That’s part of it. The place was far below market.
  - TillE 5y ago
    Rents have absolutely skyrocketed over the past 15 years or so (3-4x), despite mild controls, so no not really.
- pacetherace 5y ago
  1:1200 is probably worse than most selective universities. I know it is not apples to apples comparison but it’s the first thing that came to my mind.
jean-malo 5y ago
In highly competitive markets the good apartments don't even make it to the websites that aggregate listings before someone takes them. The ones that do are either very expensive or will have 20+ people showing up the next day (sometimes the same day) for a visit. It's crazy and while this article is very neat I'm not sure a bot that scrapes listing in batch is the answer.
- foobarian 5y ago
  A long time ago I knew someone who's parents found them an excellent apartment at a great price in Boston. Great neighborhood, 2BR, nicely kept, off-street parking, you name it. Few years later they had to move since the owner was selling the place, and went on an apartment search by themselves (using an agency). Could not find anything anywhere near nice at that price point, and settled on something a lot more expensive. I decided that their parents must have tipped/bribed the original agent to get at that listing (since none of these ever made it to any search site, not that there were many at that time).
  - judge2020 5y ago
    I don’t know the story of course, but apartments are one of the largest profit growth centers for capital firms; most metros have experienced >50% rent growth in the past decade[0], for instance. Chances are a renter will never see a rent decrease unless they downgrade in a key area (specs, amenities, location) or are moving in very short intervals. 0: https://www.self.inc/blog/cities-biggest-increase-rental-prices https://www.self.inc/blog/cities-biggest-increase-rental-pri...
- switch007 5y ago
  We only got ours because we were very nice to the agent after losing out on an apartment (thanked her, asked her to keep us in mind if anything else comes up etc). A week later she phoned us before she listed a new one, and we went to see it 3 hours later and took it. It goes to show the human aspect still matters, a lot. That one email thanking her was probably worth way way more than eg hours spent on a bot to automate applications. Many apartments also don’t even make it to an agent because people have friends they promise it to and refer them to the landlord.
spike021 5y ago
About ten years ago I was attending San Jose State University and really needed to find a 1 bedroom apartment at a decent price (lol) in the downtown SJ area since I would no longer be living in the dorms. I tried a few different ways of finding an apartment, but in the end it was one case where the old phrase "it's not what you know, it's who you know" worked in my favor. During my freshman year I had found and been working a job on campus in one of the department offices (side note: if you're a student, you're best off trying to find a student assistant opening in a department office; at least at SJSU, most paid above minimum wage, were pretty lax and could be used for some study time, and you have so many opportunities to make connections with professionals). One of my coworkers there had a family and they were living in one of these small ~8 unit apartment buildings downtown. It was a privately-owned building, not one of these newer, fancy conglomerate ones. She connected me with the management/owner and long story short, I got a decently livable place with rent control for about 1/3 of market rate due to being a lower income student and taking a unit that wasn't recently refurbished (old carpet, old paint, etc.).
sl3232323 5y ago
Why don't you just ask someone if they have an apartment? We rely so much on technology, when we could just ask a friend. Currently living in Belize for $200 a month, beachfront propety.
candiddevmike 5y ago
Government should provide apartments, free, in the center of the city, for anyone who needs one. Housing should be a human right. You can pay extra for a better place, but then landlords would actually have to compete for your business instead of the other way around.
- jfk13 5y ago
  Interesting idea. What tax rate are you ready to pay? Oh, and who defines "needs one"?
- dflock 5y ago
  I would love some city or government to try this experiment!
  - bluquark 5y ago
    The closest to it today is Finland's "Housing First" policy: https://ysaatio.fi/en/housing-first-finland https://ysaatio.fi/en/housing-first-finland But even in Finland, housing is only sometimes free and then on a net basis (rent can be canceled out by a means-tested housing benefit).
- traceroute66 5y ago
  > Government should provide apartments, free, in the center of the city, for anyone who needs one. During COVID lockdowns I watched lots of international TV drama series on Netflix. One of those "random things I learnt on Netflix" was that they (sort of) offer this in the UAE if you've lived & worked there for 10+ years.[1] Its basically means-tested housing. So they offer anything from interest-free 25 year loans to fully subsidised housing. [1] https://www.tamm.abudhabi/en/aspects-of-life/benefits-for-emiratis/landhomeallocation/Grants/emirati-housing-programmes-in-abu-dhabi https://www.tamm.abudhabi/en/aspects-of-life/benefits-for-em...
  - sofixa 5y ago
    The UAE are in a uniquely blessed situation where they have a small population, not a lot of land, and a ton of valuable resources. Not every country could pull this off without significantly reworking how it's tax/budget/administration work. That being said, there's subsidised, including to 100%, housing in France too. And cities are legally required to build affordable housing for those schemes, so it's not "it exists on paper, but nothing has been built in decades so nobody can get in".
johnchristopher 5y ago
Why not downloading the html page and scrape that file instead of using puppeteer (which uses a headless chrome I think) ?
- mattrighetti 5y ago
  I didn't have an easy solution to grab that JSON variable directly by scraping HTML. So I chose Puppeteer because it simulates the browser (yes, chrome) and therefore I could just get the JSON object with a bunch of lines as if I was using the browser's console. It's slow, but I think it's the most straightforward way to do it. But if you could elaborate more on your approach I will definitely try that out!
  - grose 5y ago
    I had a similar problem that I solved with goquery and otto. You can use goquery to traverse the DOM and otto to execute the script fragment. Then just grab the data from otto's VM. Your scraping being slow and using Chrome might be a blessing in disguise though. If you aren't careful you can get detected as a bot and banned from the site. https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery https://github.com/robertkrimen/otto https://github.com/robertkrimen/otto
Victerius 5y ago
Cheap. Spacious. In a desirable zip code. Pick two.
- raphar 5y ago
  You can make it into a theorem! Cheap, Area code, Palatial. Pick two. Nevermind, someone beat us to it.
Jeaye 5y ago
I did a similar thing to help me land an apartment 5 years ago, using Clojure: https://blog.jeaye.com/2017/02/28/clojure-apartments/ https://blog.jeaye.com/2017/02/28/clojure-apartments/ However, my approach didn't include these map visuals; that's a great idea and a nice execution!
asiachick 5y ago
I wish all apartment listing included a floor plan. I could easily reject 90% of apartments without having to go personally look at it if they's show me a floor plan. It's a little surprising to me they aren't more common. In Japan they're often missing other things like lots of pictures but at least the number 1 requirement to list an apartment is a floor plan. https://suumo.jp/jj/chintai/ichiran/FR301FC001/?ar=030&bs=040&ra=014&ae=02201&cb=0.0&ct=9999999&et=9999999&md=07&cn=9999999&mb=0&mt=9999999&shkr1=03&shkr2=03&shkr3=03&shkr4=03&fw2=&ek=022040640&rn=0220 https://suumo.jp/jj/chintai/ichiran/FR301FC001/?ar=030&bs=04... I wish Craigslist would require one or somehow strongly suggest that the person offering the apartment would get better matches and have less of their own time wasted if they'd add a floor plan. I also wish they'd add some examples of good and bad pictures. So many people take pictures with their phone and the field of view is so small and lighting so bad the pictures are basically useless. I'm not saying they need pro-equipment but my guess is there are at least some guidelines that would be helpful. How about no portrait mode pics as just one example that would likely lead to more useful images.
- YouMeWeThem 5y ago
  In Japan they're also often missing the "willing to rent to foreigners" flag. Using aggregate sites like Suumo to find places you like ends up being a waste of time when so many places won't even consider you.
mihaaly 5y ago
"creating a pipeline with different tools and languages to scrape data from a website with the use of Puppeteer, load data into a SQLite database using Go and visualizing all the data with Python and Folium." So much sophistication based on at least partly incomplete, filtered/distorted, and inaccurate data. So much important aspects that cannot be in an ad or intentionally left out and can only be extracted by painstaking efforts. Even the initial filtering out could be unreliable then.
madrox 5y ago
I did something similar 10 years ago during my move to SF. I needed to find an apartment within two weeks, so I built a crawler to do complex filtering and notify me immediately. In my case, the tool was ultimately useless because I discovered the way SF housing worked for nice apartments was 1) Openings are posted on Friday 2) Saturday, open house collects applications 3) Deal is signed on Sunday and post taken down. Anything my crawler picked up was stuff that was overpriced or in bad condition. For all my sophistication, the dataset was just the bad stuff. Apartment hunting is a great example of needing to know the underlying behaviors of your dataset before you use it to make decisions.
zoover2020 5y ago
Moien! Great article. Something tells me you're joining a big online book store ;-). Tell me when you're in town for a beer, would love to hear how this unfolds.
baby 5y ago
I find the flow in the US the simplest: 1. go to craigslist 2. play with the filter 3. add keywords in the search bar like "rent control" and "-studio" 4. go to the maps page 5. bookmark the page 6. check every day In the US/France, there's too much fragmentation and real estate agents In China I would go to the lady selling phone cards in the street and managed to visit a lot of apartments like that.
- asdff 5y ago
  If you log in with craigslist you can save the search instead of having to bookmark and manually recheck.
ilovefood 5y ago
Ah... this brings back memories. A few years ago I wrote this article https://funnybretzel.com/datamining-a-flat-in-munich/ https://funnybretzel.com/datamining-a-flat-in-munich/ when I was searching a flat in Munich. Some time has passed since then and meanwhile I've enhanced the process quite a lot, just like OP I'm currently searching for a place (a house) and I'm using quite the little bot farm to massively automate things. I'll post the results if it converges to something useful. Meanwhile, all the luck to OP and excellent write-up! :)
mreiner 5y ago
Interesting details brought up here about location specifics. At least in Zurich most ads contain an address. I fed it to google maps api to get the time by bike to my workplace which gave us some cheaper options we wouldn't have considered before. We also ended up taking/getting the one with the worst pictures. Imagine someone would scrape all platforms for your city of choice and aggregate those with some value added and maybe automatically apply with your personal documents to make sure your application gets in first even when you're busy, what would it be worth to you? Despite local differences it might still scale.
egberts1 5y ago
This is the prime example of a phenomenal approach to aggregation of various data using various languages, in this case on about hunting for a decent apartment. Absolutely stellar form of data science! Keep up the great work!
dayvid 5y ago
If I have time, I always prefer to walk around the neighborhood I want to live in an see if there’s signs or hear from people if a place is available informally. That’s where you get the best deals. If you find the seller, you usually get a better deal. If the seller finds you, …
01100011 5y ago
Renting is fundamentally broken. You submit precious personal info over to a random stranger along with payment to check your credit. Last time I checked, this fee is $25-50 USD. You've now given them enough information to steal your identity. You also may need to do this multiple times if you're rejected. The person receiving this information rarely has identity protection measures in place and you receive no guarantees that they've destroyed this information. A much better solution would be to develop an escrow service which acts as a middleman between renters and landlords. Aspiring renters could sign up with the escrow and submit their information once. Their one-time fee would cover a fixed number of queries by potential landlords. The queries could include such information as income validation, criminal history, job history, rental history, etc. But these queries could be approved by the renter in advance. This sort of query process allows the renter to control the types of information given out. The queries could be structured to allow pass/fail results only, so that landlords do not receive intimate details but only know whether or not a renter is acceptable based on their criteria. Once you have a system like that, you can open it up to all sorts of validation queries. Think of it as a general personal information escrow service. This idea has been bouncing around in my head for over a decade now but I'm more of a low-level systems engineer so I never got around to trying to implement it. Seems like a decent startup idea though.
- deleted 5y ago
  [deleted]
- turtlebits 5y ago
  A service similar to what you’re proposing called Smartmove already exists. The renter pays them for the application + credit/background/etc checks. You don’t have to handle payment or take SSNs. Unfortunately pass/fail doesn’t work due to nuance in income/credit/rental history.
- rodonn 5y ago
  Zillow does have a product like this: pay one application fee and you can reuse the background check across as many apartments as you apply to in a month. The biggest downside is that it only works for landlords that chose to accept the service.
4midori 5y ago
It's sad that this process is so broken that it takes a skilled coder to roll their own, just to get good, organized information. We were promised jetpacks!
lucb1e 5y ago
Actual title: Scraping apartment offers to show them on a map (since the current title just conveys some general topic of apartment need, with no indication if it's about "my partner died and it's interesting to notice how large a whole house is" or something)
deleted 5y ago
[deleted]
de-asis-kevin 5y ago
How many major cities in the world do we have right now where prices of rent did not go up? I'm really curious to know the breakdown of home/building ownership in the world, and how much the person makes in that city. Curious how much property is owned by individuals vs companies.
svnee 5y ago
I shared your link with some colleagues working at atHome and their CEO replied that your approach was interesting. Once you are in Luxembourg, feel free to reach out, I’ll buy you a beer to console you for the expensive apartment prices ;) Always interested in what people are up to (both as an entrepreneur as well as a politican)
pipeline_peak 5y ago
> TL;DR: I hate apartment hunting, I’ve tried to make it as interesting as possible by creating a pipeline with different tools and languages to scrape data….. Grow up, learn to take on boring tasks without playing with toys
asdff 5y ago
Despite all the fancy new websites that crop up every couple of years in the apartment rental space, imo the best place to find an apartment remains craigslist. I've only ever leased spaces through there. For the mom and pop landlords who you really want as your landlord, they might get a new client every 10 years. 10 years ago, they were using craiglist, and it works exactly the same as today, so that's what they continue to use. If it gets them a tenant its good enough to not bother learning some other new website. For the large corporolandlords who have those bougie amenities that perhaps you crave, they will be posting on all the major websites including craigslist anyhow, so you might as well use craigslist for that use case too if only to use a less bloated website.
pbasista 5y ago
Tangential. Information on the internet is often intentionally incomplete or difficult to filter. For instance, it is impossible to filter apps in the Google Play store by the presence of ads or in-app purchases within them. Why? Because Google wants it that way. In order to perform such a filtered search, one needs to create a custom crawler, then parse the data and then filter them. If anything, with all the ads and SEO games taking place, searching for the relevant information has become _more_ difficult than in the early days of Google. It is 2022 and we still do not have a universal search system which would allow everyone to quickly and efficiently perform more sophisticated searches. The most skilled and determined people are capable of doing so by creating various single-purpose data collection and data mining tools themselves. But even for them it takes a non-trivial time and effort to produce something usable. There is nothing out there (that I know of) which would be easily applicable to everyday needs.
- joshspankit 5y ago
  Further tangent: Many of the public search boxes are backed by elasticsearch or other similar “recommendation” tools which are optimized for speed and point-someone-in-the-right-direction accuracy. Since we no longer have exact text matching or many of the advanced operators (such as `not`), I’d argue that for sophisticated searches search boxes have actually gotten significantly worse.
- javajosh 5y ago
  The OP points the way, I think, to what will happen in practice. Three trends will meet: 1, more apps live in the browser, 2, search programs that combine data in the browser get formalized/paramterized, 3, user friendly methods arise to make writing ad hoc search programs very easy (e.g. voice driven, context aware AI).
nomy99 5y ago
I am so lucky to have found a room in a descent apartment in a nice part of NYC. As a software engineer making high income, I still almost got homeless in the apartment hunting. Insane!
joeman1000 5y ago
Monospaced fonts are not nice for prose.
rtpg 5y ago
Kinda funny side story: I tried scraping Homes (the Japanese equivalent to this site) for a small subsection of data, basically instantly got IP banned and could not use the site consistently until I outright moved to a new location and made a new internet contract Some people take protecting their data seriously
thiht 5y ago
I did the same thing when apartment hunting in Rennes, France! And this is how I found my current apartment :) There’s great value in building scrapers specialized to a particular city, country-wide aggregators are not enough.
WaffleIronMaker 5y ago
The author might be able to benefit from using Promise.all() to perform his async searches concurrently. Also, I would recommend closing each page when you are done with it, using page.close().
- mattrighetti 5y ago
  I've received a PR suggesting just that [0], my concern is > Only thing that concerns me is that it will spawn n parallel requests to the website which could consume a lot of memory since we're talking about spawning a new tab with Puppeteer (Chrome :sigh:) for each link. [0]: https://github.com/mattrighetti/athome-scraper/pull/1 https://github.com/mattrighetti/athome-scraper/pull/1

350 points by mattrighetti 5y ago

May be I am too old fashioned but I have never been able to select apartments like this mathematically . Sure I have used methods like this to shortlist the apartments, but I still need to visit them and feel for the place before commiting to one place.

mattrighetti 5y ago
Nah you’re doing it right! I’m just using the tool to select the best ones to eventually visit and maybe rent

ChuckNorris89 5y ago

At least OP had the full address of those apartment on that website, so he could plot them on the map. Here in Austria, most leeches, sorry, realtors, avoid putting the street in the ad to prevent you from trying to contact the property directly and bypassing their fees, so apartment hunting is even more of a headache. Still, amazing work from OP.

dazbradbury 5y ago
There are genuine privacy concerns at play here, for example there may be existing tenants/owners who don't want people randomly knocking on the door because someone saw an ad online. Or getting a bunch of junk mail because there is an extra data point about the property. I guess the variety of concerns will vary from country to country depending on local laws / existing open data though.
- bluefirebrand 5y ago
  I don't think it is reasonable to expect to be able to list real estate for rent or sale, and also expect privacy with regards to the address.
  - giraffe_lady 5y ago
    Yeah but this is a "the customer isn't the user" situation. The tenant wants and deserves privacy, and it's their privacy that can be compromised by the listing. The owner doesn't really have a reason to care, I don't think, and may even be incentivized in some situations to reveal information the tenants would rather they not.
    - fennecfoxen 5y ago
      What information? “This apartment exists at LOCATION and will be available for rent at DATE?” And you think the tenant is entitled to stop the world from knowing that because the tenant is residing at that location before DATE? Goodness. I should give the First Amendment a hug.
      - giraffe_lady 5y ago
        I didn't make any of those claims at all. I just pointed out that if there is a privacy issue, it's not the lister's privacy at stake.
        bluefirebrand 5y ago
        I'm still not sure renters really have a reasonable expectation of privacy when the place they are living in is listed for rent or sale by someone else. In most places renters are only granted 24 hours notice to have complete strangers come walk through their living spaces while the space is shown off. They have no real right to refuse to allow that. If people start knocking on their door asking to come look at the place they are free to refuse, or ignore the doorbell or anything else. Of course they have a freedom against being harassed, so if people persist they can get police involved. That seems like a separate issue to me from having their address posted online in a rental listing.
        ChuckNorris89 5y ago
        >I'm still not sure renters really have a reasonable expectation of privacy when the place they are living in is listed for rent or sale by someone else In some EU countries they do. The landlord need to ask permission to have visitors at your place and the tenant has the right to refuse.
est31 5y ago
When I searched for my apartment, I was able to reverse engineer the home's address sometimes when the general vicinity was specified, then looking at the pictures, and comparing them with Google street view. This was a manual process however and only worked if enough outside views were available. Despite the time effort, you still spent less time on it per house than making an appointment for a viewing, getting there, and looking at it. If Google allows you to download the street views for an entire city, one might build an ML solution to bring up candidates at least. Similarly, in the city where I studied for university, I immediately knew where the house was located just from outside pictures of it in the ad, at least for the districts of the city I was familiar with.
- fennecfoxen 5y ago
  I use a similar process to look at interesting job listings sent to me unsolicited by recruiters. You can anonymize the company but it only takes a few unique phrases to locate the posting with Google.
- alex-nt 5y ago
  I am looking for an apartment in eastern Europe. I was shocked to see listings with the windows covered in Microsoft Paint. I was able to find the location in some instances by finding the same property in other listings (the images were still slightly altered). This is the amateur move, what I see more often is image alteration so that the light coming through the windows makes it impossible to see what is outside. The same apartment is usually posted by 5-10 agencies, pictures and description almost identical, with different location. You get different responses depending who you call. I hope to end my suffering this year and buy something.
  - est31 5y ago
    Oh it is definitely possible to avoid this. It's kind of an arms race. The overexposed window effect might be reversed with computational photography for example. Maybe one can just lower brightness in an image editor and discover that there is a residual of the outside left. You might use "looking around a corner" like tricks on white walls, etc etc.
tecleandor 5y ago
Oh, around here (Spain) usually they don't put the address in the ad so you can't cross check the information with the national registry and find that the apartment is 20% smaller than claimed.
sswezey 5y ago
Most of the apartments on that site don't have the actual addresses, instead just have the city/town. It's the same issue you've experienced.
micheloosterhof 5y ago
I've searched for houses in multiple places around the world and the unwillingness to provide the full addresses is often due to agent competition. More precise whether the seller has one or more selling agents. In places like The Netherlands, a seller generally works with one seller's agent only, and there's no incentive to hide full addresses. Funda.nl has a huge amount of detail, from high quality photos, to floor plans. On the contrary, in Singapore, a seller often has multiple agents. The agents compete with each other and would rather not share any information at all publicly to avoid other agents using their information/photos, or buyer's bypassing them. It leads to propertyguru.com.sg with no addresses, bad photos, all photos watermarked with the agent pictures. As little information as possible, just enough to draw you into an in-person visit.
andbberger 5y ago
can you register as a broker on the austrian version of MLS to get the listings

denysvitali 5y ago

Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible). I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here. I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust. Whilst the application is not available to the public (yet), all of the libraries I've created are now available. I should have both a Rust and a Golang version for most / all of them. On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites. [0]: https://github.com/denysvitali/sbb-api-rs https://github.com/denysvitali/sbb-api-rs [1]: https://github.com/denysvitali/homegate-rs https://github.com/denysvitali/homegate-rs [2]: https://github.com/denysvitali/flatfox-rs https://github.com/denysvitali/flatfox-rs [3]: https://github.com/denysvitali/go-sbb-api https://github.com/denysvitali/go-sbb-api

vageli 5y ago
> I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust. How did you convert an address to long/lat for postgis?
- denysvitali 5y ago
  I was lucky, the data also included lat/lng and the address for most (if not all) the entries. If I would have been confronted with such a situation though, I would have probably used some Geocoding API like Google Maps's geocoding [0] or OSM's Nominatim [1]. [0]: https://developers.google.com/maps/documentation/geocoding/overview https://developers.google.com/maps/documentation/geocoding/o... [1]: https://wiki.openstreetmap.org/wiki/Nominatim https://wiki.openstreetmap.org/wiki/Nominatim
- mattrighetti 5y ago

haswell 5y ago

I'm in the middle of an apartment search. I was reflecting the other day on how much searching for an apartment has changed over the last 10 years. Conclusion: it hasn't changed much at all. It's still necessary to search on a myriad of sites. Most of the time, by the time you inquire, a place is gone. Or the ad was misleading and that "$2000 2BR" is actually "$2000 for the cheapest Studio we have, but we also have 2BR for $3.5K". Finding places that match my criteria still requires careful inspection of the description. Checking a filter box for "Parking" excludes places that only mention parking in the description. With all of the modernization and streamlining that has happened in other parts of our lives, apartment hunting remains unstructured, messy, frustrating, stressful. Aggregators like Padmapper helped for awhile, but these days, it seems like most listings are posted by scammy middlemen just trying to make a commission on the listing, and now the site actively makes the process even harder. Is hunting for an apartment doomed to be this messy forever? It seems like an area that is ripe for disruption or a better option, but what would it take to actually solve this? --- With all of that said, this post is exactly what I needed right now. I've been toying with a very similar idea to improve the efficiency of my search process and this is good inspiration.

deleted 5y ago
[deleted]
Etheryte 5y ago
As someone who just jumped through all the hoops to find a new apartment, the pain is real. I think the main issue is that the incentives are misaligned from the perspective of the seeker. Where I'm currently at (the Netherlands), the market is dominated by agents. Using an agent makes sense from the perspective of the apartment owner — they will take care of managing the ads, initial screening, arranging viewings etc. Given how many people contact a single viewing, you definitely don't want them hitting your personal inbox or phone. From the agent's perspective, it's good to drive people to their company's website rather than one of the big aggregate ones. Having more people visit it means they're more likely to do business with that company specifically. Getting people to visit your company's site is pretty straightforward: they put up ads there earlier, often so much earlier that by the time you see an ad on an aggregate site, it's already fully booked for viewings. And that's how we end up with dozens of different company sites, each crappy in their own way, and in the middle of it all, people trying to find a home pulling their hair out, refreshing a bunch of different websites every few hours.

vageli 5y ago

This is a great project and reminds me of a small project I put together when searching for an off-grid property. Price, property size, and distance to home were key constraints and using something like follium would have made it much nicer than the tabular format I used. Thanks for sharing!

dangus 5y ago

I appreciate the talent and ingenuity behind this apartment hunting project. I just wonder if OP ever thought of contacting a broker? I would have done that before I wrote a single line of code. Scraping all this data is really great, but a human broker is going to know the local area and the other human nuances of apartment searching in the local area. Plus, brokers already have access to their own software that’s different than what’s on the public Internet, along with human connections. Why reinvent the wheel? My last apartment came up as an option before it even hit any sort of public property advertisement platform, and that was thanks to our broker working directly with the landlord.

mattrighetti 5y ago
I didn't actually, I'm trying to save as much money as possible at the moment and while a broker would be a good solution it would be a bit pricey for me. Maybe when I'm older and willing to spend money on relocating that would be a better solution for sure!
- dangus 5y ago
  A lot of brokers work on commission that they get from the landlord. At least where I live, you don’t really pay for a broker as a buyer/renter, but I admittedly don’t know how that works in Luxembourg. A lot of brokers specialize on ex-pats, too! To expand on my point about knowing the local area, brokers know things like “what kind of person likes to live here?” Is it a young and trendy area or a quiet family neighborhood? (Admittedly, that information can be based on biases rather than data.) It’s something to think about. I know you might be looking to save money, but where you live is going to be a major purchase decision. Plus, spending time on your own apartment hunting platform isn’t technically free, your time is valuable. In any event, I’m sure with the kind of skills and dedication you have you’ll find a place in no time.
  - jrockway 5y ago
    > A lot of brokers work on commission that they get from the landlord. It depends on the city. In NYC, the broker is pretty much always paid for by the person looking for an apartment, and it's quite a bit of money. I haven't looked for an apartment for 10 years, but I think it was 5-10% of the annual rent. (I imagine this gravy train is coming to its end, though. 10% was fine when your employer was paying for relocation, but in the All Remote future, this free money goes away.)

bilekas 5y ago

Ciao, I had the same problem moving to Italy, specifically Bologna.. Super student rich town so apartments are gone fast, I setup a crawler of the usual sites but one or two had live updates for locations that would be published through their site's websocket, after a few hours I had the rasPi subscribed and could get notifications to me as soon as they were available, bypassing the need to sit on the website. It was a pain after for bureaucracy but that was after landing an apartment.

wildmanx 5y ago

Looks to me like procrastination from doing what he actually needs to do. Which is to click on those links to find an apartment. Tech won't solve that. Obligatory xkcd: https://xkcd.com/974/ https://xkcd.com/974/

mattrighetti 5y ago
I partially agree! But atm I’m using the tool a lot, it’s working well and as expected. Let’s say that for the time it took me to develop it was definitely worth it
- wildmanx 5y ago
  Good for you! Glad you had some fun. And you are in the beginning of your career, can just as well post it to make yourself attractive for future employers. Better solve an actual problem that you had than to post some made-up nonsense-projects on Github just because people think you gotta have "a Github" when applying somewhere.
actually_a_dog 5y ago
I’d call it more of a variant of the XY problem: https://xyproblem.info/ https://xyproblem.info/

binarymax 5y ago

I love blogs like this. Showing the dirty side of data acquisition with an end result of useable software, that hacks around provider limitations to give something way better. We’ll done.

pimlottc 5y ago

It frustrates me that there are so many "information programs" like these that are computationally easy to solve but rely on the free availability of information. Sadly, companies like athome.lu have no interest in sharing this data and so here we are in 2022, still having to manually search multiple sites. In Internet olden times, we dreamed about automated agents that would run searches on your behalf, scour the web to compare prices and hunt down things you were interested in. But user-centric fantasies like this don't make fortunes, so walled gardens is what we ended up with.

barnabee 5y ago
This is why companies must be forced to make their data available. The value that could be unleashed is insane.
- judge2020 5y ago
  athome.lu is specifically selling the 'search all listings' functionality, though. If they had to make this data available they wouldn't be interested in making the data in the first place, since it'd basically be charity work for whoever else is interested in throwing that data into elasticsearch and creating a frontend for it whilst undercutting athome.
- golergka 5y ago
  How would bankrupting these companies would unleash value, exactly?
  - Gigachad 5y ago
    How is mandatory data reporting linked to bankruptcy? The more information available to the market, the better and more efficient they run.
    - deleted 5y ago
      [deleted]
    - 5y ago

abhaynayar 5y ago

I'm looking for an apartment in Bangalore right now, and hope to be done with it soon.

sofixa 5y ago

This reminds me of the small Go utility i wrote to help when i was in a similar situation. My focus was entirely on checking the location though, and see how much time it'd take to get to my various points of interest ( work addresses, places where things happen, favourite restaurants and bars, etc.). (It was just one of the things we were looking at with my SO, but it was a factor). It's all Google Maps with arguments so i should probably open source it, someone might find it useful.

aqme28 5y ago

I'm currently searching in Berlin and the market has become insane. Here's a response I got yesterday: "With more than 1200 prospective tenants, it was really difficult for the landlord to make a decision. Unfortunately, we have to inform you that the landlord has decided on other prospective tenants. Thank you for your understanding! "

baobabKoodaa 5y ago
This sounds like rent caps.
- ghaff 5y ago
  My initial reaction was that it sounds as if they're not charging nearly enough. You may want to be picky but that's ridiculous. But you may be right.
  - wildmanx 5y ago
    These 1:1200 are exceptional anecdotes that happen if you have some idealistic landlord that doesn't max out the allowed rent but rents it out at some price point that they consider reasonable, which is perhaps the rent from 20 years ago. Of course they get flooded with applications. If they'd max out, they'd still get multiple applications, but maybe 10-30 of those, half of them from people with bad credit rating. Still bad, but not as extreme as it sounds like.
  - aqme28 5y ago
    That’s part of it. The place was far below market.
- TillE 5y ago
  Rents have absolutely skyrocketed over the past 15 years or so (3-4x), despite mild controls, so no not really.
pacetherace 5y ago
1:1200 is probably worse than most selective universities. I know it is not apples to apples comparison but it’s the first thing that came to my mind.

jean-malo 5y ago

In highly competitive markets the good apartments don't even make it to the websites that aggregate listings before someone takes them. The ones that do are either very expensive or will have 20+ people showing up the next day (sometimes the same day) for a visit. It's crazy and while this article is very neat I'm not sure a bot that scrapes listing in batch is the answer.

foobarian 5y ago
A long time ago I knew someone who's parents found them an excellent apartment at a great price in Boston. Great neighborhood, 2BR, nicely kept, off-street parking, you name it. Few years later they had to move since the owner was selling the place, and went on an apartment search by themselves (using an agency). Could not find anything anywhere near nice at that price point, and settled on something a lot more expensive. I decided that their parents must have tipped/bribed the original agent to get at that listing (since none of these ever made it to any search site, not that there were many at that time).
- judge2020 5y ago
  I don’t know the story of course, but apartments are one of the largest profit growth centers for capital firms; most metros have experienced >50% rent growth in the past decade[0], for instance. Chances are a renter will never see a rent decrease unless they downgrade in a key area (specs, amenities, location) or are moving in very short intervals. 0: https://www.self.inc/blog/cities-biggest-increase-rental-prices https://www.self.inc/blog/cities-biggest-increase-rental-pri...
switch007 5y ago
We only got ours because we were very nice to the agent after losing out on an apartment (thanked her, asked her to keep us in mind if anything else comes up etc). A week later she phoned us before she listed a new one, and we went to see it 3 hours later and took it. It goes to show the human aspect still matters, a lot. That one email thanking her was probably worth way way more than eg hours spent on a bot to automate applications. Many apartments also don’t even make it to an agent because people have friends they promise it to and refer them to the landlord.

spike021 5y ago

About ten years ago I was attending San Jose State University and really needed to find a 1 bedroom apartment at a decent price (lol) in the downtown SJ area since I would no longer be living in the dorms. I tried a few different ways of finding an apartment, but in the end it was one case where the old phrase "it's not what you know, it's who you know" worked in my favor. During my freshman year I had found and been working a job on campus in one of the department offices (side note: if you're a student, you're best off trying to find a student assistant opening in a department office; at least at SJSU, most paid above minimum wage, were pretty lax and could be used for some study time, and you have so many opportunities to make connections with professionals). One of my coworkers there had a family and they were living in one of these small ~8 unit apartment buildings downtown. It was a privately-owned building, not one of these newer, fancy conglomerate ones. She connected me with the management/owner and long story short, I got a decently livable place with rent control for about 1/3 of market rate due to being a lower income student and taking a unit that wasn't recently refurbished (old carpet, old paint, etc.).

sl3232323 5y ago

Why don't you just ask someone if they have an apartment? We rely so much on technology, when we could just ask a friend. Currently living in Belize for $200 a month, beachfront propety.

candiddevmike 5y ago

Government should provide apartments, free, in the center of the city, for anyone who needs one. Housing should be a human right. You can pay extra for a better place, but then landlords would actually have to compete for your business instead of the other way around.

jfk13 5y ago
Interesting idea. What tax rate are you ready to pay? Oh, and who defines "needs one"?
dflock 5y ago
I would love some city or government to try this experiment!
- bluquark 5y ago
  The closest to it today is Finland's "Housing First" policy: https://ysaatio.fi/en/housing-first-finland https://ysaatio.fi/en/housing-first-finland But even in Finland, housing is only sometimes free and then on a net basis (rent can be canceled out by a means-tested housing benefit).
traceroute66 5y ago
> Government should provide apartments, free, in the center of the city, for anyone who needs one. During COVID lockdowns I watched lots of international TV drama series on Netflix. One of those "random things I learnt on Netflix" was that they (sort of) offer this in the UAE if you've lived & worked there for 10+ years.[1] Its basically means-tested housing. So they offer anything from interest-free 25 year loans to fully subsidised housing. [1] https://www.tamm.abudhabi/en/aspects-of-life/benefits-for-emiratis/landhomeallocation/Grants/emirati-housing-programmes-in-abu-dhabi https://www.tamm.abudhabi/en/aspects-of-life/benefits-for-em...
- sofixa 5y ago
  The UAE are in a uniquely blessed situation where they have a small population, not a lot of land, and a ton of valuable resources. Not every country could pull this off without significantly reworking how it's tax/budget/administration work. That being said, there's subsidised, including to 100%, housing in France too. And cities are legally required to build affordable housing for those schemes, so it's not "it exists on paper, but nothing has been built in decades so nobody can get in".

johnchristopher 5y ago

Why not downloading the html page and scrape that file instead of using puppeteer (which uses a headless chrome I think) ?

mattrighetti 5y ago
I didn't have an easy solution to grab that JSON variable directly by scraping HTML. So I chose Puppeteer because it simulates the browser (yes, chrome) and therefore I could just get the JSON object with a bunch of lines as if I was using the browser's console. It's slow, but I think it's the most straightforward way to do it. But if you could elaborate more on your approach I will definitely try that out!
- grose 5y ago
  I had a similar problem that I solved with goquery and otto. You can use goquery to traverse the DOM and otto to execute the script fragment. Then just grab the data from otto's VM. Your scraping being slow and using Chrome might be a blessing in disguise though. If you aren't careful you can get detected as a bot and banned from the site. https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery https://github.com/robertkrimen/otto https://github.com/robertkrimen/otto

Victerius 5y ago

Cheap. Spacious. In a desirable zip code. Pick two.

raphar 5y ago
You can make it into a theorem! Cheap, Area code, Palatial. Pick two. Nevermind, someone beat us to it.

Jeaye 5y ago

I did a similar thing to help me land an apartment 5 years ago, using Clojure: https://blog.jeaye.com/2017/02/28/clojure-apartments/ https://blog.jeaye.com/2017/02/28/clojure-apartments/ However, my approach didn't include these map visuals; that's a great idea and a nice execution!

asiachick 5y ago

I wish all apartment listing included a floor plan. I could easily reject 90% of apartments without having to go personally look at it if they's show me a floor plan. It's a little surprising to me they aren't more common. In Japan they're often missing other things like lots of pictures but at least the number 1 requirement to list an apartment is a floor plan. https://suumo.jp/jj/chintai/ichiran/FR301FC001/?ar=030&bs=040&ra=014&ae=02201&cb=0.0&ct=9999999&et=9999999&md=07&cn=9999999&mb=0&mt=9999999&shkr1=03&shkr2=03&shkr3=03&shkr4=03&fw2=&ek=022040640&rn=0220 https://suumo.jp/jj/chintai/ichiran/FR301FC001/?ar=030&bs=04... I wish Craigslist would require one or somehow strongly suggest that the person offering the apartment would get better matches and have less of their own time wasted if they'd add a floor plan. I also wish they'd add some examples of good and bad pictures. So many people take pictures with their phone and the field of view is so small and lighting so bad the pictures are basically useless. I'm not saying they need pro-equipment but my guess is there are at least some guidelines that would be helpful. How about no portrait mode pics as just one example that would likely lead to more useful images.

YouMeWeThem 5y ago
In Japan they're also often missing the "willing to rent to foreigners" flag. Using aggregate sites like Suumo to find places you like ends up being a waste of time when so many places won't even consider you.

mihaaly 5y ago

"creating a pipeline with different tools and languages to scrape data from a website with the use of Puppeteer, load data into a SQLite database using Go and visualizing all the data with Python and Folium." So much sophistication based on at least partly incomplete, filtered/distorted, and inaccurate data. So much important aspects that cannot be in an ad or intentionally left out and can only be extracted by painstaking efforts. Even the initial filtering out could be unreliable then.

madrox 5y ago

I did something similar 10 years ago during my move to SF. I needed to find an apartment within two weeks, so I built a crawler to do complex filtering and notify me immediately. In my case, the tool was ultimately useless because I discovered the way SF housing worked for nice apartments was 1) Openings are posted on Friday 2) Saturday, open house collects applications 3) Deal is signed on Sunday and post taken down. Anything my crawler picked up was stuff that was overpriced or in bad condition. For all my sophistication, the dataset was just the bad stuff. Apartment hunting is a great example of needing to know the underlying behaviors of your dataset before you use it to make decisions.

zoover2020 5y ago

Moien! Great article. Something tells me you're joining a big online book store ;-). Tell me when you're in town for a beer, would love to hear how this unfolds.

baby 5y ago

I find the flow in the US the simplest: 1. go to craigslist 2. play with the filter 3. add keywords in the search bar like "rent control" and "-studio" 4. go to the maps page 5. bookmark the page 6. check every day In the US/France, there's too much fragmentation and real estate agents In China I would go to the lady selling phone cards in the street and managed to visit a lot of apartments like that.

asdff 5y ago
If you log in with craigslist you can save the search instead of having to bookmark and manually recheck.

ilovefood 5y ago

Ah... this brings back memories. A few years ago I wrote this article https://funnybretzel.com/datamining-a-flat-in-munich/ https://funnybretzel.com/datamining-a-flat-in-munich/ when I was searching a flat in Munich. Some time has passed since then and meanwhile I've enhanced the process quite a lot, just like OP I'm currently searching for a place (a house) and I'm using quite the little bot farm to massively automate things. I'll post the results if it converges to something useful. Meanwhile, all the luck to OP and excellent write-up! :)

mreiner 5y ago

Interesting details brought up here about location specifics. At least in Zurich most ads contain an address. I fed it to google maps api to get the time by bike to my workplace which gave us some cheaper options we wouldn't have considered before. We also ended up taking/getting the one with the worst pictures. Imagine someone would scrape all platforms for your city of choice and aggregate those with some value added and maybe automatically apply with your personal documents to make sure your application gets in first even when you're busy, what would it be worth to you? Despite local differences it might still scale.

egberts1 5y ago

This is the prime example of a phenomenal approach to aggregation of various data using various languages, in this case on about hunting for a decent apartment. Absolutely stellar form of data science! Keep up the great work!

dayvid 5y ago

If I have time, I always prefer to walk around the neighborhood I want to live in an see if there’s signs or hear from people if a place is available informally. That’s where you get the best deals. If you find the seller, you usually get a better deal. If the seller finds you, …

01100011 5y ago

Renting is fundamentally broken. You submit precious personal info over to a random stranger along with payment to check your credit. Last time I checked, this fee is $25-50 USD. You've now given them enough information to steal your identity. You also may need to do this multiple times if you're rejected. The person receiving this information rarely has identity protection measures in place and you receive no guarantees that they've destroyed this information. A much better solution would be to develop an escrow service which acts as a middleman between renters and landlords. Aspiring renters could sign up with the escrow and submit their information once. Their one-time fee would cover a fixed number of queries by potential landlords. The queries could include such information as income validation, criminal history, job history, rental history, etc. But these queries could be approved by the renter in advance. This sort of query process allows the renter to control the types of information given out. The queries could be structured to allow pass/fail results only, so that landlords do not receive intimate details but only know whether or not a renter is acceptable based on their criteria. Once you have a system like that, you can open it up to all sorts of validation queries. Think of it as a general personal information escrow service. This idea has been bouncing around in my head for over a decade now but I'm more of a low-level systems engineer so I never got around to trying to implement it. Seems like a decent startup idea though.

deleted 5y ago
[deleted]
turtlebits 5y ago
A service similar to what you’re proposing called Smartmove already exists. The renter pays them for the application + credit/background/etc checks. You don’t have to handle payment or take SSNs. Unfortunately pass/fail doesn’t work due to nuance in income/credit/rental history.
rodonn 5y ago
Zillow does have a product like this: pay one application fee and you can reuse the background check across as many apartments as you apply to in a month. The biggest downside is that it only works for landlords that chose to accept the service.

4midori 5y ago

It's sad that this process is so broken that it takes a skilled coder to roll their own, just to get good, organized information. We were promised jetpacks!

lucb1e 5y ago

Actual title: Scraping apartment offers to show them on a map (since the current title just conveys some general topic of apartment need, with no indication if it's about "my partner died and it's interesting to notice how large a whole house is" or something)

deleted 5y ago

[deleted]

de-asis-kevin 5y ago

How many major cities in the world do we have right now where prices of rent did not go up? I'm really curious to know the breakdown of home/building ownership in the world, and how much the person makes in that city. Curious how much property is owned by individuals vs companies.

svnee 5y ago

I shared your link with some colleagues working at atHome and their CEO replied that your approach was interesting. Once you are in Luxembourg, feel free to reach out, I’ll buy you a beer to console you for the expensive apartment prices ;) Always interested in what people are up to (both as an entrepreneur as well as a politican)

pipeline_peak 5y ago

> TL;DR: I hate apartment hunting, I’ve tried to make it as interesting as possible by creating a pipeline with different tools and languages to scrape data….. Grow up, learn to take on boring tasks without playing with toys

asdff 5y ago

Despite all the fancy new websites that crop up every couple of years in the apartment rental space, imo the best place to find an apartment remains craigslist. I've only ever leased spaces through there. For the mom and pop landlords who you really want as your landlord, they might get a new client every 10 years. 10 years ago, they were using craiglist, and it works exactly the same as today, so that's what they continue to use. If it gets them a tenant its good enough to not bother learning some other new website. For the large corporolandlords who have those bougie amenities that perhaps you crave, they will be posting on all the major websites including craigslist anyhow, so you might as well use craigslist for that use case too if only to use a less bloated website.

pbasista 5y ago

Tangential. Information on the internet is often intentionally incomplete or difficult to filter. For instance, it is impossible to filter apps in the Google Play store by the presence of ads or in-app purchases within them. Why? Because Google wants it that way. In order to perform such a filtered search, one needs to create a custom crawler, then parse the data and then filter them. If anything, with all the ads and SEO games taking place, searching for the relevant information has become _more_ difficult than in the early days of Google. It is 2022 and we still do not have a universal search system which would allow everyone to quickly and efficiently perform more sophisticated searches. The most skilled and determined people are capable of doing so by creating various single-purpose data collection and data mining tools themselves. But even for them it takes a non-trivial time and effort to produce something usable. There is nothing out there (that I know of) which would be easily applicable to everyday needs.

joshspankit 5y ago
Further tangent: Many of the public search boxes are backed by elasticsearch or other similar “recommendation” tools which are optimized for speed and point-someone-in-the-right-direction accuracy. Since we no longer have exact text matching or many of the advanced operators (such as `not`), I’d argue that for sophisticated searches search boxes have actually gotten significantly worse.
javajosh 5y ago
The OP points the way, I think, to what will happen in practice. Three trends will meet: 1, more apps live in the browser, 2, search programs that combine data in the browser get formalized/paramterized, 3, user friendly methods arise to make writing ad hoc search programs very easy (e.g. voice driven, context aware AI).

nomy99 5y ago

I am so lucky to have found a room in a descent apartment in a nice part of NYC. As a software engineer making high income, I still almost got homeless in the apartment hunting. Insane!

joeman1000 5y ago

Monospaced fonts are not nice for prose.

rtpg 5y ago

Kinda funny side story: I tried scraping Homes (the Japanese equivalent to this site) for a small subsection of data, basically instantly got IP banned and could not use the site consistently until I outright moved to a new location and made a new internet contract Some people take protecting their data seriously

thiht 5y ago

I did the same thing when apartment hunting in Rennes, France! And this is how I found my current apartment :) There’s great value in building scrapers specialized to a particular city, country-wide aggregators are not enough.

WaffleIronMaker 5y ago

The author might be able to benefit from using Promise.all() to perform his async searches concurrently. Also, I would recommend closing each page when you are done with it, using page.close().

mattrighetti 5y ago
I've received a PR suggesting just that [0], my concern is > Only thing that concerns me is that it will spawn n parallel requests to the website which could consume a lot of memory since we're talking about spawning a new tab with Puppeteer (Chrome :sigh:) for each link. [0]: https://github.com/mattrighetti/athome-scraper/pull/1 https://github.com/mattrighetti/athome-scraper/pull/1