5 ms·

I'm working on a video course for all things browser automation, whether it's testing, scraping, auditing, deploying etc. I'm keeping the codebase open on GitH

by umaar 6y ago

I'm working on a video course for all things browser automation, whether it's testing, scraping, auditing, deploying etc. I'm keeping the codebase open on GitHub: https://github.com/umaar/learn-browser-testing/ https://github.com/umaar/learn-browser-testing/ so anyone who wants to follow along can do so for free. I've almost finished some cool content such as: - An Amazon price checker which sends you a text message when the price decreases - An Playwright script which gets to Wikipedia Philosophy (https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosophy https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosoph...) - Having an automation script constantly running on a cheap Raspberry PI

moooo99 6y ago
A good use case would be indexing real estate portals for interested buyers. I could imagine it as a relatively attractive side business, targeting real estate agencies and buyers/renters alike. But I haven't had the time to actually build it
- mxuribe 6y ago
  From my having worked ~8 years in residential real estate (and overseeing dev. and maintenance of an in-house api)...I've learned that real estate data on all those real estate sites (from zillow, to any other broker site, etc.) is very much not clean, not always correct, nor updated at a realistic interval. From a data perspective, it is extremely messy to work with. Plus real estate brokers (and related data brokers) have little incentive to clean up the data, let alone make updates more real-time. This might not be a big issue for personal apps...but just be aware of the not-so-little gaps around real estate data. Now, if someone did this as an exercise to learn about apis, scraping, web browser automation, then by all means enjoy; plenty to learn for sure! But don't expect the data to be as usuable. (Oh, and my claims of data messiness are not limited to automata, as anyone searching for homes can tell you about their disappointments when asking about a home only to hear from the agent: "Sorry, that home was sold, we/someone forgot to take the listing down...").
  - randomdude402 6y ago
    This is unfortunately the truth of the situation. I spent way too much time writing scrapers for Zillow and Redfin to guide home purchasing decisions, only to find out half the stuff listed for sale had been under contract for weeks, places listed with garage that had none, homes with pictures of pools and no pool in the data fields... just tons of errors in the data that no amount of vigilance could clean.
    - mxuribe 6y ago
      Yep, seems you've got your own scars there. :-)
    - Dylanlacey 6y ago
      The mad thing is, I would happily pay for the real estate sites to provide me with this kind of data because buying a house is so expensive and such a pain... But it would STILL suck, because the input is so dodgy.

by umaar 6y ago

moooo99 6y ago

A good use case would be indexing real estate portals for interested buyers. I could imagine it as a relatively attractive side business, targeting real estate agencies and buyers/renters alike. But I haven't had the time to actually build it

mxuribe 6y ago
From my having worked ~8 years in residential real estate (and overseeing dev. and maintenance of an in-house api)...I've learned that real estate data on all those real estate sites (from zillow, to any other broker site, etc.) is very much not clean, not always correct, nor updated at a realistic interval. From a data perspective, it is extremely messy to work with. Plus real estate brokers (and related data brokers) have little incentive to clean up the data, let alone make updates more real-time. This might not be a big issue for personal apps...but just be aware of the not-so-little gaps around real estate data. Now, if someone did this as an exercise to learn about apis, scraping, web browser automation, then by all means enjoy; plenty to learn for sure! But don't expect the data to be as usuable. (Oh, and my claims of data messiness are not limited to automata, as anyone searching for homes can tell you about their disappointments when asking about a home only to hear from the agent: "Sorry, that home was sold, we/someone forgot to take the listing down...").
- randomdude402 6y ago
  This is unfortunately the truth of the situation. I spent way too much time writing scrapers for Zillow and Redfin to guide home purchasing decisions, only to find out half the stuff listed for sale had been under contract for weeks, places listed with garage that had none, homes with pictures of pools and no pool in the data fields... just tons of errors in the data that no amount of vigilance could clean.
  - mxuribe 6y ago
    Yep, seems you've got your own scars there. :-)
  - Dylanlacey 6y ago
    The mad thing is, I would happily pay for the real estate sites to provide me with this kind of data because buying a house is so expensive and such a pain... But it would STILL suck, because the input is so dodgy.