3 ms·

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos https://github.com/pra

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos https://github.com/prakhar1989/bekanjoos

lxchase 10y ago
Agree, used it to build this as a proof of concept: https://emailprofile.herokuapp.com/ https://emailprofile.herokuapp.com/ Many sites don't consider user enumeration a bug/threat, but theoretically given enough sites, one could build a profile around a specific email address.
- LunaSea 10y ago
  Have you open-sources this package somewhere? As an amateur web security researcher this project looks really interesting.
  - lxchase 10y ago
    I haven't yet. I'm a bit torn on whether or not to release it given the potential privacy implications. With more sites and a classification algorithm, one could say an email is a "Gender, Race, Age Range, Job Industry, Interests, etc." What makes this tool work is most sites (as an UX feature) will tell you if an account/email already exists. Whether that be an API call or a notice saying "Your password is incorrect", you'll be able to get the data you need. It was a learning experience for me to use Go to wrap each site check in its own goroutine to leverage concurrency. Quite nice.
imwally 10y ago
It really is a great package and works extremely well in the context of scraping. It's what I opted to use for an API server [1] I wrote that handles the RFC Instant Answer [2] running on DuckDuckGo. [1] https://github.com/imwally/rfcsearch https://github.com/imwally/rfcsearch [2] https://duck.co/ia/view/request_for_comments https://duck.co/ia/view/request_for_comments
sleepychu 10y ago
bekanjoos looks nice! I was thinking today about making a more generic "when this thing on that website changes" notify me sort of thing. It looks like you're running on AWS. Would you be willing to share how much the bill for that runs to?

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos https://github.com/prakhar1989/bekanjoos

lxchase 10y ago

Agree, used it to build this as a proof of concept: https://emailprofile.herokuapp.com/ https://emailprofile.herokuapp.com/ Many sites don't consider user enumeration a bug/threat, but theoretically given enough sites, one could build a profile around a specific email address.

LunaSea 10y ago
Have you open-sources this package somewhere? As an amateur web security researcher this project looks really interesting.
- lxchase 10y ago
  I haven't yet. I'm a bit torn on whether or not to release it given the potential privacy implications. With more sites and a classification algorithm, one could say an email is a "Gender, Race, Age Range, Job Industry, Interests, etc." What makes this tool work is most sites (as an UX feature) will tell you if an account/email already exists. Whether that be an API call or a notice saying "Your password is incorrect", you'll be able to get the data you need. It was a learning experience for me to use Go to wrap each site check in its own goroutine to leverage concurrency. Quite nice.

imwally 10y ago

It really is a great package and works extremely well in the context of scraping. It's what I opted to use for an API server [1] I wrote that handles the RFC Instant Answer [2] running on DuckDuckGo. [1] https://github.com/imwally/rfcsearch https://github.com/imwally/rfcsearch [2] https://duck.co/ia/view/request_for_comments https://duck.co/ia/view/request_for_comments

sleepychu 10y ago

bekanjoos looks nice! I was thinking today about making a more generic "when this thing on that website changes" notify me sort of thing. It looks like you're running on AWS. Would you be willing to share how much the bill for that runs to?