10 ms·

I'm doing something very similar with my service: https://aytwit.com/thoughter https://aytwit.com/thoughter To the author, there is a simple trick you can pull

I'm doing something very similar with my service: https://aytwit.com/thoughter https://aytwit.com/thoughter To the author, there is a simple trick you can pull in order to make the confirmation instantaneous and avoid caching. Have you figured it out? Let me know! I also use a residential proxy service for all my profile requests, regardless of the identity provider. For some sites like Twitter, Facebook, etc. this is required, and for something like Hacker News it's simply future-proofing in case they decide to block scrapers at some point in the future. Good work!

diogenesjunior 5y ago
>for something like Hacker News it's simply future-proofing in case they decide to block scrapers at some point in the future Why are you scraping? We have an API, it's linked at the bottom of every page[0]. 0: https://github.com/HackerNews/API https://github.com/HackerNews/API
- dougk16 5y ago
  Good question, and thank you for the question. Both at the time of my initial development several years ago, and including now when I just tested it, scraping has faster response times (cache invalidations) to profile updates than the API. This saves the user of my site being frustrated that they put the code in their profile, but they keep smashing the "Check again" button to no avail. To be fair the API's cache invalidation does seem to have improved since last time I tested it. It only takes up to 5-10 seconds now, but still this is an eternity to an impatient user. Also since you imply you work for HN, I'll give you my little secret: If you randomly capitalize the letters in the HN username it invalidates the cache. Otherwise scraping the profile would indeed take as long as the API approach. I'm taking a risk with you fixing it now. :) I hope that helps clarify and I look forward to your thoughts.
  - diogenesjunior 5y ago
    I don't work for HN. I meant "we" as in "we the people of HN"
    - dougk16 5y ago
      Ah ok good, that means you won't fix the caching..."feature" that I found. :) Thanks for the discussion!
- hardwaresofton 5y ago
  Honestly I need to use this API as well -- on the list of things to do. I left myself some abstraction to have different ways of checking and the easiest was scraping but honestly hitting the API would have been just as easy. Will fix this
hardwaresofton 5y ago
Hey thanks for the helpful suggestion!

diogenesjunior 5y ago

>for something like Hacker News it's simply future-proofing in case they decide to block scrapers at some point in the future Why are you scraping? We have an API, it's linked at the bottom of every page[0]. 0: https://github.com/HackerNews/API https://github.com/HackerNews/API

dougk16 5y ago
Good question, and thank you for the question. Both at the time of my initial development several years ago, and including now when I just tested it, scraping has faster response times (cache invalidations) to profile updates than the API. This saves the user of my site being frustrated that they put the code in their profile, but they keep smashing the "Check again" button to no avail. To be fair the API's cache invalidation does seem to have improved since last time I tested it. It only takes up to 5-10 seconds now, but still this is an eternity to an impatient user. Also since you imply you work for HN, I'll give you my little secret: If you randomly capitalize the letters in the HN username it invalidates the cache. Otherwise scraping the profile would indeed take as long as the API approach. I'm taking a risk with you fixing it now. :) I hope that helps clarify and I look forward to your thoughts.
- diogenesjunior 5y ago
  I don't work for HN. I meant "we" as in "we the people of HN"
  - dougk16 5y ago
    Ah ok good, that means you won't fix the caching..."feature" that I found. :) Thanks for the discussion!
hardwaresofton 5y ago
Honestly I need to use this API as well -- on the list of things to do. I left myself some abstraction to have different ways of checking and the easiest was scraping but honestly hitting the API would have been just as easy. Will fix this

hardwaresofton 5y ago

Hey thanks for the helpful suggestion!