3 ms·

> Wouldn't help in my scenario FWIW, my site is oauth-only login. Would being able to set cookies or sending custom headers help? > Try it against articles.hs

> Wouldn't help in my scenario FWIW, my site is oauth-only login. Would being able to set cookies or sending custom headers help? > Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it. Hmm, so if you check "Explore" -> "External URLs" there's a ton of external links being checked for this URL. I'm not sure what you could do here except excluding hsreplay.net URLs from being checked. Thanks for the other tips and examples, I'm actively working on this.

scrollaway 9y ago
> Would being able to set cookies or sending custom headers help? I don't think so. And if it's for SEO purposes, I don't care; these pages won't get crawled by Google anyway. I'm ok ignoring the URLs, but it'd be nice if maybe for example you detect there's a bunch of redirects to a pattern that contains "~login~" and ask the user if they want to add the login URL to the blocklist. I didn't have much success adding it myself.
- seanwilson 9y ago
  > these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. > I didn't have much success adding it myself. The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum. A common scenario I see as well is you start a crawl, see the URLs flying by and think "oops, don't want to crawl those URLs". A cancel button would help but hopefully more can be done.
  - scrollaway 9y ago
    > Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Re URLs to ignore: Ok, I see, that wasn't clear though. My two-fold suggestion is 1. Add a way to specify plain loose matching (eg. just /account/login/, skip the url parameters, etc) and 2. let me add that pattern after the urls have already been crawled (and make sure I can see which URLs are affected). Keep in mind that this is pretty raw feedback and you know your product better than me, but I definitely don't think the URLs to ignore is usable right now. I'm off to bed, I hope all that helped. :)
    - seanwilson 9y ago
      > I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Google is likely following the redirect to the login page and seeing that the login page text isn't relevant to your search results at a guess. It's not a big deal if Google hides those pages but in Checkbot perhaps those are pages you want to examine so I'll need to think about what to do here. > Keep in mind that this is pretty raw feedback and you know your product better than me This kind of feedback is amazing so keep it coming! Knowing the first thing you thought before taking the time to fully investigate a feature is super useful because most users would be gone already if they were confused. > but I definitely don't think the URLs to ignore is usable right now. Yes, fully agree with that. I think if you're dealing with regexes, you want a way to test them as it's too easy to make a mistake.

scrollaway 9y ago

> Would being able to set cookies or sending custom headers help? I don't think so. And if it's for SEO purposes, I don't care; these pages won't get crawled by Google anyway. I'm ok ignoring the URLs, but it'd be nice if maybe for example you detect there's a bunch of redirects to a pattern that contains "~login~" and ask the user if they want to add the login URL to the blocklist. I didn't have much success adding it myself.

seanwilson 9y ago
> these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. > I didn't have much success adding it myself. The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum. A common scenario I see as well is you start a crawl, see the URLs flying by and think "oops, don't want to crawl those URLs". A cancel button would help but hopefully more can be done.
- scrollaway 9y ago
  > Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Re URLs to ignore: Ok, I see, that wasn't clear though. My two-fold suggestion is 1. Add a way to specify plain loose matching (eg. just /account/login/, skip the url parameters, etc) and 2. let me add that pattern after the urls have already been crawled (and make sure I can see which URLs are affected). Keep in mind that this is pretty raw feedback and you know your product better than me, but I definitely don't think the URLs to ignore is usable right now. I'm off to bed, I hope all that helped. :)
  - seanwilson 9y ago
    > I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Google is likely following the redirect to the login page and seeing that the login page text isn't relevant to your search results at a guess. It's not a big deal if Google hides those pages but in Checkbot perhaps those are pages you want to examine so I'll need to think about what to do here. > Keep in mind that this is pretty raw feedback and you know your product better than me This kind of feedback is amazing so keep it coming! Knowing the first thing you thought before taking the time to fully investigate a feature is super useful because most users would be gone already if they were confused. > but I definitely don't think the URLs to ignore is usable right now. Yes, fully agree with that. I think if you're dealing with regexes, you want a way to test them as it's too easy to make a mistake.