3 ms·

> these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like

> these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. > I didn't have much success adding it myself. The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum. A common scenario I see as well is you start a crawl, see the URLs flying by and think "oops, don't want to crawl those URLs". A cancel button would help but hopefully more can be done.

scrollaway 9y ago
> Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Re URLs to ignore: Ok, I see, that wasn't clear though. My two-fold suggestion is 1. Add a way to specify plain loose matching (eg. just /account/login/, skip the url parameters, etc) and 2. let me add that pattern after the urls have already been crawled (and make sure I can see which URLs are affected). Keep in mind that this is pretty raw feedback and you know your product better than me, but I definitely don't think the URLs to ignore is usable right now. I'm off to bed, I hope all that helped. :)
- seanwilson 9y ago
  > I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Google is likely following the redirect to the login page and seeing that the login page text isn't relevant to your search results at a guess. It's not a big deal if Google hides those pages but in Checkbot perhaps those are pages you want to examine so I'll need to think about what to do here. > Keep in mind that this is pretty raw feedback and you know your product better than me This kind of feedback is amazing so keep it coming! Knowing the first thing you thought before taking the time to fully investigate a feature is super useful because most users would be gone already if they were confused. > but I definitely don't think the URLs to ignore is usable right now. Yes, fully agree with that. I think if you're dealing with regexes, you want a way to test them as it's too easy to make a mistake.

scrollaway 9y ago

> Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Re URLs to ignore: Ok, I see, that wasn't clear though. My two-fold suggestion is 1. Add a way to specify plain loose matching (eg. just /account/login/, skip the url parameters, etc) and 2. let me add that pattern after the urls have already been crawled (and make sure I can see which URLs are affected). Keep in mind that this is pretty raw feedback and you know your product better than me, but I definitely don't think the URLs to ignore is usable right now. I'm off to bed, I hope all that helped. :)

seanwilson 9y ago
> I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Google is likely following the redirect to the login page and seeing that the login page text isn't relevant to your search results at a guess. It's not a big deal if Google hides those pages but in Checkbot perhaps those are pages you want to examine so I'll need to think about what to do here. > Keep in mind that this is pretty raw feedback and you know your product better than me This kind of feedback is amazing so keep it coming! Knowing the first thing you thought before taking the time to fully investigate a feature is super useful because most users would be gone already if they were confused. > but I definitely don't think the URLs to ignore is usable right now. Yes, fully agree with that. I think if you're dealing with regexes, you want a way to test them as it's too easy to make a mistake.