5 ms·

Yes :( Do you have an idea of how to remove company websites in an automated way? I didn't want to manually review all 7k indexed websites. This is the GPT pr

by phgn 3y ago

Yes :( Do you have an idea of how to remove company websites in an automated way? I didn't want to manually review all 7k indexed websites. This is the GPT prompt I used for filtering domains to add, but it gives false positives: You are an API. Return a JSON array of booleans indicating whether each provided domain is someone's personal website. Use common sense. Make sure to return false for company websites.

layer8 3y ago
Maybe change the API so that GPT can express uncertainty (make it a ternary value or even a confidence percentage), and then check the “uncertain” cases manually.
- phgn 3y ago
  Great idea, I will try this. Thank you!!
- tetha 3y ago
  Yep, most of our systems end up exposing a parameter like that to the customer. Some people only like the system to take action if the system is very sure, hate incorrect action and prefer unprocessed stuff in a queue. Other customers hate unprocessed items and prefer to cleanup incorrect actions. Takes tinkering to find the best.
netghost 3y ago
For the 404s (assuming the status code isn't a 4xx), use a URL that you strongly suspect won't exist, then you can do a comparison (levenshtein distance, bag of words, etc.) to see if it's very similar to one of about, ideas, etc. pages.
- KomoD 3y ago
  > For the 404s (assuming the status code isn't a 4xx) Most are a 4xx code, I checked myself, some may be 301/302 redirect to 4xx not being handled properly by their crawler
  - phgn 3y ago
    Good point. We're using https://crawlee.dev https://crawlee.dev, I think there's a way to handle more status codes as errors... Right now it only excludes pages based on the text content: https://github.com/lindylearn/aboutideasnow/blob/main/apps/api/src/common/filter.ts https://github.com/lindylearn/aboutideasnow/blob/main/apps/a...
- andai 3y ago
  I think openai embeddings API could be useful here. Perhaps one of the neurons responds to corporate speak.

by phgn 3y ago

layer8 3y ago

Maybe change the API so that GPT can express uncertainty (make it a ternary value or even a confidence percentage), and then check the “uncertain” cases manually.

phgn 3y ago
Great idea, I will try this. Thank you!!
tetha 3y ago
Yep, most of our systems end up exposing a parameter like that to the customer. Some people only like the system to take action if the system is very sure, hate incorrect action and prefer unprocessed stuff in a queue. Other customers hate unprocessed items and prefer to cleanup incorrect actions. Takes tinkering to find the best.

netghost 3y ago

For the 404s (assuming the status code isn't a 4xx), use a URL that you strongly suspect won't exist, then you can do a comparison (levenshtein distance, bag of words, etc.) to see if it's very similar to one of about, ideas, etc. pages.

KomoD 3y ago
> For the 404s (assuming the status code isn't a 4xx) Most are a 4xx code, I checked myself, some may be 301/302 redirect to 4xx not being handled properly by their crawler
- phgn 3y ago
  Good point. We're using https://crawlee.dev https://crawlee.dev, I think there's a way to handle more status codes as errors... Right now it only excludes pages based on the text content: https://github.com/lindylearn/aboutideasnow/blob/main/apps/api/src/common/filter.ts https://github.com/lindylearn/aboutideasnow/blob/main/apps/a...
andai 3y ago
I think openai embeddings API could be useful here. Perhaps one of the neurons responds to corporate speak.