3 ms·
Google around for a "public suffix list" library for your favorite language, it will aid you in guessing that 'example.com' could be a valid FQDN. As a crawler
by greglindahl 10y ago
Google around for a "public suffix list" library for your favorite language, it will aid you in guessing that 'example.com' could be a valid FQDN. As a crawler guy, I see common suffix list features in the kinds of url libraries that I use or write.
- tokenizerrr 10y agoDon't forget to update your library whenever a new gTLD comes out
- greglindahl 10y agoIt appears that most libraries download the list from Mozilla, but yes, you do need to ensure that it's downloaded frequently.