4 ms·
Cool! There do seem to be some text encoding issues though. For example: https://search.marginalia.nu/search?query=tim+visee https://search.marginalia.nu/searc
by timvisee 5y ago
Cool!
There do seem to be some text encoding issues though. For example: https://search.marginalia.nu/search?query=tim+visee https://search.marginalia.nu/search?query=tim+visee
- marginalia_nu 5y agoYeah I think the charset detection needs work. It understands the "Content-type: text/html;charset=utf-8" -header, and <meta charset="UTF-8"> but not <meta http-equiv="content-type" content="text/html; charset=utf-8"> It turns out HTML has a lot of corner cases. I'm constantly marveling at how web browsers hold together as well as they do.
- timvisee 5y agoThanks for your response! Hope you can implement this as well without too much trouble. I wonder if you could just assume UTF-8 to be the default these days. I imagine that to fix many other cases as well.
- marginalia_nu 5y agoHaha! I did actually assume UTF-8 at first, but being a search engine has a lot of older websites, I sadly got a lot of encoding errors doing that, too.
- soheil 5y agoMaybe just like js-heavy sites also punish non-conforming-encoding sites.