3 ms·

Thanks for the feedback! The missing characters are because the of unicode decoding in Python. If the content-encoding if not found in the html headers it will

Thanks for the feedback! The missing characters are because the of unicode decoding in Python. If the content-encoding if not found in the html headers it will replace all unknown characters. I'm working on improving that. The // is something I can fix right now. [edit] the // link bug is fixed, and it also allows https:// https:// links (which i noticed that it didn't work because i pasted in your urls)

karanbhangui 16y ago
Have you considered using this: http://chardet.feedparser.org/ http://chardet.feedparser.org/ to detect the encoding? I'm not sure if this helps, but seems like it would.
- metachris 16y ago
  Thank you for the link! Will definitely give it a try (tomorrow)!