JohannesKauf

5 ms·

1.
by JohannesKauf 2y ago
Sorry, I mixed up a few topics here: - Moved everything to a VPS - way better value for money. Extra TB of traffic only costs €1-10 with Hetzner/DigitalOcean compared to 400€ with Vercel's old pricing. - Put Cloudflare in front -
2.
by JohannesKauf 2y ago
The problem was: Doing rate limiting on the application level was not enough. Once the request hit my backend the incoming bandwidth was already consumed — and I was charged for it. I contacted Vercel's Support to block that specif
3.
by JohannesKauf 2y ago
Yeah good point, that's actually difficult. They use many `<span>` html tags to color individual words and syntax. But I wrote logic to handle that. It probably needs to be adapted at some point, but works surprisingly well. Have
4.
by JohannesKauf 2y ago
My final university project was about a clean-up-approach on the HTML nodes before sending it to the html-to-markdown converter. But that was extremely difficult and dependent on some heuristics that had to be tweaked. Your idea of compar
5.
by JohannesKauf 2y ago
There are two parts to it: 1) convert html to markdown This is what my library specifically addresses, and I believe it handles this task robustly. There was a lot of testing involved. For example, I used the CommonCrawl Dataset to automati
6.
by JohannesKauf 2y ago
Thanks, Jina actually looks quite nice for use in LLMs. I also provide a REST API [1] that you can use for free (within limits). However you have get an API Key by registering with Github (see reason below). --- The demo was previously host
7.
by JohannesKauf 2y ago
Pandoc is amazing. Especially because they support so many formats. And their html to markdown converter is (in my opinion) still the best right now. But html-to-markdown is getting close. My goal is to cover more edge cases than pandoc...
8.
by JohannesKauf 2y ago
Oh very nice to hear, thank you very much! That’s actually a great idea! I personally use Raindrop for bookmarking articles. But I can’t find stuff in the search. The other day “Embeddings are underrated” was on HN. That would actually be a
9.
by JohannesKauf 2y ago
That is unfortunately out of scope. I like the philosophy of doing one thing really well. But nowadays—with Playwright and Puppeteer—there are great choices for Browser automation.
10.
by JohannesKauf 2y ago
> built something similar for the Elixir We interact with the web so much that it’s worth having such a library in every language... Great that you took the time and wrote one for the Elixir community! Feel free to contact me if you want
11.
Show HN: HTML-to-Markdown – convert entire websites to Markdown with Golang/CLI(github.com)
360 points by JohannesKauf 2y ago|47 comments
12.
by JohannesKauf 2y ago
Cool to see another library in this space! I see that you took the test cases from Turndown. However Turndown isn’t actually that accurate. This is especially noticeable when converting entires websites. The best comparison would be against