2 ms·

Nice! > The biggest issue was the conversion of 5 years of HTML noise caused by Redactor, a WYSIWYG editor with many quirks. The converter didn't know what to

Nice! > The biggest issue was the conversion of 5 years of HTML noise caused by Redactor, a WYSIWYG editor with many quirks. The converter didn't know what to do with stray line breaks at the end of block elements, among other oddities. I had to fix a lot of stuff manually. For anybody else trying this sort of rewrite, don’t rule out automating a (headless) browser in place of digesting noisy source files. The browser is designed to normalize a lot of that awful input content to a sane DOM, and you can then extract your clean markdown/whatever representation from that normalized content tree rather than from a raw HTML model (with all its loosey goosey allowances). And if this is your own project/code, you might also introduce a few changes into your CMS template to make the scraping more reliable.

nicbou 4y ago
The input was valid. It just wasn't clean. It had empty tags, bold tags that wrapped a line break and ended at the beginning of the second line, and other oddities that don't quite translate to markdown. There were also site-specific conventions that I replaced (basically, how I handle footnotes). The nice part is that I could fix those issues with regex. I could fix hundreds of page in a few keystrokes, and only commit changes if they looked right. I also fixed thousands of little things that were impossible to fix before, because editing text files is trivial. To migrate the content, I converted the templates to output plain markdown files instead of HTML. I also had one page that was a plain text list of all entries to migrate. A script read the list and saved each rendered markdown file. I can't overstate how pleasant it is to work with plain text files. Today, I'm reviewing 450 locations with the Google Places API. I fixed a dozen businesses that moved or closed. This would have taken days before. It was to tedious to even try.

nicbou 4y ago

The input was valid. It just wasn't clean. It had empty tags, bold tags that wrapped a line break and ended at the beginning of the second line, and other oddities that don't quite translate to markdown. There were also site-specific conventions that I replaced (basically, how I handle footnotes). The nice part is that I could fix those issues with regex. I could fix hundreds of page in a few keystrokes, and only commit changes if they looked right. I also fixed thousands of little things that were impossible to fix before, because editing text files is trivial. To migrate the content, I converted the templates to output plain markdown files instead of HTML. I also had one page that was a plain text list of all entries to migrate. A script read the list and saved each rendered markdown file. I can't overstate how pleasant it is to work with plain text files. Today, I'm reviewing 450 locations with the Google Places API. I fixed a dozen businesses that moved or closed. This would have taken days before. It was to tedious to even try.