2 ms·

I just moved from Craft CMS to a DIY static site generator, Ursus. I moved 850 entries with various kinds of metadata, including various relationships between e

by nicbou 4y ago

I just moved from Craft CMS to a DIY static site generator, Ursus. I moved 850 entries with various kinds of metadata, including various relationships between entries. This is the website that pays my bills, so it's important to get it right. It took 6-8 weeks, and I was on a roadtrip for 3 of those. It includes writing a SSG from scratch and migrating both the content and the templates. I'm about to write my own recap, but to summarize it here: it went okay, and I love working with markdown. The biggest issue was the conversion of 5 years of HTML noise caused by Redactor, a WYSIWYG editor with many quirks. The converter didn't know what to do with stray line breaks at the end of block elements, among other oddities. I had to fix a lot of stuff manually. The second biggest issue was implementing responsive images with captions, which replace img tags with elaborate figure elements. I converted those to markdown manually, then wrote my own markdown extension. It wasn't that bad! I've now been editing markdown for a month, and I am loving it. It's text in files, not HTML in a database. You can transform it with an arsenal of tools from the last 5 decades, not just a crummy WYSIWYG editor. You can apply the same rigour to the written word as you would to code. You can use scripts, regex, linters, and fancy text editors with fine-tuned plugins and themes. It's awesome to have so much control over your work environment. It's also great to review changes with a git merge tool, to deploy content updates like code, to work offline on slow hardware. Oh and the server has so few moving parts now. No more WordPress updates, no MySQL CPU spikes. The whole thing is so simple. I waited a long time to move to an SSG, but it was absolutely worth it.

swatcoder 4y ago
Nice! > The biggest issue was the conversion of 5 years of HTML noise caused by Redactor, a WYSIWYG editor with many quirks. The converter didn't know what to do with stray line breaks at the end of block elements, among other oddities. I had to fix a lot of stuff manually. For anybody else trying this sort of rewrite, don’t rule out automating a (headless) browser in place of digesting noisy source files. The browser is designed to normalize a lot of that awful input content to a sane DOM, and you can then extract your clean markdown/whatever representation from that normalized content tree rather than from a raw HTML model (with all its loosey goosey allowances). And if this is your own project/code, you might also introduce a few changes into your CMS template to make the scraping more reliable.
- nicbou 4y ago
  The input was valid. It just wasn't clean. It had empty tags, bold tags that wrapped a line break and ended at the beginning of the second line, and other oddities that don't quite translate to markdown. There were also site-specific conventions that I replaced (basically, how I handle footnotes). The nice part is that I could fix those issues with regex. I could fix hundreds of page in a few keystrokes, and only commit changes if they looked right. I also fixed thousands of little things that were impossible to fix before, because editing text files is trivial. To migrate the content, I converted the templates to output plain markdown files instead of HTML. I also had one page that was a plain text list of all entries to migrate. A script read the list and saved each rendered markdown file. I can't overstate how pleasant it is to work with plain text files. Today, I'm reviewing 450 locations with the Google Places API. I fixed a dozen businesses that moved or closed. This would have taken days before. It was to tedious to even try.
ChrisMarshallNY 4y ago
> a WYSIWYG editor I have had bad, bad luck with site builder/WYSIWYG editors (and not really that much luck with Gutenberg). I'm quite fortunate, in being able to write page code/CSS. Also, my sites aren't really very active, so it's not such a big deal, for me. I know the Medium CEO likes to lurk in these threads: I'd suggest that making Medium a viable replacement for sites that give their managers a great deal of control over presentation, could be a great idea. I'm fully aware of the enormity of that task. Just figured I'd put it out there.
- nicbou 4y ago
  The WYSIWYG editor was just for the content. The templates are written by me. It's just that it still inserts weird markup, especially when pasting things. I had the worst time removing unprintable characters.

by nicbou 4y ago

swatcoder 4y ago

Nice! > The biggest issue was the conversion of 5 years of HTML noise caused by Redactor, a WYSIWYG editor with many quirks. The converter didn't know what to do with stray line breaks at the end of block elements, among other oddities. I had to fix a lot of stuff manually. For anybody else trying this sort of rewrite, don’t rule out automating a (headless) browser in place of digesting noisy source files. The browser is designed to normalize a lot of that awful input content to a sane DOM, and you can then extract your clean markdown/whatever representation from that normalized content tree rather than from a raw HTML model (with all its loosey goosey allowances). And if this is your own project/code, you might also introduce a few changes into your CMS template to make the scraping more reliable.

nicbou 4y ago
The input was valid. It just wasn't clean. It had empty tags, bold tags that wrapped a line break and ended at the beginning of the second line, and other oddities that don't quite translate to markdown. There were also site-specific conventions that I replaced (basically, how I handle footnotes). The nice part is that I could fix those issues with regex. I could fix hundreds of page in a few keystrokes, and only commit changes if they looked right. I also fixed thousands of little things that were impossible to fix before, because editing text files is trivial. To migrate the content, I converted the templates to output plain markdown files instead of HTML. I also had one page that was a plain text list of all entries to migrate. A script read the list and saved each rendered markdown file. I can't overstate how pleasant it is to work with plain text files. Today, I'm reviewing 450 locations with the Google Places API. I fixed a dozen businesses that moved or closed. This would have taken days before. It was to tedious to even try.

ChrisMarshallNY 4y ago

> a WYSIWYG editor I have had bad, bad luck with site builder/WYSIWYG editors (and not really that much luck with Gutenberg). I'm quite fortunate, in being able to write page code/CSS. Also, my sites aren't really very active, so it's not such a big deal, for me. I know the Medium CEO likes to lurk in these threads: I'd suggest that making Medium a viable replacement for sites that give their managers a great deal of control over presentation, could be a great idea. I'm fully aware of the enormity of that task. Just figured I'd put it out there.

nicbou 4y ago
The WYSIWYG editor was just for the content. The templates are written by me. It's just that it still inserts weird markup, especially when pasting things. I had the worst time removing unprintable characters.