4 ms·

I started out re-writing the core of Wordpress to Elixir around last year. Once of the decisions I had to make was with databases. I started out with flat files

• nednar 7y ago
  Just don't copy the wordpress plugin system.
• seanwilson 7y ago
  > With now over 30,000 posts in the SQLITE database, my static file generator is still absolutely fast. Don't you then miss out on human readable Git diffs when posts are changed? And now you can't edit posts with a standard editor or e.g. the GitHub web UI? Also, what language are you using? Shouldn't reading in 30,000 posts and building some stats/relationships/lookups on them be pretty fast?
  • PudgePacket 7y ago
    > Don't you then miss out on human readable Git diffs when posts are changed? And now you can't edit posts with a standard editor or e.g. the GitHub web UI? You never got those with wordpress either. Seems like they're going for some kind of hybrid. Trading some of the simplicity of standard static file generators for some of the power of a tool like wordpress. I quite like the idea. Nothing to stop them from storing historical versions in the DB, then it wouldn't be much more logic for the site generator to spit out the site at X point in time, or a diff. Whether that's a feature you'd want is another thing. > what language are you using? "re-writing the core of Wordpress to Elixir"
  • neya 7y ago
    For simple lookups and stats, they are fine, but imagine complex queries. Best use case? Related posts after every post. With 30,000 posts, your generator will slowdown drastically if it's file based as opposed to some JOIN query on an SQL database. We tried this and it didn't work. Related posts is just one example, think of big publishers' use cases - dynamic menus, ad management, etc.
• corey01 7y ago
  What ORM do you use to interact with your database?
  • neya 7y ago
    Elixir has an obsolete SQLITE ORM which works pretty well for us till this point. At some point, we will have to upgrade it or maintain it ourselves I guess.
• chrismorgan 7y ago
  > If you want to add some new attributes to the flat files, they can't be backported to the older articles if the number of articles is quite large. This isn’t a reasonable claim. In SQL databases if you want to do this, you have two options: 1. Go through and update every row to add a value for the new column—which you can do in flat files just as effectively, though almost certainly more slowly because files are comparatively slow. 2. Define a default value—which you can also do in flat files just as effectively and just as quickly. In this vein, I tend to find the problem to exist in precisely the opposite direction: you need to apply some sort of transformation to all the database entries, for example to update a markup pattern or links to a certain domain. If it’s flat files, you can use standard text-processing tools and easily do anything (presuming you’re a developer). But if it’s in a database, you probably need to go through some database driver which will probably be a pain to deal with for anything involved, and there’s a fair chance with this style of app that you won’t even have access to that and will just be completely stuck. For the likes of blogs, I hate having to deal with database-driven apps. Give me a set of files that I can work with every time. Notwithstanding all that: 30,000 posts is quite a bit as most flat-file systems go, and a well-architected SQL database certainly should perform somewhat better than a well-architected flat file system, simply because the overhead per record in a database is fundamentally lower. Things like migrations and batch operation and revision management just tend to be way more flexible on flat file systems.
  • neya 7y ago
    When I say flat files, you're probably thinking just posts, but there's more to something like Wordpress than just posts. For example, there's global site options, there's individual post options, categories, taxonomies, etc. You get the idea. Sure, all of these can be maintained with static files as well. And that's what we did initially. But, over time, it becomes a maintenance nightmare. Plus, nothing like going into a nice fluid UI and checking/unchecking checkboxes. One common argument is these interfaces can be had with flat file CMSes as well. But then, you will end up re-writing an ORM for the filesystem which goes back to my original problem - SQL databases and their ORMs already solve this problem. It's easy to think flat file CMSes when you're dealing with small files. When dealing with over 30,000 posts, you need to find that file in a directory, open it up and you'll see a totally different structure of the file compared to the most recent ones. They lack consistency. In a database, if I added a new column all old values will be prefixed with null. But the columns are the same for every entry, unlike a static file CMS. Finally, think also about complex queries. Have you seen Wordpress's queries? To generate a simple Navbar menu, you will make so many joins. Especially if you are trying to render menus under different categories (eg. on category pages). Or, how about displaying related posts? It's much easier to query an SQL database, to get something related to the current post. Good luck doing that on 30,000 files whose contents reside inside of the file itself. There's no argument here. All I'm saying is, we tried all that flat file CMSes have to offer. At a scale of 30,000 (now it's actually close to 50,000) posts, it's no joke. In fact, we plan to open source our flat file CMS'es ORM. It's great for small blogs. But for big publishers, you need to use a hybrid approach.
• jakearmitage 7y ago
  What problems did you have with concurrent writes? I just lock stuff and people have to wait. (i literally lock the page if there's someone else there)
  • neya 7y ago
    Yep, same approach we take now as well. Big publishers have multiple writers working on the same article/site design elements sometimes. It's a PITA.