3 ms·

While more efficient templating sounds like a good thing and is clearly a pain point, I'm not convinced Wikimedia is really thinking this one through. 1. Every

by sigil 14y ago

While more efficient templating sounds like a good thing and is clearly a pain point, I'm not convinced Wikimedia is really thinking this one through. 1. Every page is now a "real" program. Will separation of code and content result from this? The article hints that structured data is on the way, but I have a hard time imagining the average contributor will restrain themselves given this awesome new power. Instead I foresee bigger and badder messes. 2. Does this get us any closer to a parseable format for Wikipedia content? This is the world's knowledge. The fact that it's currently trapped in an ad hoc, PHP-inspired template "language" (there is no formal grammar for it, it's whatever some PHP code accepts as input) should be very concerning. 3. Why Lua? Javascript seemed like the obvious choice. Rendering templates server-side at Wikipedia's scale sounds incredibly challenging; it boggles the mind really. Surely most page views are from js-enabled browsers -- why not offload all that rendering to the client?

capnrefsmmat 14y ago
Lua was chosen over JavaScript for its ease of embedding. It's easy to limit how much memory the Lua interpreter uses to prevent users from DoSing Wikimedia servers with templates. V8 and SpiderMonkey do not make this easy. More details here: https://www.mediawiki.org/wiki/User:Sumanah/Lua_vs_Javascript https://www.mediawiki.org/wiki/User:Sumanah/Lua_vs_Javascrip... Wikipedia pages are massively cached: rendered pages are cached in Squid, and I believe parsed page contents are stored in memcached. Templates are only re-rendered when pages are edited, if all goes well.
gregholmberg 14y ago
Why Lua? I have been running Node.js and Lua at fairly large scale on EC2 recently. I like the V8/Lua/Luajit platforms just fine. Each has some nice strengths. With Lua, I see significantly less memory use, and moderately increased performance. As long as RAM is the expensive bottleneck for hosted servers, it looks like I will be able to get better performance (in operations per dollar) by using a smaller runtime. Why not client-side rendering? Accessibility? Cross-platform uniformity? (better answer from 'capnrefsmmat: render once, then cache massively)
neilk 14y ago
It would be impossible to move rendering to the client. Transforming wikitext into HTML is surprisingly IO-bound, since most Wikipedia pages transclude many other pages for templating and logic. It's very slow now, and you need to be close to the database for this to be remotely practical. Lua gives the community a way to replace certain horrible templates which perverted transclusion and macro-expansion to do logic. But most Wikipedia pages will still transclude a lot of other pages. Also, web browsers are not the only consumer of Wikipedia pages; you can also download PDFs and so on.
- gngeal 14y ago
  "Transforming wikitext into HTML is surprisingly IO-bound, since most Wikipedia pages transclude many other pages for templating and logic." Sounds like a case for in memory caching. Or is it that any random Wikipedia page requires logic from a random subset of a large pool of other pages? Somehow I doubt that, with Zipf's law coming to my mind. Also remember that an HTML5 front-end would be able to cache a substantial amount of this (a few megs at least) and Lua can be quite effectively compiled into JS.
  - neilk 14y ago
    MediaWiki uses lots of caching on the server, but the question is how hard it would be to move rendering to the client. For a sense of how hard this would be, try using the Special:Export page on Wikipedia. http://en.wikipedia.org/wiki/Special:Export http://en.wikipedia.org/wiki/Special:Export If you download transcluded templates, the article for Barack Obama is 781K. My experience of Special:Export is that it has some flaws that cause it to miss some things it needs to export, so the real total may be much larger. And that's just the data - one would also have to download a lot of related code, which might balloon that up to a megabyte or more. Wikitext is particularly ornery (because it's just based on grinding regular expressions against each line, it is not easy to describe with regular grammars) so you'd have to download a very large parser, with various plugins as each MediaWiki install uses them to warp how Wikitext is processed. This is assuming some optimistic scenario where MediaWiki's rendering, and all related plugins, are entirely ported to JavaScript compatible with all desired browsers. I'm not denying that, if you wanted to create a new Wikipedia from scratch today, based on JavaScript, you could probably move a lot of rendering to the browser. You would choose more browser-friendly formats, like JSON or XML, rather than making up some random text-based format, just because it was easy to type into a textarea. You would make transformational operations work in JavaScript, or be exportable to JavaScript. You could definitely get it to the point where it would be practical for quick previews while editing. For the Wikipedia that we have today, it's really hard.

by sigil 14y ago

capnrefsmmat 14y ago

Lua was chosen over JavaScript for its ease of embedding. It's easy to limit how much memory the Lua interpreter uses to prevent users from DoSing Wikimedia servers with templates. V8 and SpiderMonkey do not make this easy. More details here: https://www.mediawiki.org/wiki/User:Sumanah/Lua_vs_Javascript https://www.mediawiki.org/wiki/User:Sumanah/Lua_vs_Javascrip... Wikipedia pages are massively cached: rendered pages are cached in Squid, and I believe parsed page contents are stored in memcached. Templates are only re-rendered when pages are edited, if all goes well.

gregholmberg 14y ago

Why Lua? I have been running Node.js and Lua at fairly large scale on EC2 recently. I like the V8/Lua/Luajit platforms just fine. Each has some nice strengths. With Lua, I see significantly less memory use, and moderately increased performance. As long as RAM is the expensive bottleneck for hosted servers, it looks like I will be able to get better performance (in operations per dollar) by using a smaller runtime. Why not client-side rendering? Accessibility? Cross-platform uniformity? (better answer from 'capnrefsmmat: render once, then cache massively)

neilk 14y ago

It would be impossible to move rendering to the client. Transforming wikitext into HTML is surprisingly IO-bound, since most Wikipedia pages transclude many other pages for templating and logic. It's very slow now, and you need to be close to the database for this to be remotely practical. Lua gives the community a way to replace certain horrible templates which perverted transclusion and macro-expansion to do logic. But most Wikipedia pages will still transclude a lot of other pages. Also, web browsers are not the only consumer of Wikipedia pages; you can also download PDFs and so on.

gngeal 14y ago
"Transforming wikitext into HTML is surprisingly IO-bound, since most Wikipedia pages transclude many other pages for templating and logic." Sounds like a case for in memory caching. Or is it that any random Wikipedia page requires logic from a random subset of a large pool of other pages? Somehow I doubt that, with Zipf's law coming to my mind. Also remember that an HTML5 front-end would be able to cache a substantial amount of this (a few megs at least) and Lua can be quite effectively compiled into JS.
- neilk 14y ago
  MediaWiki uses lots of caching on the server, but the question is how hard it would be to move rendering to the client. For a sense of how hard this would be, try using the Special:Export page on Wikipedia. http://en.wikipedia.org/wiki/Special:Export http://en.wikipedia.org/wiki/Special:Export If you download transcluded templates, the article for Barack Obama is 781K. My experience of Special:Export is that it has some flaws that cause it to miss some things it needs to export, so the real total may be much larger. And that's just the data - one would also have to download a lot of related code, which might balloon that up to a megabyte or more. Wikitext is particularly ornery (because it's just based on grinding regular expressions against each line, it is not easy to describe with regular grammars) so you'd have to download a very large parser, with various plugins as each MediaWiki install uses them to warp how Wikitext is processed. This is assuming some optimistic scenario where MediaWiki's rendering, and all related plugins, are entirely ported to JavaScript compatible with all desired browsers. I'm not denying that, if you wanted to create a new Wikipedia from scratch today, based on JavaScript, you could probably move a lot of rendering to the browser. You would choose more browser-friendly formats, like JSON or XML, rather than making up some random text-based format, just because it was easy to type into a textarea. You would make transformational operations work in JavaScript, or be exportable to JavaScript. You could definitely get it to the point where it would be practical for quick previews while editing. For the Wikipedia that we have today, it's really hard.