4 ms·

Why do you think that MediaWiki is not optimized to handle thousands request per second just because it is written in PHP? MediaWiki repository size is about 1

by tored 5y ago

Why do you think that MediaWiki is not optimized to handle thousands request per second just because it is written in PHP? MediaWiki repository size is about 1.4 GB and according to phploc it has closer to a million lines of PHP code. Sounds like fools errand to rewrite it.

jiggawatts 5y ago
I looked into the topic in some detail year ago, because I copied Wikipedia's approach for an in-house CMS editor. I looked through the code, did loads of performance experiments, read through related forums posts, etc... The wiki staff themselves admitted that the current parser is inefficient, partly because of PHP, partly because the underlying grammar was not designed to be efficient, and partly because the parser itself was built up over time and wasn't easy to optimise. My approach was to write the parser and a matching "markdown inspired" format at the same time, optimised for speed. Just a handful of small tweaks to the syntax were all that was required to largely eliminate backtracking and achieve a nearly linear parsing time in most cases. If I remember correctly, I had it down to about 1-5ms for a typical 64 KB page, and then HTML generation was another 5-10ms depending on various factors. What a lot of people are missing here is that Wikis are not at all like typical "ERP" applications. The latter sometimes requires dozens of API calls and thousands of database queries to generate just one kilobyte of output HTML. Wiki is very linear, with a single 1-100KB blob of text as input, a matching 1-100KB blob of HTML as output. It all boils down to the parsing and HTML generation efficiency, nothing else matters!
- tored 5y ago
  You don't need to parse the grammar for every request and as they admitted it was an old implementation. What I understand they have a new one now. You can parse grammar on save and make an optimized compiled format where static content is already resolved. Next step is to substitute semi-static content (like author name) to a runtime format, this last format can handle dynamic data substitution like current date & time, but probably going to be rare that any substitution is needed for the runtime format (how much truly dynamic data does a wiki have?), most of the time just print it with readfile or similar. Only difficult part here is cache invalidation for the runtime format (I know, it is one of the three difficult things you can do in programming). 3000 request/second in PHP is not hard, especially if you plan for it.
  - jiggawatts 5y ago
    It's not at all hard! A typical Wikipedia page has the "chrome", which changes based on your logon, and then content, which generally does not. Cache invalidation is trivial, because there's only one atomic "blob" to track instead of the output of some complex query. This is why it amuses me that people seem to think that Wikipedia "requires" tens of millions of dollars of infrastructure to host. It doesn't require it. It's just that some things are better/easier with more kit, such as maintenance.

by tored 5y ago

jiggawatts 5y ago

I looked into the topic in some detail year ago, because I copied Wikipedia's approach for an in-house CMS editor. I looked through the code, did loads of performance experiments, read through related forums posts, etc... The wiki staff themselves admitted that the current parser is inefficient, partly because of PHP, partly because the underlying grammar was not designed to be efficient, and partly because the parser itself was built up over time and wasn't easy to optimise. My approach was to write the parser and a matching "markdown inspired" format at the same time, optimised for speed. Just a handful of small tweaks to the syntax were all that was required to largely eliminate backtracking and achieve a nearly linear parsing time in most cases. If I remember correctly, I had it down to about 1-5ms for a typical 64 KB page, and then HTML generation was another 5-10ms depending on various factors. What a lot of people are missing here is that Wikis are not at all like typical "ERP" applications. The latter sometimes requires dozens of API calls and thousands of database queries to generate just one kilobyte of output HTML. Wiki is very linear, with a single 1-100KB blob of text as input, a matching 1-100KB blob of HTML as output. It all boils down to the parsing and HTML generation efficiency, nothing else matters!

tored 5y ago
You don't need to parse the grammar for every request and as they admitted it was an old implementation. What I understand they have a new one now. You can parse grammar on save and make an optimized compiled format where static content is already resolved. Next step is to substitute semi-static content (like author name) to a runtime format, this last format can handle dynamic data substitution like current date & time, but probably going to be rare that any substitution is needed for the runtime format (how much truly dynamic data does a wiki have?), most of the time just print it with readfile or similar. Only difficult part here is cache invalidation for the runtime format (I know, it is one of the three difficult things you can do in programming). 3000 request/second in PHP is not hard, especially if you plan for it.
- jiggawatts 5y ago
  It's not at all hard! A typical Wikipedia page has the "chrome", which changes based on your logon, and then content, which generally does not. Cache invalidation is trivial, because there's only one atomic "blob" to track instead of the output of some complex query. This is why it amuses me that people seem to think that Wikipedia "requires" tens of millions of dollars of infrastructure to host. It doesn't require it. It's just that some things are better/easier with more kit, such as maintenance.