3 ms·

> Serving giant websites isn't all that hard if you're just spewing out SQL queries into html templates. It all scales in all directions with a properly thought

> Serving giant websites isn't all that hard if you're just spewing out SQL queries into html templates. It all scales in all directions with a properly thought through architecture. No. 1. Your comment makes it sound like Wikipedia is just, or mostly, serving read-only content, which is far from true. Yes, static read-only content is significantly easier to serve than dynamic, editable one, but Wikipedia is the latter. 2. Claiming it's easy to build something at this scale is "isn't all that hard" just makes me think you've never done anything similar. It reminds me of devs saying they could re-build MS Office over a weekend. It's just ignorant of the software's actual complexity. I'm not associated with Wikimedia in any way, but have worked on large-scale software projects before, and things are quite different from, say, websites only serving 100k monthly active users.

abraae 7y ago
I've never heard anyone in my life say they could rebuild MS office in a weekend. What, in your opinion, would be the work needed to go from a 100k monthly active user site to a wikipedia scale site - that would be comparable to rebuilding MS office?
- ivan_gammel 7y ago
  That seems indeed comparable given the scope and functionality of Wikimedia products.
- z3t4 7y ago
  The core parts of Office could be done on a weekend, but in order to get the same complexity and incompatibility it would take several "codemonkeys" several years to achieve.
  - journalctl 7y ago
    Silly Microsoft wasted hundreds of people and decades of time. Why didn’t you tell them?
    - z3t4 7y ago
      Software projects are usually 90% done in 1% of total time taken. And if you just solve the problem with duct tape eg. a shell script, like piping stdin to a file, or contenteditable=true in HTML, you would have a very basic word program, and if you take that route you will probably have the essential features done over a weekend. But going from that to a full Office clone would take years. The real challenge in development though is to solve real problems, eg not make solutions looking for a problem, and not implement new features ( implementing only features that solve real problems).
- rhizome 7y ago
  I've never heard anyone in my life say they could rebuild MS office in a weekend. The saying usually uses Facebook or Twitter.
nickpsecurity 7y ago
Your countering a point they didn't make. SQL to HTML templates indicated dynamic site, not static. From there, they describe surveillance and ad networks that both increase the browser workload and make it rely on 3rd-party dependencies. I thought it was a good, but snarky, point. Especially given my browsing sped up after I installed extensions that turn all that crap off.
chillydawg 7y ago
I have, actually, worked on very large and interactive websites at the very core. Notably: betfair.com which has a very busy API and website and used to be something like a 1:10 write:read ratio with multiple clusters and layers of fancy caching to keep it all coherent down to millisecond scales. Wikipedia does not need to be globally consistent like Betfair does and the ratio of writes to reads is nothing like 10%, I'd guess at one write per million reads or less. There are several pretty obvious ways to architect a site like Wikipedia for effectively unlimited scalability. The main trick is that it doesn't matter if a page is slightly stale and you can queue edits in the backend for quite some time (many seconds) without severely harming end users. Given those constraints it really isn't rocket science given the plethora of amazing tools we have to hand. What I'm NOT saying is that I could build it in a weekend. It would clearly require a few teams of skilled engineers to put it all together and, crucially, operate it. My initial comment was in the context of Wikipedia having 100 engineers, and I think it's reasonable to say that a team that size is easily capable of such a feat.
- floriol 7y ago
  Hi, I am interested in this project. Could you please provide some minor detail about the architecture, like what framework was used for serving that many requests?
  - chillydawg 7y ago
    Oracle and java and a whole lotta optimisations.

abraae 7y ago

I've never heard anyone in my life say they could rebuild MS office in a weekend. What, in your opinion, would be the work needed to go from a 100k monthly active user site to a wikipedia scale site - that would be comparable to rebuilding MS office?

ivan_gammel 7y ago
That seems indeed comparable given the scope and functionality of Wikimedia products.
z3t4 7y ago
The core parts of Office could be done on a weekend, but in order to get the same complexity and incompatibility it would take several "codemonkeys" several years to achieve.
- journalctl 7y ago
  Silly Microsoft wasted hundreds of people and decades of time. Why didn’t you tell them?
  - z3t4 7y ago
    Software projects are usually 90% done in 1% of total time taken. And if you just solve the problem with duct tape eg. a shell script, like piping stdin to a file, or contenteditable=true in HTML, you would have a very basic word program, and if you take that route you will probably have the essential features done over a weekend. But going from that to a full Office clone would take years. The real challenge in development though is to solve real problems, eg not make solutions looking for a problem, and not implement new features ( implementing only features that solve real problems).
rhizome 7y ago
I've never heard anyone in my life say they could rebuild MS office in a weekend. The saying usually uses Facebook or Twitter.

nickpsecurity 7y ago

Your countering a point they didn't make. SQL to HTML templates indicated dynamic site, not static. From there, they describe surveillance and ad networks that both increase the browser workload and make it rely on 3rd-party dependencies. I thought it was a good, but snarky, point. Especially given my browsing sped up after I installed extensions that turn all that crap off.

chillydawg 7y ago

I have, actually, worked on very large and interactive websites at the very core. Notably: betfair.com which has a very busy API and website and used to be something like a 1:10 write:read ratio with multiple clusters and layers of fancy caching to keep it all coherent down to millisecond scales. Wikipedia does not need to be globally consistent like Betfair does and the ratio of writes to reads is nothing like 10%, I'd guess at one write per million reads or less. There are several pretty obvious ways to architect a site like Wikipedia for effectively unlimited scalability. The main trick is that it doesn't matter if a page is slightly stale and you can queue edits in the backend for quite some time (many seconds) without severely harming end users. Given those constraints it really isn't rocket science given the plethora of amazing tools we have to hand. What I'm NOT saying is that I could build it in a weekend. It would clearly require a few teams of skilled engineers to put it all together and, crucially, operate it. My initial comment was in the context of Wikipedia having 100 engineers, and I think it's reasonable to say that a team that size is easily capable of such a feat.

floriol 7y ago
Hi, I am interested in this project. Could you please provide some minor detail about the architecture, like what framework was used for serving that many requests?
- chillydawg 7y ago
  Oracle and java and a whole lotta optimisations.