13 ms·

If Khan Academy uses Youtube to serve their video and uses Fastly to serve static content, what makes it hard to scale ? I mean being able to scale that easily

by polote 6y ago

If Khan Academy uses Youtube to serve their video and uses Fastly to serve static content, what makes it hard to scale ? I mean being able to scale that easily is a great thing, but is there anything worth sharing with the world in their case ?

tomnipotent 6y ago
Like the rest of the system that isn't just videos and static content?
- cameronfraser 6y ago
  It's a cookie cutter app engine stack which scales automatically
cbhl 6y ago
There's quite a bit of dynamic content, no? Things like exercise grading, and progression through the skill tree.
- jonny_eh 6y ago
  Which is powered by Google's AppEngine, which scales very easily, at least technically easily.
  - tyree731 6y ago
    s/easily/expensively/ Khan Academy doesn't have infinite money.
    - ses1984 6y ago
      what makes GAE expensive?
      - freehunter 6y ago
        Anything is expensive if you have to scale up beyond your financial ability to do so.
        merlinsbrain 6y ago
        Not sure why you’re being downvoted. Scaling systems at companies with a lot of money is very different from designing system to be reliable and performant and with reducing spend being a real goal. Some companies need to take a much harder look at their spend than others regardless of where you deploy.
      - deleted 6y ago
        [deleted]
SilasX 6y ago
"How to scale: Make it somebody else's problem."
- spyspy 6y ago
  A surprisingly legitimate solution.
  - SilasX 6y ago
    Yes, with the caveat that you may have to check that they're actually capable of handling the load and you don't get a surprise notice that "uh we can't do this, you're on your own".
    - RcouF1uZ4gsC 6y ago
      I am pretty confident that YouTube will be able to scale to handle any increased load coming from my service.
      - judge2020 6y ago
        But they might not at any time, at least without charging. With an apparent internal push for some services to become self-sustaining (see Google Maps API, Recaptcha), YouTube embeds might be next.
        toomuchtodo 6y ago
        It seems unlikely Khan Academy doesn't have the technical competency to deploy their video content to an alternate host rapidly, whether that be an object store (Backblaze) or dedicated servers with very cheap bandwidth (Hetzner, OVH, and similar), perhaps even using PeerTube. There's a reason other non profits like Wikipedia and the Internet Archive run their own hardware, networking, and connectivity to transit providers. And before the "doing that is expensive!" argument comes up, note how expensive having someone else do these things are. Lots of margin built into cloud services.
DevKoala 6y ago
Yes. In this case, they share the fact they used common sense. This is not saying that all of the other major re-architecture blog posts are flawed. However, it is good to know that scaling when using cloud vendor specific tools is as easy as advertised.
kzrdude 6y ago
I'd like to hear engineering stories from Youtube in that case. What does their operation look like? Do they ever tell?
- londons_explore 6y ago
  Make an educated guess about the design of youtubes serving infrastructure, and I'm pretty certain you're right. There's only really one sensible way to do it, and that's the way it's done.
  - kbenson 6y ago
    That's not terribly helpful to a large portion of people that don't have the requisite knowledge or experience to make an educated guess, or don't have the knowledge to eliminate most of a few educated guesses. It's sort of like someone asking how you rebuild the stock carburetor on a 1967 Mustang, and replying "Make an educated guess about the design of youtubes serving infrastructure, and I'm pretty certain you're right." For the vast majority of people, that's no use at all.
dangoor 6y ago
I need to write a blog post about this :) A lot of people seem to think of Khan Academy as a bunch of videos. Many have also seen the exercises and articles. Those things are all pretty static (though it gets more complex when you consider how much content there is and how many languages it's localized into). There's a whole bunch of dynamic behavior around that static content. Keeping track of progress to tell a learner how they're doing, plus to help recommend the next place to go in the content. Reporting on progress to parents and teachers. Letting teachers create assignments and manage their classrooms. Bubbling up information to school districts. Content pages have discussions and clarifications. There are notifications to tell students about new assignments, for example. There are connections to tests, like the SAT prep or integration with the MAP test, which involve connecting our accounts with external accounts in order to help students based on those test results. And a bunch of other stuff that isn't coming to mind right now because I'm just naming things off the top of my head. Doing all these things across a user base of millions of monthly users can get quite involved.
- VWWHFSfQ 6y ago
  Thank you! I'm curious what the cost was to run this stack for one month? In the blog post Marta said: > In the month of April, we served 30 million learners on our platform How much did it cost to run this whole stack on GCP in April 2020? Was it $150,000? More or less?
  - dangoor 6y ago
    Unfortunately, I don't think I've seen this publicly reported and I'm not in a position to decide to make that public. I will say that the amount we pay for our infrastructure, especially right now, is reduced by generous support from both Fastly and Google. It still remains a substantial expense after that support. One final note on cost: we are actively working on a Python to Go transition[1] that will reduce our hosting costs, among other benefits. [1]: http://engineering.khanacademy.org/posts/goliath.htm http://engineering.khanacademy.org/posts/goliath.htm
infogulch 6y ago
Yes, for the same reason that researchers should publish null results: all of the data is useful. Getting confirmation that a particular formulation of a strategy works or does not work is valuable in and of itself, regardless of the exact outcome. The only reason why it wouldn't be valuable would be if there were a plethora of similar reports of successfully scaling this solution, which I do not, so their experience is very welcome. To turn it around, why would you ever want someone to not share their experience with the world if they took time to write it down? It's not like you must read it; it doesn't cost you anything to exist. But if someone's experience adds to the library of human knowledge, even a little bit, why would one try to reject that?
SergeAx 6y ago
"Just use external video hosting and deploy CDN for static" sounds like a trivial advice, but I constantly see businesses here and there not following it. So, as it says in Russsian proverb, "repetition is a mother of learning".

by polote 6y ago

tomnipotent 6y ago

Like the rest of the system that isn't just videos and static content?

cameronfraser 6y ago
It's a cookie cutter app engine stack which scales automatically

cbhl 6y ago

There's quite a bit of dynamic content, no? Things like exercise grading, and progression through the skill tree.

jonny_eh 6y ago
Which is powered by Google's AppEngine, which scales very easily, at least technically easily.
- tyree731 6y ago
  s/easily/expensively/ Khan Academy doesn't have infinite money.
  - ses1984 6y ago
    what makes GAE expensive?
    - freehunter 6y ago
      Anything is expensive if you have to scale up beyond your financial ability to do so.
      - merlinsbrain 6y ago
        Not sure why you’re being downvoted. Scaling systems at companies with a lot of money is very different from designing system to be reliable and performant and with reducing spend being a real goal. Some companies need to take a much harder look at their spend than others regardless of where you deploy.
    - deleted 6y ago
      [deleted]

SilasX 6y ago

"How to scale: Make it somebody else's problem."

spyspy 6y ago
A surprisingly legitimate solution.
- SilasX 6y ago
  Yes, with the caveat that you may have to check that they're actually capable of handling the load and you don't get a surprise notice that "uh we can't do this, you're on your own".
  - RcouF1uZ4gsC 6y ago
    I am pretty confident that YouTube will be able to scale to handle any increased load coming from my service.
    - judge2020 6y ago
      But they might not at any time, at least without charging. With an apparent internal push for some services to become self-sustaining (see Google Maps API, Recaptcha), YouTube embeds might be next.
      - toomuchtodo 6y ago
        It seems unlikely Khan Academy doesn't have the technical competency to deploy their video content to an alternate host rapidly, whether that be an object store (Backblaze) or dedicated servers with very cheap bandwidth (Hetzner, OVH, and similar), perhaps even using PeerTube. There's a reason other non profits like Wikipedia and the Internet Archive run their own hardware, networking, and connectivity to transit providers. And before the "doing that is expensive!" argument comes up, note how expensive having someone else do these things are. Lots of margin built into cloud services.

DevKoala 6y ago

Yes. In this case, they share the fact they used common sense. This is not saying that all of the other major re-architecture blog posts are flawed. However, it is good to know that scaling when using cloud vendor specific tools is as easy as advertised.

kzrdude 6y ago

I'd like to hear engineering stories from Youtube in that case. What does their operation look like? Do they ever tell?

londons_explore 6y ago
Make an educated guess about the design of youtubes serving infrastructure, and I'm pretty certain you're right. There's only really one sensible way to do it, and that's the way it's done.
- kbenson 6y ago
  That's not terribly helpful to a large portion of people that don't have the requisite knowledge or experience to make an educated guess, or don't have the knowledge to eliminate most of a few educated guesses. It's sort of like someone asking how you rebuild the stock carburetor on a 1967 Mustang, and replying "Make an educated guess about the design of youtubes serving infrastructure, and I'm pretty certain you're right." For the vast majority of people, that's no use at all.

dangoor 6y ago

I need to write a blog post about this :) A lot of people seem to think of Khan Academy as a bunch of videos. Many have also seen the exercises and articles. Those things are all pretty static (though it gets more complex when you consider how much content there is and how many languages it's localized into). There's a whole bunch of dynamic behavior around that static content. Keeping track of progress to tell a learner how they're doing, plus to help recommend the next place to go in the content. Reporting on progress to parents and teachers. Letting teachers create assignments and manage their classrooms. Bubbling up information to school districts. Content pages have discussions and clarifications. There are notifications to tell students about new assignments, for example. There are connections to tests, like the SAT prep or integration with the MAP test, which involve connecting our accounts with external accounts in order to help students based on those test results. And a bunch of other stuff that isn't coming to mind right now because I'm just naming things off the top of my head. Doing all these things across a user base of millions of monthly users can get quite involved.

VWWHFSfQ 6y ago
Thank you! I'm curious what the cost was to run this stack for one month? In the blog post Marta said: > In the month of April, we served 30 million learners on our platform How much did it cost to run this whole stack on GCP in April 2020? Was it $150,000? More or less?
- dangoor 6y ago
  Unfortunately, I don't think I've seen this publicly reported and I'm not in a position to decide to make that public. I will say that the amount we pay for our infrastructure, especially right now, is reduced by generous support from both Fastly and Google. It still remains a substantial expense after that support. One final note on cost: we are actively working on a Python to Go transition[1] that will reduce our hosting costs, among other benefits. [1]: http://engineering.khanacademy.org/posts/goliath.htm http://engineering.khanacademy.org/posts/goliath.htm

infogulch 6y ago

Yes, for the same reason that researchers should publish null results: all of the data is useful. Getting confirmation that a particular formulation of a strategy works or does not work is valuable in and of itself, regardless of the exact outcome. The only reason why it wouldn't be valuable would be if there were a plethora of similar reports of successfully scaling this solution, which I do not, so their experience is very welcome. To turn it around, why would you ever want someone to not share their experience with the world if they took time to write it down? It's not like you must read it; it doesn't cost you anything to exist. But if someone's experience adds to the library of human knowledge, even a little bit, why would one try to reject that?

SergeAx 6y ago

"Just use external video hosting and deploy CDN for static" sounds like a trivial advice, but I constantly see businesses here and there not following it. So, as it says in Russsian proverb, "repetition is a mother of learning".