7 ms·

When this feature launched it was literally an engineer sitting at home watching TV and entering the scores into a Google Spreadsheet, which Search pulled via G

When this feature launched it was literally an engineer sitting at home watching TV and entering the scores into a Google Spreadsheet, which Search pulled via Google Sheets API and stuck into a formatted OneBox. I think they've since taken the engineer out of the loop and worked out real data distribution deals with the sports leagues. The thing is - from a product perspective, this is the right way to do it. At the time of its launch (2013, IIRC), about 90% of new features launched on the SRP didn't survive live experiment phase (i.e. nobody wanted them), and about 98% didn't survive their first birthday (if people did want them, it was only transient, or they didn't want them enough to justify ongoing maintenance). This was one of the lucky ones. It's also somewhat pointless to optimize this case - from a product perspective, polling is better for the user (websockets have some edge cases around firewalls and network changes that make them a bit less reliable), and I guarantee that the bandwidth used by this feature is rounding error for Google. They spend many times more hosting JQuery for free than this feature consumes. Sometimes the dead-simple answer is the right one, and rather than paying someone to do the hard bits, you're better off not doing the hard bits at all.

matt_oriordan 7y ago
I appreciate when features are in test phase that it makes sense to keep it simple. But this is in production, and there are many dead-simple solutions that do this efficiently, long polling, SSE, etc. This is Google after all.
- Raidion 7y ago
  To implement this, you have to have a developer who is doing something else switch to this, get up to speed, and make a change. This involves risk and opportunity cost. The question for enterprise software development isn't "Will this make this thing better?", but "Is making this change more valuable than the other things on the priority list?"
  - matt_oriordan 7y ago
    Sure, but I'm never suggested front-end developers should build new tech every time they want a new feature. But there are solutions to do things better now, so it seems odd to ignore that. It's a bit like saying React/Angular/Vue/any other frontend framework won't make things tangibly better, so why bother. Or optimising page size won't make things better, so why bother. Google actively invests in telling everyone else to optimize their websites and even prioritizes fast sites over slow sites (https://developers.google.com/speed https://developers.google.com/speed). Why shouldn't they care about their performance too?
    - askmike 7y ago
      This isn't just a frontend optimization that can be applied in isolation by a good dev. With WS or SSE there now needs to be server written & supported to push this data over. On top of all the networking issues that come with these streaming protocols. > Why shouldn't they care about their performance too? They definitely do this for everything they push out that runs on billions of devices for people all over the world. How many people use Google to keep track of the live score of "the Laver Cup tennis championship in Australia?" I'd be surprised if that's more than 100. But Google has these numbers, and based on that they've chosen not to optimize this.
      - nostrademons 7y ago
        There's also the issue of optimizing for users of sports fans but pessimizing for the general population. Say that they do this "right" and include a real-time websockets client that has all the proper reconnect logic, uses a minified protocol, has a proper real-time stack on the backend with all the appropriate DDoS protection, encryption, CORS, etc. All the client logic is extra bytes that need to be shipped on every search request. Or they could split the bundle and lazy-load it, which is still extra bytes, but fewer of them, and adds complexity. The server code needs to be audited for security; one compromised frontend server and the damage to Google is already more than the entire value of the feature. One black-swan websearch outage (caused, say, by a misconfiguration leading to the load balancers getting slammed by websocket connections, or a misconfiguration causing the DDoS servers to think websearch is actually undergoing a DDoS when it's just normal websocket traffic) is already more than the entire value of the feature. The bar for simplicity and reliability has to be pretty high for a feature that's going to be used by 0.0001% of users, because if there are any negative effects at all on mainstream websearch traffic, that feature should never have been launched.
- deleted 7y ago
  [deleted]
- cbhl 7y ago
  Those solutions are not dead-simple at scale: they consume resources (such as file handles) on the server and they have to be supported by middle-boxes (such as proxies and reverse-proxies). Google gets to see the actual error rates when these solutions are deployed in production. If you want traffic to get through to end-users, then it has to look like what network administrators expect to get to the long tail. The end result: QUIC is encapsulated in UDP, DNS-over-HTTPS, and MPEG-DASH live streaming (making streaming video look like HTTP, and making a new request every few seconds). If you are not at Google scale, please, yes, use the newer, simpler technology.
  - wahern 7y ago
    > MPEG-DASH live streaming IMO that was more dictated by server-side complexity. A multimedia streaming server, especially for live video, that is optimized end-to-end for robustness and efficiency can be a very complex beast. Streaming transcoding, metadata injection, buffering, etc are difficult on their own. Doing them together in a unified fashioned is especially difficult. Solutions like GStreamer that abstract this away and make it conceptually simple are also extremely inefficient at scale. (Take it from someone who wrote an entire end-to-end solution that could do M:N codec, format, and transport transcoding, per-user streaming ad injection, and a bunch of other stuff. I could do 5k+ live audio streams, each with unique ad injection/splicing, with most of the pipeline occurring on a single E3 Xeon core--the socket writes to 5,000 clients from a single thread and the resulting ethernet interrupts that caused that CPU to max out. Reusing solutions like GStreamer or FFmpeg directly in your pipeline you'd be extremely lucky to get 1/10 of that, and more likely 1/100 at best.) MPEG-DASH permits the incoming media processor to simply dump a bunch of files to disk. The client facing service can then serve those up however it wishes--it can be completely dumb or incredibly smart, but importantly both halves can evolve on their own. You could do codec and rate transcoding as a third, middle component. And you could break these components down even further, as "everything is a file" is the most basic IPC interface we have. This makes iterative development much easier. IME, MPEG-DASH is pretty much what you'd get if you began with stodgy old (relatively) commercial broadcast equipment using nasty broadcast framing standards and organically built a streaming pipeline from scratch, without ever having the benefit/burden of knowing how to do performant streaming server development, or even how existing solutions worked. The multiple files non-sense was never necessary for client compatibility or network filter bypassing. Apple had solved that with RTSP-over-HTTP tunneling years prior, as did Flash with its clunkier RTMP solutions later.
wil421 7y ago
Google, just like any other company, has to balance between something working and working the right way. The Google managers are probably just fine with polling and, as you said, the bandwidth isn’t even a rounding error. Who sits at the Google search page for longer than a minute or two waiting for scores to update? They probably aren’t waiting for more than a couple polls.
- chipperyman573 7y ago
  I do this all the time when I want the live scores but don't want to watch a game
xg15 7y ago
> and I guarantee that the bandwidth used by this feature is rounding error for Google. You're points are of course correct, but I want to note that bandwidth goes both ways. They may not care that they are wasting their own bandwidth - but they are wasting the users' bandwidth as well. If the page was kept open for some time on e.g. a metered connection, this could cause some annoying surprises. From Google's perspective, the descisions make a lot of sense, but it's not simply cost/benefit, it's also not considering externalised cost.
- kllrnohj 7y ago
  > but they are wasting the users' bandwidth as well. If the page was kept open for some time on e.g. a metered connection, this could cause some annoying surprises. Over a 5 minute window we're talking 68KiB before compression. The article made the (poor imo) decision to omit compression, claiming it will affect all endpoints. But of course it won't affect all endpoints equally, since the payloads themselves are not equal. But let's go with this 68KiB for now as the worst case. That means if you left this open for a solid 2 hours it'd consume a grand total of less than 1.7MiB. Even on a metered connection it'd be damn hard to single this out as anything but a rounding error.
  - flanbiscuit 7y ago
    > 1.7MiB That's way less than so many web pages deliver on initial load.
  - xg15 7y ago
    Hmm, that's a good point.

matt_oriordan 7y ago

I appreciate when features are in test phase that it makes sense to keep it simple. But this is in production, and there are many dead-simple solutions that do this efficiently, long polling, SSE, etc. This is Google after all.

Raidion 7y ago
To implement this, you have to have a developer who is doing something else switch to this, get up to speed, and make a change. This involves risk and opportunity cost. The question for enterprise software development isn't "Will this make this thing better?", but "Is making this change more valuable than the other things on the priority list?"
- matt_oriordan 7y ago
  Sure, but I'm never suggested front-end developers should build new tech every time they want a new feature. But there are solutions to do things better now, so it seems odd to ignore that. It's a bit like saying React/Angular/Vue/any other frontend framework won't make things tangibly better, so why bother. Or optimising page size won't make things better, so why bother. Google actively invests in telling everyone else to optimize their websites and even prioritizes fast sites over slow sites (https://developers.google.com/speed https://developers.google.com/speed). Why shouldn't they care about their performance too?
  - askmike 7y ago
    This isn't just a frontend optimization that can be applied in isolation by a good dev. With WS or SSE there now needs to be server written & supported to push this data over. On top of all the networking issues that come with these streaming protocols. > Why shouldn't they care about their performance too? They definitely do this for everything they push out that runs on billions of devices for people all over the world. How many people use Google to keep track of the live score of "the Laver Cup tennis championship in Australia?" I'd be surprised if that's more than 100. But Google has these numbers, and based on that they've chosen not to optimize this.
    - nostrademons 7y ago
      There's also the issue of optimizing for users of sports fans but pessimizing for the general population. Say that they do this "right" and include a real-time websockets client that has all the proper reconnect logic, uses a minified protocol, has a proper real-time stack on the backend with all the appropriate DDoS protection, encryption, CORS, etc. All the client logic is extra bytes that need to be shipped on every search request. Or they could split the bundle and lazy-load it, which is still extra bytes, but fewer of them, and adds complexity. The server code needs to be audited for security; one compromised frontend server and the damage to Google is already more than the entire value of the feature. One black-swan websearch outage (caused, say, by a misconfiguration leading to the load balancers getting slammed by websocket connections, or a misconfiguration causing the DDoS servers to think websearch is actually undergoing a DDoS when it's just normal websocket traffic) is already more than the entire value of the feature. The bar for simplicity and reliability has to be pretty high for a feature that's going to be used by 0.0001% of users, because if there are any negative effects at all on mainstream websearch traffic, that feature should never have been launched.
deleted 7y ago
[deleted]
cbhl 7y ago
Those solutions are not dead-simple at scale: they consume resources (such as file handles) on the server and they have to be supported by middle-boxes (such as proxies and reverse-proxies). Google gets to see the actual error rates when these solutions are deployed in production. If you want traffic to get through to end-users, then it has to look like what network administrators expect to get to the long tail. The end result: QUIC is encapsulated in UDP, DNS-over-HTTPS, and MPEG-DASH live streaming (making streaming video look like HTTP, and making a new request every few seconds). If you are not at Google scale, please, yes, use the newer, simpler technology.
- wahern 7y ago
  > MPEG-DASH live streaming IMO that was more dictated by server-side complexity. A multimedia streaming server, especially for live video, that is optimized end-to-end for robustness and efficiency can be a very complex beast. Streaming transcoding, metadata injection, buffering, etc are difficult on their own. Doing them together in a unified fashioned is especially difficult. Solutions like GStreamer that abstract this away and make it conceptually simple are also extremely inefficient at scale. (Take it from someone who wrote an entire end-to-end solution that could do M:N codec, format, and transport transcoding, per-user streaming ad injection, and a bunch of other stuff. I could do 5k+ live audio streams, each with unique ad injection/splicing, with most of the pipeline occurring on a single E3 Xeon core--the socket writes to 5,000 clients from a single thread and the resulting ethernet interrupts that caused that CPU to max out. Reusing solutions like GStreamer or FFmpeg directly in your pipeline you'd be extremely lucky to get 1/10 of that, and more likely 1/100 at best.) MPEG-DASH permits the incoming media processor to simply dump a bunch of files to disk. The client facing service can then serve those up however it wishes--it can be completely dumb or incredibly smart, but importantly both halves can evolve on their own. You could do codec and rate transcoding as a third, middle component. And you could break these components down even further, as "everything is a file" is the most basic IPC interface we have. This makes iterative development much easier. IME, MPEG-DASH is pretty much what you'd get if you began with stodgy old (relatively) commercial broadcast equipment using nasty broadcast framing standards and organically built a streaming pipeline from scratch, without ever having the benefit/burden of knowing how to do performant streaming server development, or even how existing solutions worked. The multiple files non-sense was never necessary for client compatibility or network filter bypassing. Apple had solved that with RTSP-over-HTTP tunneling years prior, as did Flash with its clunkier RTMP solutions later.

wil421 7y ago

Google, just like any other company, has to balance between something working and working the right way. The Google managers are probably just fine with polling and, as you said, the bandwidth isn’t even a rounding error. Who sits at the Google search page for longer than a minute or two waiting for scores to update? They probably aren’t waiting for more than a couple polls.

chipperyman573 7y ago
I do this all the time when I want the live scores but don't want to watch a game

xg15 7y ago

> and I guarantee that the bandwidth used by this feature is rounding error for Google. You're points are of course correct, but I want to note that bandwidth goes both ways. They may not care that they are wasting their own bandwidth - but they are wasting the users' bandwidth as well. If the page was kept open for some time on e.g. a metered connection, this could cause some annoying surprises. From Google's perspective, the descisions make a lot of sense, but it's not simply cost/benefit, it's also not considering externalised cost.

kllrnohj 7y ago
> but they are wasting the users' bandwidth as well. If the page was kept open for some time on e.g. a metered connection, this could cause some annoying surprises. Over a 5 minute window we're talking 68KiB before compression. The article made the (poor imo) decision to omit compression, claiming it will affect all endpoints. But of course it won't affect all endpoints equally, since the payloads themselves are not equal. But let's go with this 68KiB for now as the worst case. That means if you left this open for a solid 2 hours it'd consume a grand total of less than 1.7MiB. Even on a metered connection it'd be damn hard to single this out as anything but a rounding error.
- flanbiscuit 7y ago
  > 1.7MiB That's way less than so many web pages deliver on initial load.
- xg15 7y ago
  Hmm, that's a good point.