6 ms·

How Twitter deploys its widgets JavaScript

54 points by doppp 10y ago

ec109685 10y ago
This has now been posted 5 times: https://news.ycombinator.com/from?site=blog.twitter.com https://news.ycombinator.com/from?site=blog.twitter.com
- dang 10y ago
  But not discussed yet, so the reposts are ok. Please see https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html. (We do say a 'small number' of reposts, which 5 is probably over, but not that much over.)
  - mianos 10y ago
    Probably because no one cares.
    - dang 10y ago
      That probably underestimates the randomness of what gets noticed here.
luvz2code 10y ago
How does this work along with browser caches ?
- bpicolo 10y ago
  Cache-control headers. Browsers won't cache indefinitely unless you ask them to.
  - Kiro 10y ago
    Am I insane or are cache-control headers unreliable? I always set cache-control: no-cache on my S3 files but still get complaints about people getting old versions with broken pages as result.
    - chipperyman573 10y ago
      Some CDNs don't always listen to those (I've noticed cloudflare especially has a problem with them if I don't add the files to the ignore list in their control panel), maybe that's it?
    - jgalt212 10y ago
      I don't rely on them. We set query parameters on new JS code to definitively bust caches. src="foo.js?v=1", then src="foo.js?v=2", etc. I'm sure people consider this a hack, so I'm open to hearing other suggestions.
      - dcherman 10y ago
        It's not a hack at all for resources you control - in fact, it's encouraged to do so in order to use very long lived Expires headers while still being able to deploy new versions of your app. It doesn't work for unversioned library code though where you want to automatically upgrade all users of your code as is the case with Twitter it looks like.
        tarancato 10y ago
        Problem with these is that many caching proxies consider all resources with a query string to be dynamically generated and won't cache them at all, which in the case of js/css/etc is bad
        dcherman 10y ago
        Sure, but that's easily solvable by versioning assets in the filename itself. That's how the `filerev` task works for both Grunt and Gulp.
- ricardobeat 10y ago
  Had the same thought. Looking at the headers they serve the file with a mere 30m expire time.
  - ptolts 10y ago
    I was wondering if the experimental group gets a cache time of 0 so they can immediately be rolled back.
- gcb0 10y ago
  they probably set no-cache, which gives them the worst performance. and don't even use it for their benefit since they decide what to serve based on dns, which is always cached in the os and the browser. found that write up extremely comical.
rhizome 10y ago
"Many bugs manifest themselves during deploys, especially when there are a large number of code changes." I'm intrigued by this hole in the process. At Twitter scale, how is it that these bugs make it past testing, QA, CI, and staging? I know the Dijkstra quote[1], but still: Twitter AFAIK has as organized and talented a code practice as any company of that size needs in order to do any programming and maintenance, and still they get the kinds of problems that even freelance clients gripe about costing them thousand$/hour. I also realize that I'm playing a little No True Scotsman here, but again: still. 1. http://www.brainyquote.com/quotes/quotes/e/edsgerdijk201165.html http://www.brainyquote.com/quotes/quotes/e/edsgerdijk201165....
squiggy22 10y ago
I have recently come across a bug in relation to this particular javascript widget. Once they dropped the requirement for widget IDs here: https://blog.twitter.com/2016/embedding-twitter-timelines-just-got-a-lot-easier https://blog.twitter.com/2016/embedding-twitter-timelines-ju... Widgets no longer worked. So basically if you had widget IDs on data elements in HTML, the code no longer executed. Forced upgrade of the frontend code to match the JS requirements.
markdown 10y ago
Do they clean up and minify first? The devs at Pinterest and TripAdvisor are jerks about this. Try following this one: <script src="https://www.jscache.com/wejs?wtype=certificateOfExcellence&uniq=847&locationId=307144&lang=en_US&year=2016&display_version=2"></script> After the redirects, they serve you a file that is unminified and with 20% comments. I'm not sure if the engineers at these places hate their users, or are just incompetent. EDIT: They do: http://platform.twitter.com/widgets.js http://platform.twitter.com/widgets.js
- kalleboo 10y ago
  While not minifying is a bit unprofessional, I'm not sure wasting 1.4 KB counts as "hating your users" in this world of large retina images.
  - markdown 10y ago
    > "hating your users" You're right, I went to far. Not hatred, disregard and carelessness. Contempt maybe. It would take them a couple of hours at most (testing included) to add minification to their build process if they have one (a this stage, one can't be too sure since they don't follow basic web dev best-practice). > 1.4 KB That is for one badge. This is even more madness; they make you download the same js file from a different URL for each badge you want to display. Resorts like to show as many badges as they have to show that they've been highly ranked for a long time. A client of mine who wants to show her little resort has been the topped rank in her area for the past five years has to add 7KB of JS to do it. > This world of large retina images Not outside the first world, and tripadvisor is a global company supporting multinationals and tiny little 2-hut resorts alike. It's very likely they made the same assumptions that you did... which is basically my point... that they have no respect for their users (resorts and web devs). A little internet cafe in Timbuktu doesn't have 7KB of data to waste.
edtechdev 10y ago
You can no longer embed their widgets on platforms that allow iframes but not script tags, like most learning management systems.

54 points by doppp 10y ago

This has now been posted 5 times: https://news.ycombinator.com/from?site=blog.twitter.com https://news.ycombinator.com/from?site=blog.twitter.com

dang 10y ago
But not discussed yet, so the reposts are ok. Please see https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html. (We do say a 'small number' of reposts, which 5 is probably over, but not that much over.)
- mianos 10y ago
  Probably because no one cares.
  - dang 10y ago
    That probably underestimates the randomness of what gets noticed here.

luvz2code 10y ago

How does this work along with browser caches ?

bpicolo 10y ago
Cache-control headers. Browsers won't cache indefinitely unless you ask them to.
- Kiro 10y ago
  Am I insane or are cache-control headers unreliable? I always set cache-control: no-cache on my S3 files but still get complaints about people getting old versions with broken pages as result.
  - chipperyman573 10y ago
    Some CDNs don't always listen to those (I've noticed cloudflare especially has a problem with them if I don't add the files to the ignore list in their control panel), maybe that's it?
  - jgalt212 10y ago
    I don't rely on them. We set query parameters on new JS code to definitively bust caches. src="foo.js?v=1", then src="foo.js?v=2", etc. I'm sure people consider this a hack, so I'm open to hearing other suggestions.
    - dcherman 10y ago
      It's not a hack at all for resources you control - in fact, it's encouraged to do so in order to use very long lived Expires headers while still being able to deploy new versions of your app. It doesn't work for unversioned library code though where you want to automatically upgrade all users of your code as is the case with Twitter it looks like.
      - tarancato 10y ago
        Problem with these is that many caching proxies consider all resources with a query string to be dynamically generated and won't cache them at all, which in the case of js/css/etc is bad
        dcherman 10y ago
        Sure, but that's easily solvable by versioning assets in the filename itself. That's how the `filerev` task works for both Grunt and Gulp.
ricardobeat 10y ago
Had the same thought. Looking at the headers they serve the file with a mere 30m expire time.
- ptolts 10y ago
  I was wondering if the experimental group gets a cache time of 0 so they can immediately be rolled back.
gcb0 10y ago
they probably set no-cache, which gives them the worst performance. and don't even use it for their benefit since they decide what to serve based on dns, which is always cached in the os and the browser. found that write up extremely comical.

rhizome 10y ago

"Many bugs manifest themselves during deploys, especially when there are a large number of code changes." I'm intrigued by this hole in the process. At Twitter scale, how is it that these bugs make it past testing, QA, CI, and staging? I know the Dijkstra quote[1], but still: Twitter AFAIK has as organized and talented a code practice as any company of that size needs in order to do any programming and maintenance, and still they get the kinds of problems that even freelance clients gripe about costing them thousand$/hour. I also realize that I'm playing a little No True Scotsman here, but again: still. 1. http://www.brainyquote.com/quotes/quotes/e/edsgerdijk201165.html http://www.brainyquote.com/quotes/quotes/e/edsgerdijk201165....

squiggy22 10y ago

I have recently come across a bug in relation to this particular javascript widget. Once they dropped the requirement for widget IDs here: https://blog.twitter.com/2016/embedding-twitter-timelines-just-got-a-lot-easier https://blog.twitter.com/2016/embedding-twitter-timelines-ju... Widgets no longer worked. So basically if you had widget IDs on data elements in HTML, the code no longer executed. Forced upgrade of the frontend code to match the JS requirements.

markdown 10y ago

Do they clean up and minify first? The devs at Pinterest and TripAdvisor are jerks about this. Try following this one: <script src="https://www.jscache.com/wejs?wtype=certificateOfExcellence&uniq=847&locationId=307144&lang=en_US&year=2016&display_version=2"></script> After the redirects, they serve you a file that is unminified and with 20% comments. I'm not sure if the engineers at these places hate their users, or are just incompetent. EDIT: They do: http://platform.twitter.com/widgets.js http://platform.twitter.com/widgets.js

kalleboo 10y ago
While not minifying is a bit unprofessional, I'm not sure wasting 1.4 KB counts as "hating your users" in this world of large retina images.
- markdown 10y ago
  > "hating your users" You're right, I went to far. Not hatred, disregard and carelessness. Contempt maybe. It would take them a couple of hours at most (testing included) to add minification to their build process if they have one (a this stage, one can't be too sure since they don't follow basic web dev best-practice). > 1.4 KB That is for one badge. This is even more madness; they make you download the same js file from a different URL for each badge you want to display. Resorts like to show as many badges as they have to show that they've been highly ranked for a long time. A client of mine who wants to show her little resort has been the topped rank in her area for the past five years has to add 7KB of JS to do it. > This world of large retina images Not outside the first world, and tripadvisor is a global company supporting multinationals and tiny little 2-hut resorts alike. It's very likely they made the same assumptions that you did... which is basically my point... that they have no respect for their users (resorts and web devs). A little internet cafe in Timbuktu doesn't have 7KB of data to waste.

edtechdev 10y ago

You can no longer embed their widgets on platforms that allow iframes but not script tags, like most learning management systems.