7 ms·
Major Incident at Fastly
- jgrahamc 6y ago#hugops
- thunderbong 6y agoIs it just me or is the text practically non readable with very light font on a white background?
- mobilio 6y agoThey definitely need to work on accessibility.
- kordlessagain 6y agoIt's a status page, so do they need to fix the font color or the actual problem first?
- exikyut 6y agoI guess the litmus test is that it doesn't become inaccessible when infrastructure goes down.
- Sodman 6y agoThey obviously need to fix the actual problem first, but they only just found out about the new "actual problem". Surely somebody somewhere has actually looked at this status page before now, so this should have been a known issue already.
- kps 6y agoHTML is accessible by default, so nobody really has to work on accessiblity; they just have to refrain from working on breaking it. Unfortunately that seems too difficult for many web designers.
- gabrielsroka 6y agoI thought maybe that was the major incident. All their design people have left and messed up their CSS on the way out.
- hanoz 6y agoI haven't seen so much barely visible grey text since the last Hacker News thread discussing current affairs.
- uncledave 6y ago“It’s the wild colour scheme that freaks me,” said Zaphod whose love affair with this ship had lasted almost three minutes into the flight, “Every time you try to operate on of these weird black controls that are labelled in black on a black background, a little black light lights up black to let you know you’ve done it. What is this? Some kind of galactic hyperhearse?”
- gabrielsroka 6y agoI just ran the Chrome Lighthouse accessibility report again. They increased the contrast and went from 74 to 90. Well done!
- fastlyjen 6y agoI am the head of Product Design at Fastly. We updated the design of the page to include text with the proper color contrast to improve accessibility. The colors are all now WCAG AA compliant. Thank you for the feedback.
- gabrielsroka 6y agoI'm a little embarrassed by my earlier comment. Thank you for taking this in stride and for updating your site.
- josefresco 6y agoIt's not just you, there are 73 contrast errors (WCAG): https://wave.webaim.org/report#/https://status.fastly.com/incidents/1rbr326kjsmt https://wave.webaim.org/report#/https://status.fastly.com/in...
- jshprentz 6y agoAs compared with 139 contrast errors on this HN page as of 2021-01-12 19:04 GMT: https://wave.webaim.org/report#/https://news.ycombinator.com/item?id=25748442 https://wave.webaim.org/report#/https://news.ycombinator.com...
- user5994461 6y agoThe 139 contrast errors are all the headers "jshprentz" "12 minutes ago" not having enough contrast.
- rectang 6y agoThat "12 minutes ago" has insufficient contrast is still a valid critique from an accessibility standpoint, even if the design choice seems reasonable at first. There will be sight-impaired people who struggle to consume those "age" snippets, as opposed to perceiving them as legible but de-emphasized.
- zdragnar 6y agoI don't think the point was that they didn't matter, but structurally, it is 2 errors repeated 70 times as opposed to 140 distinct errors- might even just be one if they are lumped together as "post label" rather than "name" and "time".
- thelean12 6y agoHN has always had horrible UX and no desire to fix it. This tracks and does not excuse the original site at all.
- pdevr 6y agoVery true - I did Inspect Element and changed #808080 to #202020 without even thinking. I guess I am by now used to these low contrast designs :-)
- TheAnswerMan 6y agoUnreadable gray text on white background is the latest in Web Designer fashion.
- ec109685 6y agoLuckily I think it’s on the way out, thank goodness. I agree that was an awful trend.
- m3kw9 6y agoThe font colour they choose was named “really don’t want people to know but still have to”
- deleted 6y ago[deleted]
- silicon2401 6y agoI just don't understand how UI/UX designers, managers, whoever it was, managed to convince the world that grey on white makes any sense. Pure trend-following
- subsubzero 6y agoWhen asked why going with a light on light colored web page rendering readability to almost nil, the Designer spoke at length eloquently stating obscure mission statements, having the site born at the intersection of arts, activism and the global community at large. Finally they lastly mentioned "because its the right thing to do". /s
- codeduck 6y agoIs this a cry for help? :)
- vbezhenar 6y agoDesigners just want their work to be unique and eye-catching. They don’t care about readability. They care about user remembering that page. If most pages have good contrast, designers will want to use light contrast to stand different. If light contrast would become the norm, they will reinvent hard contrast. It all goes in cycle.
- dylan604 6y agoI remember that page as something I can't read and will tell everyone I know about how horrible it is. Is that really what "designers" want?
- foo_barrio 6y agoIf you have something else other than text that you want to be the focus of the page like a photo, reducing the contrast on the text allows the photos "pop" more if it using the range of light and dark. My personal color schemes are all dark background and light grey text. So even in my world of IDEs, consoles, gvim, sublime and emacs where text is king, I prefer having less than 100% contrast between background/text. I am not using any kind of custom color scheme so at least 1 other person prefers this as well!
- nvarsj 6y agoDark mode browser extension is a wonderful thing.
- 411111111111111 6y agoI was wondering what everyone was taking about. I totally forgot I installed the dark reader extension on my phone... Thanks for reminding me!
- thatguy0900 6y agoI'm 25 and had to use reader mode for this site, never seen a site this bad before
- shrimp_emoji 6y agohttps://www.contrastrebellion.com/ https://www.contrastrebellion.com/
- HomeDeLaPot 6y agoYep, it's next to unreadable. Who thought this was a good idea?
- josephorjoe 6y agoIt gives it that really cool look of weatherworn text chiseled into stone. What you should do is spread some parchment over it and get some charcoal and rub the charcoal on the parchment over the chiseled letters... oh. yeah. sorry...
- karmakaze 6y agoIf you turn up the brightness to max when your eyes hurt the text is legible.
- cdubzzz 6y agoIs this major (per title)? Anyone experiencing it? I’ve got two sites in different affected regions that do not seem to be affected at all.
- arecurrence 6y agoIt's intermittent but I'm seeing it on sites that are using fastly (EG: terraform runs).
- ctrlrsf 6y agoNoticed a few 503 and 502s around 10:40AM EST from different POPs.
- bussierem 6y agoPyPi is hosted through Fastly, and pip installs were all failing in our automated builds this morning, so I would call that pretty major.
- wcfields 6y agoThings like this can sink entire companies. I'm at a livestreaming service / production company that uses Vimeo as our primary. An outage like this when (as an example) "Procter & Gamble" is giving their yearly sales conference via a livestream due to COVID (instead of a large corporate event at a convention center) can cause a company to loose them as a client.
- move-on-by 6y agoAh, that explains the minor hiccup I just saw on my Github Pages. I thought it was odd that Github status was all green.
- BrentOzar 6y agoAffected live streaming on both Mux.com and Vimeo.com (the two streaming companies I use for my training classes, hahaha - so much for redundancy with paid companies. Had to switch to YouTube.)
- wcfields 6y agoUgh, on pins and needles over here. We have a handful of Vimeo livestreams going on right now.
- andrewbarba 6y agoWith Mux you can enable "Redundant Streams" (multiple CDN's used in HLS manifest) which I think solves the issue here. We are also a big Mux customer but admittedly haven't made the full switch to redundant streams, although our testing was very promising. Since their ingest is all GCP I don't think that was affected by this, and it would have just been delivery. https://www.mux.com/blog/survive-cdn-failures-with-redundant-streams/ https://www.mux.com/blog/survive-cdn-failures-with-redundant...
- mmcclure 6y agoThanks for the kind words, Andrew! Redundant streams would definitely help mitigate the problem, but we're also days away from shipping a feature that would have allowed us to seamlessly switch over to another partner in a situation like this. C'est la vie.
- therealwardo 6y agothere is a good chance that redundant streams would have helped here assuming that the player could load the manifest to fail over, but that manifest sadly hits Fastly. the problem with redundancy is that it really only works if you are redundant all the way up and down the serving stack. in our standard serving path (even without redundant streams) we automatically select the optimal CDN for chunk serving which does take into account availability, but during this incident we measured Fastly as about 5% less available than normal which wasn't enough to trigger a full automatic drain off of them. turns out, a 5% failure rate when you are serving 50 chunks to enable a video playback means 1 of them will likely fail and you get a playback error or in the best case, rebuffering. sadly, Mux's manifest servers have not been redundant because the edge logic is a pile of Fastly's VCL. as mmcclure says, we're working on adding another CDN that can run our manifest serving logic so that we can be fully redundant. I am really looking forward to a standard and portable edge platform that makes redundancy at this application layer easier. wasm here we come?
- floatingatoll 6y ago“A fix has been implemented”: for those just now seeing this, you’ve probably read this post unnecessarily :)
- paulluuk 6y agoI noticed that pythonhosted.org was shaky, which meant that `pip install` failed over and over again on different packages. CircleCI and Github were also affected.
- 0xbadcafebee 6y agoFastly: if you could add context to the incident as you track it, it would help customers and users feel better. Also, linking a public postmortem to the incident would be great. It doesn't inspire confidence as a customer if we can't see that you are working on identifying the cause and are preventing it from recurring. (We don't need to see the full RCA or other process, just an overview of the problem, the cause, the fix, and the follow-up. This is a great way to iterate & improve on your RCA process)
- philip1209 6y agoAnd yet, their stock is up +5.32% today.