10 ms·

Coordinating the Superbowl's visual fidelity with Elixir

644 points by lawik 2y ago

JSR_FDED 2y ago
It’s always surprised me how the world of digital video is a cousin of IT yet is impenetrable to people outside the video industry. How they refer to resolutions, colors, networking, storage is (almost deliberately?) different.
- walrus01 2y ago
  People who only ever work with 'consumer' video equipment need extra training and a back-to-basics set of reading material to understand things like the difference between a 420 and 422 color space, or why serious cinema cameras record video ungraded, or what the color grading process in a post-production workflow looks like (and the different aesthetic choices of grading that might be possible). That's before even getting into things like raw yuv/y4m uncompressed video, or very-high-bitrate barely compressed video, generating proxy footage to work with in an editor because the raw is too much of a firehose of data to handle even on a serious workstation.... I would say that unless you have a professional reason, there's very little benefit to the average end-user to do a deep dive into it. If your intention is to spend $7000 on a RED camera and then $13,000 on lenses, gimbal, cage, follow focus, matte box, memory cards etc to make a small and cost effective single camera production package, then by all means, dig into it.
  - keane 2y ago
    4:2:0 vs 4:2:2 for anyone curious: https://youtu.be/7JYZDnenaGc?feature=shared&t=101 https://youtu.be/7JYZDnenaGc?feature=shared&t=101 and https://www.red.com/red-101/video-chroma-subsampling https://www.red.com/red-101/video-chroma-subsampling
  - dist-epoch 2y ago
    Grading is abused so much these days, it's like a curse. You have a pristine video chain, only to turn it all into yellow/blue at the end.
    - davidbou 2y ago
      There's a notable difference between shading and grading. Shading is for the TV industry where you adjust all cameras to match perfectly the exposure, tone curve and colors. So when switching between camera angles you don't notice any difference in skin tone or detail, and the green of the grass and blue of the sky are all the same. Also a very important point is to get the color of the sponsor logos right, that would be where to start sometimes... There's less creativity here, you have mainly to follow the standards like ITU-R BT.709 or for HDR HLG and ITU-R BT.2020. Grading is the creative process of adding a look to your production, which is usually handled in post production but there are now ways to do it live, although by using similar tools as the post production software. And they still re-do it in post production. This is used live for concerts and fashion shows. There is a significant distinction between shading and grading. Shading is essential in the TV industry, where the goal is to ensure all cameras are perfectly matched in exposure, tone curve, and colors. This ensures seamless transitions between camera angles, maintaining consistency in skin tones, fine details, and the color of grass and sky. A crucial aspect of shading is accurately reproducing sponsor logos' colors, which can sometimes be the starting point as that's where the money comes from. Creativity plays a lesser role here, as the focus is on following industry standards such as ITU-R BT.709 for SDR or ITU-R BT.2020 and HLG for HDR. Grading, on the other hand, is a creative process meant to give a distinctive look to a production . Traditionally done in post-production, it can also now be applied in real time using tools similar to those found in post-production software. Despite this, it is often still refined further in post. Live grading is commonly used for events such as concerts and fashion shows, where you want to look different from TV productions.
      - aalam 2y ago
        TIL about shading, and am surprised how less I've seen this term in grading tutorials. While different, I feel like shading is something that should be learnt before grading. PS You might have pasted two different answer drafts above. Paras 1,4 and 2,5 deliver similar information
- deleted 2y ago
  [deleted]
- davidbou 2y ago
  This gives an idea of the parameters we cover for roughly 200 different models of broadcast cameras we might have so far. These are only to tweak the image quality which is the job of the video engineer (vision engineer in UK). We usually don't cover all the other functions a camera has, which could be more intended for the camera operator himself. The difficulty is to bring some consistency with so many different cameras and protocols. https://pastebin.com/cgeG2r0k https://pastebin.com/cgeG2r0k
  - noisy_boy 2y ago
    Do you "normalize" the parameters to some intermediate config so that everything behind that just needs to work with that uniform intermediate config? What about settings that are unique to a given device?
    - davidbou 2y ago
      That was the idea—we started by normalizing all the standard parameters found in most cameras. The challenge came when we had to incorporate brand-specific parameters, many of which are only used by a single manufacturer. Operators also weren’t keen on having values changed from what the camera itself provided, as some settings serve as familiar reference points. For example, they know the right detail enhancement values to use for football or studio work. So, we kept normalization for the key functions where it made sense, but for other parameters, we now try to stay as close as possible to the camera’s native values. As for the topics on MQTT, they function as a kind of universal API—at least internally. Some partners and customers are already using them to automate certain functions. However, we haven’t officially released anything yet, as we wouldn’t be able to guarantee stability or prevent changes at this stage.
      - noisy_boy 2y ago
        I have noticed that you and your team's answers are detailed and insightful - much appreciated.
ram_rar 2y ago
Great to see Elixir gaining traction in mission-critical broadcast systems! I wonder, how much of Cyanview's reliability comes from Elixir specifically versus just good implementation of MQTT? and is there any specific Elixir features were essential that couldn't be replicated in other languages?
- zaik 2y ago
  > couldn't be replicated in other languages? All programming languages can do any task. It's about how easy they make that task for you.
  - zwnow 2y ago
    Yea and with that i'd think it would be a pain in the ass trying to replicate BEAM behavior in different langs
  - jdufawdfas 2y ago
    [flagged]
  - thibaut_barrere 2y ago
    This is true in general but only until it gets false. For instance, Elixir supports compilation targeting GPUs (within exactly the same language, not a fork). Most languages do not allow that (and for most it would be fairly hard to implement).
  - goatlover 2y ago
    For any finite, computable task, as long as the language has access to the hardware that can perform the task in practical time, assuming the language doesn't present any compilation or memory issues to take advantage of said hardware in practical time for the task to be worth computing.
- ghislainle 2y ago
  Main developer here. We use MQTT a lot, it is really a central piece of our architecture, but Elixir brings a lot of benefits regarding the handling of many processes which are often loosely coupled. The BEAM and OTP offer a sane approach to concurrency and Elixir is a nice language on top. Here is what I find the most important benefits: - good process isolation, even the heap is per process. This allows us to have robust and mature code running along more experimental features without the fear of everything going down. And you still have easy communication between processes - supervision tree allows easy process management. I also created a special supervisor with different restart strategies. The language allows this and then, it integrates as any other supervisor. With network connections being broken and later reconnected, the resilience of our system is tested regularly, like a physical chaos monkey - the immutability as implemented by the BEAM greatly simplifies how to write concurrent code. Inside a process, you don't need to worry about the data changing under you, no other process can change your state. So no more mutex/critical sections (or very little need). You can still have deadlock though, so it is not a silver bullet
arakutourisam 2y ago
[dead]
notepad0x90 2y ago
What is being used in similar broadcast setups outside of this Superbowl?
- imjonse 2y ago
  According to the article this software is used for all major sporting events.
- davidbou 2y ago
  Major events use it for all kind of soecialty cameras as they aready have the technology for the main studio cameras. So we had to develop solutions for everything that was not working. And major productions have budgets for all kind of new toys:mini-cams, drones, cable cams, now cinematic look from small mirrorless cameras, slow motion, etc. That opened up a whole lot of possibilities to be creative but you have to be as reliable as the main cameras and aim for the best image quality. Now the same products are used for very small productions that don't have the budget for any studio camera (look typically at 50k+ for a camera without lens). In that case we try to provide a similar user experience and functions but with much more ffordable cameras. Finaly more and more live productions are now handled using cine style cameras which don't have the standard broadcast remote panels and that's another area we cover, by combining camera control with control of many external boxes like external motors to drive manual lenses or 3D Lut video processors. Applications are on fashion shows, concerts, theater, Churches, studio shows, even corporate. In the end Elixir is used for a lot of small processes which handle very low level control protocols. And then on top of that add a high-level of communication between devices either on local networks or over the cloud.
  - mcintyre1994 2y ago
    > Now the same products are used for very small productions that don't have the budget for any studio camera Just out of curiosity, what would be examples of very small productions here? Would an independent YouTube channel with great production quality be using this?
    - 2y ago
laserbeam 2y ago
Of course! Of course you have to do color correction on all the different cameras pointed from different angles at a sports event. I absolutely love reading about hard problems that are invisible to most people.
- abrookewood 2y ago
  Yes, it's one of those super-niche & super important functions that are obvious once you know about them, but would never think about otherwise.
  - myst 2y ago
    Why is it super-important?
    - GiorgioG 2y ago
      Because events using many cameras without this type of setup would cause jarring visual differences when switching between one camera/view to another during the broadcast.
      - HeatrayEnjoyer 2y ago
        Why wasn't it jarring in early Superbowl games?
        lgeorget 2y ago
        Fewer cameras, lower resolution, poor color rendering even with one camera anyway?
        devmor 2y ago
        And less color fidelity on receiving displays I would imagine!
        davidbou 2y ago
ZeWaka 2y ago
> Without any marketing, it earned a reputation among seasoned professionals and became a staple at the world’s top live events. Sounds like the entertainment industry. Everyone truly knows everyone, especially when you're working on the same show with the same crew year after year. It's definitely a family of sorts.
- pharrington 2y ago
  "Without any marketing" is also an obvious lie, since Cyanview has a storefront website and marketing posts on Linkedin.
  - caseyohara 2y ago
    Also this article seems more like an ad for Cyanview than for Elixir. Smells like content marketing to me.
    - mtndew4brkfst 2y ago
      Most of the Elixir in Production posts on the official blog (including this one) come off this way, IMO: https://elixir-lang.org/blog/categories.html#Elixir%20in%20Production https://elixir-lang.org/blog/categories.html#Elixir%20in%20P... It's pretty much just appeal to authority. "These people are successful and they used Elixir, why don't you?"
    - lawik 2y ago
      Really? I can spill all the juicy details as the main author and instigator. Cyanview reached out to me to help find a dev a while back. Hearing about their customers I knew it would be a decently big splash for Elixir. I was surprised that they were unknown and had this succcess with big household name clients. I like them. I like their whole deal. Small team, punching above their weight. Hardware, software, FPGAs and live broadcasts. The story has so much to it. David and team have been great sports in sharing their story. Fundamentally I care more about Elixir adoption though, I reached out to the Elixir team and offered to interview them and write something up. A case study about successful Elixir production deployments is definitely content marketing. But for Elixir. It is a very common question when mentioning a less common language. "Who uses this?" I thought it was a very interesting case. Glad to have it documented. The style of a case study won't suit everyone. I suppose "without any marketing, before _this_" would have been funny.
xlii 2y ago
I won’t let a friend start development in Elixir. Let me get it out: I love BEAM. OTP is awesome and one of the best systems in its kind. I was completely enamored with Elixir years ago as a modern Erlang which excited me to the bone. It’s no longer the case. When you get into non-trivial things there are many sharp edges and paper cuts. Some from the top of my head: - it’s impossible to disable warnings - test runs are often highly verbose because libraries ignore them and (as discussed somewhere in forums) warnings are deemed useful so they can’t be disable - the only way to catch some, important ones, in CI is to use "warnings-as-errors" … - so one cannot use deprecation flags because it’s also a warning - when having non trivial ecosystem one cannot selectively deprecate and get errors, this has to be a human process (remember to replace…) - when doing umbrella tests on non compiled code seed influences order of compilation - this order of compilation influences outcome and can lead to buggy code - dependency compilation is not parallelized - takes a lot of time and uses 10% of CPU - compilation process can break and Elixir isn’t aware that it was interrupted - this means that it doesn’t try to compile properly app/dependency but instead tries to release crippled one - documentation is hard to use - searching shows random versions and there isn’t even a link to „open newest” - searching in docs often not finds the keywords you can actually see - a lot of knowledge is implicit (try checking if you can dynamically add children to a Supervisor) - sidebar with new ExDocs break for some reason so there is no navigation - there is no way to browse navigation outside this broken ExDocs which outputs only HTML and LSP - LSP is afterthought, there are few but neither works well - Dialyzer/dialyxyr typespecs are useless most of the time - Squiggly arrow works weird (i.e. ~0.3 might catch 0.99) - my colleague recently mentioned Renovate not picking it up I could go on and on. I’m doing plenty of research so I’m working with various languages including „niche” ones like Prolog, OCaml, Clojure, Cuelang. Recently I’ve been developing tooling in Go and many core systems are developed in Rust in Elixir, and I work on the latter often. In principle Elixir is awesome, but has the worst developer experience of all. Sometimes it takes 4h to prepare and push release. Tooling I’m working on can do the same in 5 minutes - I’m parallelizing processes in containers, making idempotent output artifacts and heuristic failure detection to retry on flakiness. When switching between Go and OCaml you can sense how tooling cares for me and my time. Often I bounce off forums where people’s need are shrugged off as non-essential, treating those who came as uneducated juniors (because who in sane mind would like to have a parallel dependencies compilation or disable compilation time warning). There is nothing better than BEAM, but (for me) Elixir got much worse over the years.
pinoy420 2y ago
[dead]
sschueller 2y ago
Someone tracked every single camera shot during the halftime show: https://www.youtube.com/watch?v=YXNWfFtgbNI https://www.youtube.com/watch?v=YXNWfFtgbNI
- jcalx 2y ago
  You can also see what it looks like from the control room on Hamish Hamilton's YouTube channel, with the AD calling out shots and all: https://m.youtube.com/watch?v=gfjWjkTP4p8 https://m.youtube.com/watch?v=gfjWjkTP4p8. (Hamish Hamilton has directed every Super Bowl halftime show since 2010.)
- oplav 2y ago
  John DeMarsico directs the SNY broadcasts for the NY Mets and sometimes posts behind the scenes for how all the cameras come together into a production. I think they are pretty interesting to watch. https://x.com/SNYtv/status/1832250958258036871 https://x.com/SNYtv/status/1832250958258036871
deleted 2y ago
[deleted]
innocentoldguy 2y ago
I have implemented Elixir in critical financial applications, B2B growth intelligence applications, fraud detection applications, scan-and-go shopping applications, and several others. In every case, like the engineering team in this article demonstrates, the developer experience and end results have exceeded expectations. If you haven’t used Elixir, you should give it a try. Edit: Fixed an editing error.
- roughly 2y ago
  Elixir and Erlang have always garnered a lot of respect and praise - I’m always curious why they’re not more widely used (I’m no exception - despite hearing great things for literal decades, I’ve never actually picked it up to try for a project).
  - AlchemistCamp 2y ago
    I think most people tend to stick with what they learn first or hop to very similar languages. Schools generally taught Java and then more recently Python and JS, all of which are relatively similar. Unless someone who knows those three languages is curious or encounters a particular problem that motivates them to explore, they're unlikely to pick up an immutable, functional language.
    - innocentoldguy 2y ago
      I think you’re right. I only picked up Elixir about 10 years ago after getting frustrated with Python’s GIL and Java’s cumbersomeness, and feeling that object oriented programming over complicates things and never lived up to its hype. I have never looked back. Elixir is an absolute joy to use. It simplifies multi-threaded programming, pattern-matching makes code easier to understand and maintain, and it is magnitudes faster to code in than Java. For me, Elixir’s version of functional programming provides the ease of development that OOP promised and failed to deliver. In my opinion, Elixir is software engineering’s best kept secret.
  - solid_fuel 2y ago
    I've thought about this a lot, and I think that part of what hurts Erlang/Elixir adoption is the scale of the OTP. It brings a ton of fantastic tools, like supervision trees, process linking, ETS, application environments & config management, releases, and more. In some ways it's closer to adopting a new OS than a new programming language. That's what I love about Elixir, but it means that selling it is more like convincing a developer who knows and uses CSV to switch to Postgres. There's a ton of advantages to storing data in a relational DB instead of flat files, but now you have to define a schema up front, deal with table and row locking, figure out that VACUUM thing, etc. When you're just setting out to learn a new language, trying to understand a new OS on top hurts adoption.
DerekL 2y ago
The title is misspelled, should be “Super Bowl”, two words.
frankfrank13 2y ago
Really cool piece, this jumped out to me: > The devices in a given location communicate and coordinate on the network over a custom MQTT protocol. Over a hundred cameras without issue on a single Remote Control Panel (RCP), implemented on top of Elixir’s network stack. Makes sense! MQTT is, if I understand right, built on TCP. Idk if I would have found the same solution, but its seemingly a good one
eggy 2y ago
Would Gleam be practical for a similar application aside from the OTP/BEAM runtime? I am guessing you'd have to leverage Elixir libraries that are not present for Gleam yet, and you might have slower compile times due to static typing, but you'd catch runtime errors sooner. Would it be more of a debugging vs. fast dynamic iteration trade-off? I am looking to settle on either Gleam or Elixir. I liked Gleam's original ML syntax before, but I like static typing. Thoughts? I am replacing C with Zig, and I am brushing up on my assembly by adding ARM to my x64 skill set.
- widdershins 2y ago
  Gleam has a subset of OTP functionality already [1]. It also compiles extremely quickly. I haven't made any huge projects yet, but I've used some fairly chunky libraries and everything compiles super quick. [1] https://github.com/gleam-lang/otp https://github.com/gleam-lang/otp
  - nesarkvechnep 2y ago
    It’s subpar at the moment.
- AlchemistCamp 2y ago
  > you'd catch runtime errors sooner I don’t think there’s any evidence whatsoever that you would catch runtime bugs sooner with Gleam than with Elixir (or Erlang). Erlang’s record for reliability is stronger than many statically typed languages, including even Java. There is a certain class of errors static types can prevent but there’s a much larger set of those it can’t. To make the case for a language like TS/Java/Swift/Golang or Gleam actually resulting in fewer runtime defects than Erlang or Elixir, I’d want to see some real world data.
  - eggy 2y ago
    It depends on what “sooner” means to you. Gleam catches more before the code runs; Elixir catches them when they happen but recovers gracefully. If you’re paranoid about bugs reaching users, I would think Gleam’s your pick, no? If you trust your tests and love dynamic freedom, Elixir should be fine. I don't have much experience with either language. I did more in Erlang 8 years ago, but not much. I am on the edge of choosing Gleam over Elixir. It's mainly subjective: I prefer the syntax in Gleam, although I liked the original ML-like syntax when it first came out.
markb139 2y ago
30 odd years ago, part of my role was to colour balance cameras in a studio environment. We didn’t need computers - but at most there were only 5 cameras :)
RKFADU_UOFCCLEL 2y ago
[dead]
brcmthrowaway 2y ago
Wait, is Elixir actually accessing color pixel data in realtime?
- davidbou 2y ago
  No, it deals with metadata: control and status as explained in a previous reply https://news.ycombinator.com/item?id=43479094#43482362 https://news.ycombinator.com/item?id=43479094#43482362 Elixir does some computations as well but when we had to compute 3D luts based on video processing algorithms, Ghislain had to write them in C to be fast enough for our needs on embedded hardware.
hackburg 2y ago
[dead]

644 points by lawik 2y ago

It’s always surprised me how the world of digital video is a cousin of IT yet is impenetrable to people outside the video industry. How they refer to resolutions, colors, networking, storage is (almost deliberately?) different.

walrus01 2y ago
People who only ever work with 'consumer' video equipment need extra training and a back-to-basics set of reading material to understand things like the difference between a 420 and 422 color space, or why serious cinema cameras record video ungraded, or what the color grading process in a post-production workflow looks like (and the different aesthetic choices of grading that might be possible). That's before even getting into things like raw yuv/y4m uncompressed video, or very-high-bitrate barely compressed video, generating proxy footage to work with in an editor because the raw is too much of a firehose of data to handle even on a serious workstation.... I would say that unless you have a professional reason, there's very little benefit to the average end-user to do a deep dive into it. If your intention is to spend $7000 on a RED camera and then $13,000 on lenses, gimbal, cage, follow focus, matte box, memory cards etc to make a small and cost effective single camera production package, then by all means, dig into it.
- keane 2y ago
  4:2:0 vs 4:2:2 for anyone curious: https://youtu.be/7JYZDnenaGc?feature=shared&t=101 https://youtu.be/7JYZDnenaGc?feature=shared&t=101 and https://www.red.com/red-101/video-chroma-subsampling https://www.red.com/red-101/video-chroma-subsampling
- dist-epoch 2y ago
  Grading is abused so much these days, it's like a curse. You have a pristine video chain, only to turn it all into yellow/blue at the end.
  - davidbou 2y ago
    There's a notable difference between shading and grading. Shading is for the TV industry where you adjust all cameras to match perfectly the exposure, tone curve and colors. So when switching between camera angles you don't notice any difference in skin tone or detail, and the green of the grass and blue of the sky are all the same. Also a very important point is to get the color of the sponsor logos right, that would be where to start sometimes... There's less creativity here, you have mainly to follow the standards like ITU-R BT.709 or for HDR HLG and ITU-R BT.2020. Grading is the creative process of adding a look to your production, which is usually handled in post production but there are now ways to do it live, although by using similar tools as the post production software. And they still re-do it in post production. This is used live for concerts and fashion shows. There is a significant distinction between shading and grading. Shading is essential in the TV industry, where the goal is to ensure all cameras are perfectly matched in exposure, tone curve, and colors. This ensures seamless transitions between camera angles, maintaining consistency in skin tones, fine details, and the color of grass and sky. A crucial aspect of shading is accurately reproducing sponsor logos' colors, which can sometimes be the starting point as that's where the money comes from. Creativity plays a lesser role here, as the focus is on following industry standards such as ITU-R BT.709 for SDR or ITU-R BT.2020 and HLG for HDR. Grading, on the other hand, is a creative process meant to give a distinctive look to a production . Traditionally done in post-production, it can also now be applied in real time using tools similar to those found in post-production software. Despite this, it is often still refined further in post. Live grading is commonly used for events such as concerts and fashion shows, where you want to look different from TV productions.
    - aalam 2y ago
      TIL about shading, and am surprised how less I've seen this term in grading tutorials. While different, I feel like shading is something that should be learnt before grading. PS You might have pasted two different answer drafts above. Paras 1,4 and 2,5 deliver similar information
deleted 2y ago
[deleted]
davidbou 2y ago
This gives an idea of the parameters we cover for roughly 200 different models of broadcast cameras we might have so far. These are only to tweak the image quality which is the job of the video engineer (vision engineer in UK). We usually don't cover all the other functions a camera has, which could be more intended for the camera operator himself. The difficulty is to bring some consistency with so many different cameras and protocols. https://pastebin.com/cgeG2r0k https://pastebin.com/cgeG2r0k
- noisy_boy 2y ago
  Do you "normalize" the parameters to some intermediate config so that everything behind that just needs to work with that uniform intermediate config? What about settings that are unique to a given device?
  - davidbou 2y ago
    That was the idea—we started by normalizing all the standard parameters found in most cameras. The challenge came when we had to incorporate brand-specific parameters, many of which are only used by a single manufacturer. Operators also weren’t keen on having values changed from what the camera itself provided, as some settings serve as familiar reference points. For example, they know the right detail enhancement values to use for football or studio work. So, we kept normalization for the key functions where it made sense, but for other parameters, we now try to stay as close as possible to the camera’s native values. As for the topics on MQTT, they function as a kind of universal API—at least internally. Some partners and customers are already using them to automate certain functions. However, we haven’t officially released anything yet, as we wouldn’t be able to guarantee stability or prevent changes at this stage.
    - noisy_boy 2y ago
      I have noticed that you and your team's answers are detailed and insightful - much appreciated.

ram_rar 2y ago

Great to see Elixir gaining traction in mission-critical broadcast systems! I wonder, how much of Cyanview's reliability comes from Elixir specifically versus just good implementation of MQTT? and is there any specific Elixir features were essential that couldn't be replicated in other languages?

zaik 2y ago
> couldn't be replicated in other languages? All programming languages can do any task. It's about how easy they make that task for you.
- zwnow 2y ago
  Yea and with that i'd think it would be a pain in the ass trying to replicate BEAM behavior in different langs
- jdufawdfas 2y ago
  [flagged]
- thibaut_barrere 2y ago
  This is true in general but only until it gets false. For instance, Elixir supports compilation targeting GPUs (within exactly the same language, not a fork). Most languages do not allow that (and for most it would be fairly hard to implement).
- goatlover 2y ago
  For any finite, computable task, as long as the language has access to the hardware that can perform the task in practical time, assuming the language doesn't present any compilation or memory issues to take advantage of said hardware in practical time for the task to be worth computing.
ghislainle 2y ago
Main developer here. We use MQTT a lot, it is really a central piece of our architecture, but Elixir brings a lot of benefits regarding the handling of many processes which are often loosely coupled. The BEAM and OTP offer a sane approach to concurrency and Elixir is a nice language on top. Here is what I find the most important benefits: - good process isolation, even the heap is per process. This allows us to have robust and mature code running along more experimental features without the fear of everything going down. And you still have easy communication between processes - supervision tree allows easy process management. I also created a special supervisor with different restart strategies. The language allows this and then, it integrates as any other supervisor. With network connections being broken and later reconnected, the resilience of our system is tested regularly, like a physical chaos monkey - the immutability as implemented by the BEAM greatly simplifies how to write concurrent code. Inside a process, you don't need to worry about the data changing under you, no other process can change your state. So no more mutex/critical sections (or very little need). You can still have deadlock though, so it is not a silver bullet

arakutourisam 2y ago

[dead]

notepad0x90 2y ago

What is being used in similar broadcast setups outside of this Superbowl?

imjonse 2y ago
According to the article this software is used for all major sporting events.
davidbou 2y ago
Major events use it for all kind of soecialty cameras as they aready have the technology for the main studio cameras. So we had to develop solutions for everything that was not working. And major productions have budgets for all kind of new toys:mini-cams, drones, cable cams, now cinematic look from small mirrorless cameras, slow motion, etc. That opened up a whole lot of possibilities to be creative but you have to be as reliable as the main cameras and aim for the best image quality. Now the same products are used for very small productions that don't have the budget for any studio camera (look typically at 50k+ for a camera without lens). In that case we try to provide a similar user experience and functions but with much more ffordable cameras. Finaly more and more live productions are now handled using cine style cameras which don't have the standard broadcast remote panels and that's another area we cover, by combining camera control with control of many external boxes like external motors to drive manual lenses or 3D Lut video processors. Applications are on fashion shows, concerts, theater, Churches, studio shows, even corporate. In the end Elixir is used for a lot of small processes which handle very low level control protocols. And then on top of that add a high-level of communication between devices either on local networks or over the cloud.
- mcintyre1994 2y ago
  > Now the same products are used for very small productions that don't have the budget for any studio camera Just out of curiosity, what would be examples of very small productions here? Would an independent YouTube channel with great production quality be using this?
  - 2y ago

laserbeam 2y ago

Of course! Of course you have to do color correction on all the different cameras pointed from different angles at a sports event. I absolutely love reading about hard problems that are invisible to most people.

abrookewood 2y ago
Yes, it's one of those super-niche & super important functions that are obvious once you know about them, but would never think about otherwise.
- myst 2y ago
  Why is it super-important?
  - GiorgioG 2y ago
    Because events using many cameras without this type of setup would cause jarring visual differences when switching between one camera/view to another during the broadcast.
    - HeatrayEnjoyer 2y ago
      Why wasn't it jarring in early Superbowl games?
      - lgeorget 2y ago
        Fewer cameras, lower resolution, poor color rendering even with one camera anyway?
        devmor 2y ago
        And less color fidelity on receiving displays I would imagine!
      - davidbou 2y ago

ZeWaka 2y ago

> Without any marketing, it earned a reputation among seasoned professionals and became a staple at the world’s top live events. Sounds like the entertainment industry. Everyone truly knows everyone, especially when you're working on the same show with the same crew year after year. It's definitely a family of sorts.

pharrington 2y ago
"Without any marketing" is also an obvious lie, since Cyanview has a storefront website and marketing posts on Linkedin.
- caseyohara 2y ago
  Also this article seems more like an ad for Cyanview than for Elixir. Smells like content marketing to me.
  - mtndew4brkfst 2y ago
    Most of the Elixir in Production posts on the official blog (including this one) come off this way, IMO: https://elixir-lang.org/blog/categories.html#Elixir%20in%20Production https://elixir-lang.org/blog/categories.html#Elixir%20in%20P... It's pretty much just appeal to authority. "These people are successful and they used Elixir, why don't you?"
  - lawik 2y ago
    Really? I can spill all the juicy details as the main author and instigator. Cyanview reached out to me to help find a dev a while back. Hearing about their customers I knew it would be a decently big splash for Elixir. I was surprised that they were unknown and had this succcess with big household name clients. I like them. I like their whole deal. Small team, punching above their weight. Hardware, software, FPGAs and live broadcasts. The story has so much to it. David and team have been great sports in sharing their story. Fundamentally I care more about Elixir adoption though, I reached out to the Elixir team and offered to interview them and write something up. A case study about successful Elixir production deployments is definitely content marketing. But for Elixir. It is a very common question when mentioning a less common language. "Who uses this?" I thought it was a very interesting case. Glad to have it documented. The style of a case study won't suit everyone. I suppose "without any marketing, before _this_" would have been funny.

xlii 2y ago

I won’t let a friend start development in Elixir. Let me get it out: I love BEAM. OTP is awesome and one of the best systems in its kind. I was completely enamored with Elixir years ago as a modern Erlang which excited me to the bone. It’s no longer the case. When you get into non-trivial things there are many sharp edges and paper cuts. Some from the top of my head: - it’s impossible to disable warnings - test runs are often highly verbose because libraries ignore them and (as discussed somewhere in forums) warnings are deemed useful so they can’t be disable - the only way to catch some, important ones, in CI is to use "warnings-as-errors" … - so one cannot use deprecation flags because it’s also a warning - when having non trivial ecosystem one cannot selectively deprecate and get errors, this has to be a human process (remember to replace…) - when doing umbrella tests on non compiled code seed influences order of compilation - this order of compilation influences outcome and can lead to buggy code - dependency compilation is not parallelized - takes a lot of time and uses 10% of CPU - compilation process can break and Elixir isn’t aware that it was interrupted - this means that it doesn’t try to compile properly app/dependency but instead tries to release crippled one - documentation is hard to use - searching shows random versions and there isn’t even a link to „open newest” - searching in docs often not finds the keywords you can actually see - a lot of knowledge is implicit (try checking if you can dynamically add children to a Supervisor) - sidebar with new ExDocs break for some reason so there is no navigation - there is no way to browse navigation outside this broken ExDocs which outputs only HTML and LSP - LSP is afterthought, there are few but neither works well - Dialyzer/dialyxyr typespecs are useless most of the time - Squiggly arrow works weird (i.e. ~0.3 might catch 0.99) - my colleague recently mentioned Renovate not picking it up I could go on and on. I’m doing plenty of research so I’m working with various languages including „niche” ones like Prolog, OCaml, Clojure, Cuelang. Recently I’ve been developing tooling in Go and many core systems are developed in Rust in Elixir, and I work on the latter often. In principle Elixir is awesome, but has the worst developer experience of all. Sometimes it takes 4h to prepare and push release. Tooling I’m working on can do the same in 5 minutes - I’m parallelizing processes in containers, making idempotent output artifacts and heuristic failure detection to retry on flakiness. When switching between Go and OCaml you can sense how tooling cares for me and my time. Often I bounce off forums where people’s need are shrugged off as non-essential, treating those who came as uneducated juniors (because who in sane mind would like to have a parallel dependencies compilation or disable compilation time warning). There is nothing better than BEAM, but (for me) Elixir got much worse over the years.

pinoy420 2y ago

[dead]

sschueller 2y ago

Someone tracked every single camera shot during the halftime show: https://www.youtube.com/watch?v=YXNWfFtgbNI https://www.youtube.com/watch?v=YXNWfFtgbNI

jcalx 2y ago
You can also see what it looks like from the control room on Hamish Hamilton's YouTube channel, with the AD calling out shots and all: https://m.youtube.com/watch?v=gfjWjkTP4p8 https://m.youtube.com/watch?v=gfjWjkTP4p8. (Hamish Hamilton has directed every Super Bowl halftime show since 2010.)
oplav 2y ago
John DeMarsico directs the SNY broadcasts for the NY Mets and sometimes posts behind the scenes for how all the cameras come together into a production. I think they are pretty interesting to watch. https://x.com/SNYtv/status/1832250958258036871 https://x.com/SNYtv/status/1832250958258036871

deleted 2y ago

[deleted]

innocentoldguy 2y ago

I have implemented Elixir in critical financial applications, B2B growth intelligence applications, fraud detection applications, scan-and-go shopping applications, and several others. In every case, like the engineering team in this article demonstrates, the developer experience and end results have exceeded expectations. If you haven’t used Elixir, you should give it a try. Edit: Fixed an editing error.

roughly 2y ago
Elixir and Erlang have always garnered a lot of respect and praise - I’m always curious why they’re not more widely used (I’m no exception - despite hearing great things for literal decades, I’ve never actually picked it up to try for a project).
- AlchemistCamp 2y ago
  I think most people tend to stick with what they learn first or hop to very similar languages. Schools generally taught Java and then more recently Python and JS, all of which are relatively similar. Unless someone who knows those three languages is curious or encounters a particular problem that motivates them to explore, they're unlikely to pick up an immutable, functional language.
  - innocentoldguy 2y ago
    I think you’re right. I only picked up Elixir about 10 years ago after getting frustrated with Python’s GIL and Java’s cumbersomeness, and feeling that object oriented programming over complicates things and never lived up to its hype. I have never looked back. Elixir is an absolute joy to use. It simplifies multi-threaded programming, pattern-matching makes code easier to understand and maintain, and it is magnitudes faster to code in than Java. For me, Elixir’s version of functional programming provides the ease of development that OOP promised and failed to deliver. In my opinion, Elixir is software engineering’s best kept secret.
- solid_fuel 2y ago
  I've thought about this a lot, and I think that part of what hurts Erlang/Elixir adoption is the scale of the OTP. It brings a ton of fantastic tools, like supervision trees, process linking, ETS, application environments & config management, releases, and more. In some ways it's closer to adopting a new OS than a new programming language. That's what I love about Elixir, but it means that selling it is more like convincing a developer who knows and uses CSV to switch to Postgres. There's a ton of advantages to storing data in a relational DB instead of flat files, but now you have to define a schema up front, deal with table and row locking, figure out that VACUUM thing, etc. When you're just setting out to learn a new language, trying to understand a new OS on top hurts adoption.

DerekL 2y ago

The title is misspelled, should be “Super Bowl”, two words.

frankfrank13 2y ago

Really cool piece, this jumped out to me: > The devices in a given location communicate and coordinate on the network over a custom MQTT protocol. Over a hundred cameras without issue on a single Remote Control Panel (RCP), implemented on top of Elixir’s network stack. Makes sense! MQTT is, if I understand right, built on TCP. Idk if I would have found the same solution, but its seemingly a good one

eggy 2y ago

Would Gleam be practical for a similar application aside from the OTP/BEAM runtime? I am guessing you'd have to leverage Elixir libraries that are not present for Gleam yet, and you might have slower compile times due to static typing, but you'd catch runtime errors sooner. Would it be more of a debugging vs. fast dynamic iteration trade-off? I am looking to settle on either Gleam or Elixir. I liked Gleam's original ML syntax before, but I like static typing. Thoughts? I am replacing C with Zig, and I am brushing up on my assembly by adding ARM to my x64 skill set.

widdershins 2y ago
Gleam has a subset of OTP functionality already [1]. It also compiles extremely quickly. I haven't made any huge projects yet, but I've used some fairly chunky libraries and everything compiles super quick. [1] https://github.com/gleam-lang/otp https://github.com/gleam-lang/otp
- nesarkvechnep 2y ago
  It’s subpar at the moment.
AlchemistCamp 2y ago
> you'd catch runtime errors sooner I don’t think there’s any evidence whatsoever that you would catch runtime bugs sooner with Gleam than with Elixir (or Erlang). Erlang’s record for reliability is stronger than many statically typed languages, including even Java. There is a certain class of errors static types can prevent but there’s a much larger set of those it can’t. To make the case for a language like TS/Java/Swift/Golang or Gleam actually resulting in fewer runtime defects than Erlang or Elixir, I’d want to see some real world data.
- eggy 2y ago
  It depends on what “sooner” means to you. Gleam catches more before the code runs; Elixir catches them when they happen but recovers gracefully. If you’re paranoid about bugs reaching users, I would think Gleam’s your pick, no? If you trust your tests and love dynamic freedom, Elixir should be fine. I don't have much experience with either language. I did more in Erlang 8 years ago, but not much. I am on the edge of choosing Gleam over Elixir. It's mainly subjective: I prefer the syntax in Gleam, although I liked the original ML-like syntax when it first came out.

markb139 2y ago

30 odd years ago, part of my role was to colour balance cameras in a studio environment. We didn’t need computers - but at most there were only 5 cameras :)

RKFADU_UOFCCLEL 2y ago

[dead]

brcmthrowaway 2y ago

Wait, is Elixir actually accessing color pixel data in realtime?

davidbou 2y ago
No, it deals with metadata: control and status as explained in a previous reply https://news.ycombinator.com/item?id=43479094#43482362 https://news.ycombinator.com/item?id=43479094#43482362 Elixir does some computations as well but when we had to compute 3D luts based on video processing algorithms, Ghislain had to write them in C to be fast enough for our needs on embedded hardware.

hackburg 2y ago

[dead]