6 ms·

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rat

by slt2021 2y ago

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promotion and salary/rsu bump, ofc. People need to realize that code is a liability, anything that is not the business critical stuff that earns/makes $$$ for the company is a distraction and resource sink.

ripped_britches 2y ago
100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.
jefurii 2y ago
This sounds like the beginning of a sales pitch.
makeset 2y ago
> anything that is not the business critical stuff That's an important qualifier. For skilled teams in performance-critical domains, the inflection point where any outside code becomes a low-quality/low-control liability is not that far.
YawningAngel 2y ago
Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff
- slt2021 2y ago
  airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively. And they develop them in a way that works for many customers and use cases, not just netflix. But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into custom cron?
  - hulahoof 2y ago
    But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.
    - artwr 2y ago
      Agreed. To be fair, I doubt Maestro will take off like Airflow did. Airflow filled a void of an easier orchestrator for Big Data with a prettier UI than the competitors of the time (Oozie, Luigi), implementing some UX patterns which had been tested at scale at Facebook with data swarm. The field is quite a bit more crowded now.
      - turtle4 2y ago
        Seems like you have some experience with the orchestrator offerings. Airflow still the way to go, or would you recommend something else for someone just starting down the path of selecting and implementing a data orchestrator?
        reinhardt 2y ago
        I haven't used Airflow for years but it used to be quite clunky, not sure how much it's improved since. I'd look into Prefect and/or Dagster first, both are more modern alternatives built with Airflow's shortcomings in mind.
bluepizza 2y ago
> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Code is not a liability. Code is what makes a company move its gears.
- delecti 2y ago
  Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.
  - bilalq 2y ago
    But thinking of those trucks primarily as a liability is exactly the kind of mindset that leads to companies minimizing their liabilities instead of maximizing their potential.
    - shermantanktop 2y ago
      Especially when the cost of minimizing (long hours, unsafe conditions) is not felt by decision makers, and may not materialize for a while, but the benefits of maximizing their potential is felt directly and immediately. Incentives are everything. That's why managers are so careful when applying them to their own jobs.
  - slt2021 2y ago
    For trucking company owning and developing trucks makes sense. But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business
alfalfasprout 2y ago
I very much disagree with this take-- and the more I've experienced throughout my career the more I'm sure of it. Companies spend an IMMENSE amount of time and effort adapting sometimes subpar off the shelf solutions to fit their infra and pay an ongoing tax w/ increasing tech debt trying to support them. Often something bespoke and smaller + more tailored would unlock significantly more productivity if the investment is made consciously. Any code that is written has both assets and liabilities. But to claim it is a distraction and resource sink is a very, very bad take. Every decision to build something in-house needs to be done thoughtfully and deliberately.
bhawks 2y ago
> with long history of maintenance and improvement, That is a huge load bearing statement. Do you plan on any contributions back to the community yourself? Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.
- deleted 2y ago
  [deleted]
- slt2021 2y ago
  If I needed container orchestration I would use k8s. I can improve it and even propose patches/bugs or chip into opensource maintainers fund. I wont write my own orchestrator, especialy being in a streaming business. Thats what I meant, doesnt even necessarily Build-vs-Buy, but rather Use-Open-Source-and-Contribute or Reinvent-the-wheel-for-L6-promo-and-then-opensource ?? Would the world be better with 10 workflow orchestrator systems or one mature?
  - bhawks 2y ago
    Netflix is building a workflow orchestrator not a container orchestrator. The viable alternative would be Airflow or maybe something like Temporal. K8s alone isn't going to meet the need in this case. Does the world need another workflow orchestrator? Who knows - some folks at Netflix seem willing to pay a handful of engineers $ to do so. Good luck to them
why-el 2y ago
I am confused by this comment: > open source stuff with long history of maintenance and improvement improvement and maintenance is continent on usage, and having been used at Netflix, this project is in a better place to have already faced whatever bug you are worried about (and let's be real, 99% of applications wont ever get the luck to exercise code paths sophisticated enough to find bugs Netflix has not found already). You might be unnecessarily projecting here. You don't have evidence to support that open sourcing this might have been for any other reason than it is simply good for the community to have.
cortesoft 2y ago
Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source tools supposed to be created in the first place? This is the process, someone has to do it
- rjh29 2y ago
  Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
  - darkwater 2y ago
    Absolutely this. It literally happened to me with a Netflix OSS we were using at work. I found a bug that was biting us, opened a ticket with a PR attached with a possible fix and got an answer after a few months "ah yeah we fixed this in our internal version time ago, thanks, will merge it now".
  - gregoriol 2y ago
    Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and this company doesn't have any reason to review and fix code that is not part of their business.
    - mhitza 2y ago
      > Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.
ldjkfkdsjnv 2y ago
I was going to say this. I never mess with random libraries like this, always so much pain.
beanjuiceII 2y ago
Off the shelf come with its own set of burdens, not always sunshine rainbows and loli's
wodenokoto 2y ago
Aren’t most of those things developed in house at tech giants and later open sourced?
archerx 2y ago
This is a naive view, other people’s code is even more of a liability. Look at crowdstrike and opensource infiltrations. Using opensource software doesn’t magically grant you security nor stability.
jcgrillo 2y ago
> People need to realize that code is a liability Code that you own and intimately understand is less of a liability than some 3rd party dependency (paid or free). Stitching together a patchwork of dependencies is not likely the optimal result. The more aligned your codebase is with the problem you're trying to solve the better, and if functionality is core to your business better to own than borrow or rent.
MetaWhirledPeas 2y ago
> code is a liability 3rd parties are also a liability. Pick your poison. Trust in unknown individuals, trust in megacorps, or trust your own people. Choosing wisely is why people get paid the big bucks.
paxys 2y ago
> I would rather use off-the-shelf open source stuff with long history of maintenance and improvement Where do you think this "off-the-shelf open source stuff" comes from exactly?

ripped_britches 2y ago

100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.

jefurii 2y ago

This sounds like the beginning of a sales pitch.

makeset 2y ago

> anything that is not the business critical stuff That's an important qualifier. For skilled teams in performance-critical domains, the inflection point where any outside code becomes a low-quality/low-control liability is not that far.

YawningAngel 2y ago

Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff

slt2021 2y ago
airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively. And they develop them in a way that works for many customers and use cases, not just netflix. But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into custom cron?
- hulahoof 2y ago
  But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.
  - artwr 2y ago
    Agreed. To be fair, I doubt Maestro will take off like Airflow did. Airflow filled a void of an easier orchestrator for Big Data with a prettier UI than the competitors of the time (Oozie, Luigi), implementing some UX patterns which had been tested at scale at Facebook with data swarm. The field is quite a bit more crowded now.
    - turtle4 2y ago
      Seems like you have some experience with the orchestrator offerings. Airflow still the way to go, or would you recommend something else for someone just starting down the path of selecting and implementing a data orchestrator?
      - reinhardt 2y ago
        I haven't used Airflow for years but it used to be quite clunky, not sure how much it's improved since. I'd look into Prefect and/or Dagster first, both are more modern alternatives built with Airflow's shortcomings in mind.

bluepizza 2y ago

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Code is not a liability. Code is what makes a company move its gears.

delecti 2y ago
Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.
- bilalq 2y ago
  But thinking of those trucks primarily as a liability is exactly the kind of mindset that leads to companies minimizing their liabilities instead of maximizing their potential.
  - shermantanktop 2y ago
    Especially when the cost of minimizing (long hours, unsafe conditions) is not felt by decision makers, and may not materialize for a while, but the benefits of maximizing their potential is felt directly and immediately. Incentives are everything. That's why managers are so careful when applying them to their own jobs.
- slt2021 2y ago
  For trucking company owning and developing trucks makes sense. But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business

alfalfasprout 2y ago

I very much disagree with this take-- and the more I've experienced throughout my career the more I'm sure of it. Companies spend an IMMENSE amount of time and effort adapting sometimes subpar off the shelf solutions to fit their infra and pay an ongoing tax w/ increasing tech debt trying to support them. Often something bespoke and smaller + more tailored would unlock significantly more productivity if the investment is made consciously. Any code that is written has both assets and liabilities. But to claim it is a distraction and resource sink is a very, very bad take. Every decision to build something in-house needs to be done thoughtfully and deliberately.

bhawks 2y ago

> with long history of maintenance and improvement, That is a huge load bearing statement. Do you plan on any contributions back to the community yourself? Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.

deleted 2y ago
[deleted]
slt2021 2y ago
If I needed container orchestration I would use k8s. I can improve it and even propose patches/bugs or chip into opensource maintainers fund. I wont write my own orchestrator, especialy being in a streaming business. Thats what I meant, doesnt even necessarily Build-vs-Buy, but rather Use-Open-Source-and-Contribute or Reinvent-the-wheel-for-L6-promo-and-then-opensource ?? Would the world be better with 10 workflow orchestrator systems or one mature?
- bhawks 2y ago
  Netflix is building a workflow orchestrator not a container orchestrator. The viable alternative would be Airflow or maybe something like Temporal. K8s alone isn't going to meet the need in this case. Does the world need another workflow orchestrator? Who knows - some folks at Netflix seem willing to pay a handful of engineers $ to do so. Good luck to them

why-el 2y ago

I am confused by this comment: > open source stuff with long history of maintenance and improvement improvement and maintenance is continent on usage, and having been used at Netflix, this project is in a better place to have already faced whatever bug you are worried about (and let's be real, 99% of applications wont ever get the luck to exercise code paths sophisticated enough to find bugs Netflix has not found already). You might be unnecessarily projecting here. You don't have evidence to support that open sourcing this might have been for any other reason than it is simply good for the community to have.

cortesoft 2y ago

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source tools supposed to be created in the first place? This is the process, someone has to do it

rjh29 2y ago
Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
- darkwater 2y ago
  Absolutely this. It literally happened to me with a Netflix OSS we were using at work. I found a bug that was biting us, opened a ticket with a PR attached with a possible fix and got an answer after a few months "ah yeah we fixed this in our internal version time ago, thanks, will merge it now".
- gregoriol 2y ago
  Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and this company doesn't have any reason to review and fix code that is not part of their business.
  - mhitza 2y ago
    > Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.

ldjkfkdsjnv 2y ago

I was going to say this. I never mess with random libraries like this, always so much pain.

beanjuiceII 2y ago

Off the shelf come with its own set of burdens, not always sunshine rainbows and loli's

wodenokoto 2y ago

Aren’t most of those things developed in house at tech giants and later open sourced?

archerx 2y ago

This is a naive view, other people’s code is even more of a liability. Look at crowdstrike and opensource infiltrations. Using opensource software doesn’t magically grant you security nor stability.

jcgrillo 2y ago

> People need to realize that code is a liability Code that you own and intimately understand is less of a liability than some 3rd party dependency (paid or free). Stitching together a patchwork of dependencies is not likely the optimal result. The more aligned your codebase is with the problem you're trying to solve the better, and if functionality is core to your business better to own than borrow or rent.

MetaWhirledPeas 2y ago

> code is a liability 3rd parties are also a liability. Pick your poison. Trust in unknown individuals, trust in megacorps, or trust your own people. Choosing wisely is why people get paid the big bucks.

paxys 2y ago

> I would rather use off-the-shelf open source stuff with long history of maintenance and improvement Where do you think this "off-the-shelf open source stuff" comes from exactly?