12 ms·

I started my career, and have spent most of it, working in places where the production network was (almost) airgapped from the entire internet (MPAA accredited

I started my career, and have spent most of it, working in places where the production network was (almost) airgapped from the entire internet (MPAA accredited facilities). I would say that the general quality of software and and robustness when it comes to dependencies is so much greater in these places. If you want to use some library, it’s up to you to get it, and its dependencies, check the versions are compatible and package it up and build it all internally. Yup, it’s work... Do you need this library? Is it actually any good? Is the license compatible with our usage? This is all basic code quality stuff that’s often completely overlooked when people can just pull in whatever junk from whatever trendy repo is the hotness nowadays. And then when that goes down/bankrupt - it’s up to you to fix something you’ve no control over.

jrsj 6y ago
This is probably better from a quality perspective and also probably not worth the time it would take in 90% of projects. That being said I'm amazed how much production software depends on multiple libraries that are developed and maintained by a single person as a hobby.
- madeofpalk 6y ago
  Yeah like as a frontend/web developer, most of what we do is make what are essentially Wordpress themes for some company that really doesn’t matter or do much important.
- ClikeX 6y ago
  That one person (sometimes) puts in more effort into that single library then I've seen some agencies put into a client project.
  - wwweston 6y ago
    Intrinsic vs extrinsic motivation. It's pretty common for someone building a library to do so for the utility, scratching a real itch. Not universal (people build libraries to be someone who built a library, too), but common. It's very common for agencies to be building client projects in order to bill for it. Sometimes there's a special alignment where the client is also primarily interested in spending an allotted budget so long as there's a plausibly adequate deliverable.
  - typon 6y ago
    It's hard for me to think of open source libraries that are developed and maintained by mega corporations that I actually prefer to use over libraries made by small indie developers. The only one that I can think of is pytorch, but thats kind of unfair since it was acquired by Facebook not developed from scratch by them.
    - tracker1 6y ago
      For me, the biggest is React... aside from that, not much really.
    - srtjstjsj 6y ago
      Golang is pretty well liked. And Guava in Java land, and Abseil in C++. It's not too surprising that the stuff big companies make for their needs isn't as useful for small indie devs as stuff small indie devs make.
- alibarber 6y ago
  Sure, everything's a tradeoff - but sometimes I see things like that as frontloading the pain of when bndlrrr.io or whatever goes down in the middle of the night and your client is angry at you. But yeah, like everything in software, it's a spectrum and 'best practice' and the 'right way' are highly dependent on the context.
- benhurmarcel 6y ago
  https://xkcd.com/2347/ https://xkcd.com/2347/
bob1029 6y ago
We provide software that runs within very secure financial networks, and have some extreme constraints regarding what sorts of 3rd party code we can pull in. We are having to do a vendor swap on some document parsing code because one of our clients scanned a dependency and found it could be vulnerable in a context that it would never be exposed to in our application. These types of things make it really risky to go out and build your enterprise on top of someone else's idea of a good time. Virtually everything we do is in-house on top of first party platform libraries - i.e. `System.`, `Microsoft.`, etc. We exclusively use SQLite for persistence in order to reduce attack surface. Our deliverables to our customers consist of a single binary package that is signed and traceable all the way through our devops tool chain, which is also developed in-house for this express purpose of tightly enforcing software release processes. This approach is certainly slower than vendoring out everything to the 7 winds, but there are many other advantages. Every developer knows how everything works up and down the entire vertical since its all sitting inside one happy solution just an F12 away. Being able to see a true, enterprise-wide reference count above a particular property or method is like a drug to me at this point. We are definitely over the hill and reaping dividends for building our own stack. It did take 3-4 years though. Most organizations cannot afford to do what we did.
- postpawl 6y ago
  Couldn’t this end up being disaster with a large codebase and even a small amount of turnover? Are you really advocating that someone should write all their dependencies themselves even if they can afford it?
  - alibarber 6y ago
    It depends on the context. Yeah it'd be lovely to have your system connected to the internet - but no, our clients wouldn't give us money if you do that, and we need money. So, maybe yes in this case, start writing some quality, well thought out maintainable libraries (that can include audited third party code), and just bill it. In my case, the cost to the client of a team of devs working on that was less than the cost to them of the risk of a film leaking... [Edit] - But what I have found through experience is that the code that was written under these constraints seemed to be better, more secure and robust, than without them. YMMV.
ex_amazon_sde 6y ago
Ex-Amazon SDE here. The same happens in FAANGS: tons of software is written and served internally. It's not NIH syndrome, usually. It's about having control over the whole software supply chain for security, reliability, licensing compliance and general quality.
- Twirrim 6y ago
  As another ex-Amazon... there's a whole bunch of NIH going on there too. The way I saw it, it tended to be split three ways: 1) NIH. Almost always the problems that need solved are interesting, and engineers are naturally chomping at the bit to solve them. Added bonus you can potentially make a name for yourself. This happens way more than it should, in cases that don't meet the other two ways I saw. Solving problems that have already been solved very effectively and efficiently, in a mature low friction fashion. 2) It doesn't scale to needs. A lot of software just doesn't scale to the requirements of the platform. It's hard to understate just how much traffic and work a lot of Amazon infrastructure has to handle. Most software doesn't scale that well because it's not run in so big an environment. We're using some well known commercial software at my current employers (because it works, has a good reputation, and did everything we need), that is experiencing major scaling issues because we're literally orders of magnitude larger than any of their other customers. We're seeing stuff they've never had to deal with before. We're not even close to Amazon's scale for this particular type of software. 3) Need to control the entire software stack, have the ability to drastically modify it to meet the changing demands placed on it. A lot of public software is written to meet one need, and it rarely changes that drastically over time. That's not what the consumers want, even though needs change over time. Change your software too much and you'll lose your existing users that fundamentally need what the software is providing. You can see the boom and fall of it all with so many projects. Take a look at what's happened with Chef and Puppet, for a quick off-the-top-of-my-head example.
  - srtjstjsj 6y ago
    4) the most important reason: open source stuff isn't built to integrate with the decades old custom stack cruft
  - 6y ago
echelon 6y ago
Why is MPAA air-gapped? Are they worried about leaks or accidentally using copyrighted material? Seems arbitrary, but I bet the rationale is fascinating. Could you go into more detail? I'd love hearing a little about your work experience, the industry, process, etc.
- boulos 6y ago
  Disclosure: I work on Google Cloud. Your intuition is right: they're afraid of the content leaking. For cloud providers, this results in ... amusingly long documents. Here's GCP's at 110 pages [1], while the AWS folks were clever and used landscape mode for theirs [2] so that it's only 59 pages :). [1] https://cloud.google.com/files/gcp-mpaa-compliancemapping.pdf https://cloud.google.com/files/gcp-mpaa-compliancemapping.pd... [2] https://d1.awsstatic.com/whitepapers/compliance/AWS_Alignment_with_Motion_Picture_of_America_Association.pdf https://d1.awsstatic.com/whitepapers/compliance/AWS_Alignmen...
  - srtjstjsj 6y ago
    Good to know that Spider-Man movies are better protected than my personal health information.
  - echelon 6y ago
    That's fascinating! These companies protect their assets as if release were an existential risk. I suppose it would impact their bottom line to some extent. It'll be interesting to see in the future when content is cheap to produce. I predict a complete shift away from this.
    - redis_mlc 6y ago
      > These companies protect their assets as if release were an existential risk. I guess you're not familiar with the MPAA. They're perenially headed by one of the US' top lawyers/lobbyists, at a salary of millions per year. They're the group behind the continual assault on the "analog hole", the promoters of "must use Silverlight for DRM", etc. https://en.wikipedia.org/wiki/Motion_Picture_Association https://en.wikipedia.org/wiki/Motion_Picture_Association
rapind 6y ago
Are you saying that when you need to add a space character to the beginning of your strings you write it yourself?!
- alibarber 6y ago
  Yes, but after getting approval from the board I was able to get a dispensation to use https://isevenapi.xyz/ https://isevenapi.xyz/ for some of our calculation services.
cs02rm0 6y ago
I've spent much of my career on similarly, perhaps a little more, restricted networks and found the opposite. Dependencies are painful to pull in and only pulled in when a dev needs a specific version. The internal repos end up being a missing version nightmare where people cobble together whatever works with what's available. Where feature and security upgrades go ignored, left to rot like the brains of the devs who struggle to keep up with what's available in the real world. Many of those networks I've worked on are becoming more permeable at the edges because the cost of the air gap outweighs any benefits.
tyldum 6y ago
Even Cisco's web-based firewall management interface uses Google Analytics. Granted, it will work just as bad regardless of reachability, so there's that.
dvdkon 6y ago
I have to ask, why does the MPAA accredit airgapped facilities? I know movies are a big business, but that seems a little extreme.
- mikeryan 6y ago
  It’s not airgapping per se. The large studios want to ensure they can send out their content pre-release to the various third-party vendors working on a project. Ensuring the vendors meet MPAA guidelines is a mechanism they use to ensure this. It’s not technically an accreditation but it’s usually contractually enforced if you want to work with a large studio. You can actually read the whole thing: https://www.motionpictures.org/wp-content/uploads/2020/07/MPA-Best-Practices-Common-Guidelines-V4.07-Final.pdf https://www.motionpictures.org/wp-content/uploads/2020/07/MP...
- srtjstjsj 6y ago
  Lots of people like to get copies of IP worth many millions of dollars.
lmm 6y ago
I'd say just the opposite. I worked on a 10-million LoC codebase where third-party libraries had to be individually brought in by an internal owner. I suspect most of those 10 million lines of code ended up being ad hoc, informally-specified, bug-ridden, slow implementations of half of some external library.
dikei 6y ago
Even when you package your dependencies with your code, unless you have policies in place and actively enforcing it, there's no preventing people from bloating the project with all kind of junks libraries. I've had to deal with legacy projects that include multiple versions of the same libraries, all of them being shaded/relocated so they don't conflict. The result, however, is bloated binaries that takes 30 minutes to build.
physicsguy 6y ago
True to some degree, but you can also end up with very old dependencies which have vulnerabilities if someone is not keeping track of it all.
baud147258 6y ago
Where I'm working, all the dependencies have to be approved before use and are stored locally (build machines have no internet access), but I haven't found that the quality of the software is better than the other places where I've worked. I think it's more a matter of work 'culture' (not sure if that's the right word, perhaps expectations would be a better fit?), with the idea that we ship once it's good enough and that's it.
- baud147258 6y ago
  Though I think it's still a good thing that we're doing it, but it's not enough on its own to get good quality.

jrsj 6y ago

This is probably better from a quality perspective and also probably not worth the time it would take in 90% of projects. That being said I'm amazed how much production software depends on multiple libraries that are developed and maintained by a single person as a hobby.

madeofpalk 6y ago
Yeah like as a frontend/web developer, most of what we do is make what are essentially Wordpress themes for some company that really doesn’t matter or do much important.
ClikeX 6y ago
That one person (sometimes) puts in more effort into that single library then I've seen some agencies put into a client project.
- wwweston 6y ago
  Intrinsic vs extrinsic motivation. It's pretty common for someone building a library to do so for the utility, scratching a real itch. Not universal (people build libraries to be someone who built a library, too), but common. It's very common for agencies to be building client projects in order to bill for it. Sometimes there's a special alignment where the client is also primarily interested in spending an allotted budget so long as there's a plausibly adequate deliverable.
- typon 6y ago
  It's hard for me to think of open source libraries that are developed and maintained by mega corporations that I actually prefer to use over libraries made by small indie developers. The only one that I can think of is pytorch, but thats kind of unfair since it was acquired by Facebook not developed from scratch by them.
  - tracker1 6y ago
    For me, the biggest is React... aside from that, not much really.
  - srtjstjsj 6y ago
    Golang is pretty well liked. And Guava in Java land, and Abseil in C++. It's not too surprising that the stuff big companies make for their needs isn't as useful for small indie devs as stuff small indie devs make.
alibarber 6y ago
Sure, everything's a tradeoff - but sometimes I see things like that as frontloading the pain of when bndlrrr.io or whatever goes down in the middle of the night and your client is angry at you. But yeah, like everything in software, it's a spectrum and 'best practice' and the 'right way' are highly dependent on the context.
benhurmarcel 6y ago
https://xkcd.com/2347/ https://xkcd.com/2347/

bob1029 6y ago

We provide software that runs within very secure financial networks, and have some extreme constraints regarding what sorts of 3rd party code we can pull in. We are having to do a vendor swap on some document parsing code because one of our clients scanned a dependency and found it could be vulnerable in a context that it would never be exposed to in our application. These types of things make it really risky to go out and build your enterprise on top of someone else's idea of a good time. Virtually everything we do is in-house on top of first party platform libraries - i.e. `System.`, `Microsoft.`, etc. We exclusively use SQLite for persistence in order to reduce attack surface. Our deliverables to our customers consist of a single binary package that is signed and traceable all the way through our devops tool chain, which is also developed in-house for this express purpose of tightly enforcing software release processes. This approach is certainly slower than vendoring out everything to the 7 winds, but there are many other advantages. Every developer knows how everything works up and down the entire vertical since its all sitting inside one happy solution just an F12 away. Being able to see a true, enterprise-wide reference count above a particular property or method is like a drug to me at this point. We are definitely over the hill and reaping dividends for building our own stack. It did take 3-4 years though. Most organizations cannot afford to do what we did.

postpawl 6y ago
Couldn’t this end up being disaster with a large codebase and even a small amount of turnover? Are you really advocating that someone should write all their dependencies themselves even if they can afford it?
- alibarber 6y ago
  It depends on the context. Yeah it'd be lovely to have your system connected to the internet - but no, our clients wouldn't give us money if you do that, and we need money. So, maybe yes in this case, start writing some quality, well thought out maintainable libraries (that can include audited third party code), and just bill it. In my case, the cost to the client of a team of devs working on that was less than the cost to them of the risk of a film leaking... [Edit] - But what I have found through experience is that the code that was written under these constraints seemed to be better, more secure and robust, than without them. YMMV.

ex_amazon_sde 6y ago

Ex-Amazon SDE here. The same happens in FAANGS: tons of software is written and served internally. It's not NIH syndrome, usually. It's about having control over the whole software supply chain for security, reliability, licensing compliance and general quality.

Twirrim 6y ago
As another ex-Amazon... there's a whole bunch of NIH going on there too. The way I saw it, it tended to be split three ways: 1) NIH. Almost always the problems that need solved are interesting, and engineers are naturally chomping at the bit to solve them. Added bonus you can potentially make a name for yourself. This happens way more than it should, in cases that don't meet the other two ways I saw. Solving problems that have already been solved very effectively and efficiently, in a mature low friction fashion. 2) It doesn't scale to needs. A lot of software just doesn't scale to the requirements of the platform. It's hard to understate just how much traffic and work a lot of Amazon infrastructure has to handle. Most software doesn't scale that well because it's not run in so big an environment. We're using some well known commercial software at my current employers (because it works, has a good reputation, and did everything we need), that is experiencing major scaling issues because we're literally orders of magnitude larger than any of their other customers. We're seeing stuff they've never had to deal with before. We're not even close to Amazon's scale for this particular type of software. 3) Need to control the entire software stack, have the ability to drastically modify it to meet the changing demands placed on it. A lot of public software is written to meet one need, and it rarely changes that drastically over time. That's not what the consumers want, even though needs change over time. Change your software too much and you'll lose your existing users that fundamentally need what the software is providing. You can see the boom and fall of it all with so many projects. Take a look at what's happened with Chef and Puppet, for a quick off-the-top-of-my-head example.
- srtjstjsj 6y ago
  4) the most important reason: open source stuff isn't built to integrate with the decades old custom stack cruft
- 6y ago

echelon 6y ago

Why is MPAA air-gapped? Are they worried about leaks or accidentally using copyrighted material? Seems arbitrary, but I bet the rationale is fascinating. Could you go into more detail? I'd love hearing a little about your work experience, the industry, process, etc.

boulos 6y ago
Disclosure: I work on Google Cloud. Your intuition is right: they're afraid of the content leaking. For cloud providers, this results in ... amusingly long documents. Here's GCP's at 110 pages [1], while the AWS folks were clever and used landscape mode for theirs [2] so that it's only 59 pages :). [1] https://cloud.google.com/files/gcp-mpaa-compliancemapping.pdf https://cloud.google.com/files/gcp-mpaa-compliancemapping.pd... [2] https://d1.awsstatic.com/whitepapers/compliance/AWS_Alignment_with_Motion_Picture_of_America_Association.pdf https://d1.awsstatic.com/whitepapers/compliance/AWS_Alignmen...
- srtjstjsj 6y ago
  Good to know that Spider-Man movies are better protected than my personal health information.
- echelon 6y ago
  That's fascinating! These companies protect their assets as if release were an existential risk. I suppose it would impact their bottom line to some extent. It'll be interesting to see in the future when content is cheap to produce. I predict a complete shift away from this.
  - redis_mlc 6y ago
    > These companies protect their assets as if release were an existential risk. I guess you're not familiar with the MPAA. They're perenially headed by one of the US' top lawyers/lobbyists, at a salary of millions per year. They're the group behind the continual assault on the "analog hole", the promoters of "must use Silverlight for DRM", etc. https://en.wikipedia.org/wiki/Motion_Picture_Association https://en.wikipedia.org/wiki/Motion_Picture_Association

rapind 6y ago

Are you saying that when you need to add a space character to the beginning of your strings you write it yourself?!

alibarber 6y ago
Yes, but after getting approval from the board I was able to get a dispensation to use https://isevenapi.xyz/ https://isevenapi.xyz/ for some of our calculation services.

cs02rm0 6y ago

I've spent much of my career on similarly, perhaps a little more, restricted networks and found the opposite. Dependencies are painful to pull in and only pulled in when a dev needs a specific version. The internal repos end up being a missing version nightmare where people cobble together whatever works with what's available. Where feature and security upgrades go ignored, left to rot like the brains of the devs who struggle to keep up with what's available in the real world. Many of those networks I've worked on are becoming more permeable at the edges because the cost of the air gap outweighs any benefits.

tyldum 6y ago

Even Cisco's web-based firewall management interface uses Google Analytics. Granted, it will work just as bad regardless of reachability, so there's that.

dvdkon 6y ago

I have to ask, why does the MPAA accredit airgapped facilities? I know movies are a big business, but that seems a little extreme.

mikeryan 6y ago
It’s not airgapping per se. The large studios want to ensure they can send out their content pre-release to the various third-party vendors working on a project. Ensuring the vendors meet MPAA guidelines is a mechanism they use to ensure this. It’s not technically an accreditation but it’s usually contractually enforced if you want to work with a large studio. You can actually read the whole thing: https://www.motionpictures.org/wp-content/uploads/2020/07/MPA-Best-Practices-Common-Guidelines-V4.07-Final.pdf https://www.motionpictures.org/wp-content/uploads/2020/07/MP...
srtjstjsj 6y ago
Lots of people like to get copies of IP worth many millions of dollars.

lmm 6y ago

I'd say just the opposite. I worked on a 10-million LoC codebase where third-party libraries had to be individually brought in by an internal owner. I suspect most of those 10 million lines of code ended up being ad hoc, informally-specified, bug-ridden, slow implementations of half of some external library.

dikei 6y ago

Even when you package your dependencies with your code, unless you have policies in place and actively enforcing it, there's no preventing people from bloating the project with all kind of junks libraries. I've had to deal with legacy projects that include multiple versions of the same libraries, all of them being shaded/relocated so they don't conflict. The result, however, is bloated binaries that takes 30 minutes to build.

physicsguy 6y ago

True to some degree, but you can also end up with very old dependencies which have vulnerabilities if someone is not keeping track of it all.

baud147258 6y ago

Where I'm working, all the dependencies have to be approved before use and are stored locally (build machines have no internet access), but I haven't found that the quality of the software is better than the other places where I've worked. I think it's more a matter of work 'culture' (not sure if that's the right word, perhaps expectations would be a better fit?), with the idea that we ship once it's good enough and that's it.

baud147258 6y ago
Though I think it's still a good thing that we're doing it, but it's not enough on its own to get good quality.