12 ms·

Tools you’d miss if you left a company

170 points by kogir 7y ago

i386 7y ago
If you’re not building complicated new systems and just gluing things together, you’re not doing engineering? 1) definitely gate keeping 2) good engineering is about what not to build as much as it is what to build.
- nemothekid 7y ago
  I agree - shouldn't you feel most proud of the products you've built that help actual people who pay you, rather than your coworkers for which those internal tools wouldn't need to exist if not for your customers. I'd rather have engineers working on products that customers would miss.
  - adamsea 7y ago
    Everyone is motivated by different things. It makes sense that some people are motivated by the craft of software development, which overlaps with, but is not the same as, product development or running a successful business.
- vector_spaces 7y ago
  Yes, exactly. The view that such work just constitutes writing "glue" is super reductive. So much time is also spent empathizing with others: talking to stakeholders, understanding existing processes, determining requirements, and simplifying. So if we can reduce the solution to glue instead of needing to write a complicated system, that's hands down a win.
  - i386 7y ago
    Any engineer who comes up with a solution that requires less new code being written is a more valuable engineer IMO. Less code is less time spent on building then maintaining code for all eternity. If you go through the build/buy/partner decision, and decide to buy or partner with development comprising of glue code - then that's probably the optimal solution for your org. And you made the right engineering choice.
    - patrec 7y ago
      > Any engineer who comes up with a solution that requires less new code being written is a more valuable engineer IMO Of course less code is good, all things being equal. But taking that idea literally, basically no tooling code gets ever written and people develop some sort of learned helplessness around dysfunctional workflows that are just about feasible but fantastically wasteful of engineering time. This tends to happen a lot, also because tooling is often perceived by management as an unnecessary luxury.
      - i386 7y ago
        Agree, In my own experience it’s only orgs with a min 50 to 100 developers who can afford to have a single developer (it was me!) out working on productivity tools.
  - patrec 7y ago
    Just because you're not doing much (any?) interesting engineering doesn't mean you're not creating a lot of value. But it probably does mean that if you really deeply care about engineering in itself you should reconsider your employment options. But learning when you can just glue and how is a super-valuable skill, even if some of your work involves pushing the envelope. The worst engineers are not those who just glue stuff together, it's the people who cv-drivenly develop 100x too slow, resource intensive and broken buzzword bingo clusterfucks that could be solved, correctly and cheaply, with a page of shell script.
- foobarian 7y ago
  Yup, it sure is nice to have a monopoly and be able to throw money at 100+ engineer teams to invent cool new tech from scratch. I'd like to see you try this at a startup.
  - i386 7y ago
    Daddy AdWords always has a spare $100M
crispinb 7y ago
If you can't come up with anything, it's possible you're just beyond repair and too snarky to think reasonably Or perhaps that we've mostly worked in crap dev jobs/companies? There are easily enough of them around to encompass a career.
- sophiebits 7y ago
  She explains that in the second half of the paragraph you quoted.
  - crispinb 7y ago
    You're right. The internet has done nothing for my reading comprehension.
yoz-y 7y ago
I don't know. Having an "80% solution" but not needing to support or fix it is not such a bad price to pay. Also this seems to be a bit skewed towards large networked solutions.
thesehands 7y ago
Wanted to see what Facebook Scuba was, and found this comment thread that really highlights the point of this blog. https://news.ycombinator.com/item?id=13463016 https://news.ycombinator.com/item?id=13463016
- dehrmann 7y ago
  I think it's somewhat like Apache Druid.
  - packetslave 7y ago
    Honeycomb.io is a good place to look. It has a strong Scuba-like smell (unsurprising since founders are from Facebook (by way of Parse). Basically: structured log events with dozens or hundreds of columns, no write time aggregation, and arbitrary folding and slicing of data for investigation and debugging.
analog31 7y ago
I'd miss the engineering machine shop. Those facilities only get developed by accident, for instance if a company gets rid of its professional machinist but keeps the equipment, or something like that. The machines and collections of tooling take a long time to curate. Any modern manager would refuse to allow that much stuff to be purchased if they were asked, and believe in the magic of outsourcing everything.
- mistrial9 7y ago
  At Apple Inc in the Sculley days (long ago) there was a lab in .. Monroe? .. the guys there were super orderly.. sports dads with good jobs.. that sort of thing.. I asked one day, what do you do here? They had so much precision equipment in one place.. the guy said that the did compliance testing for standards? and, some performance/spec testing? He looked slightly proud, and said "they had won Industry Awards" .. a year later they were gone. All the equipment, and I suspect most of the guys. The VP name was Moon? seemed like outsourcing and unwise to me..
- crtlaltdel 7y ago
  yeah in the olden days i had access to a full metal shop, silkscreen, smt and all the components i could ever want
deleted 7y ago
[deleted]
jackie_treehorn 7y ago
I just joined apple and I miss all the Linux tools
- andrethegiant 7y ago
  Yeah but you have Radar. Didn't realize how much I liked it until after I left
  - i386 7y ago
    I suspect only people at Apple like Radar ;)
  - __initbrian__ 7y ago
    TIL antbear is another name for an aardvark https://nshipster.com/bug-reporting/ https://nshipster.com/bug-reporting/
- mr_toad 7y ago
  I guess you can’t install Homebrew on work machines?
stereolambda 7y ago
I don't agree without reservations but like how against the grain this currently is. Outsourcing everything does seem like a sign of immediate hyperoptimization and missing building a more lasting engineering value. Do as little as possible to extract some margin. There is business wisdom in this but indeed such companies would seem more brittle. Sometimes it's like there's an ecosystem of plumbing companies selling to other plumbing companies without much visible external cash intake. That being said, I think most of the things she mentions are not customer facing but still pretty differentiating to the businesses. Another perspective is that the first crop of truly successful Internet companies were in the position to solve the big backend problems with running big Internet companies in-house. Nowadays getting to the scale while building some fundamental in-house stuff makes less economic sense (you'd lose to someone who doesn't). It's probably better to have some unique technology but more closely related to the specific business.
Insanity 7y ago
At one of the places I worked, a relatively large university-hospital, there was an entire Java-UI framework build based on C#s XAML. But it could also experimentally be compiled to html/css/js. Writing any type of UI in Swing is just painful - but this system made it so much nicer. It also had a WYSIWYG editor that you could interop with actually writing the code. Plus you could write either the 'markup language' or the "Java" flavour directly. Making complex responsive UIs was a breeze. It's a shame it did not get open sourced - but it was tied into a great amount of internal tooling. (The entire codebase was 28+ million lines of code when I left). It's one of the few non-open source libraries I've worked with that really impressed me.
- IggleSniggle 7y ago
  I am very curious what the university-hospital was, if you can share it.
  - Insanity 7y ago
    A Belgian one (Leuven) ;-)
wpeterson 7y ago
This article is dangerous in romanticizing the “not invented here” culture at many big tech companies and seems rooted more in the 90s than present day. The world of open source tooling and easily re-usable SAAS offerings means everyone has access to the best tools, whether you’re a small startup or a big company. Anyone who longs for internal, corporate tooling baffles me when they can use things that actually have polish, user experience and likely better implementations under the hood. Companies should spend their time/energy building things unique to their problem domain, not weak also-ran corporate tooling.
- barrkel 7y ago
  The mistake you're making is thinking that the public tools are the best tools. There are reasons that make them the best for many, if not most, companies: more investment, more mindshare, easier to hire employees with prior experience, and so on. But there are also costs in having a wobbly stack of glued together stuff, especially if the parts aren't quite right for your goal. Sometimes the best tool is more focused, more vertically integrated, in a different language or for a different operating system because those choices integrate better with the rest of your stuff. The constraints of the company are also part of the problem domain. Using the wrong tools can be quite expensive, and the public tools may all be just a bit wrong in a way that compounds.
  - kelnos 7y ago
    Where I work, the majority of our tools were built in-house, mainly because we started (2008) before there were good open source or even paid options for most of it all. As good options started to appear, we found that we couldn't adopt them, because there was a mismatch in concepts/fundamentals between what we'd built and what was out there. We've evaluated a lot of things, and for many of them, we end up realizing that integrating them with our systems would require a hard fork, and so we'd lose most of the benefit of using it. Frankly, it sucks. Our tools are mostly very good, but it took us a long time to get there, and the internal fights over getting funding to really invest in our internal platform have been exhausting for all involved. I get that it's not zero work or zero time to use something off the shelf, but as someone who has been playing in the grass on the other side of that particular fence, it takes a lot of work to keep that grass green.
stochastastic 7y ago
This really resonates with me. I have seen the “buy before you build” attitude turn into “buy because we can’t build”. I know correlation is not causation and there are always plenty of other things going on... but it sure seems like the pride in our work dropped off and we were on the wrong side of every tipping point.
- eropple 7y ago
  I've seen this at a few places now. Learned helplessness is definitely a thing and a lot of it comes down to an inability or unwillingness to hire broadly. When you're hiring Bolt Turner #3, they might know a little bit about turning bolts (but might not, either) but expect everything that isn't a turnable bolt to be handed to them. But the screw turners and the pipe welders you've also hired don't think a working ecosystem of tooling is their responsibility either and they're not really equipped to build it either, so the only option becomes to buy. And at that point you lack the breadth and the depth of knowledge to effectively choose the right thing to buy, and the outcome is as if by dart board. There's a reason to hire specialists in this industry, but we conflate one-track ponies with specialists and it's to our detriment.
alexhutcheson 7y ago
For 99% of engineering organizations, internal engineering tooling is not your source of competitive advantage. Your advantage in the market comes from understanding the needs of your target user better than anyone else, and building a product that meets those needs. For every piece of infrastructure or developer tooling you need, you have to make a buy vs. build decision based on cost/benefit analysis. Your cost to "buy" needs to include the upfront cost of developer time to configure the software for your needs, and the ongoing support cost to make sure it continues to work. When you include this, even open source software isn't "free", although it may be cheap. Your cost to "build" includes the upfront engineering time to design & code, ongoing support time to keep it working, and opportunity costs incurred because until the system is actually ready, no one can use it, and they have to either meet their needs in some other way or simply block until the tool is complete. Very large engineering organizations like Facebook, Netflix, etc. can amortize the cost of tool and infrastructure development over thousands of developers and an enormous infrastructure budget. If a tool would make most of Facebook's developer's 3% more efficient, than it's worth staffing a team to build that tool. If better caching infrastructure could reduce Netflix's network usage by 3%, then it's worth staffing a team to build that infrastructure. If you're a small- or medium-sized business, then that math doesn't pencil out. A 3% (or even 10%) developer efficiency improvement doesn't even pay for the opportunity cost of one developer. A 10% decrease in a single dimension of your AWS bill probably doesn't pay for the developer time either. If you're at a mega-corp with a huge engineering staff: Yes, absolutely invest in tools that make your engineers more productive, especially if you've identified truly unique needs that can't be met by off-the-shelf software. If you're at a smaller business: Focus your developer energy on your user-facing problems, and almost always prefer to "buy" instead of "build" (where "buy" includes using open source software).
- pm90 7y ago
  I do not think it’s that simple. I understand the arguments for buying vs build. However, if you’re a technical heavy organization, relying on buying things also has its costs. You’re constrained on your product offerings because of the thing you bought. The decision is not that simple and requires a lot of work and foresight. As such, my guiding principle has been to consider the layer at which the thing you’re trying to build/buy fits. For most companies it doesn’t make much sense to build data centers and any cloud vendor will do just fine. Infrastructure is therefore somewhat easier... almost always buy. But if your product requires some kind of integration that is critical for customers, I would think hard about it before buying. The costs of integrating a bought tool are also overlooked in most of these conversations. External tools are marketed well but don’t always deliver on their promises of being easy to use or setup. Fundamentally I genuinely believe that if your product is tech heavy you absolutely need a CTO or a founding engineer who has a solid experience in building things. There is no replacement or magic pill to replace that just yet. And still, many fall into this trap.
lidHanteyk 7y ago
I prefer Kubernetes and Prometheus to Borg and Borgmon. Better interoperability, portability, and documentation. Does any other Xoogler have thoughts on this? The author didn't mention it, but I also prefer Capn to Proto3.
- packetslave 7y ago
  Kubernetes has a lot going for it, but it isn't (yet) anywhere close to Borg in terms of scalability. 5000 maximum hosts in a cluster is a drop in the bucket for a Borg cluster. Google was publicly talking about 10,000 machines as the median size almost 5 years ago. As for Prometheus, I like the rules engine VERY much (unsurprising, since they basically reimplemented borgmon, there). I'd take Viceroy any day over Grafana, though.
  - meddlepal 7y ago
    I have deep experience with operating Kubernetes and no experience with Google's Borg but I imagine Borg is significantly more challenging to operate? Kubernetes is designed to be a workload scheduler that a three or four person team can operate.
    - packetslave 7y ago
      I imagine if you're Borg SRE, you'd say "yes, it's significantly more challenging to operate", but for the tens of thousands of other engineers? Not really, although there was a LOT of copypasta in terms of building new configs for services.
ThrowAwayGlen 7y ago
ime, for many companies, standard, off-the-shelf tooling that's a mix of OSS and paid services works great. And one benefit is that when you leave, you can bring whatever tools you loved into the next company. The reason Facebook and Google built their own is that they're at such a huge scale both in terms of usage and number of engineers, that none of the off-the-shelf things work for them. For most companies that's not the case.
wink 7y ago
This whole premise falls completely flat if you've only ever worked at small and/or young companies: > but which didn't exist on the outside world. > You're not allowed to include things you or your friends built If you work for a service provider, you probably have zero to not much internal tooling, because you're usually solving the customer's problems, not your company's. When I did, we open sourced everything cool we built - sometimes tried and true defaults, or sensible Vagrantboxes or something like that. (Today that would be Dockerfiles, I guess) - and I consider myself lucky to have worked at a smaller company that was close to the FOSS scene, and not something like Accenture. Companies like ThoughtWorks might be an outlier here. You're unlikely to build something so immensely cool in your first 1-2 years in a startup that it will change your live (as a developer) completely. (This is not about your company's product, but internal developer tooling, so if your startup does developer tools as a product, you're the 1% where this would apply). And finally, if your startup grows.. you might probably still not be the next Google or Facebook where the internal team providing nice things to your fleet of developers is bigger than some companies. Internal tooling will often be seen more of a cost factor than a tool for productivity enhancement. And often it's actually the case. In my current company we just have so many software development teams creating parts of our products.. so many languages (with good reason) - the only internal things I could think about that would even make sense for all of them would be some code review tool, or anything targeting source control or maybe CI. Anything else would have to be really workflow-centric and language-agnostic. Anything regarding the mentioned borg/k8s/BigTable would be completely useless, for example. That said, at my last company, I think we created some pretty cool things that made a lot of things a lot easier for the team. But we were two small teams, so point to "we built it ourselves, for ourselves" applies. Also it probably wasn't so revolutionary in the greater sense, but everything made sense for that company and I'm often still sad if I have to solve some similar problem these days and don't have my old setup (more than a specific tool).
- rachelbythebay 7y ago
  Counterpoint: Rackspace. The line internally was "service, not servers". They developed CORE which was their customer tracking and computer provisioning system. When RightNowWeb stopped working for them, they built the CATS ticketing system on top of that. Those were nice tools which worked quite well most of the time. They worked in any browser on any OS, unlike some of the ActiveX-only stuff they tried to replace it with. This is not the Rackspace you probably know. This is the teeny little one that nobody had heard of outside of a few advertisements in Linux Journal (and the occasional rant about spammers and malware hosting). This is them back before the whole cloud thing - over fifteen years ago.
cwyers 7y ago
Not everyone finds BigCorp problems to be the interesting problems. Not everyone wants to live in the Bay Area. There's a lot of jobs in the world that aren't at FAANG, and it's okay if the right job for someone is at a place whose entire dev shop is smaller than the team at Google that maintains Blaze. If you find yourself in a job like that, it is absolutely the right call to invest in the best commodity tools you can (where best is a local, not a global, maxima) and focus on the things you can do that add value to your org.
- paulie_a 7y ago
  There seems to be an elitism when it comes to working for faang. there are a ton of people that have zero interest in building advertising engines or living in the Bay area. Quite frankly my min salary for having to live there sacrifice working on something interesting with be mid to high 6 figures so I could save and move the hell away after a few years. I'd consider it a life sacrifice to endure the bay area and culture.
  - hashhar 7y ago
    I agree with your points but just want to add my anecdotal experience here from India that not all companies with great internal tooling are unicorns. They generally tend to be the slow but steadily moving companies who take their time and think about sustainability rather than 10x growth.
    - lifeisstillgood 7y ago
      Yes. And I might suggest that the unicorns with great tooling just lucked out and has engineers who built great stuff irrespective of management
  - Youden 7y ago
    I feel similarly but I find it to be the Bay Area that's the problem rather than FAANG. All those startups and unicorns being in one place makes people see the Bay Area as some kind of software engineering Mecca and I think engineers based there subconsciously see themselves as superior for being a part of it, especially given the high pay. If you go to another city or even better another country, even in the same company, things are very different. Having done so, I would never return to the Bay Area except for the reason you mentioned: to pile up enough money to leave it again.
novok 7y ago
I’ve been at places where they should of bought vs build, because they don’t put enough staff on it
tmpz22 7y ago
Doesn't it take months to train new developers in these mystical internal tools? I remember hearing specifically that Google expects 5+ months of training until a developer is productive. How many companies can afford to spend ~50k+/hire on on-boarding ((100k/yr * 5 months) + benefits)? What is the opportunity cost of holding so many people in training mode for this time period? Different companies have different problems. Big companies need big expensive tools to help with their problems. Little companies usually get by with smaller more portable tools. Let's not use this to demean people at small companies with some false sense of superiority please...
- lainga 7y ago
  Given the background of the blog's author, I wouldn't hold my breath
  - nostrebored 7y ago
    What, a successful engineer with tenure at two of the most successful business operating at scale?
- nbm 7y ago
  It takes a few hours to learn to use the tools mentioned in the article (and many similar ones) to a level of competence - part of the appeal is how easy they are to use. It takes a few hours to learn how to get new data into and out of them as well. Both are the sort of thing taught in 45 minutes presentations and workshops during the Bootcamp training at FB, and then most people can just explore the rest themselves over time in normal usage. What takes months is feeling like you understand the most important tools and aspects of the environment to be as effective in them as those around you, or to match how you felt in an environment where the scope is smaller. It’s not like you’re sitting in classes for months - you’re doing your job (maybe even more effectively than normal due to the availability of these tools), but every once in a while you’re encountering something where you need to ask someone else or explore or watch a video about how to do something, and that might break your momentum.
- bagacrap 7y ago
doggydogs94 7y ago
If the tool is valuable to you, make sure you know exactly how it works. Then you can recreate it, for yourself at least, whenever you go.
- downerending 7y ago
  Also, if you wrote it (as often happens in my case), put some effort into getting it open-sourced if possible, so that you can just use it elsewhere later. These days, I very rarely enjoy using tools that aren't open source, or at least generic and very widely available. I simply don't want to spend my limited brain cells learning a piece of software that I won't be able to use in the future. And in any case, large, internally developed tools are usually awful. Unless I wrote them. ;-)
superbatfish 7y ago
Couldn’t it be argued that all software is built with more “glue” than non-glue? That’s just the nature of the beast. If a user wants to, say, view a record from a remote database in her web browser, how many separate layers of code does that involve? (Several.) How many of them aren’t “glue”? (I dunno, like two?) Perhaps it’s true that most glue is poorly written, but doesn’t that imply that writing good glue is, in fact, difficult, and therefore it’s as hard — or maybe harder - than building “real” things?
yibg 7y ago
Nothing wrong with writing glue, if that’s what the business / company / market needs. Having said that, I do miss the data that’s available at FAANG companies (at least ones I’ve been at). I could dig up all sorts of system and business data, and really get good insight into both what my application and the users are doing. Tends to lead to a lot better and more clear decisions and thus helps to build the right thing.
bgdnyxbjx 7y ago
This topic is one of the primary reasons I enjoy working at MS these days. The tech stack is full of OSS or industry standard tools, not a bunch of internal NIH things. My current stack is: React, Node, Kubernetes, Linux (for dev and deploy), Git for sc, GitHub, vs code. I can’t even think of some internal tool that I depend on at the moment.
- ddlutz 7y ago
  That depends entirely on what part of MS you are in right? There is cosmos (internal, not exactly the same as external cosmos), autopilot for infra, mdm for metrics, mds for logs, icm for alerts, etc.
  - bgdnyxbjx 7y ago
    Yeh, you’re right. I was thinking more of this and remember all the alerting and incident management stuff. Then I realized that I hate those tools and would happily leave them behind :)
- zepolen 7y ago
  Github is open source?
  - deleted 7y ago
    [deleted]
  - cnst 7y ago
    Like it or, GitHub is ubiquitous enough that noone really cares. If you do care, there's plenty of OSS alternatives that aren't much different in their UI than GitHub itself. You can easily put GitHub on your resume, and everyone would know what you're talking about. Of course, by itself, it's not much of an experience, but, surprisingly, it is often listed as a requirement and/or nice-to-have for certain SWE job posts.
kelnos 7y ago
I'm always so disappointed that we're all so closed when it comes to tooling. It's fortunate that some tools get open-sourced, but it's also astounding the amount of time and money that is wasted building the same tools over and over at each company. Sure, sometimes paying for a hosted solution for your CI, deployments, container platform, data pipeline, etc. isn't the most ideal route, depending on your business, but I'd much rather focus on building actual product over reinventing the deployment system wheel for the Nth time. > People from Google probably miss Borg (and lament the whole Kubernetes thing) or BigTable. I've never worked at Google, and thus have no experience with Borg, but Kubernetes is exactly what I want to see companies open-sourcing, or, if they must, selling. I don't see what there is to lament. Being able to start a new company and not have to reinvent and rebuild your own deployment/container platform is a huge gift. For the most part I find this blog to be insightful and fun to read, but this post IMO completely misses the mark. Given that I think the author works a lot on the operations side of the house, I guess I'm not surprised. Internal platform tools are there to serve the needs of the product teams; they don't exist just to make someone feel good about their ability to build something instead of buying instead, which she unfairly looks down on so disdainfully, as if it's not real engineering. Part of doing engineering is putting your ego aside when it's more cost-effective and robust to use someone else's solution instead of rolling your own.
- godot 7y ago
  I feel like the author's point and your point would be applicable a lot more say 5 to 8 years ago. There has been so many SaaS startups in the past 8 years or so where many things that used to require intense internal tooling are now available as a SaaS.
adreamingsoul 7y ago
I honestly miss the internal tooling at AWS. The build system and CI/CD pipelines used internally are truely amazing.
- burpsnard 7y ago
  Brazil? isn't it mostly there in work-alike saas?
SergeAx 7y ago
I have to strongly disagree. When doing business you are solving a business problems. Facebook, Google, Microsoft as business has grown so much they have problems of scale no open tools are able to handle. On the other hand, any kind of smaller company doesn't have those business problems, but a cost of having and maintaining in-house solutions of that scale will just effectively topple and sink them. This is why I'm always frowning during job interviews with engineers telling stories about they migrations from monolith to microservices and putting it all into k8s, while their headcount in the firs tens and their DAU in about 10000 users. Disclaimer: I worked in Yandex, virtually Russian Google. Loads and operations scale there are comparable with second rank international tech giants. Yandex has and have lots of internal tools to handle that load and scale, but I've almost never really needed them in smaller companies. Maybe "nice to have" couple of times.
- cnst 7y ago
  It's also a lot about the marketability of your skills. The only reason the OP can boast about the experience with these proprietary tools is because they're used by FAANG. No other reason — it's just an indirect and inconspicuous boasting about having worked at FAANG. If it was yet another proprietary non-OSS language used by one of the investment companies that has no direct analogues at any of their competitors — good luck boasting on your resume that you have skills in programming languages that noone cares about. Proprietary software sucks. Just because it's written and used internally by FAANG doesn't make it suck any less. When you're evaluating whether or not to join a company, presence of such tools should be more of a red flag, if anything. Same goes with software-as-a-service. Tools should be OSS, and easily portable between the workplaces. Another case in point — it kind of works both ways — many companies start around with Erlang and ejabberd, because it's a really great tool, but then, after having issues finding developers who can understand it properly, they eventually, one way or another, convert their stack back to the more common languages; I recall this has happened with Facebook Messenger and Heroku, and also at a random startup I was interviewing with during the course of my interview (they just hired a manager who knew Java — and, what do you know, he decides to set a direction to eventually replace all Erlang with Java).
zhte415 7y ago
I've recently branched out on my own. Never had great tooling, was in government or a F50 where chains of approvals for the slightest changes were cursed by all, including those doing the approving. But, there are some organisational things I miss: Having someone/a team to ensure network uptime is fantastic and that have the scale and experience to anticipate and mitigate potential trouble before it comes up. Likewise all infra in general. Having a legal department to ask "Can I do this?" where often the answer may be "We don't know" but that's still an answer. A security force, property, accounting, providing peace of mind. Someone making sure the projector's setup, tiny things that add up quickly. There are downsides to this, such 'organisation' often grinds things to a glacial pace, hence why I'm breaking out by myself now, but... not tooling, I can take care of that myself and I'm incredibly thankful for the SaaS ecosystem that would have made this impossible even a decade ago, or at least very inconvenient with 3rd party consultants providing such services for high fees and little more. There are a lot of things an organisation provides that we can take for granted, and having that experience provides a great bonus in expectations of robustness not only in technology but in all processes of a business. Tooling is, in the end, in the mind.
pjmlp 7y ago
The only thing I miss are teammates and some of the office moments we had together, but those luckily survive employers and we get to meet in other contexts. I never been attached to work tools, or technology fads, things are ephemeral, best practices of today are the legacy codebase that no one wants to work on tomorrow.
bullen 7y ago
I solved this problem 10 years ago by building my own open-source stack and only working with that stack: https://github.com/tinspin https://github.com/tinspin You should build your own tools, that you bring with you, that is what all artisans have done for all time. No reason to change that just because you can use the newest buss-word every week.
vermooten 7y ago
I'm very disappointed if I move from a company that uses macs, slack, Confluence to PCs, MS Teams (horrible) and SharePoint (worse).
- schappim 7y ago
  Why is MS Teams horrible? We have an engineer who is trying to convince our team to move from Slack to MS Teams. My only gripe is the requirement for a M$ login account (we are a Google Apps house).
sershe 7y ago
One problem that I've seen in one such large company, and heard of from some people in others (FAANGs), is that there's no incentive to maintain these internal tools. Sure, they often start out as sort of amazing and there's a ship email and for 1-3 years a strong team, but you are unlikely to be promoted a lot polishing UI, fixing bugs and adding features or even support for new backend systems... so, eventually there's a skeleton crew maintaining an old tool, everybody complains how it's slow and sucks and behind the times, and then a brand new replacement is built that is going to fix everything. There's a ship email, people gradually switch to it, rinse, repeat.
heelix 7y ago
The only thing I'd miss from a larger company is the travel budget. The 'devil may care' attitude to drop a couple grand on a short window flight... compared to travel with the small company where that meeting could break the company. Double goes for the in-house travel agent who sorted everything while in route - Jane... you were missed.
caseymarquis 7y ago
Every time I make a neat tool I open source it using the unlicense so I can use it elsewhere with no attribution. I'm happy to work at a company where this is possible. This doesn't apply to business critical code a competitor might want, just libraries and tools for making generic 'stuff'. Typically, I make the tools at home first and decide I want to use them at work, but I eventually end up patching them at work for company purposes.
SZJX 7y ago
This article makes little sense. "Engineering" by definition is some sort of "glue" work that reuses existing libraries/wheels and builds solutions out of them to address specific problems. I as an engineer am proud of this glue work that I do, just like the carpenters, plumbers, cathedral builders, doctors etc. If you really want to be unique and invent something, you should be in the academia and are probably in the wrong place to begin with. I have a friend who worked in the TensorFlow team and still quit to pursue a PhD. In his words "all engineering jobs are boring" from his perspective. I don't think being in a BigCorp or not makes much difference at all.

170 points by kogir 7y ago

i386 7y ago

If you’re not building complicated new systems and just gluing things together, you’re not doing engineering? 1) definitely gate keeping 2) good engineering is about what not to build as much as it is what to build.

nemothekid 7y ago
I agree - shouldn't you feel most proud of the products you've built that help actual people who pay you, rather than your coworkers for which those internal tools wouldn't need to exist if not for your customers. I'd rather have engineers working on products that customers would miss.
- adamsea 7y ago
  Everyone is motivated by different things. It makes sense that some people are motivated by the craft of software development, which overlaps with, but is not the same as, product development or running a successful business.
vector_spaces 7y ago
Yes, exactly. The view that such work just constitutes writing "glue" is super reductive. So much time is also spent empathizing with others: talking to stakeholders, understanding existing processes, determining requirements, and simplifying. So if we can reduce the solution to glue instead of needing to write a complicated system, that's hands down a win.
- i386 7y ago
  Any engineer who comes up with a solution that requires less new code being written is a more valuable engineer IMO. Less code is less time spent on building then maintaining code for all eternity. If you go through the build/buy/partner decision, and decide to buy or partner with development comprising of glue code - then that's probably the optimal solution for your org. And you made the right engineering choice.
  - patrec 7y ago
    > Any engineer who comes up with a solution that requires less new code being written is a more valuable engineer IMO Of course less code is good, all things being equal. But taking that idea literally, basically no tooling code gets ever written and people develop some sort of learned helplessness around dysfunctional workflows that are just about feasible but fantastically wasteful of engineering time. This tends to happen a lot, also because tooling is often perceived by management as an unnecessary luxury.
    - i386 7y ago
      Agree, In my own experience it’s only orgs with a min 50 to 100 developers who can afford to have a single developer (it was me!) out working on productivity tools.
- patrec 7y ago
  Just because you're not doing much (any?) interesting engineering doesn't mean you're not creating a lot of value. But it probably does mean that if you really deeply care about engineering in itself you should reconsider your employment options. But learning when you can just glue and how is a super-valuable skill, even if some of your work involves pushing the envelope. The worst engineers are not those who just glue stuff together, it's the people who cv-drivenly develop 100x too slow, resource intensive and broken buzzword bingo clusterfucks that could be solved, correctly and cheaply, with a page of shell script.
foobarian 7y ago
Yup, it sure is nice to have a monopoly and be able to throw money at 100+ engineer teams to invent cool new tech from scratch. I'd like to see you try this at a startup.
- i386 7y ago
  Daddy AdWords always has a spare $100M

crispinb 7y ago

If you can't come up with anything, it's possible you're just beyond repair and too snarky to think reasonably Or perhaps that we've mostly worked in crap dev jobs/companies? There are easily enough of them around to encompass a career.

sophiebits 7y ago
She explains that in the second half of the paragraph you quoted.
- crispinb 7y ago
  You're right. The internet has done nothing for my reading comprehension.

yoz-y 7y ago

I don't know. Having an "80% solution" but not needing to support or fix it is not such a bad price to pay. Also this seems to be a bit skewed towards large networked solutions.

thesehands 7y ago

Wanted to see what Facebook Scuba was, and found this comment thread that really highlights the point of this blog. https://news.ycombinator.com/item?id=13463016 https://news.ycombinator.com/item?id=13463016

dehrmann 7y ago
I think it's somewhat like Apache Druid.
- packetslave 7y ago
  Honeycomb.io is a good place to look. It has a strong Scuba-like smell (unsurprising since founders are from Facebook (by way of Parse). Basically: structured log events with dozens or hundreds of columns, no write time aggregation, and arbitrary folding and slicing of data for investigation and debugging.

analog31 7y ago

I'd miss the engineering machine shop. Those facilities only get developed by accident, for instance if a company gets rid of its professional machinist but keeps the equipment, or something like that. The machines and collections of tooling take a long time to curate. Any modern manager would refuse to allow that much stuff to be purchased if they were asked, and believe in the magic of outsourcing everything.

mistrial9 7y ago
At Apple Inc in the Sculley days (long ago) there was a lab in .. Monroe? .. the guys there were super orderly.. sports dads with good jobs.. that sort of thing.. I asked one day, what do you do here? They had so much precision equipment in one place.. the guy said that the did compliance testing for standards? and, some performance/spec testing? He looked slightly proud, and said "they had won Industry Awards" .. a year later they were gone. All the equipment, and I suspect most of the guys. The VP name was Moon? seemed like outsourcing and unwise to me..
crtlaltdel 7y ago
yeah in the olden days i had access to a full metal shop, silkscreen, smt and all the components i could ever want

deleted 7y ago

[deleted]

jackie_treehorn 7y ago

I just joined apple and I miss all the Linux tools

andrethegiant 7y ago
Yeah but you have Radar. Didn't realize how much I liked it until after I left
- i386 7y ago
  I suspect only people at Apple like Radar ;)
- __initbrian__ 7y ago
  TIL antbear is another name for an aardvark https://nshipster.com/bug-reporting/ https://nshipster.com/bug-reporting/
mr_toad 7y ago
I guess you can’t install Homebrew on work machines?

stereolambda 7y ago

I don't agree without reservations but like how against the grain this currently is. Outsourcing everything does seem like a sign of immediate hyperoptimization and missing building a more lasting engineering value. Do as little as possible to extract some margin. There is business wisdom in this but indeed such companies would seem more brittle. Sometimes it's like there's an ecosystem of plumbing companies selling to other plumbing companies without much visible external cash intake. That being said, I think most of the things she mentions are not customer facing but still pretty differentiating to the businesses. Another perspective is that the first crop of truly successful Internet companies were in the position to solve the big backend problems with running big Internet companies in-house. Nowadays getting to the scale while building some fundamental in-house stuff makes less economic sense (you'd lose to someone who doesn't). It's probably better to have some unique technology but more closely related to the specific business.

Insanity 7y ago

At one of the places I worked, a relatively large university-hospital, there was an entire Java-UI framework build based on C#s XAML. But it could also experimentally be compiled to html/css/js. Writing any type of UI in Swing is just painful - but this system made it so much nicer. It also had a WYSIWYG editor that you could interop with actually writing the code. Plus you could write either the 'markup language' or the "Java" flavour directly. Making complex responsive UIs was a breeze. It's a shame it did not get open sourced - but it was tied into a great amount of internal tooling. (The entire codebase was 28+ million lines of code when I left). It's one of the few non-open source libraries I've worked with that really impressed me.

IggleSniggle 7y ago
I am very curious what the university-hospital was, if you can share it.
- Insanity 7y ago
  A Belgian one (Leuven) ;-)

wpeterson 7y ago

This article is dangerous in romanticizing the “not invented here” culture at many big tech companies and seems rooted more in the 90s than present day. The world of open source tooling and easily re-usable SAAS offerings means everyone has access to the best tools, whether you’re a small startup or a big company. Anyone who longs for internal, corporate tooling baffles me when they can use things that actually have polish, user experience and likely better implementations under the hood. Companies should spend their time/energy building things unique to their problem domain, not weak also-ran corporate tooling.

barrkel 7y ago
The mistake you're making is thinking that the public tools are the best tools. There are reasons that make them the best for many, if not most, companies: more investment, more mindshare, easier to hire employees with prior experience, and so on. But there are also costs in having a wobbly stack of glued together stuff, especially if the parts aren't quite right for your goal. Sometimes the best tool is more focused, more vertically integrated, in a different language or for a different operating system because those choices integrate better with the rest of your stuff. The constraints of the company are also part of the problem domain. Using the wrong tools can be quite expensive, and the public tools may all be just a bit wrong in a way that compounds.
- kelnos 7y ago
  Where I work, the majority of our tools were built in-house, mainly because we started (2008) before there were good open source or even paid options for most of it all. As good options started to appear, we found that we couldn't adopt them, because there was a mismatch in concepts/fundamentals between what we'd built and what was out there. We've evaluated a lot of things, and for many of them, we end up realizing that integrating them with our systems would require a hard fork, and so we'd lose most of the benefit of using it. Frankly, it sucks. Our tools are mostly very good, but it took us a long time to get there, and the internal fights over getting funding to really invest in our internal platform have been exhausting for all involved. I get that it's not zero work or zero time to use something off the shelf, but as someone who has been playing in the grass on the other side of that particular fence, it takes a lot of work to keep that grass green.

stochastastic 7y ago

This really resonates with me. I have seen the “buy before you build” attitude turn into “buy because we can’t build”. I know correlation is not causation and there are always plenty of other things going on... but it sure seems like the pride in our work dropped off and we were on the wrong side of every tipping point.

eropple 7y ago
I've seen this at a few places now. Learned helplessness is definitely a thing and a lot of it comes down to an inability or unwillingness to hire broadly. When you're hiring Bolt Turner #3, they might know a little bit about turning bolts (but might not, either) but expect everything that isn't a turnable bolt to be handed to them. But the screw turners and the pipe welders you've also hired don't think a working ecosystem of tooling is their responsibility either and they're not really equipped to build it either, so the only option becomes to buy. And at that point you lack the breadth and the depth of knowledge to effectively choose the right thing to buy, and the outcome is as if by dart board. There's a reason to hire specialists in this industry, but we conflate one-track ponies with specialists and it's to our detriment.

alexhutcheson 7y ago

For 99% of engineering organizations, internal engineering tooling is not your source of competitive advantage. Your advantage in the market comes from understanding the needs of your target user better than anyone else, and building a product that meets those needs. For every piece of infrastructure or developer tooling you need, you have to make a buy vs. build decision based on cost/benefit analysis. Your cost to "buy" needs to include the upfront cost of developer time to configure the software for your needs, and the ongoing support cost to make sure it continues to work. When you include this, even open source software isn't "free", although it may be cheap. Your cost to "build" includes the upfront engineering time to design & code, ongoing support time to keep it working, and opportunity costs incurred because until the system is actually ready, no one can use it, and they have to either meet their needs in some other way or simply block until the tool is complete. Very large engineering organizations like Facebook, Netflix, etc. can amortize the cost of tool and infrastructure development over thousands of developers and an enormous infrastructure budget. If a tool would make most of Facebook's developer's 3% more efficient, than it's worth staffing a team to build that tool. If better caching infrastructure could reduce Netflix's network usage by 3%, then it's worth staffing a team to build that infrastructure. If you're a small- or medium-sized business, then that math doesn't pencil out. A 3% (or even 10%) developer efficiency improvement doesn't even pay for the opportunity cost of one developer. A 10% decrease in a single dimension of your AWS bill probably doesn't pay for the developer time either. If you're at a mega-corp with a huge engineering staff: Yes, absolutely invest in tools that make your engineers more productive, especially if you've identified truly unique needs that can't be met by off-the-shelf software. If you're at a smaller business: Focus your developer energy on your user-facing problems, and almost always prefer to "buy" instead of "build" (where "buy" includes using open source software).

pm90 7y ago
I do not think it’s that simple. I understand the arguments for buying vs build. However, if you’re a technical heavy organization, relying on buying things also has its costs. You’re constrained on your product offerings because of the thing you bought. The decision is not that simple and requires a lot of work and foresight. As such, my guiding principle has been to consider the layer at which the thing you’re trying to build/buy fits. For most companies it doesn’t make much sense to build data centers and any cloud vendor will do just fine. Infrastructure is therefore somewhat easier... almost always buy. But if your product requires some kind of integration that is critical for customers, I would think hard about it before buying. The costs of integrating a bought tool are also overlooked in most of these conversations. External tools are marketed well but don’t always deliver on their promises of being easy to use or setup. Fundamentally I genuinely believe that if your product is tech heavy you absolutely need a CTO or a founding engineer who has a solid experience in building things. There is no replacement or magic pill to replace that just yet. And still, many fall into this trap.

lidHanteyk 7y ago

I prefer Kubernetes and Prometheus to Borg and Borgmon. Better interoperability, portability, and documentation. Does any other Xoogler have thoughts on this? The author didn't mention it, but I also prefer Capn to Proto3.

packetslave 7y ago
Kubernetes has a lot going for it, but it isn't (yet) anywhere close to Borg in terms of scalability. 5000 maximum hosts in a cluster is a drop in the bucket for a Borg cluster. Google was publicly talking about 10,000 machines as the median size almost 5 years ago. As for Prometheus, I like the rules engine VERY much (unsurprising, since they basically reimplemented borgmon, there). I'd take Viceroy any day over Grafana, though.
- meddlepal 7y ago
  I have deep experience with operating Kubernetes and no experience with Google's Borg but I imagine Borg is significantly more challenging to operate? Kubernetes is designed to be a workload scheduler that a three or four person team can operate.
  - packetslave 7y ago
    I imagine if you're Borg SRE, you'd say "yes, it's significantly more challenging to operate", but for the tens of thousands of other engineers? Not really, although there was a LOT of copypasta in terms of building new configs for services.

ThrowAwayGlen 7y ago

ime, for many companies, standard, off-the-shelf tooling that's a mix of OSS and paid services works great. And one benefit is that when you leave, you can bring whatever tools you loved into the next company. The reason Facebook and Google built their own is that they're at such a huge scale both in terms of usage and number of engineers, that none of the off-the-shelf things work for them. For most companies that's not the case.

wink 7y ago

This whole premise falls completely flat if you've only ever worked at small and/or young companies: > but which didn't exist on the outside world. > You're not allowed to include things you or your friends built If you work for a service provider, you probably have zero to not much internal tooling, because you're usually solving the customer's problems, not your company's. When I did, we open sourced everything cool we built - sometimes tried and true defaults, or sensible Vagrantboxes or something like that. (Today that would be Dockerfiles, I guess) - and I consider myself lucky to have worked at a smaller company that was close to the FOSS scene, and not something like Accenture. Companies like ThoughtWorks might be an outlier here. You're unlikely to build something so immensely cool in your first 1-2 years in a startup that it will change your live (as a developer) completely. (This is not about your company's product, but internal developer tooling, so if your startup does developer tools as a product, you're the 1% where this would apply). And finally, if your startup grows.. you might probably still not be the next Google or Facebook where the internal team providing nice things to your fleet of developers is bigger than some companies. Internal tooling will often be seen more of a cost factor than a tool for productivity enhancement. And often it's actually the case. In my current company we just have so many software development teams creating parts of our products.. so many languages (with good reason) - the only internal things I could think about that would even make sense for all of them would be some code review tool, or anything targeting source control or maybe CI. Anything else would have to be really workflow-centric and language-agnostic. Anything regarding the mentioned borg/k8s/BigTable would be completely useless, for example. That said, at my last company, I think we created some pretty cool things that made a lot of things a lot easier for the team. But we were two small teams, so point to "we built it ourselves, for ourselves" applies. Also it probably wasn't so revolutionary in the greater sense, but everything made sense for that company and I'm often still sad if I have to solve some similar problem these days and don't have my old setup (more than a specific tool).

rachelbythebay 7y ago
Counterpoint: Rackspace. The line internally was "service, not servers". They developed CORE which was their customer tracking and computer provisioning system. When RightNowWeb stopped working for them, they built the CATS ticketing system on top of that. Those were nice tools which worked quite well most of the time. They worked in any browser on any OS, unlike some of the ActiveX-only stuff they tried to replace it with. This is not the Rackspace you probably know. This is the teeny little one that nobody had heard of outside of a few advertisements in Linux Journal (and the occasional rant about spammers and malware hosting). This is them back before the whole cloud thing - over fifteen years ago.

cwyers 7y ago

Not everyone finds BigCorp problems to be the interesting problems. Not everyone wants to live in the Bay Area. There's a lot of jobs in the world that aren't at FAANG, and it's okay if the right job for someone is at a place whose entire dev shop is smaller than the team at Google that maintains Blaze. If you find yourself in a job like that, it is absolutely the right call to invest in the best commodity tools you can (where best is a local, not a global, maxima) and focus on the things you can do that add value to your org.

paulie_a 7y ago
There seems to be an elitism when it comes to working for faang. there are a ton of people that have zero interest in building advertising engines or living in the Bay area. Quite frankly my min salary for having to live there sacrifice working on something interesting with be mid to high 6 figures so I could save and move the hell away after a few years. I'd consider it a life sacrifice to endure the bay area and culture.
- hashhar 7y ago
  I agree with your points but just want to add my anecdotal experience here from India that not all companies with great internal tooling are unicorns. They generally tend to be the slow but steadily moving companies who take their time and think about sustainability rather than 10x growth.
  - lifeisstillgood 7y ago
    Yes. And I might suggest that the unicorns with great tooling just lucked out and has engineers who built great stuff irrespective of management
- Youden 7y ago
  I feel similarly but I find it to be the Bay Area that's the problem rather than FAANG. All those startups and unicorns being in one place makes people see the Bay Area as some kind of software engineering Mecca and I think engineers based there subconsciously see themselves as superior for being a part of it, especially given the high pay. If you go to another city or even better another country, even in the same company, things are very different. Having done so, I would never return to the Bay Area except for the reason you mentioned: to pile up enough money to leave it again.

novok 7y ago

I’ve been at places where they should of bought vs build, because they don’t put enough staff on it

tmpz22 7y ago

Doesn't it take months to train new developers in these mystical internal tools? I remember hearing specifically that Google expects 5+ months of training until a developer is productive. How many companies can afford to spend ~50k+/hire on on-boarding ((100k/yr * 5 months) + benefits)? What is the opportunity cost of holding so many people in training mode for this time period? Different companies have different problems. Big companies need big expensive tools to help with their problems. Little companies usually get by with smaller more portable tools. Let's not use this to demean people at small companies with some false sense of superiority please...

lainga 7y ago
Given the background of the blog's author, I wouldn't hold my breath
- nostrebored 7y ago
  What, a successful engineer with tenure at two of the most successful business operating at scale?
nbm 7y ago
It takes a few hours to learn to use the tools mentioned in the article (and many similar ones) to a level of competence - part of the appeal is how easy they are to use. It takes a few hours to learn how to get new data into and out of them as well. Both are the sort of thing taught in 45 minutes presentations and workshops during the Bootcamp training at FB, and then most people can just explore the rest themselves over time in normal usage. What takes months is feeling like you understand the most important tools and aspects of the environment to be as effective in them as those around you, or to match how you felt in an environment where the scope is smaller. It’s not like you’re sitting in classes for months - you’re doing your job (maybe even more effectively than normal due to the availability of these tools), but every once in a while you’re encountering something where you need to ask someone else or explore or watch a video about how to do something, and that might break your momentum.
bagacrap 7y ago

doggydogs94 7y ago

If the tool is valuable to you, make sure you know exactly how it works. Then you can recreate it, for yourself at least, whenever you go.

downerending 7y ago
Also, if you wrote it (as often happens in my case), put some effort into getting it open-sourced if possible, so that you can just use it elsewhere later. These days, I very rarely enjoy using tools that aren't open source, or at least generic and very widely available. I simply don't want to spend my limited brain cells learning a piece of software that I won't be able to use in the future. And in any case, large, internally developed tools are usually awful. Unless I wrote them. ;-)

superbatfish 7y ago

Couldn’t it be argued that all software is built with more “glue” than non-glue? That’s just the nature of the beast. If a user wants to, say, view a record from a remote database in her web browser, how many separate layers of code does that involve? (Several.) How many of them aren’t “glue”? (I dunno, like two?) Perhaps it’s true that most glue is poorly written, but doesn’t that imply that writing good glue is, in fact, difficult, and therefore it’s as hard — or maybe harder - than building “real” things?

yibg 7y ago

Nothing wrong with writing glue, if that’s what the business / company / market needs. Having said that, I do miss the data that’s available at FAANG companies (at least ones I’ve been at). I could dig up all sorts of system and business data, and really get good insight into both what my application and the users are doing. Tends to lead to a lot better and more clear decisions and thus helps to build the right thing.

bgdnyxbjx 7y ago

This topic is one of the primary reasons I enjoy working at MS these days. The tech stack is full of OSS or industry standard tools, not a bunch of internal NIH things. My current stack is: React, Node, Kubernetes, Linux (for dev and deploy), Git for sc, GitHub, vs code. I can’t even think of some internal tool that I depend on at the moment.

ddlutz 7y ago
That depends entirely on what part of MS you are in right? There is cosmos (internal, not exactly the same as external cosmos), autopilot for infra, mdm for metrics, mds for logs, icm for alerts, etc.
- bgdnyxbjx 7y ago
  Yeh, you’re right. I was thinking more of this and remember all the alerting and incident management stuff. Then I realized that I hate those tools and would happily leave them behind :)
zepolen 7y ago
Github is open source?
- deleted 7y ago
  [deleted]
- cnst 7y ago
  Like it or, GitHub is ubiquitous enough that noone really cares. If you do care, there's plenty of OSS alternatives that aren't much different in their UI than GitHub itself. You can easily put GitHub on your resume, and everyone would know what you're talking about. Of course, by itself, it's not much of an experience, but, surprisingly, it is often listed as a requirement and/or nice-to-have for certain SWE job posts.

kelnos 7y ago

I'm always so disappointed that we're all so closed when it comes to tooling. It's fortunate that some tools get open-sourced, but it's also astounding the amount of time and money that is wasted building the same tools over and over at each company. Sure, sometimes paying for a hosted solution for your CI, deployments, container platform, data pipeline, etc. isn't the most ideal route, depending on your business, but I'd much rather focus on building actual product over reinventing the deployment system wheel for the Nth time. > People from Google probably miss Borg (and lament the whole Kubernetes thing) or BigTable. I've never worked at Google, and thus have no experience with Borg, but Kubernetes is exactly what I want to see companies open-sourcing, or, if they must, selling. I don't see what there is to lament. Being able to start a new company and not have to reinvent and rebuild your own deployment/container platform is a huge gift. For the most part I find this blog to be insightful and fun to read, but this post IMO completely misses the mark. Given that I think the author works a lot on the operations side of the house, I guess I'm not surprised. Internal platform tools are there to serve the needs of the product teams; they don't exist just to make someone feel good about their ability to build something instead of buying instead, which she unfairly looks down on so disdainfully, as if it's not real engineering. Part of doing engineering is putting your ego aside when it's more cost-effective and robust to use someone else's solution instead of rolling your own.

godot 7y ago
I feel like the author's point and your point would be applicable a lot more say 5 to 8 years ago. There has been so many SaaS startups in the past 8 years or so where many things that used to require intense internal tooling are now available as a SaaS.

adreamingsoul 7y ago

I honestly miss the internal tooling at AWS. The build system and CI/CD pipelines used internally are truely amazing.

burpsnard 7y ago
Brazil? isn't it mostly there in work-alike saas?

SergeAx 7y ago

I have to strongly disagree. When doing business you are solving a business problems. Facebook, Google, Microsoft as business has grown so much they have problems of scale no open tools are able to handle. On the other hand, any kind of smaller company doesn't have those business problems, but a cost of having and maintaining in-house solutions of that scale will just effectively topple and sink them. This is why I'm always frowning during job interviews with engineers telling stories about they migrations from monolith to microservices and putting it all into k8s, while their headcount in the firs tens and their DAU in about 10000 users. Disclaimer: I worked in Yandex, virtually Russian Google. Loads and operations scale there are comparable with second rank international tech giants. Yandex has and have lots of internal tools to handle that load and scale, but I've almost never really needed them in smaller companies. Maybe "nice to have" couple of times.

cnst 7y ago
It's also a lot about the marketability of your skills. The only reason the OP can boast about the experience with these proprietary tools is because they're used by FAANG. No other reason — it's just an indirect and inconspicuous boasting about having worked at FAANG. If it was yet another proprietary non-OSS language used by one of the investment companies that has no direct analogues at any of their competitors — good luck boasting on your resume that you have skills in programming languages that noone cares about. Proprietary software sucks. Just because it's written and used internally by FAANG doesn't make it suck any less. When you're evaluating whether or not to join a company, presence of such tools should be more of a red flag, if anything. Same goes with software-as-a-service. Tools should be OSS, and easily portable between the workplaces. Another case in point — it kind of works both ways — many companies start around with Erlang and ejabberd, because it's a really great tool, but then, after having issues finding developers who can understand it properly, they eventually, one way or another, convert their stack back to the more common languages; I recall this has happened with Facebook Messenger and Heroku, and also at a random startup I was interviewing with during the course of my interview (they just hired a manager who knew Java — and, what do you know, he decides to set a direction to eventually replace all Erlang with Java).

zhte415 7y ago

I've recently branched out on my own. Never had great tooling, was in government or a F50 where chains of approvals for the slightest changes were cursed by all, including those doing the approving. But, there are some organisational things I miss: Having someone/a team to ensure network uptime is fantastic and that have the scale and experience to anticipate and mitigate potential trouble before it comes up. Likewise all infra in general. Having a legal department to ask "Can I do this?" where often the answer may be "We don't know" but that's still an answer. A security force, property, accounting, providing peace of mind. Someone making sure the projector's setup, tiny things that add up quickly. There are downsides to this, such 'organisation' often grinds things to a glacial pace, hence why I'm breaking out by myself now, but... not tooling, I can take care of that myself and I'm incredibly thankful for the SaaS ecosystem that would have made this impossible even a decade ago, or at least very inconvenient with 3rd party consultants providing such services for high fees and little more. There are a lot of things an organisation provides that we can take for granted, and having that experience provides a great bonus in expectations of robustness not only in technology but in all processes of a business. Tooling is, in the end, in the mind.

pjmlp 7y ago

The only thing I miss are teammates and some of the office moments we had together, but those luckily survive employers and we get to meet in other contexts. I never been attached to work tools, or technology fads, things are ephemeral, best practices of today are the legacy codebase that no one wants to work on tomorrow.

bullen 7y ago

I solved this problem 10 years ago by building my own open-source stack and only working with that stack: https://github.com/tinspin https://github.com/tinspin You should build your own tools, that you bring with you, that is what all artisans have done for all time. No reason to change that just because you can use the newest buss-word every week.

vermooten 7y ago

I'm very disappointed if I move from a company that uses macs, slack, Confluence to PCs, MS Teams (horrible) and SharePoint (worse).

schappim 7y ago
Why is MS Teams horrible? We have an engineer who is trying to convince our team to move from Slack to MS Teams. My only gripe is the requirement for a M$ login account (we are a Google Apps house).

sershe 7y ago

One problem that I've seen in one such large company, and heard of from some people in others (FAANGs), is that there's no incentive to maintain these internal tools. Sure, they often start out as sort of amazing and there's a ship email and for 1-3 years a strong team, but you are unlikely to be promoted a lot polishing UI, fixing bugs and adding features or even support for new backend systems... so, eventually there's a skeleton crew maintaining an old tool, everybody complains how it's slow and sucks and behind the times, and then a brand new replacement is built that is going to fix everything. There's a ship email, people gradually switch to it, rinse, repeat.

heelix 7y ago

The only thing I'd miss from a larger company is the travel budget. The 'devil may care' attitude to drop a couple grand on a short window flight... compared to travel with the small company where that meeting could break the company. Double goes for the in-house travel agent who sorted everything while in route - Jane... you were missed.

caseymarquis 7y ago

Every time I make a neat tool I open source it using the unlicense so I can use it elsewhere with no attribution. I'm happy to work at a company where this is possible. This doesn't apply to business critical code a competitor might want, just libraries and tools for making generic 'stuff'. Typically, I make the tools at home first and decide I want to use them at work, but I eventually end up patching them at work for company purposes.

SZJX 7y ago

This article makes little sense. "Engineering" by definition is some sort of "glue" work that reuses existing libraries/wheels and builds solutions out of them to address specific problems. I as an engineer am proud of this glue work that I do, just like the carpenters, plumbers, cathedral builders, doctors etc. If you really want to be unique and invent something, you should be in the academia and are probably in the wrong place to begin with. I have a friend who worked in the TensorFlow team and still quit to pursue a PhD. In his words "all engineering jobs are boring" from his perspective. I don't think being in a BigCorp or not makes much difference at all.