8 ms·

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” commen

I think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.

hashkb 6y ago
They are a publicly traded company. They have a team dedicated to engineering support. A better article would include a management and hiring postmortem. It's shocking, really. Humility is nice, but competency is also nice.
- ponker 6y ago
  They did a billion dollars of revenue last year, their management and hiring systems seem to be getting the job done.
  - sk5t 6y ago
    Yep--although multiple unpleasant experiences with pinterest have spurred me to permaban it from search engine results and smite it with network filters, somewhat wasteful CI/CD pipelines have clearly not prevented the company from flourishing.
- Aeolun 6y ago
  I don’t know, I consider myself fairly competent but I’d never even considered that. It’s just not so relevant until your repo is multiple gigabytes big. Still, I’ll see if it works for our pipelines, and we can get our clone from 20s to 1s
  - hashkb 6y ago
    Good teams profile everything. This team's only goal is to support other engineers. Build time is a huge issue for every ops team. Missing this for so long is wasted money that's easy to calculate. We can be nice to people while still having high standards. It's a missed opportunity for a deeper postmortem, and it's bland content at best.
    - deleted 6y ago
      [deleted]
    - deleted 6y ago
      [deleted]
    - jacquesm 6y ago
      I think you live in a highly theoretical parallel universe. The one I occupy is the one where 'good teams' profile those things that take too long. Take yourself as an example: in spite of the wide availability of free certificates you are still hosting your domain without using a secure transport layer. Some would take that as incompetence. Others would assume you have more stuff on your plate rather than that you don't have high standards.
      - sdoering 6y ago
        And others would assume, that there are other philosophies out there regarding ssl everywhere. So the question is, who's POV has more validity and logical rigor attached to it. I actually can't see any side winning here on a purely logical level. Only on an ideological level. At least as long as we are talking about consuming public information. Am I in favor of the aggressiveness of OP in other posts? No. Am I using SSL myself. Hell yes. Nonetheless, I understand that there are people who feel that consuming public information like on a private homepage is nothing that necessitates using SSL. Even if I myself have a different ideology/value set governing my decision. I once heard the comparison that it is like the difference of sending a letter and sending a picture postcard. Not sure if I buy into that, but I can't argue against it on a purely rational basis.
        hashkb 6y ago
        Yes, it's by choice. It's read only, public information. We don't set cookies or anything. We take security very seriously. But we don't take anything too seriously. Edit: by the way, persuade me that there's an upside and I'll turn it on.
      - dmurray 6y ago
        > The one I occupy is the one where 'good teams' profile those things that take too long. Deciding which things take too long is profiling. Maybe you do it in your head or with pencil and paper instead of using a software approach but I think your position aligns with "good teams profile everything".
- argc 6y ago
  This is neither incompetence nor surprising. Maybe you’ve only worked at large companies who have had time to optimize things for years (and even then, I see grotesque software decisions at my large company quite often). Try accepting that software is often written poorly optimized on the first pass, for good reason, and learn to celebrate the wins without needing to shame someone.
  - hashkb 6y ago
    This is Pinterest. Every org I've worked at has been smaller. There's space between shame and ignoring mistakes. The purpose of this post is not to educate. There's nothing in here that anyone can use to improve. It's just marketing.
    - phist_mcgee 6y ago
      Well I learned something from this article, and I thought I had a good handle on CI/CD, so either I am incredibly stupid and shouldn't be reading these 'nothing' articles, or maybe there is so much to learn it's impossible to know it all.
      - sytringy05 6y ago
        If you dont have massive repos, then this is the sort of thing that not often a big problem. Also - if you are using things like gitlab runners, you might be in the same AZ and even large repos clones are fast. And it is impossible to know it all, I like these articles just for the differing ways people work
        phist_mcgee 6y ago
        It sounds like GP has a bit of a chip on their shoulder when it comes to feeling like you're not worthy if you don't know everything.
    - stjohnswarts 6y ago
      Are you just an angry ex Pinterest employee? There is something to learn here and a reminder to pay more attention even when you're knee deep in other tasks. Besides the obvious feature/limitation of git that the author points out.
- swsieber 6y ago
  People praise my git skills at work (among other things) when they come to me for git help. My response is always the same: I've just run into these bugs more often than they.
- fizixer 6y ago
  Fully agree, and the mindless HN downvote sheep bandwagon is in full effect. Working at Pinterest and acting humbled by learning basic stuff on the job? I play the world's smallest violin for how hard and stressful your work is. Poor babies. Somebody give this engineering team a participation trophy.
- jedberg 6y ago
  Why are you so angry about this? You've commented throughout this post about how this is boring and the Pinterest team is incompetent. Why? I found it quite interesting. I've been working in deployments for over 20 years at some pretty big places, and never really though about this before. I now have a new tool in my toolbox, and I'm quite happy about it.
  - hashkb 6y ago
    In general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry... The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering posts to brag about fixing our own mistakes. It's a terrible shame and I speak up about it to remind everyone there was a time when RTFM would be the only response to this. Edit: rate limited but one last thing: are we this forgiving of Equifax when they oopsie our data? Seeing this would immediately make me wonder if anything I have shared with Pinterest is safe. That's why they owe us a postmortem and not a thirst trap.
    - pavel_lishin 6y ago
      > The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. I have not noticed, in the past decade, any move away from tech screens whatsoever.
      - uglycoyote 6y ago
        Are we talking about technological screens, like LCD screens here?
        xdavidliu 6y ago
        I believe they mean screens in the sense of technical interviews to assess candidates' technical ability.
- kevinventullo 6y ago
  A team discovers a major efficiency win requiring minimal engineering effort and your response is to... punish them?
- Hnrobert42 6y ago
  Shocking? Incompetent? A hiring postmortem? Really? AYFKM?
projektfu 6y ago
I'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on StackOverflow? Hire a consultant?
- pushrax 6y ago
  I would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git. Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.
  - marta_morena_28 6y ago
    It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.
    - oefrha 6y ago
      Yep, I’m shocked that it had to be bloated to 40min before they even thought about fixing it. Anyone who has used Jenkins for nontrivial builds must have had the experience of staring at the slowly expanding session log screen? It doesn’t take any “profiling” to realize git clone’s taking forever.
    - pushrax 6y ago
      It takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.
bald42 6y ago
I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.
- mschuster91 6y ago
  Ephemeral CI runners. I have the same problem at work - 4GB repository that is redownloaded on every single pipeline run. Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene. An ephemeral runner using docker:dind however? It simply gets removed.
- ForHackernews 6y ago
  I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.
  - DougBTX 6y ago
    It is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.

hashkb 6y ago

They are a publicly traded company. They have a team dedicated to engineering support. A better article would include a management and hiring postmortem. It's shocking, really. Humility is nice, but competency is also nice.

ponker 6y ago
They did a billion dollars of revenue last year, their management and hiring systems seem to be getting the job done.
- sk5t 6y ago
  Yep--although multiple unpleasant experiences with pinterest have spurred me to permaban it from search engine results and smite it with network filters, somewhat wasteful CI/CD pipelines have clearly not prevented the company from flourishing.
Aeolun 6y ago
I don’t know, I consider myself fairly competent but I’d never even considered that. It’s just not so relevant until your repo is multiple gigabytes big. Still, I’ll see if it works for our pipelines, and we can get our clone from 20s to 1s
- hashkb 6y ago
  Good teams profile everything. This team's only goal is to support other engineers. Build time is a huge issue for every ops team. Missing this for so long is wasted money that's easy to calculate. We can be nice to people while still having high standards. It's a missed opportunity for a deeper postmortem, and it's bland content at best.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - jacquesm 6y ago
    I think you live in a highly theoretical parallel universe. The one I occupy is the one where 'good teams' profile those things that take too long. Take yourself as an example: in spite of the wide availability of free certificates you are still hosting your domain without using a secure transport layer. Some would take that as incompetence. Others would assume you have more stuff on your plate rather than that you don't have high standards.
    - sdoering 6y ago
      And others would assume, that there are other philosophies out there regarding ssl everywhere. So the question is, who's POV has more validity and logical rigor attached to it. I actually can't see any side winning here on a purely logical level. Only on an ideological level. At least as long as we are talking about consuming public information. Am I in favor of the aggressiveness of OP in other posts? No. Am I using SSL myself. Hell yes. Nonetheless, I understand that there are people who feel that consuming public information like on a private homepage is nothing that necessitates using SSL. Even if I myself have a different ideology/value set governing my decision. I once heard the comparison that it is like the difference of sending a letter and sending a picture postcard. Not sure if I buy into that, but I can't argue against it on a purely rational basis.
      - hashkb 6y ago
        Yes, it's by choice. It's read only, public information. We don't set cookies or anything. We take security very seriously. But we don't take anything too seriously. Edit: by the way, persuade me that there's an upside and I'll turn it on.
    - dmurray 6y ago
      > The one I occupy is the one where 'good teams' profile those things that take too long. Deciding which things take too long is profiling. Maybe you do it in your head or with pencil and paper instead of using a software approach but I think your position aligns with "good teams profile everything".
argc 6y ago
This is neither incompetence nor surprising. Maybe you’ve only worked at large companies who have had time to optimize things for years (and even then, I see grotesque software decisions at my large company quite often). Try accepting that software is often written poorly optimized on the first pass, for good reason, and learn to celebrate the wins without needing to shame someone.
- hashkb 6y ago
  This is Pinterest. Every org I've worked at has been smaller. There's space between shame and ignoring mistakes. The purpose of this post is not to educate. There's nothing in here that anyone can use to improve. It's just marketing.
  - phist_mcgee 6y ago
    Well I learned something from this article, and I thought I had a good handle on CI/CD, so either I am incredibly stupid and shouldn't be reading these 'nothing' articles, or maybe there is so much to learn it's impossible to know it all.
    - sytringy05 6y ago
      If you dont have massive repos, then this is the sort of thing that not often a big problem. Also - if you are using things like gitlab runners, you might be in the same AZ and even large repos clones are fast. And it is impossible to know it all, I like these articles just for the differing ways people work
      - phist_mcgee 6y ago
        It sounds like GP has a bit of a chip on their shoulder when it comes to feeling like you're not worthy if you don't know everything.
  - stjohnswarts 6y ago
    Are you just an angry ex Pinterest employee? There is something to learn here and a reminder to pay more attention even when you're knee deep in other tasks. Besides the obvious feature/limitation of git that the author points out.
swsieber 6y ago
People praise my git skills at work (among other things) when they come to me for git help. My response is always the same: I've just run into these bugs more often than they.
fizixer 6y ago
Fully agree, and the mindless HN downvote sheep bandwagon is in full effect. Working at Pinterest and acting humbled by learning basic stuff on the job? I play the world's smallest violin for how hard and stressful your work is. Poor babies. Somebody give this engineering team a participation trophy.
jedberg 6y ago
Why are you so angry about this? You've commented throughout this post about how this is boring and the Pinterest team is incompetent. Why? I found it quite interesting. I've been working in deployments for over 20 years at some pretty big places, and never really though about this before. I now have a new tool in my toolbox, and I'm quite happy about it.
- hashkb 6y ago
  In general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry... The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering posts to brag about fixing our own mistakes. It's a terrible shame and I speak up about it to remind everyone there was a time when RTFM would be the only response to this. Edit: rate limited but one last thing: are we this forgiving of Equifax when they oopsie our data? Seeing this would immediately make me wonder if anything I have shared with Pinterest is safe. That's why they owe us a postmortem and not a thirst trap.
  - pavel_lishin 6y ago
    > The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. I have not noticed, in the past decade, any move away from tech screens whatsoever.
    - uglycoyote 6y ago
      Are we talking about technological screens, like LCD screens here?
      - xdavidliu 6y ago
        I believe they mean screens in the sense of technical interviews to assess candidates' technical ability.
kevinventullo 6y ago
A team discovers a major efficiency win requiring minimal engineering effort and your response is to... punish them?
Hnrobert42 6y ago
Shocking? Incompetent? A hiring postmortem? Really? AYFKM?

projektfu 6y ago

I'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on StackOverflow? Hire a consultant?

pushrax 6y ago
I would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git. Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.
- marta_morena_28 6y ago
  It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.
  - oefrha 6y ago
    Yep, I’m shocked that it had to be bloated to 40min before they even thought about fixing it. Anyone who has used Jenkins for nontrivial builds must have had the experience of staring at the slowly expanding session log screen? It doesn’t take any “profiling” to realize git clone’s taking forever.
  - pushrax 6y ago
    It takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.

bald42 6y ago

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

mschuster91 6y ago
Ephemeral CI runners. I have the same problem at work - 4GB repository that is redownloaded on every single pipeline run. Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene. An ephemeral runner using docker:dind however? It simply gets removed.
ForHackernews 6y ago
I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.
- DougBTX 6y ago
  It is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.