4 ms·

>> Amazon internal tooling (is that 'industry standard' enough?). I'm not sure, maybe? When I say industry-standard tools I mean "Best you can purchase." So I

>> Amazon internal tooling (is that 'industry standard' enough?). I'm not sure, maybe? When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn't hold a candle) Some problems are harder at scale (feature toggles interactions just get harder with size), but some of them are just a mess of their own making (restarting taking 30m, butterfly rules having >20m+ delays, eden not being optimized to work with buck, which needs to read tons of files out of tons of directories)

vlovich123 3y ago
GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. I found the tools at Meta made it a lot smoother and faster to write and review code. Also, their automated flaky test detection, suppression, & ticket issuance/resolution is no joke. One of my favorite features. Their feature flagging system was also on point. Also, Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you. Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them). I agree the observability tools I interfaced within Meta was subpar a few years ago, but I think you're being ungenerous to take that and extend it to the coding tooling specifically.
- erik_seaberg 3y ago
  Suppressing flaky tests is a little terrifying. Google would just retry them a few times, which is slow but only delays requesting a code review.
  - vlovich123 3y ago
    It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot). https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/ https://engineering.fb.com/2020/12/10/developer-tools/probab...
    - erik_seaberg 3y ago
      If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
      - vlovich123 3y ago
        What makes you think that’s not the case?
        erik_seaberg 3y ago
        Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
        vlovich123 3y ago
        I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on. > Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.
- jarjoura 3y ago
  Phabricator w/ "arc & jf" were my bae. Wish they would fully open source all of it or expose a public version of it to compete against GitHub.
- zug_zug 3y ago
  >> GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. ??
  - vlovich123 3y ago
    Completely consistent and unified UX and text editing (GitHub has this, atlassian doesn't). All text editing is rich multi-modal that is basically what you see in Facebook posts (i.e. you can add a bunch of images easily, it all previews immediately) w/ programmer-friendly extensions (e.g. code formatting, markdown w/ live preview etc). The ticketing system had a rich tagging system (which GitHub has). The notifications system was completely unified and extremely useful - Workplace activity, relevant ticket updates, PRs to review, reviews, etc - all relevant and timely. From what I've seen, GitHub tends towards just being a useless uncoordinated firehose. Finally code reviews. Having a code review for every commit is a bit to get used to but their tooling makes it super easy to post a stack (i.e. PR). Each commit in the PR can be independently reviewed & merged. This means you can land improvements progressively as code reviews complete (i.e. short cleanup commits go in faster while larger commits are reviewed longer) This also encourages people to size the commits appropriately & to be standalone.
- rsdbdr203 3y ago
  >Their feature flagging system was also on point. Which one... there are like 3 (or more)? The ones I worked with were the single biggest cause of SEVs. I wrote some tooling around trying to make it better, but it honestly needed to be completely replaced. >Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you. Yea? I found it horrible to have to read through post-after-post-after-post to try and keep up-to-date with what's going on. Not my way of ingesting information. >Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them). 100% disagree... telling my dog to bark at the neighbor because I want something from them is better than Workplace chat. Constantly switching between internal beta and production to try and get the features I wanted, with something stable. Go ahead and send me a link to that message in a chat... oh yea, you can't :-|

vlovich123 3y ago

GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. I found the tools at Meta made it a lot smoother and faster to write and review code. Also, their automated flaky test detection, suppression, & ticket issuance/resolution is no joke. One of my favorite features. Their feature flagging system was also on point. Also, Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you. Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them). I agree the observability tools I interfaced within Meta was subpar a few years ago, but I think you're being ungenerous to take that and extend it to the coding tooling specifically.

erik_seaberg 3y ago
Suppressing flaky tests is a little terrifying. Google would just retry them a few times, which is slow but only delays requesting a code review.
- vlovich123 3y ago
  It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot). https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/ https://engineering.fb.com/2020/12/10/developer-tools/probab...
  - erik_seaberg 3y ago
    If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
    - vlovich123 3y ago
      What makes you think that’s not the case?
      - erik_seaberg 3y ago
        Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
        vlovich123 3y ago
        I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on. > Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.
jarjoura 3y ago
Phabricator w/ "arc & jf" were my bae. Wish they would fully open source all of it or expose a public version of it to compete against GitHub.
zug_zug 3y ago
>> GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. ??
- vlovich123 3y ago
  Completely consistent and unified UX and text editing (GitHub has this, atlassian doesn't). All text editing is rich multi-modal that is basically what you see in Facebook posts (i.e. you can add a bunch of images easily, it all previews immediately) w/ programmer-friendly extensions (e.g. code formatting, markdown w/ live preview etc). The ticketing system had a rich tagging system (which GitHub has). The notifications system was completely unified and extremely useful - Workplace activity, relevant ticket updates, PRs to review, reviews, etc - all relevant and timely. From what I've seen, GitHub tends towards just being a useless uncoordinated firehose. Finally code reviews. Having a code review for every commit is a bit to get used to but their tooling makes it super easy to post a stack (i.e. PR). Each commit in the PR can be independently reviewed & merged. This means you can land improvements progressively as code reviews complete (i.e. short cleanup commits go in faster while larger commits are reviewed longer) This also encourages people to size the commits appropriately & to be standalone.
rsdbdr203 3y ago
>Their feature flagging system was also on point. Which one... there are like 3 (or more)? The ones I worked with were the single biggest cause of SEVs. I wrote some tooling around trying to make it better, but it honestly needed to be completely replaced. >Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you. Yea? I found it horrible to have to read through post-after-post-after-post to try and keep up-to-date with what's going on. Not my way of ingesting information. >Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them). 100% disagree... telling my dog to bark at the neighbor because I want something from them is better than Workplace chat. Constantly switching between internal beta and production to try and get the features I wanted, with something stable. Go ahead and send me a link to that message in a chat... oh yea, you can't :-|