4 ms·

> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc… This gives the lie t

> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc… This gives the lie to Jeff Bezos' whole "Day 1" philosophy. If you're at a Day 1 company you don't spend 40% of your time fighting with the crappy internal tools.

dilyevsky 4y ago
Not sure what the Amzn situation is but something I've noticed at few previous gigs that there're some folks who claim they spent significant time wrestling with tooling but when you actually go in and make them formulate the issue it's either something really obvious or a 10-min google/github search like 9/10 times. These also weren't coming from people who I considered lazy or unintelligent (well, for the most part) so I can't really explain it logically. I suspect it may have something to do with motivation or cultural issues ("not my job" sort of folks).
- zug_zug 4y ago
  I've seen people complain about it who were wrong, but I've also seen people fail to complain about it when they really should have been. I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
  - dilyevsky 4y ago
    Your example does not sound as outrageous as you seem to think - I can imagine a number of scenarios where this would be totally acceptable solution assuming we’re talking about software engineers here. Consider the fact that most startups don’t even have dedicated teams running CI unlike at fangs and it’s up to devs to set everything up. Yet I don’t really hear about folks complaining about internal tooling at early startups
    - zug_zug 4y ago
      The fact that you don't think that's ridiculous is a great example of terrible tooling be accepted as "normal." Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for. But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day). If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
    - robertlagrant 4y ago
      If you're smaller you'd hopefully use something like Datadog, where you can easily ingest logs and have alert triggers from log lines. It's probably harder at bigger firms, where internal politics get in the way of having nice things.
- adamwk 4y ago
  When it’s an internal tool you can’t google it. You search through internal wikis that haven’t been updated in years
  - dilyevsky 4y ago
    When I worked at google you could =) j/k moma was terrible but most issues were easily searchable via a combination of email list/bug tracker/codesearch
  - gruffle 4y ago
    Or you reach out to the owning teams/oncalls and half the time they have no idea how their own code/product works for anything nontrivial because most of it was written several generations of attrition ago.
pinewurst 4y ago
I think what "Day 1" means at Amazon is that you're always still finding your footing, until you quit.
- difflens 4y ago
  Haha, I found this quite funny :) I'm sure there's an element of truth to it considering how complicated the code base must be for the various services
mandeepj 4y ago
> This gives the lie to Jeff Bezos' whole "Day 1" philosophy. You can also interpret it like - “always start from scratch. You don’t have anything”
munchler 4y ago
"Day 1" is just motivational poster nonsense, anyway. No company can reinvent itself every day.

dilyevsky 4y ago

Not sure what the Amzn situation is but something I've noticed at few previous gigs that there're some folks who claim they spent significant time wrestling with tooling but when you actually go in and make them formulate the issue it's either something really obvious or a 10-min google/github search like 9/10 times. These also weren't coming from people who I considered lazy or unintelligent (well, for the most part) so I can't really explain it logically. I suspect it may have something to do with motivation or cultural issues ("not my job" sort of folks).

zug_zug 4y ago
I've seen people complain about it who were wrong, but I've also seen people fail to complain about it when they really should have been. I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
- dilyevsky 4y ago
  Your example does not sound as outrageous as you seem to think - I can imagine a number of scenarios where this would be totally acceptable solution assuming we’re talking about software engineers here. Consider the fact that most startups don’t even have dedicated teams running CI unlike at fangs and it’s up to devs to set everything up. Yet I don’t really hear about folks complaining about internal tooling at early startups
  - zug_zug 4y ago
    The fact that you don't think that's ridiculous is a great example of terrible tooling be accepted as "normal." Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for. But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day). If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
  - robertlagrant 4y ago
    If you're smaller you'd hopefully use something like Datadog, where you can easily ingest logs and have alert triggers from log lines. It's probably harder at bigger firms, where internal politics get in the way of having nice things.
adamwk 4y ago
When it’s an internal tool you can’t google it. You search through internal wikis that haven’t been updated in years
- dilyevsky 4y ago
  When I worked at google you could =) j/k moma was terrible but most issues were easily searchable via a combination of email list/bug tracker/codesearch
- gruffle 4y ago
  Or you reach out to the owning teams/oncalls and half the time they have no idea how their own code/product works for anything nontrivial because most of it was written several generations of attrition ago.

pinewurst 4y ago

I think what "Day 1" means at Amazon is that you're always still finding your footing, until you quit.

difflens 4y ago
Haha, I found this quite funny :) I'm sure there's an element of truth to it considering how complicated the code base must be for the various services

mandeepj 4y ago

> This gives the lie to Jeff Bezos' whole "Day 1" philosophy. You can also interpret it like - “always start from scratch. You don’t have anything”

munchler 4y ago

"Day 1" is just motivational poster nonsense, anyway. No company can reinvent itself every day.