4 ms·

Another random selection: * When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Chances are the next reorg is com

Another random selection: * When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Chances are the next reorg is coming faster than the lifetime of the identifier and renaming is often hard. * The industry leading tools will contain bugs. From Linux kernel to deploy tooling, there are bugs everywhere. Part of your job is to identify and work around them until upstream patches make it to you if ever. * Maintaining a patched fork is usually more expensive than setting up a workaround * Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time. * Bought SaaS will break production in the middle of the night. Your own team will have the best context and motivation to fix/workaround them. When choosing a vendor, include the visibility into their internal monitoring as a factor for disaster recovery (exported metrics and logs of their control plane for example)

vladvasiliu 4y ago
> * Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time. If only they'd tell you. We had this exact issue on AWS. Seemingly random packet drops. Metrics on both clients and servers were ok, latency specifically was very low when it worked. Call up support "yeah, you're running into our connection limit". "Oh. What's that limit?" "yeah, I can't tell you that". His solution was that, since this was somehow related to connection tracking in the security group, I could set this to allow all/all, and set up filtering at the NACL level. Turns out I could do it for this particular issue. This was before there was a possibility to monitor this [0]. Called up our customer manager. "Let me check". A few days later, "yeah, that's not something we divulge". --- [0] For those who don't know, it's now possible to keep an eye on refused connections (at least on Linux). https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html#network-performance-metrics https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitori... -> conntrack_allowance_exceeded
- toast0 4y ago
  Ran into that one too, but my service rep didn't mention the possibility of configuring connectionless firewall rules. I'm still bitter many years later.
  - vladvasiliu 4y ago
    It wasn't the service rep who suggested that, but the support guy after I opened a ticket. I think this is an internal policy issue because he actually gave me the impression of trying to be helpful. The service rep has been, overall, useless, when he wasn't actually less informed than I was (my knowledge being based on the AWS website).
anilakar 4y ago
> When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Naming in general is hard. If you name stuff based on location, use an identifier that won't change, like provider datacenter names, street addresses or customer building codes, not the current tenant or purpose of use. For products, come up with an internal product/project name and stick to it in everything that is not immediately visible to the customer. At one point you could see the current and three previous names of our product if you popped out an iframe and opened the inspector (logo with name, page title, URL and prefixed log messages). > Maintaining a patched fork is usually more expensive than setting up a workaround When your bosses demand additional features a single customer requested, you absolutely have to make them understand that the functionality must be added to the main product.
- throwawaaarrgh 4y ago
  My naming convention is like this: - anything my company doesn't create or own, is just called whatever it is - logical components that aren't specific to the org chart can be whatever you want - anything org chart specific gets a randomly generated code name. an internal website allows you to register and look up any code name across the company. this allows you to find who the hell owns server "ws-prod" in a 6 year old account that nobody seems to maintain. instead you find "peanutcar-ws-prod" and can then look up who registered "peanutcar". (this also prevents the bu-org-group-product-subproduct-env mess that eventually runs over character limits) - doesn't matter if the rest of the company doesn't do it, I do it for what I manage. later on if it gets adopted, fine, but if not, at least I won't ever have to rename my crap.

vladvasiliu 4y ago

> * Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time. If only they'd tell you. We had this exact issue on AWS. Seemingly random packet drops. Metrics on both clients and servers were ok, latency specifically was very low when it worked. Call up support "yeah, you're running into our connection limit". "Oh. What's that limit?" "yeah, I can't tell you that". His solution was that, since this was somehow related to connection tracking in the security group, I could set this to allow all/all, and set up filtering at the NACL level. Turns out I could do it for this particular issue. This was before there was a possibility to monitor this [0]. Called up our customer manager. "Let me check". A few days later, "yeah, that's not something we divulge". --- [0] For those who don't know, it's now possible to keep an eye on refused connections (at least on Linux). https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html#network-performance-metrics https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitori... -> conntrack_allowance_exceeded

toast0 4y ago
Ran into that one too, but my service rep didn't mention the possibility of configuring connectionless firewall rules. I'm still bitter many years later.
- vladvasiliu 4y ago
  It wasn't the service rep who suggested that, but the support guy after I opened a ticket. I think this is an internal policy issue because he actually gave me the impression of trying to be helpful. The service rep has been, overall, useless, when he wasn't actually less informed than I was (my knowledge being based on the AWS website).

anilakar 4y ago

> When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Naming in general is hard. If you name stuff based on location, use an identifier that won't change, like provider datacenter names, street addresses or customer building codes, not the current tenant or purpose of use. For products, come up with an internal product/project name and stick to it in everything that is not immediately visible to the customer. At one point you could see the current and three previous names of our product if you popped out an iframe and opened the inspector (logo with name, page title, URL and prefixed log messages). > Maintaining a patched fork is usually more expensive than setting up a workaround When your bosses demand additional features a single customer requested, you absolutely have to make them understand that the functionality must be added to the main product.

throwawaaarrgh 4y ago
My naming convention is like this: - anything my company doesn't create or own, is just called whatever it is - logical components that aren't specific to the org chart can be whatever you want - anything org chart specific gets a randomly generated code name. an internal website allows you to register and look up any code name across the company. this allows you to find who the hell owns server "ws-prod" in a 6 year old account that nobody seems to maintain. instead you find "peanutcar-ws-prod" and can then look up who registered "peanutcar". (this also prevents the bu-org-group-product-subproduct-env mess that eventually runs over character limits) - doesn't matter if the rest of the company doesn't do it, I do it for what I manage. later on if it gets adopted, fine, but if not, at least I won't ever have to rename my crap.