17 ms·

Common Infrastructure Errors I've Made

246 points by eandre 5y ago

grafelic 5y ago
> You spring back to the present day, almost bolting out of your chair to object, "Don't do X!". Your colleagues are startled by your intense reaction, but they haven't seen the horrors you have. They may be startled, but they almost certainly won't listen. The purgatory nature of IT work culture ensures this repetitive pattern.
- doctor_eval 5y ago
  That’s my experience too. The older I get the more frustrated I am that people don’t want to learn from my mistakes. It’s not that I’m a jerk about it. Most of the time it’s business types saying “you don’t need to do all that stuff - just deploy it like that, it’ll be fine”. And then it’s not fine - and also, it’s somehow my fault. It seems to be more about personal aggrandisement (“I’m the boss and my word is law”) rather than trying to build a great business together. I’m pretty over it.
  - worik 5y ago
    It is easier to right than to read. It is easier to talk than to listen. It is easier to build new than adapt. All three of those statements are false. But all three are seductive
    - cstoner 5y ago
      > It is easier to right than to read. I just wanted to point out that I found that particular error really funny given the context.
      - worik 5y ago
        Leaving it!
SkipperCat 5y ago
Not sure if I agree about the Python jab. I've seen "pip install ...." run flawlessly more times than I've had breakfast cereal, and I eat a lot of cereal. I kinda agree on his first point about migrating stuff to the cloud but if you've done your deployments on like-like platforms (on prem containers to cloud containers) its not that bad.
- jddioeow 5y ago
  I use python professionally and can't make sense of this comment. I'm doing my thing, running my ls, cd, greps and cats, and now I need to run my python cli.... And so I do pip install in which env exactly? The system one? Or do I have to create a venv just to install my python cli, and have to enable it every time I wanna run the cli? It's easy to say "just pip install", but I can't see how the details make sense in the context of a cli. Explain please?
  - password4321 5y ago
    https://asdf-vm.com/ https://asdf-vm.com/ to the rescue! Well, maybe it's the latest in a long line of options...
    - volker48 5y ago
      I started using asdf instead of homebrew for installing and managing Python and I haven't looked back. Asdf is infinitely better than homebrew python.
  - maleldil 5y ago
    You can use pipx, which will create a new venv for each tool it installs. You call the program normally afterwards, and it will use the correct interpreter. There's not much wrong with just installing the tool to the global (user) interpreter, though.
  - SkipperCat 5y ago
    Let's take the AWS CLI as an example. Run "pip3 install awscli --upgrade --user" and you're done. Drop the "--upgrade --user" and you've installed it globally. Easy Peezy. Sure, you can do a lot with venv, Anaconda and whatnot, but if you have a well written package, it can be very portable without the need of environments.
tex0 5y ago
That's a seriously good and honest list. Thank you.
privacyonsec 5y ago
> Nobody knows how to correctly install and package Python apps Anybody tried PyInstaller ? it packages the whole Python project, dependencies included into a single exactable binary
tommiegannert 5y ago
> Soon I was auditing new services for "multi-cloud compatibility", ensuring that instead of using the premade SDKs from AWS, we maintained our own. I wonder if a useful middle-ground is to have lint checker rules to enforce using a blessed subset of a cloud provider's SDK. So that some thought/effort must be put into using a new feature?
raffraffraff 5y ago
Especially the alerts thing. I think every company I've ever worked for made the mistake of ignoring alert spam. If an alert doesn't require human action, then it should be a log or a metric. And by all means plot it on a graph (the metric that triggered the alarm, or in the case of a boolean test result, the frequency of the failure). Look at the graph during real incidents if you want. Talk about it at the monthly meeting. But don't generate an alert that people should ignore. You're playing Russian Roulette.
cloudengineer94 5y ago
I work with Azure and my issue is always problems with the Azure Keyvault.. At a personal deployment, it's always passwords I always forget to save them because I'm juggling with tons of things at once lol.
hdjjhhvvhga 5y ago
> If you are in AWS, don't pretend that there is a real need for your applications to be deployable to multiple clouds. If AWS disappeared tomorrow, yes you would need to migrate your applications. But the probability of AWS outliving your company is high Well, it's not about AWS shutting down at all! It's about them having complete control over your infrastructure, so they dictate the terms. This has many consequences: (1) they can raise prices and you can do absolutely nothing about it, (2) since you chose AWS with its dynamic pricing instead of flat-rate dedicated servers, each expansion (traffic, new services) is a cost for you. This means at some point you will realize you will save sick amounts of money if you switch to bare metal (as several notable companies have done). Except that at this point it's really difficult because you have to basically start from zero so the inertia basically pulls you into continuing this vicious cycle. So this is just a straw-man argument. Really, I haven't heard anyone saying "but Amazon can go out of business", it's just ridiculous.
odiroot 5y ago
The post has really great advice but... > Don't write internal cli tools in python Disagree completely with this. This has been probably the biggest overall boost for both engineers and operators at a few companies, I worked at. You deliver fast, it's easy to debug, and requires no compilation -- which is usually a bigger hassle than any Python-specific problem. It gets really important if you have operators on Linux/Windows/Mac.
- w-j-w 5y ago
  Have you tried go lately? It needs to "compile", but the compiler is so fast that the combined compile/run is essentially the same experience as running a script. goreleaser can also make cross compilation extremely easy.
gorgoiler 5y ago
Late to the party, but here’s a solution to “Python packaging” that works fantastically for all my stuff and requires four lines of setup and (only) one magical invocation — pip: $ cat dogclock/__init__.py import arrow # Example dependency def bark(): print(*(["woof"] * arrow.get().hour)) $ cat scripts/dogclock #!/usr/bin/env python3 import dogclock dogclock.bark() $ cat setup.py from setuptools import setup setup( install_requires=['arrow'], packages=['dogclock'], scripts=['scripts/dogclock']) $ pip3 install . … $ dogclock woof woof woof woof woof woof
jl6 5y ago
> Don't Design for Multiple Cloud Providers This has its own sub-antipattern: “Just put your application in a container, then it will run anywhere!”
dijit 5y ago
Regarding being cloud provider agnostic: it’s not always for fault tolerance, there can be a couple different reasons. 1) it gives your company a stronger bargaining position with the cloud provider. Granted, my companies tend to have extremely high spend- but being able to shave a dozen or so percent off your bill is enough to hire another 50 engineers in my org. 2) you may end up hitting some kind of unarguable problem. These could be business driven (my CEO doesn’t like yours!), technical (GCP is not supported by $vendor) or political (you need to make a China version of your product, no GCP in China!) Everything is trade offs. AWS never worked for us because the technical implementation of their hypervisor was not affined to CPU cores of the machine, meaning you often compete with other VMs on memory bandwidth. — but AWS works in China (kinda). So my solutions support both GCP and AWS as slightly less supported backup.
- sneak 5y ago
  The guy also recommends using several proprietary AWS services in other points. Then he goes on to advocate against designing for cloud flexibility. This almost feels like AWS marketing.
- kgeist 5y ago
  >you may end up hitting some kind of unarguable problem. Another example: there's multiple countries (for example, here in Russia) where personal data must be stored in data centers located in the country's borders and not every country has a AWS datacenter on its soil .
- cconstantine 5y ago
  I'd add another reason: Devs need to be able to run stuff locally sometimes. It's neat having a serverless single page app that is hosted in s3, served through cloudfront, with lambda's that post messages to sqs queues that are read by god knows what else, but what happens when there's a bug? How do you test it? You can throw more cloud at it and give each dev a way to build their own copy of the stack, but that's even more work to manage. Maybe localstack behaves the same, but can you integrate it with your test framework? I never took a hard "we must never use aws-only services" approach, but having the ability to run something locally was a huge plus. Postgres RDS? Totally fine, you don't need amazon to run postgres. Redshift? Worth the lock-in given the performance. Lambda? Eh, probably not, given that we already have a streamlined way to host a webapp.
betaby 5y ago
0) Don't write software - outsource it to someone smarter.
- barbazoo 5y ago
  -1) Don't outsize writing software. Let someone smarter outsource it for you.
  - doctor_eval 5y ago
    It’s the business equivalent of an AbstractButtonFactoryFactoryFactory.
- charcircuit 5y ago
  They don't even need to be smarter. They just need to value their time less than you do.
klodolph 5y ago
> Don't migrate an application from the datacenter to the cloud Reading the actual text of this one I get a different impression, but I'm still not sure I agree with this one. Applications can be radically different from each other in terms of how they are run. At one company, we ran a simple application as SaaS for our customers or gave them packages to run on-prem. We'd stack something like seven SaaS customers on a single set of hardware (front-ends and DB servers). The cloud offering was a no-brainer, you can just migrate customers one by one to AWS or whatever, or spin up a new customer on AWS instead of in our colocation center. Applications have a very wide range of operational complexity. Some applications are total beasts--you ask a new engineer to set up a test environment as part of on-boarding and it takes them a week. Some applications are very svelte, like a single JAR file + PostgreSQL database. The operational complexity (complexity of running the software) doesn't always correspond to the complexity of the code itself or its featureset.
- shoo 5y ago
  > I've been involved now in three attempts to do large-scale migrations of applications written for a specific datacenter to the cloud and every time I have crashed upon the rocks of undocumented assumptions about the environment I've only participated in a single on-prem to cloud migration. Some parts of the migration were easy, e.g. moving a postgres DB that was running on some on-prem linux server to run in AWS RDS. Some parts were rather unpleasant: where you discover that a bunch of the application code that runs in worker processes assumes it has access to a shared CIFS network share that can be used for communication throug the filesystem, and absolute file paths to on-prem CIFS network share locations are stored in metadata throughout the database. So then your available moves for how to migrate the application code and migrate the CIFS network share and migrate the data in the database all become somewhat tangled together.
- AyyWS 5y ago
  I helped migrate an app from on-prem to cloud. During the migration we found that the app needed a locally installed oracleDB. Well, it violates on-prem best practices and cloud best practices. I think migrating just exposes all the shortcuts baked into a "craplication."
timwis 5y ago
Bummer about Python :/ it’s my go-to for CLI tools, but I’ve seen that problem too.. pipenv helps, but I wonder if there’s a better way to package them so they’re more future proof.. or do I really need to learn go?
- GVRV 5y ago
  If the standard tech stack at your organisation includes Python, there's no reason why you shouldn't write CLI tools in Python. Packaging and distribution is only a problem for organisations that do not usually deal with Python.
- i_like_apis 5y ago
  IMO we loose a lot with go: having to compile, loosing the interactive shell, etc. Best case you work with a lot of people who know how to install python and use pip. Many people whine on boards, but it's not that complicated, especially with python 3.
  - cconstantine 5y ago
    > install python and use pip Now you've given your users 2 issues completely unrelated to the problem they're trying to solve. If you can't give your users a single simple command, or a single file to download your tool is too complicated to install.
    - p_l 5y ago
      Not to mention all kinds of python version issues, especially given that botched 2-to-3 migration means there's a lot more Python 2 than there should be.
      - worik 5y ago
        Yes. Except. If I have a Python2 tool I paid $X for, and now I need to to change it for $Y because, because, because no reason. Tough luck! Stop "whining" and write a cheque!
travisd 5y ago
> Don't write internal cli tools in python 100%. I stopped writing anything that had to be deployed (basically everything except Jupyter notebooks for data stuff) in Python because it’s truly a nightmare. Go and goreleaser is great for writing a CLI (and if it’s public, it can auto generate binaries and upload to GitHub, create a Homebrew/Scoop bucket, etc)
- apple4ever 5y ago
  I've written tons of internal tools with Python, and I have no problem getting it deployed. Just keep them up to date with changes and always use system provided packages, and there is no issue.
- throwaway984393 5y ago
  For the past 3 years I've written almost all of my users' system tools in Bash v3. Nobody has yet reported a bug to me. Runs fine on Linux, Mac, and WSL, on every architecture. Single file, easy for anyone to read and edit, gets the job done. I don't even have to care that it's unpopular because there are no installation steps. Just 'bash foo.sh'. Sometimes that means they need to download some Go app that my script uses, but that's much better than me having to write and support all that other code. If I have to do weird things with data structures or complex logic, I reach for Python, but I do it one of two ways: 1) so stupid that "python foo.py" works (no external deps), or 2) publish it to pypi so the user can just 'pip install foobar'.
  - exdsq 5y ago
    I love bash. It's probably my favourite language for productivity. I spent some time a while back trying to write a Lisp in bash, and have written a ~1kloc performance testing tool to get results while an entire team worked on a Haskell implementation.
  - raffraffraff 5y ago
    I'm also a huge bash fan, and agree that distributing Python tools is a mess. But "Runs fine on Linux, Mac and WSL" is a stretch. Bash on Mac OS is a ball of shit unless you're installing a newer GNU bash via homebrew. Mac OS bash is just too old. I can't even remember the list of stuff that breaks because I stopped trying. If the target audience is mostly Mac users I'll just write zsh to begin with.
torton 5y ago
> Don't write internal cli tools in python A lot of the advice is good but I take an issue with this one. With poetry and docker, packaging Python apps for easy consumption is a non-issue. Same for Ruby. If you can get your team to standardize on poetry, you might not even need containers -- but, honestly, running these tools from CI or automation (anywhere) is so useful that you probably want container versions anyway. Golang is not a good fit for exploratory CLIs that work with complex data structures and are written for one-off, low-CPU consumption purposes -- not for scale-up API services. Just having an interactive shell (or `pry` in Ruby, those two are identical for the purpose) saved me probably weeks of time. Trying to unit test any moderately complex API surface brings me to tears when I compare it to trivial object mocking in something like Ruby. Python (or Ruby) are ideal for this and have excellent frameworks for CLI tools.
- p_l 5y ago
  Ruby is pretty good in this, but Python I'd heavily argue against. Maybe in another decade when they manage to restabilize what they wrought - it used to be easy. Python gets more and more complex the further away the people running the tool are from fancy recentish distros (Fedora, Ubuntu, Arch) or special constrained environments (Nix, running the CLI in docker). The moment you have to deal with unspecified RHEL version (6 is still reasonably common) or derivative, or Mac or Windows, kiss any expectation of python packaging being nice "bye bye". Unless of course you have a platform team that can handle the packaging and distribution for you, but then it probably falls a bit under "constrained environment".
  - zrail 5y ago
    Ruby is also a pain in the butt, fwiw. It's just slightly better because gems are standardized, but you still need something like rbenv to manage Ruby versions, and gods help you if you need openssl.
    - p_l 5y ago
      In my experience the major difference is that rbenv feels more of a convenience in upgrading ruby on truly outdated systems (or when you can't use prepackaged for reasons), and that for majority work minor version differences are at most one line in Gemspec away. I really, really can't say that about Python.
iechoz6H 5y ago
> Don't run your own Kubernetes cluster If we ran our cluster in the cloud we'd be on the hook for hundreds of thousands of dollars of additional costs due to the high throughput of our service. There are always exceptions to any list of rules.
- ericxtang 5y ago
  Yeah the bandwidth tax on the cloud makes some use cases impossible. What do you do instead? We use Ansible to bootstrap k3s. It works, but we have to build a lot more stuff related to monitoring, alerting, routing, etc.
  - iechoz6H 5y ago
    Kubespray on bare metal in a leased data centre with a dedicated link [1] 1. https://www.jisc.ac.uk/janet https://www.jisc.ac.uk/janet
zebraflask 5y ago
Nobody wants to mention "don't roll your own security"? That's a 101 kind of question - very easy to feel clever when you try it as an amateur, nightmarish when (really not if, when) you get it wrong. That is one area where I think you want to outsource that to specialists.
- doctor_eval 5y ago
  I absolutely agree with this, but it think it’s good for hackers to have a play with it just to go down the rabbit hole a bit. Build some toy security… but don’t deploy it.
- deleted 5y ago
  [deleted]
- worik 5y ago
  I have lost work by recommanding on hiring experts to review sensitive code I would be tasked with writing. Glad of that.
raz32dust 5y ago
> If you are in AWS, don't pretend that there is a real need for your applications to be deployable to multiple clouds. Isn't the reason people do this to make sure they have leverage in case AWS increases prices in the future? I can see how cloud providers have probably made it extremely difficult to design for multiple clouds and so this effort might not be worth it but the reason at least seems justifiable.
streetcat1 5y ago
Hopefully you will have enough money to pay AWS once they raise their prices. I am not sure what will happen if you will not?
nickjj 5y ago
I think Python still has a place for CLI tools, both internal and external. If you can get away with a zero dependency Python script then there's no struggle. You can download the single Python file and run it, that's it. It works without any ceremony and just about every major system has Python 3.x installed by default. I'd say it's even easier than a compiled Go binary because you don't need to worry about building it for a specific OS or CPU architecture and then instructing users on which one to download. Argparse (part of the Python standard library) is also quite good for making quick work out of setting up CLI commands, flags, validation, etc.. There's a number of tasks where using Python instead of Bash is easier. I tend to switch between both based on what I'm doing.
- worik 5y ago
  Python2 or Python3? If you say "Python3 of course, is this 2002 or something?" what do I do with all my Python2 scripts?
  - nickjj 5y ago
    Python 3. The 2 vs 3 era ended quite some ago. Most major operating systems that aren't end of life have Python 3.6+ installed by default giving you access to nice things like f-strings. A lot of Python 2.x scripts will work with Python 3. If they don't then it's on you to fix them since Python 2.x was officially labeled end of life almost 2 years ago from today. On the bright side, your Python 2.7 script had a good run. 2.7 was released back in 2010, so having ~10 years of not having to worry about anything is pretty good! Chances are we'll get the same experience or longer with Python 3.6+, we already at the 5 year mark for Python 3.6.
    - worik 5y ago
      > A lot of Python 2.x scripts will work with Python 3. Does not inspire confidence!
worik 5y ago
> Don't Design for Multiple Cloud Providers Designing for portability is important. Otherwise you expose yourself to dreadful uncertainties. "AWS will not disappear". That is probably true. The average business can take this risk (and if you are huge you are not listening to me). But AWS might raise its prices to a point they are getting all your profit. DO you trust Amazon? Really? The particular AWS feature you depended on with the tight coupling "Don't Design for Multiple Cloud Providers" implies may get deprecated. What then? This is as old as the hills: Design in layers. Have an AWS layer. If AWS goes away, quadruples their fees, deprecates your services, or you are hit with USA sanctions then there is a layer that has to be rewritten. Old wisdom. Use it.
- SkipperCat 5y ago
  Parler was on AWS and they got booted off. Because of that, they collapsed. If they had two cloud deployments they could have survived. AWS will never go away, but they can make your small (or medium) size business go away pretty quick. And just a side note, Parler was toxic and I shed no tears about their demise...
  - iudqnolq 5y ago
    But Parler's risk analysis should have included "we are unusually toxic so we may face unusual deplatforming". If a large number of normal companies are kicked of AWS someone will build a compat shim.
    - SkipperCat 5y ago
      I think Parler drank so much of the kool-aid that they thought they were "true patriots and no true American would dare be de-platform their website". They're ideologues not pragmatists. Kara Swisher had a very interesting interview with their CEO, John Matze. Its a good listen. https://www.nytimes.com/2021/01/07/opinion/sway-kara-swisher-john-matze.html https://www.nytimes.com/2021/01/07/opinion/sway-kara-swisher...
- worik 5y ago
  "meaning developers were constantly reading about some great new feature of AWS they weren't able to use or try out" That is a feature! Bleeding edge bleeds.
gumby 5y ago
> Don't migrate an application from the datacenter to the cloud Eh, the salesman told me it would be seamless while we were watching the football game from his company’s box. And they are the experts: it’s their cloud! I’m gonna tell the team to do it this way when I get back to the office. I think they just like running hardware and aren’t thinking of our balance sheet.
maleldil 5y ago
> Don't write internal cli tools in python What if your team is Python-based? Why would I write a CLI tool to be used by other Python programmers in Go or Rust, when some of them know neither? It doesn't matter that you know Go and can generate all possible binaries; eventually, someone else will have to make a change in your tool. It will already be difficult for them to understand a new codebase, so you don't need to make it harder by also exposing them to another language.

246 points by eandre 5y ago

> You spring back to the present day, almost bolting out of your chair to object, "Don't do X!". Your colleagues are startled by your intense reaction, but they haven't seen the horrors you have. They may be startled, but they almost certainly won't listen. The purgatory nature of IT work culture ensures this repetitive pattern.

doctor_eval 5y ago
That’s my experience too. The older I get the more frustrated I am that people don’t want to learn from my mistakes. It’s not that I’m a jerk about it. Most of the time it’s business types saying “you don’t need to do all that stuff - just deploy it like that, it’ll be fine”. And then it’s not fine - and also, it’s somehow my fault. It seems to be more about personal aggrandisement (“I’m the boss and my word is law”) rather than trying to build a great business together. I’m pretty over it.
- worik 5y ago
  It is easier to right than to read. It is easier to talk than to listen. It is easier to build new than adapt. All three of those statements are false. But all three are seductive
  - cstoner 5y ago
    > It is easier to right than to read. I just wanted to point out that I found that particular error really funny given the context.
    - worik 5y ago
      Leaving it!

SkipperCat 5y ago

Not sure if I agree about the Python jab. I've seen "pip install ...." run flawlessly more times than I've had breakfast cereal, and I eat a lot of cereal. I kinda agree on his first point about migrating stuff to the cloud but if you've done your deployments on like-like platforms (on prem containers to cloud containers) its not that bad.

jddioeow 5y ago
I use python professionally and can't make sense of this comment. I'm doing my thing, running my ls, cd, greps and cats, and now I need to run my python cli.... And so I do pip install in which env exactly? The system one? Or do I have to create a venv just to install my python cli, and have to enable it every time I wanna run the cli? It's easy to say "just pip install", but I can't see how the details make sense in the context of a cli. Explain please?
- password4321 5y ago
  https://asdf-vm.com/ https://asdf-vm.com/ to the rescue! Well, maybe it's the latest in a long line of options...
  - volker48 5y ago
    I started using asdf instead of homebrew for installing and managing Python and I haven't looked back. Asdf is infinitely better than homebrew python.
- maleldil 5y ago
  You can use pipx, which will create a new venv for each tool it installs. You call the program normally afterwards, and it will use the correct interpreter. There's not much wrong with just installing the tool to the global (user) interpreter, though.
- SkipperCat 5y ago
  Let's take the AWS CLI as an example. Run "pip3 install awscli --upgrade --user" and you're done. Drop the "--upgrade --user" and you've installed it globally. Easy Peezy. Sure, you can do a lot with venv, Anaconda and whatnot, but if you have a well written package, it can be very portable without the need of environments.

tex0 5y ago

That's a seriously good and honest list. Thank you.

privacyonsec 5y ago

> Nobody knows how to correctly install and package Python apps Anybody tried PyInstaller ? it packages the whole Python project, dependencies included into a single exactable binary

tommiegannert 5y ago

> Soon I was auditing new services for "multi-cloud compatibility", ensuring that instead of using the premade SDKs from AWS, we maintained our own. I wonder if a useful middle-ground is to have lint checker rules to enforce using a blessed subset of a cloud provider's SDK. So that some thought/effort must be put into using a new feature?

raffraffraff 5y ago

Especially the alerts thing. I think every company I've ever worked for made the mistake of ignoring alert spam. If an alert doesn't require human action, then it should be a log or a metric. And by all means plot it on a graph (the metric that triggered the alarm, or in the case of a boolean test result, the frequency of the failure). Look at the graph during real incidents if you want. Talk about it at the monthly meeting. But don't generate an alert that people should ignore. You're playing Russian Roulette.

cloudengineer94 5y ago

I work with Azure and my issue is always problems with the Azure Keyvault.. At a personal deployment, it's always passwords I always forget to save them because I'm juggling with tons of things at once lol.

hdjjhhvvhga 5y ago

> If you are in AWS, don't pretend that there is a real need for your applications to be deployable to multiple clouds. If AWS disappeared tomorrow, yes you would need to migrate your applications. But the probability of AWS outliving your company is high Well, it's not about AWS shutting down at all! It's about them having complete control over your infrastructure, so they dictate the terms. This has many consequences: (1) they can raise prices and you can do absolutely nothing about it, (2) since you chose AWS with its dynamic pricing instead of flat-rate dedicated servers, each expansion (traffic, new services) is a cost for you. This means at some point you will realize you will save sick amounts of money if you switch to bare metal (as several notable companies have done). Except that at this point it's really difficult because you have to basically start from zero so the inertia basically pulls you into continuing this vicious cycle. So this is just a straw-man argument. Really, I haven't heard anyone saying "but Amazon can go out of business", it's just ridiculous.

odiroot 5y ago

The post has really great advice but... > Don't write internal cli tools in python Disagree completely with this. This has been probably the biggest overall boost for both engineers and operators at a few companies, I worked at. You deliver fast, it's easy to debug, and requires no compilation -- which is usually a bigger hassle than any Python-specific problem. It gets really important if you have operators on Linux/Windows/Mac.

w-j-w 5y ago
Have you tried go lately? It needs to "compile", but the compiler is so fast that the combined compile/run is essentially the same experience as running a script. goreleaser can also make cross compilation extremely easy.

gorgoiler 5y ago

Late to the party, but here’s a solution to “Python packaging” that works fantastically for all my stuff and requires four lines of setup and (only) one magical invocation — pip: $ cat dogclock/__init__.py import arrow # Example dependency def bark(): print(*(["woof"] * arrow.get().hour)) $ cat scripts/dogclock #!/usr/bin/env python3 import dogclock dogclock.bark() $ cat setup.py from setuptools import setup setup( install_requires=['arrow'], packages=['dogclock'], scripts=['scripts/dogclock']) $ pip3 install . … $ dogclock woof woof woof woof woof woof

jl6 5y ago

> Don't Design for Multiple Cloud Providers This has its own sub-antipattern: “Just put your application in a container, then it will run anywhere!”

dijit 5y ago

Regarding being cloud provider agnostic: it’s not always for fault tolerance, there can be a couple different reasons. 1) it gives your company a stronger bargaining position with the cloud provider. Granted, my companies tend to have extremely high spend- but being able to shave a dozen or so percent off your bill is enough to hire another 50 engineers in my org. 2) you may end up hitting some kind of unarguable problem. These could be business driven (my CEO doesn’t like yours!), technical (GCP is not supported by $vendor) or political (you need to make a China version of your product, no GCP in China!) Everything is trade offs. AWS never worked for us because the technical implementation of their hypervisor was not affined to CPU cores of the machine, meaning you often compete with other VMs on memory bandwidth. — but AWS works in China (kinda). So my solutions support both GCP and AWS as slightly less supported backup.

sneak 5y ago
The guy also recommends using several proprietary AWS services in other points. Then he goes on to advocate against designing for cloud flexibility. This almost feels like AWS marketing.
kgeist 5y ago
>you may end up hitting some kind of unarguable problem. Another example: there's multiple countries (for example, here in Russia) where personal data must be stored in data centers located in the country's borders and not every country has a AWS datacenter on its soil .
cconstantine 5y ago
I'd add another reason: Devs need to be able to run stuff locally sometimes. It's neat having a serverless single page app that is hosted in s3, served through cloudfront, with lambda's that post messages to sqs queues that are read by god knows what else, but what happens when there's a bug? How do you test it? You can throw more cloud at it and give each dev a way to build their own copy of the stack, but that's even more work to manage. Maybe localstack behaves the same, but can you integrate it with your test framework? I never took a hard "we must never use aws-only services" approach, but having the ability to run something locally was a huge plus. Postgres RDS? Totally fine, you don't need amazon to run postgres. Redshift? Worth the lock-in given the performance. Lambda? Eh, probably not, given that we already have a streamlined way to host a webapp.

betaby 5y ago

0) Don't write software - outsource it to someone smarter.

barbazoo 5y ago
-1) Don't outsize writing software. Let someone smarter outsource it for you.
- doctor_eval 5y ago
  It’s the business equivalent of an AbstractButtonFactoryFactoryFactory.
charcircuit 5y ago
They don't even need to be smarter. They just need to value their time less than you do.

klodolph 5y ago

> Don't migrate an application from the datacenter to the cloud Reading the actual text of this one I get a different impression, but I'm still not sure I agree with this one. Applications can be radically different from each other in terms of how they are run. At one company, we ran a simple application as SaaS for our customers or gave them packages to run on-prem. We'd stack something like seven SaaS customers on a single set of hardware (front-ends and DB servers). The cloud offering was a no-brainer, you can just migrate customers one by one to AWS or whatever, or spin up a new customer on AWS instead of in our colocation center. Applications have a very wide range of operational complexity. Some applications are total beasts--you ask a new engineer to set up a test environment as part of on-boarding and it takes them a week. Some applications are very svelte, like a single JAR file + PostgreSQL database. The operational complexity (complexity of running the software) doesn't always correspond to the complexity of the code itself or its featureset.

shoo 5y ago
> I've been involved now in three attempts to do large-scale migrations of applications written for a specific datacenter to the cloud and every time I have crashed upon the rocks of undocumented assumptions about the environment I've only participated in a single on-prem to cloud migration. Some parts of the migration were easy, e.g. moving a postgres DB that was running on some on-prem linux server to run in AWS RDS. Some parts were rather unpleasant: where you discover that a bunch of the application code that runs in worker processes assumes it has access to a shared CIFS network share that can be used for communication throug the filesystem, and absolute file paths to on-prem CIFS network share locations are stored in metadata throughout the database. So then your available moves for how to migrate the application code and migrate the CIFS network share and migrate the data in the database all become somewhat tangled together.
AyyWS 5y ago
I helped migrate an app from on-prem to cloud. During the migration we found that the app needed a locally installed oracleDB. Well, it violates on-prem best practices and cloud best practices. I think migrating just exposes all the shortcuts baked into a "craplication."

timwis 5y ago

Bummer about Python :/ it’s my go-to for CLI tools, but I’ve seen that problem too.. pipenv helps, but I wonder if there’s a better way to package them so they’re more future proof.. or do I really need to learn go?

GVRV 5y ago
If the standard tech stack at your organisation includes Python, there's no reason why you shouldn't write CLI tools in Python. Packaging and distribution is only a problem for organisations that do not usually deal with Python.
i_like_apis 5y ago
IMO we loose a lot with go: having to compile, loosing the interactive shell, etc. Best case you work with a lot of people who know how to install python and use pip. Many people whine on boards, but it's not that complicated, especially with python 3.
- cconstantine 5y ago
  > install python and use pip Now you've given your users 2 issues completely unrelated to the problem they're trying to solve. If you can't give your users a single simple command, or a single file to download your tool is too complicated to install.
  - p_l 5y ago
    Not to mention all kinds of python version issues, especially given that botched 2-to-3 migration means there's a lot more Python 2 than there should be.
    - worik 5y ago
      Yes. Except. If I have a Python2 tool I paid $X for, and now I need to to change it for $Y because, because, because no reason. Tough luck! Stop "whining" and write a cheque!

travisd 5y ago

> Don't write internal cli tools in python 100%. I stopped writing anything that had to be deployed (basically everything except Jupyter notebooks for data stuff) in Python because it’s truly a nightmare. Go and goreleaser is great for writing a CLI (and if it’s public, it can auto generate binaries and upload to GitHub, create a Homebrew/Scoop bucket, etc)

apple4ever 5y ago
I've written tons of internal tools with Python, and I have no problem getting it deployed. Just keep them up to date with changes and always use system provided packages, and there is no issue.
throwaway984393 5y ago
For the past 3 years I've written almost all of my users' system tools in Bash v3. Nobody has yet reported a bug to me. Runs fine on Linux, Mac, and WSL, on every architecture. Single file, easy for anyone to read and edit, gets the job done. I don't even have to care that it's unpopular because there are no installation steps. Just 'bash foo.sh'. Sometimes that means they need to download some Go app that my script uses, but that's much better than me having to write and support all that other code. If I have to do weird things with data structures or complex logic, I reach for Python, but I do it one of two ways: 1) so stupid that "python foo.py" works (no external deps), or 2) publish it to pypi so the user can just 'pip install foobar'.
- exdsq 5y ago
  I love bash. It's probably my favourite language for productivity. I spent some time a while back trying to write a Lisp in bash, and have written a ~1kloc performance testing tool to get results while an entire team worked on a Haskell implementation.
- raffraffraff 5y ago
  I'm also a huge bash fan, and agree that distributing Python tools is a mess. But "Runs fine on Linux, Mac and WSL" is a stretch. Bash on Mac OS is a ball of shit unless you're installing a newer GNU bash via homebrew. Mac OS bash is just too old. I can't even remember the list of stuff that breaks because I stopped trying. If the target audience is mostly Mac users I'll just write zsh to begin with.

torton 5y ago

> Don't write internal cli tools in python A lot of the advice is good but I take an issue with this one. With poetry and docker, packaging Python apps for easy consumption is a non-issue. Same for Ruby. If you can get your team to standardize on poetry, you might not even need containers -- but, honestly, running these tools from CI or automation (anywhere) is so useful that you probably want container versions anyway. Golang is not a good fit for exploratory CLIs that work with complex data structures and are written for one-off, low-CPU consumption purposes -- not for scale-up API services. Just having an interactive shell (or `pry` in Ruby, those two are identical for the purpose) saved me probably weeks of time. Trying to unit test any moderately complex API surface brings me to tears when I compare it to trivial object mocking in something like Ruby. Python (or Ruby) are ideal for this and have excellent frameworks for CLI tools.

p_l 5y ago
Ruby is pretty good in this, but Python I'd heavily argue against. Maybe in another decade when they manage to restabilize what they wrought - it used to be easy. Python gets more and more complex the further away the people running the tool are from fancy recentish distros (Fedora, Ubuntu, Arch) or special constrained environments (Nix, running the CLI in docker). The moment you have to deal with unspecified RHEL version (6 is still reasonably common) or derivative, or Mac or Windows, kiss any expectation of python packaging being nice "bye bye". Unless of course you have a platform team that can handle the packaging and distribution for you, but then it probably falls a bit under "constrained environment".
- zrail 5y ago
  Ruby is also a pain in the butt, fwiw. It's just slightly better because gems are standardized, but you still need something like rbenv to manage Ruby versions, and gods help you if you need openssl.
  - p_l 5y ago
    In my experience the major difference is that rbenv feels more of a convenience in upgrading ruby on truly outdated systems (or when you can't use prepackaged for reasons), and that for majority work minor version differences are at most one line in Gemspec away. I really, really can't say that about Python.

iechoz6H 5y ago

> Don't run your own Kubernetes cluster If we ran our cluster in the cloud we'd be on the hook for hundreds of thousands of dollars of additional costs due to the high throughput of our service. There are always exceptions to any list of rules.

ericxtang 5y ago
Yeah the bandwidth tax on the cloud makes some use cases impossible. What do you do instead? We use Ansible to bootstrap k3s. It works, but we have to build a lot more stuff related to monitoring, alerting, routing, etc.
- iechoz6H 5y ago
  Kubespray on bare metal in a leased data centre with a dedicated link [1] 1. https://www.jisc.ac.uk/janet https://www.jisc.ac.uk/janet

zebraflask 5y ago

Nobody wants to mention "don't roll your own security"? That's a 101 kind of question - very easy to feel clever when you try it as an amateur, nightmarish when (really not if, when) you get it wrong. That is one area where I think you want to outsource that to specialists.

doctor_eval 5y ago
I absolutely agree with this, but it think it’s good for hackers to have a play with it just to go down the rabbit hole a bit. Build some toy security… but don’t deploy it.
deleted 5y ago
[deleted]
worik 5y ago
I have lost work by recommanding on hiring experts to review sensitive code I would be tasked with writing. Glad of that.

raz32dust 5y ago

> If you are in AWS, don't pretend that there is a real need for your applications to be deployable to multiple clouds. Isn't the reason people do this to make sure they have leverage in case AWS increases prices in the future? I can see how cloud providers have probably made it extremely difficult to design for multiple clouds and so this effort might not be worth it but the reason at least seems justifiable.

streetcat1 5y ago

Hopefully you will have enough money to pay AWS once they raise their prices. I am not sure what will happen if you will not?

nickjj 5y ago

I think Python still has a place for CLI tools, both internal and external. If you can get away with a zero dependency Python script then there's no struggle. You can download the single Python file and run it, that's it. It works without any ceremony and just about every major system has Python 3.x installed by default. I'd say it's even easier than a compiled Go binary because you don't need to worry about building it for a specific OS or CPU architecture and then instructing users on which one to download. Argparse (part of the Python standard library) is also quite good for making quick work out of setting up CLI commands, flags, validation, etc.. There's a number of tasks where using Python instead of Bash is easier. I tend to switch between both based on what I'm doing.

worik 5y ago
Python2 or Python3? If you say "Python3 of course, is this 2002 or something?" what do I do with all my Python2 scripts?
- nickjj 5y ago
  Python 3. The 2 vs 3 era ended quite some ago. Most major operating systems that aren't end of life have Python 3.6+ installed by default giving you access to nice things like f-strings. A lot of Python 2.x scripts will work with Python 3. If they don't then it's on you to fix them since Python 2.x was officially labeled end of life almost 2 years ago from today. On the bright side, your Python 2.7 script had a good run. 2.7 was released back in 2010, so having ~10 years of not having to worry about anything is pretty good! Chances are we'll get the same experience or longer with Python 3.6+, we already at the 5 year mark for Python 3.6.
  - worik 5y ago
    > A lot of Python 2.x scripts will work with Python 3. Does not inspire confidence!

worik 5y ago

> Don't Design for Multiple Cloud Providers Designing for portability is important. Otherwise you expose yourself to dreadful uncertainties. "AWS will not disappear". That is probably true. The average business can take this risk (and if you are huge you are not listening to me). But AWS might raise its prices to a point they are getting all your profit. DO you trust Amazon? Really? The particular AWS feature you depended on with the tight coupling "Don't Design for Multiple Cloud Providers" implies may get deprecated. What then? This is as old as the hills: Design in layers. Have an AWS layer. If AWS goes away, quadruples their fees, deprecates your services, or you are hit with USA sanctions then there is a layer that has to be rewritten. Old wisdom. Use it.

SkipperCat 5y ago
Parler was on AWS and they got booted off. Because of that, they collapsed. If they had two cloud deployments they could have survived. AWS will never go away, but they can make your small (or medium) size business go away pretty quick. And just a side note, Parler was toxic and I shed no tears about their demise...
- iudqnolq 5y ago
  But Parler's risk analysis should have included "we are unusually toxic so we may face unusual deplatforming". If a large number of normal companies are kicked of AWS someone will build a compat shim.
  - SkipperCat 5y ago
    I think Parler drank so much of the kool-aid that they thought they were "true patriots and no true American would dare be de-platform their website". They're ideologues not pragmatists. Kara Swisher had a very interesting interview with their CEO, John Matze. Its a good listen. https://www.nytimes.com/2021/01/07/opinion/sway-kara-swisher-john-matze.html https://www.nytimes.com/2021/01/07/opinion/sway-kara-swisher...
worik 5y ago
"meaning developers were constantly reading about some great new feature of AWS they weren't able to use or try out" That is a feature! Bleeding edge bleeds.

gumby 5y ago

> Don't migrate an application from the datacenter to the cloud Eh, the salesman told me it would be seamless while we were watching the football game from his company’s box. And they are the experts: it’s their cloud! I’m gonna tell the team to do it this way when I get back to the office. I think they just like running hardware and aren’t thinking of our balance sheet.

maleldil 5y ago

> Don't write internal cli tools in python What if your team is Python-based? Why would I write a CLI tool to be used by other Python programmers in Go or Rust, when some of them know neither? It doesn't matter that you know Go and can generate all possible binaries; eventually, someone else will have to make a change in your tool. It will already be difficult for them to understand a new codebase, so you don't need to make it harder by also exposing them to another language.