8 ms·

My personal theory of modern software development is that the #1 and #2 things you can to do improve software quality are: 1) Make sure devs can run a realisti

by habosa 6y ago

My personal theory of modern software development is that the #1 and #2 things you can to do improve software quality are: 1) Make sure devs can run a realistic version of the whole stack locally. 2) Drive incremental build times and effort on the whole stack to zero. With today's world of frameworks, dependencies, and microservices it's too hard to just "think through" a change. You need to try it, adjust, and try it again. Reducing friction on the iteration cycle makes this task pleasant and increases the likelihood of discovering the best solution. When this cycle is painful devs are likely to stop at the first working solution (or even partially working).

gnrlst 6y ago
I find 1) especially challenging with microservices. What technologies do you use/recommend to help with this?
- Aeolun 6y ago
  Docker compose and very beefy dev machines.
- rutthenut 6y ago
  This is where a Docker/K8s containerised setup can help. Not trivial to get to, but for deve/test environments it can be really beneficial. Can also mean the same clean starting point for every series of tests, if new containers are spun up each time through.
  - bosie 6y ago
    how do you combine this approach with some external services (i.e. databases aren't run in docker but AWS RDS or some part of the setup is using AWS lambdas)?
    - majewsky 6y ago
      Even if you have your prod database on RDS, you might be able to spin up a local instance for testing. My Postgres-using applications have a script that sets up and runs a local Postgres instance for the duration of the test. It takes about 2 seconds on first run, and less than noticeable time when the DB has already been initialized. https://github.com/sapcc/keppel/blob/master/testing/with-postgres-db.sh https://github.com/sapcc/keppel/blob/master/testing/with-pos...
      - bosie 6y ago
        Is the RDS Postgres identical to whatever you have installed in a docker? How would you do this with Aurora?
        majewsky 6y ago
        > Is the RDS Postgres identical to whatever you have installed in a docker? I don't use AWS myself, so not sure. But if it's not the same, doesn't that destroy their value proposition?
        bosie 6y ago
        not necessarily. If the positives of the RDS environment are more important to me than running my own database on EC2 in a docker with the latest features, i am totally fine with giving up some DB related features.
        smoe 6y ago
        We have been using RDS for production and normal Postgres during development for about 5 years and the only thing you have to watch out for we found is that RDS supports only certain extensions and they are fairly slow making new Postgres major versions available. So e.g. if you want to use TimescaleDB you either have to run it yourself or use a proprietary AWS alternative. It is just a Postgres running, but you have limited control about its configuration. So as long as you make sure during development that you are not doing something with the database that RDS doesn't support you should be fine. I don't know about Aurora, but I reckon it is similar. Standard Postgres features are likely going to work fine, but more advanced things are not going to be available on Aurora.
    - luxuryballs 6y ago
      https://docs.aws.amazon.com/lambda/latest/dg/images-test.html https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...
    - throwaway894345 6y ago
      Not to mention all of the EKS/GKE integrations that pretty much everyone on the platform uses. Things like EFS backed persistent volumes, ALB backed ingress, ELB backed service load balancers, etc. You need some way to abstract over the different stacks because you can’t just deploy your YAML to any Kubernetes distribution and have it work properly.
    - railsgirls112 6y ago
      https://github.com/localstack/localstack https://github.com/localstack/localstack Is a pretty good testing framework that we use at my workplace for a dockerized microservice setup. Requires some config but has mocks for most of the big services (specifically RDS looks to be in their paid tier, but we get along w/ the free version just fine.)
- fouc 6y ago
  Not that many companies actually need microservices. Consider going back to a more monolithic setup
ajsnigrutin 6y ago
I'd be happy if the developers were given an "average" device for testing, and not the newest-shiniest-flagship one. I get it why for development it's beneficial for a test device to be as fast as possible (because there less time is wasted for (eg.) an app to start after a change in code), but a lot of modern applications don't do more than they did 10 years ago, but eat up a lot more resources for the same output.
- muskox2 6y ago
  I find this is true for phones also (or even moreso). Google maps is a bad offender, getting slower every year despite doing essentially the same thing.
- iudqnolq 6y ago
  Google (allegedly, in public) does this, although for some NIH-smelling reason they've invented their own representative low-end phone instead of just buying some.
- pitay 6y ago
  I'd like it if they tested their distributable product on a hard drive sometimes, not just SSD all the time. I've come across a game that had major stutter problems, very noticeable on a HDD but still noticeable on SSD, just noticeable on a ramdisk, problem was frequent saving level data to disk and having some things block while they did that. The developer got lots of reports about performance, they said that people should upgrade their computer, game was 2d and graphically simple, the developer did not understand why people were having performance problems and didn't really care, it is essentially abandoned at this point and still for sale. Another game also had major stutter problems which would last a few seconds each, turns out it was reading a file (which should be read once at startup) over 100 times a second almost all the time it is open. Submitted bug reports in both places they support with log output from the file monitoring I did. No response to either of them. Firefox generally runs fine, if the HDD where your profile is located is saturated with IO then pages will not load until Firefox is able to write to the disk, this is with disk cache turned off. This isn't unexpected behaviour and not a bug like I would consider the previous two examples, but it is noticeable. I really would like developers to test on a HDD even if only every few releases, and would also like them to use a monitoring tool on their software sometimes. This is just so their products don't have really bad behaviour on the system it is running on which the developers are completely oblivious to.
wildmanx 6y ago
At least as important for software quality is: 3) Make extension of thorough automatic testing in all levels of the testing hierarchy as effortless as possible. Nobody is going to bother with thorough testing if it takes a day to add a simple test case for the new minor thing they just added in the last half an hour. Boom, test hole, disaster waiting to happen next time somebody refactors something. No matter how great you are doing with 1) and 2).
ericHosick 6y ago
> 1) Make sure devs can run a realistic version of the whole stack locally. This idea is greatly under-appreciated in the industry.
- jermaustin1 6y ago
  I'd love to be able to do this where I work, but we have a configuration of our software requires a minimum of 16 machines to work properly. And that is full blown machines, not just tiny dockers or whatever, but our AWS bill is in the 10s of thousands just for our 1 test environment for that setup. That test environment is undersized massively from the production environment, so we can't even do real world load tests against it. It doesn't have the disaster recovery enabled (which would double its size and adds a f5 big-ip load/proxy server in front. We also have a configuration that is more like a single server mode, which is what we primarily develop against on our local setups, but it isn't the exact same functionally, and there are always defects when deploying to the other configuration.
cellularmitosis 6y ago
I’ve been working on native (Swift) components within a React Native app, and the single-line edit/rebuild time was about a minute, which was just killing me. Creating a separate Xcode project which includes only the Swift code dropped the edit/rebuild cycle down to about 6 seconds.
rory 6y ago
This is definitely an issue with microservices, but how do frameworks contribute? I've moved more toward monolithic frameworks over the last year (Next.js and Phoenix specifically), and it feels to me like local setup is easier and build times are shorter than when I'm rolling more of my own setup.
- habosa 6y ago
  I agree frameworks help with the dev setup through all the cool hot-reloading tricks, I just meant because of their complexity it's one more thing you have to keep running in testing. I don't want to go back to jQuery in a <script> tag but that had 0 startup cost.
aidenn0 6y ago
I recently was asked what I liked about common lisp, and one thing I struggled to put into words was how good the #2 experience is. SBCL is an implementation that is AOT compiled to machine code, yet the incremental build time is shorter than even python because you can recompile a single (file|function|system) without restarting your program. Python fights you if you even want to re-load a single imported module (IIRC there are ways to do it, but it doesn't seem to be a typical way of developing in python).
phendrenad2 6y ago
> Make sure devs can run a realistic version of the whole stack locally. And through the magic of docker-compose, they can! Now just let me wait while 33 local containers spin up so I can unit test this one-line fix...

by habosa 6y ago

gnrlst 6y ago

I find 1) especially challenging with microservices. What technologies do you use/recommend to help with this?

Aeolun 6y ago
Docker compose and very beefy dev machines.
rutthenut 6y ago
This is where a Docker/K8s containerised setup can help. Not trivial to get to, but for deve/test environments it can be really beneficial. Can also mean the same clean starting point for every series of tests, if new containers are spun up each time through.
- bosie 6y ago
  how do you combine this approach with some external services (i.e. databases aren't run in docker but AWS RDS or some part of the setup is using AWS lambdas)?
  - majewsky 6y ago
    Even if you have your prod database on RDS, you might be able to spin up a local instance for testing. My Postgres-using applications have a script that sets up and runs a local Postgres instance for the duration of the test. It takes about 2 seconds on first run, and less than noticeable time when the DB has already been initialized. https://github.com/sapcc/keppel/blob/master/testing/with-postgres-db.sh https://github.com/sapcc/keppel/blob/master/testing/with-pos...
    - bosie 6y ago
      Is the RDS Postgres identical to whatever you have installed in a docker? How would you do this with Aurora?
      - majewsky 6y ago
        > Is the RDS Postgres identical to whatever you have installed in a docker? I don't use AWS myself, so not sure. But if it's not the same, doesn't that destroy their value proposition?
        bosie 6y ago
        not necessarily. If the positives of the RDS environment are more important to me than running my own database on EC2 in a docker with the latest features, i am totally fine with giving up some DB related features.
      - smoe 6y ago
        We have been using RDS for production and normal Postgres during development for about 5 years and the only thing you have to watch out for we found is that RDS supports only certain extensions and they are fairly slow making new Postgres major versions available. So e.g. if you want to use TimescaleDB you either have to run it yourself or use a proprietary AWS alternative. It is just a Postgres running, but you have limited control about its configuration. So as long as you make sure during development that you are not doing something with the database that RDS doesn't support you should be fine. I don't know about Aurora, but I reckon it is similar. Standard Postgres features are likely going to work fine, but more advanced things are not going to be available on Aurora.
  - luxuryballs 6y ago
    https://docs.aws.amazon.com/lambda/latest/dg/images-test.html https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...
  - throwaway894345 6y ago
    Not to mention all of the EKS/GKE integrations that pretty much everyone on the platform uses. Things like EFS backed persistent volumes, ALB backed ingress, ELB backed service load balancers, etc. You need some way to abstract over the different stacks because you can’t just deploy your YAML to any Kubernetes distribution and have it work properly.
  - railsgirls112 6y ago
    https://github.com/localstack/localstack https://github.com/localstack/localstack Is a pretty good testing framework that we use at my workplace for a dockerized microservice setup. Requires some config but has mocks for most of the big services (specifically RDS looks to be in their paid tier, but we get along w/ the free version just fine.)
fouc 6y ago
Not that many companies actually need microservices. Consider going back to a more monolithic setup

ajsnigrutin 6y ago

I'd be happy if the developers were given an "average" device for testing, and not the newest-shiniest-flagship one. I get it why for development it's beneficial for a test device to be as fast as possible (because there less time is wasted for (eg.) an app to start after a change in code), but a lot of modern applications don't do more than they did 10 years ago, but eat up a lot more resources for the same output.

muskox2 6y ago
I find this is true for phones also (or even moreso). Google maps is a bad offender, getting slower every year despite doing essentially the same thing.
iudqnolq 6y ago
Google (allegedly, in public) does this, although for some NIH-smelling reason they've invented their own representative low-end phone instead of just buying some.
pitay 6y ago
I'd like it if they tested their distributable product on a hard drive sometimes, not just SSD all the time. I've come across a game that had major stutter problems, very noticeable on a HDD but still noticeable on SSD, just noticeable on a ramdisk, problem was frequent saving level data to disk and having some things block while they did that. The developer got lots of reports about performance, they said that people should upgrade their computer, game was 2d and graphically simple, the developer did not understand why people were having performance problems and didn't really care, it is essentially abandoned at this point and still for sale. Another game also had major stutter problems which would last a few seconds each, turns out it was reading a file (which should be read once at startup) over 100 times a second almost all the time it is open. Submitted bug reports in both places they support with log output from the file monitoring I did. No response to either of them. Firefox generally runs fine, if the HDD where your profile is located is saturated with IO then pages will not load until Firefox is able to write to the disk, this is with disk cache turned off. This isn't unexpected behaviour and not a bug like I would consider the previous two examples, but it is noticeable. I really would like developers to test on a HDD even if only every few releases, and would also like them to use a monitoring tool on their software sometimes. This is just so their products don't have really bad behaviour on the system it is running on which the developers are completely oblivious to.

wildmanx 6y ago

At least as important for software quality is: 3) Make extension of thorough automatic testing in all levels of the testing hierarchy as effortless as possible. Nobody is going to bother with thorough testing if it takes a day to add a simple test case for the new minor thing they just added in the last half an hour. Boom, test hole, disaster waiting to happen next time somebody refactors something. No matter how great you are doing with 1) and 2).

ericHosick 6y ago

> 1) Make sure devs can run a realistic version of the whole stack locally. This idea is greatly under-appreciated in the industry.

jermaustin1 6y ago
I'd love to be able to do this where I work, but we have a configuration of our software requires a minimum of 16 machines to work properly. And that is full blown machines, not just tiny dockers or whatever, but our AWS bill is in the 10s of thousands just for our 1 test environment for that setup. That test environment is undersized massively from the production environment, so we can't even do real world load tests against it. It doesn't have the disaster recovery enabled (which would double its size and adds a f5 big-ip load/proxy server in front. We also have a configuration that is more like a single server mode, which is what we primarily develop against on our local setups, but it isn't the exact same functionally, and there are always defects when deploying to the other configuration.

cellularmitosis 6y ago

I’ve been working on native (Swift) components within a React Native app, and the single-line edit/rebuild time was about a minute, which was just killing me. Creating a separate Xcode project which includes only the Swift code dropped the edit/rebuild cycle down to about 6 seconds.

rory 6y ago

This is definitely an issue with microservices, but how do frameworks contribute? I've moved more toward monolithic frameworks over the last year (Next.js and Phoenix specifically), and it feels to me like local setup is easier and build times are shorter than when I'm rolling more of my own setup.

habosa 6y ago
I agree frameworks help with the dev setup through all the cool hot-reloading tricks, I just meant because of their complexity it's one more thing you have to keep running in testing. I don't want to go back to jQuery in a <script> tag but that had 0 startup cost.

aidenn0 6y ago

I recently was asked what I liked about common lisp, and one thing I struggled to put into words was how good the #2 experience is. SBCL is an implementation that is AOT compiled to machine code, yet the incremental build time is shorter than even python because you can recompile a single (file|function|system) without restarting your program. Python fights you if you even want to re-load a single imported module (IIRC there are ways to do it, but it doesn't seem to be a typical way of developing in python).

phendrenad2 6y ago

> Make sure devs can run a realistic version of the whole stack locally. And through the magic of docker-compose, they can! Now just let me wait while 33 local containers spin up so I can unit test this one-line fix...