5 ms·

I love systems which don't prevent foot guns, and go bang spectacularly when I foot gun myself. It's good that you know what you like. Just make sure that none

by zaphar 5y ago

I love systems which don't prevent foot guns, and go bang spectacularly when I foot gun myself. It's good that you know what you like. Just make sure that none of your software is run by me because when your software goes bang spectacularly and foot guns me instead of you I emphatically don't like it.

bayindirh 5y ago
Neither do I. That's because all my code is tested for both leaks and all scenarios. None of my code ever have gone bang in production. To be honest, no service I have written ever restarted outside system reboots or configuration changes. Having systems with no guardrails doesn't equate to having bad code automatically.
- tzmudzin 5y ago
  > my code is tested for both leaks and all scenarios That‘s a bold claim. Reeks of hubris though. Healthy scepticism would convince me a lot more that we can trust your products. You may also consider if maintaining that test suite for „all cases” is a good investment of your time.
  - bayindirh 5y ago
    Actually, there's no hubris. I'd rather put the money where my mouth is. Let's take an example which I worked on for almost a decade (and still working on). How can I make sure that I have no leaks? 1. I design software by hand. I design construction and destruction chains beforehand. 2. I implement the modules one by one, create some test suites, plug to valgrind, make sure that it has no leaks. 3. Chain the modules together, re-run the tests. 4. For every component and chain which passes the test, "Seal" the unit. Any change requires whole set of tests again. For this set of components, since components doesn't change, test suites are also "sealed". After every build, I have a CI/CD pipeline which runs a series of tests including, unit, module and end to end scenarios. Test the result with ground truth up to 32 significant digits. If something doesn't hold, flag the build and fail. I also keep timing values for certain states, and they shouldn't deviate much. Remember, we need speed. We should have invalid inputs. However these inputs shouldn't reach to the processing state and just be marked invalid and thrown out. Apply the above pipeline. Implement, test and seal. Since the stack is stable, and this is an "old school" C++ code, I don't need to migrate libraries and other stuff around much. So, the test suite I've written doesn't need maintenance unless the seal is broken. However, with the feature I'm implementing, I need to break a couple of these seals, but no biggie. Extend a function, write a couple more unit tests, run the test suite and hammer the code, seal it again. Usual tests will go on, of course. End to end valgrind tests are done periodically, with not every build. It takes around ~12 hours to complete that with full tracing and reporting. I'd rather be methodical and give the code I've written a torturous shake down, instead of saying this looks good and move on. I trust myself with the code I write, but not so blindly to go over the top and say that "I'm the one". Instead I do my best, but I believe that I'm the worst coder around here, so I test to break my code. Not to validate.
    - tzmudzin 5y ago
      Well done, and I respect your strict method. I still doubt you cover "all cases". Even for a simple problem (e.g. given three integers interpreted as side lengths, decide if they result in a equilateral, isosceles, or scalene triangle), the number of cases will be daunting (in our case: 65, see Robert V. Binder, "Testing Object-Oriented Systems. Models, Patterns and Tools"). Despite the effort you chose to invest into your framework, I still doubt you achieve anything close to this depth within your system and the underlying stack.
- johannes1234321 5y ago
  so gow donyou test it for such cases as discussed here? Like a power outage while writing to disk? Maybe you were lucky and didn't hit thst in production, yet.
  - andai 5y ago
    This could be simulated with a VM, or perhaps (maybe the underlying OS abstractions would alter the result) by simply killing the process?
    - bayindirh 5y ago
      We pull the plugs out of the systems while they're running (sometimes knowingly, sometimes unknowingly). "kill -9" is almost the same thing in most cases.
      - mst 5y ago
        Sounds like you're doing something similar to FoundationDB's Sinkhole thing. I rarely get to work on projects where we have the budget to destruction test things that hard, but I am absolutely in love with the concept.
    - johannes1234321 5y ago
      Nope. The actual harddrive then still has power and will finish its work.
  - bayindirh 5y ago
    We operate with a cattle and pet model [0], and we use a lot of services on that cattle servers. Their power can go out, disks can go bad, hardware can fry. We've seen a lot of things. As an HPC center, we absolutely hammer what we have. Even manufacturers baffled how we can fry our servers. We're eating memory controllers in one generation, and on board NICs were being cooked on others for example. XFS and EXT4 can handle a lot of abuse, incl. power loss without any problems during a heavy write. Regardless of the services we run, we didn't loss any data during a power outage, and for us, power outage means "power outage during full load". Making sure that your scientific computation can continue from the point you left it is a big business. Nobody wants to lose three weeks or a month just because a server gave out its magic smoke. [0]: https://www.engineyard.com/blog/pets-vs-cattle/ https://www.engineyard.com/blog/pets-vs-cattle/
- nemothekid 5y ago
  The systems aren't bad, the sqlite developers are just holding it wrong.
  - bayindirh 5y ago
    No, never. SQLite is one of the things I use most. Similarly, I respect the project and developers behind it a lot. On the other hand, I don't accept calling something damned because it's old, or has quirks or both. This is the same API (and set of standards) I work with, and I had my fair share of problems with it too. However, I accept that no API/Standard is perfect, and work my life around it. Also we have a huge ecosystem built around it, and while it's not perfect, it's working so far. And yes, I refuse to give my freedom to make errors, crash and burn spectacularly in the name of ease of use and abstractions. Because I need that performance, and want to be able to reach to hardware without all these layers and safety nets. We have abstracted safety nets above POSIX level, and anyone can use that if they want.
- IMTDb 5y ago
  > None of my code ever have gone bang in production Should be read "None of my code ever have gone bang in production, so far, that I am aware of"

by zaphar 5y ago

bayindirh 5y ago

Neither do I. That's because all my code is tested for both leaks and all scenarios. None of my code ever have gone bang in production. To be honest, no service I have written ever restarted outside system reboots or configuration changes. Having systems with no guardrails doesn't equate to having bad code automatically.

tzmudzin 5y ago
> my code is tested for both leaks and all scenarios That‘s a bold claim. Reeks of hubris though. Healthy scepticism would convince me a lot more that we can trust your products. You may also consider if maintaining that test suite for „all cases” is a good investment of your time.
- bayindirh 5y ago
  Actually, there's no hubris. I'd rather put the money where my mouth is. Let's take an example which I worked on for almost a decade (and still working on). How can I make sure that I have no leaks? 1. I design software by hand. I design construction and destruction chains beforehand. 2. I implement the modules one by one, create some test suites, plug to valgrind, make sure that it has no leaks. 3. Chain the modules together, re-run the tests. 4. For every component and chain which passes the test, "Seal" the unit. Any change requires whole set of tests again. For this set of components, since components doesn't change, test suites are also "sealed". After every build, I have a CI/CD pipeline which runs a series of tests including, unit, module and end to end scenarios. Test the result with ground truth up to 32 significant digits. If something doesn't hold, flag the build and fail. I also keep timing values for certain states, and they shouldn't deviate much. Remember, we need speed. We should have invalid inputs. However these inputs shouldn't reach to the processing state and just be marked invalid and thrown out. Apply the above pipeline. Implement, test and seal. Since the stack is stable, and this is an "old school" C++ code, I don't need to migrate libraries and other stuff around much. So, the test suite I've written doesn't need maintenance unless the seal is broken. However, with the feature I'm implementing, I need to break a couple of these seals, but no biggie. Extend a function, write a couple more unit tests, run the test suite and hammer the code, seal it again. Usual tests will go on, of course. End to end valgrind tests are done periodically, with not every build. It takes around ~12 hours to complete that with full tracing and reporting. I'd rather be methodical and give the code I've written a torturous shake down, instead of saying this looks good and move on. I trust myself with the code I write, but not so blindly to go over the top and say that "I'm the one". Instead I do my best, but I believe that I'm the worst coder around here, so I test to break my code. Not to validate.
  - tzmudzin 5y ago
    Well done, and I respect your strict method. I still doubt you cover "all cases". Even for a simple problem (e.g. given three integers interpreted as side lengths, decide if they result in a equilateral, isosceles, or scalene triangle), the number of cases will be daunting (in our case: 65, see Robert V. Binder, "Testing Object-Oriented Systems. Models, Patterns and Tools"). Despite the effort you chose to invest into your framework, I still doubt you achieve anything close to this depth within your system and the underlying stack.
johannes1234321 5y ago
so gow donyou test it for such cases as discussed here? Like a power outage while writing to disk? Maybe you were lucky and didn't hit thst in production, yet.
- andai 5y ago
  This could be simulated with a VM, or perhaps (maybe the underlying OS abstractions would alter the result) by simply killing the process?
  - bayindirh 5y ago
    We pull the plugs out of the systems while they're running (sometimes knowingly, sometimes unknowingly). "kill -9" is almost the same thing in most cases.
    - mst 5y ago
      Sounds like you're doing something similar to FoundationDB's Sinkhole thing. I rarely get to work on projects where we have the budget to destruction test things that hard, but I am absolutely in love with the concept.
  - johannes1234321 5y ago
    Nope. The actual harddrive then still has power and will finish its work.
- bayindirh 5y ago
  We operate with a cattle and pet model [0], and we use a lot of services on that cattle servers. Their power can go out, disks can go bad, hardware can fry. We've seen a lot of things. As an HPC center, we absolutely hammer what we have. Even manufacturers baffled how we can fry our servers. We're eating memory controllers in one generation, and on board NICs were being cooked on others for example. XFS and EXT4 can handle a lot of abuse, incl. power loss without any problems during a heavy write. Regardless of the services we run, we didn't loss any data during a power outage, and for us, power outage means "power outage during full load". Making sure that your scientific computation can continue from the point you left it is a big business. Nobody wants to lose three weeks or a month just because a server gave out its magic smoke. [0]: https://www.engineyard.com/blog/pets-vs-cattle/ https://www.engineyard.com/blog/pets-vs-cattle/
nemothekid 5y ago
The systems aren't bad, the sqlite developers are just holding it wrong.
- bayindirh 5y ago
  No, never. SQLite is one of the things I use most. Similarly, I respect the project and developers behind it a lot. On the other hand, I don't accept calling something damned because it's old, or has quirks or both. This is the same API (and set of standards) I work with, and I had my fair share of problems with it too. However, I accept that no API/Standard is perfect, and work my life around it. Also we have a huge ecosystem built around it, and while it's not perfect, it's working so far. And yes, I refuse to give my freedom to make errors, crash and burn spectacularly in the name of ease of use and abstractions. Because I need that performance, and want to be able to reach to hardware without all these layers and safety nets. We have abstracted safety nets above POSIX level, and anyone can use that if they want.
IMTDb 5y ago
> None of my code ever have gone bang in production Should be read "None of my code ever have gone bang in production, so far, that I am aware of"