3 ms·

Questioning why something is not enough isn’t the point of my question. The environment meeting requirements could be part of the certification process. Just l

Questioning why something is not enough isn’t the point of my question. The environment meeting requirements could be part of the certification process. Just like if you mine crypto currencies, it works on any computer, and the result can be checked but not forged.

viraptor 2y ago
You can't do that. You know the test, you know the expected answers. You can trivially forge the test being run. There's no way you can get around it for deterministic tests. You could write only nondeterministic tests and require a unique answer every time... but that gets rid of any other time gains you got from the system.
- viraptor 2y ago
  Actually no, even nondeterministic tests won't work, because you can just generate the answers assuming the changed source, but submit it for the change you actually wanted. Unless you DRM the whole machine successfully (good luck), there's no way to do it.
  - jiehong 2y ago
    Fair enough. I guess if the testing framework of the language used would take into account the hash of the current file and imported sources and generate a signature based on that, it could detect on which content it ran, and certify this, somehow. But I guess it's your DRM idea: the testing lib should not be tampered with in the first place.
    - viraptor 2y ago
      That's an area of TOCTOU issues. You can either hash the file and then replace it before running, or replace the code which was supposed to do the hashing with a constant value you want, or just run in a VM, freeze it, and replace the hash in memory at the time you want. Unless you perfectly control everything: drive, system, application - you can't validate anything. That is unless you want to run in a signed trusted computing enclave, but those have at least one security failure a year in practice.
re-thc 2y ago
> The environment meeting requirements could be part of the certification process. Have you seen some of the stacks used these days? 90% of the time it isn't even replicable locally. And even if it did it'd break 1/2 the time. - There are endless SaaS products that won't give an environment per developer and the shared environment is different to production - There are so many cloud-based constructs like SES, S3, SQS, etc that replicating it all locally turns into a giant mess - You are forced to run a local kubernetes cluster taking a long time to start and forever to debug The list goes on. The better place to start is that people need to stop with this outsource mentality, simplify things and go back to the basics.

viraptor 2y ago

You can't do that. You know the test, you know the expected answers. You can trivially forge the test being run. There's no way you can get around it for deterministic tests. You could write only nondeterministic tests and require a unique answer every time... but that gets rid of any other time gains you got from the system.

viraptor 2y ago
Actually no, even nondeterministic tests won't work, because you can just generate the answers assuming the changed source, but submit it for the change you actually wanted. Unless you DRM the whole machine successfully (good luck), there's no way to do it.
- jiehong 2y ago
  Fair enough. I guess if the testing framework of the language used would take into account the hash of the current file and imported sources and generate a signature based on that, it could detect on which content it ran, and certify this, somehow. But I guess it's your DRM idea: the testing lib should not be tampered with in the first place.
  - viraptor 2y ago
    That's an area of TOCTOU issues. You can either hash the file and then replace it before running, or replace the code which was supposed to do the hashing with a constant value you want, or just run in a VM, freeze it, and replace the hash in memory at the time you want. Unless you perfectly control everything: drive, system, application - you can't validate anything. That is unless you want to run in a signed trusted computing enclave, but those have at least one security failure a year in practice.

re-thc 2y ago

> The environment meeting requirements could be part of the certification process. Have you seen some of the stacks used these days? 90% of the time it isn't even replicable locally. And even if it did it'd break 1/2 the time. - There are endless SaaS products that won't give an environment per developer and the shared environment is different to production - There are so many cloud-based constructs like SES, S3, SQS, etc that replicating it all locally turns into a giant mess - You are forced to run a local kubernetes cluster taking a long time to start and forever to debug The list goes on. The better place to start is that people need to stop with this outsource mentality, simplify things and go back to the basics.