7 ms·

The unfortunate aspect is that this only works for code you control. If you have code you need to run, but has no control over (such as a database library), th

by chii 3y ago

The unfortunate aspect is that this only works for code you control. If you have code you need to run, but has no control over (such as a database library), then you will have issues. For example, you want to test that an entry in a caching library is indeed expiring entries like you wanted. How do you change the library to consume a time interface, if that library doesn't expose it to you?

theamk 3y ago
In C++, you normally don't -- there are low-level function hooking calls or ptrace api, but those are too complex to be used in unit tests. The approaches I have seen used in practical projects is: Ignore unit testing of time-related parts, rely on integration tests. Have unit tests be super slow so that actual system clock can be used. Get occasional false positive failures when your build system scheduled unit tests in parallel with particularly complex compilation tasks. Wrap everything in your own wrapper, providing "production" and "testing" version of class, which "production" calling into the 3rd party library while "testing" calls into your implementation that uses debug time. This is non-trivial as "testing" implementation might be quite complex, but may be worth it for some common functionality like timers.
randomdata 3y ago
You don't. Testing third-party code is beyond the purview of first-party tests. If you have first-party code that is adjacent to a third-party library in a way that the library hinders testing of that first-party code, you would substitute a (minimal) implementation that you control in for the third-party library while under test. If the library is open source or similarly code accessible, you would fork it or coordinate control with its stakeholders to see the testing of it in that codebase (and refactor it as necessary to support that). This is where the library needs to ensure it does what the documentation claims. It is not the concern of your application. Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using.
- MaulingMonkey 3y ago
  > You don't. Testing third-party code is beyond the purview of first-party tests. Testing third-party code is often well within the purview of first-party integration testing. Aside from validating your code, and your understanding of the documentation, these sometimes incidentally catch third-party bugs. Working around bugs until upstream fixes their code may also be within the purview of first-party code... and perhaps catching regressions even if they've supposedly been fixed. > Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using. We unit test our code because we can't even trust ourselves - nevermind our coworkers, nevermind third party vendors. Granted, unit tests for third party libraries are often worth upstreaming, and you may trust upstream enough to delete your downstream copies of those unit tests once you do...
  - randomdata 3y ago
    > Testing third-party code is often well within the purview of first-party integration testing. Not really. While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented.
    - dragonwriter 3y ago
      > The value proposition of testing is that it documents [...] No, the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Documentation of anything is not the central value proposition of testing.
MaulingMonkey 3y ago
> For example, you want to test that an entry in a caching library is indeed expiring entries like you wanted. How do you change the library to consume a time interface, if that library doesn't expose it to you? Intercepting/hooking system APIs might be an option in some cases. You might even be able to use something someone else wrote. Some prior art for date/time stuff specifically (which I haven't used and thus can't vouch for:) • https://manpages.ubuntu.com/manpages/trusty/man1/datefudge.1.html https://manpages.ubuntu.com/manpages/trusty/man1/datefudge.1... • https://www.nirsoft.net/utils/run_as_date.html https://www.nirsoft.net/utils/run_as_date.html More dynamic / finer grained hooking (e.g. perhaps targeting a single thread, within a function's scope, because you're not writing a command line tool that quickly exits) is likely to require a more manual touch.
deleted 3y ago
[deleted]
jcul 3y ago
I recently came across libfaketime. It intercepts time related system calls, so you can set a time of your choosing, starting from program startup. $ faketime 'last friday 5 pm' sh -c "/bin/date; sleep 5; /bin/date" Fri 15 Dec 2023 17:00:00 GMT Fri 15 Dec 2023 17:00:05 GMT

by chii 3y ago

theamk 3y ago

In C++, you normally don't -- there are low-level function hooking calls or ptrace api, but those are too complex to be used in unit tests. The approaches I have seen used in practical projects is: Ignore unit testing of time-related parts, rely on integration tests. Have unit tests be super slow so that actual system clock can be used. Get occasional false positive failures when your build system scheduled unit tests in parallel with particularly complex compilation tasks. Wrap everything in your own wrapper, providing "production" and "testing" version of class, which "production" calling into the 3rd party library while "testing" calls into your implementation that uses debug time. This is non-trivial as "testing" implementation might be quite complex, but may be worth it for some common functionality like timers.

randomdata 3y ago

You don't. Testing third-party code is beyond the purview of first-party tests. If you have first-party code that is adjacent to a third-party library in a way that the library hinders testing of that first-party code, you would substitute a (minimal) implementation that you control in for the third-party library while under test. If the library is open source or similarly code accessible, you would fork it or coordinate control with its stakeholders to see the testing of it in that codebase (and refactor it as necessary to support that). This is where the library needs to ensure it does what the documentation claims. It is not the concern of your application. Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using.

MaulingMonkey 3y ago
> You don't. Testing third-party code is beyond the purview of first-party tests. Testing third-party code is often well within the purview of first-party integration testing. Aside from validating your code, and your understanding of the documentation, these sometimes incidentally catch third-party bugs. Working around bugs until upstream fixes their code may also be within the purview of first-party code... and perhaps catching regressions even if they've supposedly been fixed. > Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using. We unit test our code because we can't even trust ourselves - nevermind our coworkers, nevermind third party vendors. Granted, unit tests for third party libraries are often worth upstreaming, and you may trust upstream enough to delete your downstream copies of those unit tests once you do...
- randomdata 3y ago
  > Testing third-party code is often well within the purview of first-party integration testing. Not really. While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented.
  - dragonwriter 3y ago
    > The value proposition of testing is that it documents [...] No, the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Documentation of anything is not the central value proposition of testing.

MaulingMonkey 3y ago

> For example, you want to test that an entry in a caching library is indeed expiring entries like you wanted. How do you change the library to consume a time interface, if that library doesn't expose it to you? Intercepting/hooking system APIs might be an option in some cases. You might even be able to use something someone else wrote. Some prior art for date/time stuff specifically (which I haven't used and thus can't vouch for:) • https://manpages.ubuntu.com/manpages/trusty/man1/datefudge.1.html https://manpages.ubuntu.com/manpages/trusty/man1/datefudge.1... • https://www.nirsoft.net/utils/run_as_date.html https://www.nirsoft.net/utils/run_as_date.html More dynamic / finer grained hooking (e.g. perhaps targeting a single thread, within a function's scope, because you're not writing a command line tool that quickly exits) is likely to require a more manual touch.

deleted 3y ago

[deleted]

jcul 3y ago

I recently came across libfaketime. It intercepts time related system calls, so you can set a time of your choosing, starting from program startup. $ faketime 'last friday 5 pm' sh -c "/bin/date; sleep 5; /bin/date" Fri 15 Dec 2023 17:00:00 GMT Fri 15 Dec 2023 17:00:05 GMT