8 ms·
To add to all the excellent answers in this thread: unit tests are massively overrated. It is good to know that your base64 encoding function is tested for all
by sph 4y ago
To add to all the excellent answers in this thread: unit tests are massively overrated.
It is good to know that your base64 encoding function is tested for all corner cases, but integration and behaviour tests for the external interface/API are more important than exercising an internal implementation detail.
What is the external interface of the kernel? What is its surface area? A kernel is so central and massive the only way to test its complete contract with the user (space) is... just to run stuff and see if it breaks.
TDD has some good ideas but for a while it had turned in a religion. While tests are great to have, a good and underrated integration testing system is just for someone to run your software. If no one complains, either no one is using it, or the software is doing its work. Do you really need tests for the read(2) syscall when Linux is running on a billion devices, and that syscall is called some 10^12 times per second globally?
https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM
- bluepoint 4y agoThank you! In a previous job I was doing the equivalent of integration testing and the very young collaborators were bashing me and demanded that I contribute to small random and useless (according to them even) tests that were not really testing anything.
- kwhitefoot 4y ago> If no one complains, either no one is using it, or the software is doing its work. Or it crashed and they couldn't be bothered to report it. Looks the same as no one used it but has a different cause.
- sph 4y agoIf one person globally used a feature, it crashed and they didn't report it (so it was not critical for them), it's a massive waste of time and energy to commit and maintain a test case for it. In fact, perhaps that feature has no place existing at all in the first place and should be dropped. You're underrating the fact that Linux's massive and diverse user base is a great filter for bugs. Only the weirdest heisenbugs can survive this filter, and because of their nature, they wouldn't be tested anyway.
- michaelt 4y agoWe're supposed to be reporting Linux crashes? I always assumed the fact I can reliably break Linux on Laptop A with Docking Station B by doing Action C was pointless to report, because how is anyone supposed to reproduce it if they don't own Laptop A and Docking Station B?
- sph 4y agoCan you reliably break Linux on Laptop A with Docking Station B? Then go here: https://bugzilla.kernel.org https://bugzilla.kernel.org (and follow the big yellow notice) There's a good chance it's a firmware or hardware bug Linux cannot fix, but it's worth a shot. But to answer your question, Laptop A and Docking Station B probably interface with each other with the same two chips/subsystems present in other devices. And if they work with Linux, the maintainers of these drivers, often the manufacturers, are publicly listed and are the ones that will try and troubleshoot it with you.
- baq 4y agoSometimes it’s just obvious like a bad assumption.
- zitterbewegung 4y agoIsn't SQLite a counterexample? They have more code that has tests than their actually running code. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- omginternets 4y agoIsn't the public API quite narrow, though?
- dahfizz 4y agoThe API is massive. Just the number build parameters and boot parameters is very high. There is also a ton of things you can tune at runtime in /sys/proc . Not to mention all the syscalls and the subtle ways they all interact with each other.
- omginternets 4y agoSurely it’s miles smaller than the public Linux API.
- interactivecode 4y agosounds like someone should write tests for that
- iLoveOncall 4y ago> They have more code that has tests than their actually running code. I'd say that's very common and not exclusive to some specific project. I made a change of about 30 lines of code recently, and it had 140 lines of tests (and I, by far, do not cover every situation). I think I've rarely encountered well tested pieces of code that were not way smaller than the tests that came with them.
- ilyt 4y agoSQLite doesn't have hardware drivers (that IIRC are still majority of kernel bugs) that need actual hardware to test (as hardware mocks are at best mildly useful, coz real hardware can be... weird) And unit testing is relatively useless, what would be useful are end to end tests that start with userspace API but that's much harder task, altho hopefully those tests wouldn't need to be changed that often.
- tetek 4y agoI can't upvote this enough. It breaks my heart when I see a manager with some engineering background, whose idea of better is to make the code coverage number higher.
- SamuelAdams 4y agoIs it weird if I would be *thrilled* to work with a manager like that? People here have great arguments against tests. I agree that they’re not always useful. Aiming for 100% code coverage is a waste of time. But if a manager wants easily measurable metrics to hit and they have the budget and bandwidth for it, why should I complain? Writing tests is easy, predictable work. My rate is my rate and if this is how management dictates I spend my time, I’ll take an easy couple weeks and paychecks any day.
- IgorPartola 4y ago> Writing tests is easy, predictable work. But is it useful work? This is the problem: engineers have at some point decided that keeping busy writing tests is better than finding other ways to write correct code. Now that TDD has died down as a religion the current fad became strong typing. I will repeat it again and again, I have yet to see an error caused by me passing an int where an array-type was expected in Python or JavaScript. I have been doing this for a long time. This too shall pass and we will see posts like “Show HN: new lightweight dialect of XYZ without the burden of types”.
- Veuxdo 4y agoNot for nothing, but static type checking isn't some kind of fad. It's been the rule, not the exception, since the 1980s at least.
- IgorPartola 4y agoFor some languages it’s a whole lot more important than others. Python and JavaScript are memory-safe so you don’t need to worry about writing a bigger object into a too-small memory allocation. You will discover that you reversed the order for parameters to a function by running your code with much less effort than annotating all your code. This type annotation is entirely useless: def get_user(email: str, create_if_not_exists: bool)
- waprin 4y agoYeah even in the webdev world, the zeitgeist has shifted from "write as many tests as possible, 100% code coverage, in fact 100% branching code coverage. Use an absurd amount of mocks to achieve this" to "write tests, not too many. mostly integration". (Guillermo Rauch tweet) And thank god because 100% code coverage always felt like an exercise in obedience to dumb process over good judgement.
- sph 4y agoTo add to Guillermo's mantra, a good suggestion from the video I posted is "write tests exercising the internal logic before a big refactor. Then remove them." Because otherwise, the more tests you have against implementation details, the more times your tests will break just because you've moved some code around. Tests need to go red when there is a bug, not because you've renamed a couple of private methods and now the mocks are broken.
- arkh 4y agoTo go farther: tests should not break during a refactoring. They're here to let you know if your refactoring broke something. They should even be green when you decide to scratch your software to rewrite it in another language. Code coverage? Use it to determine what is dead code you can remove: if your tests don't go through some code, that code is useless.
- GTP 4y agoI agree with the first part of your comment, but not the second. Achieving 100% code coverage is very hard and sometimes unreasonable. E.g. How do you test a GUI? I think that often it is better to write tests for the application logic and test a GUI by using it.
- anthonypasq 4y agothis is simply not possible. if you move a method from one class to another every unit test breaks.
- 3a2d29 4y ago
- artursapek 4y ago> that syscall is called some 10^12 times per second globally? wow. that blows my mind. is that the most frequently invoked piece of code in the world?
- sph 4y agoProbably the context switch logic (save registers, switch to ring 0 mode, etc.) in the kernel might be. That's called for every syscall, up to thousands of times per second per active core. It would be cool to know which is the most frequently called piece of code in the world. Maybe something that hasn't been changed in decades.
- artursapek 4y agoI'm not qualified to do it, but someone should write a blog post figuring this out
- fiedzia 4y ago> Do you really need tests for the read(2) syscall when Linux is running on a billion devices, and that syscall is called some 10^12 times per second globally? Yes, you do, because otherwise you will find that some change or new hardware will break it.
- Veuxdo 4y agoIf the read(2) implementation itself never changes, then unit tests of it shouldn't ever break, I think...
- sph 4y agoIf some hardware breaks it, the hardware must be seriously broken. If a change breaks it, some one will probably have called read(2) a few times before that kernel is shipped to you. A change is filtered through multiple developer and the kernel has some kind of smoke and consistency tests it is subject to. Also there are people and companies that compile their kernel from the master branch and run software on it. No one's gonna ship Linux with a broken read syscall. And if there were tests, one could still ship a broken read syscall anyway. Tests do not guarantee the absence of bugs.
- Cyph0n 4y ago> Also there are people and companies that compile their kernel from the master branch and run software on it. Genuinely curious: why would a company ever do this?
- sph 4y agoI was thinking of Red Hat (for example), that pays kernel developers and regularly tests and backports features for its paying customers. That means having a finger of the pulse of the kernel tree, and trying changes to decide whether it might be worth to port them to their supported kernel branch (which lags behind master).
- Cyph0n 4y ago
- emilengler 4y ago> It is good to know that your base64 encoding function is tested for all corner cases, but integration and behaviour tests for the external interface/API are more important than exercising an internal implementation detail. I think that this heavily depends upon the part that is tested or not tested. For certain parts in the kernel, such as the firewall, I truly believe, that such tests cases (including corner ones) shall be present.
- thunky 4y ago> Do you really need tests for the read(2) syscall when Linux is running on a billion devices, and that syscall is called some 10^12 times per second globally? Maybe not, but isn't it true that any new code that is being added to the kernel has been run on exactly 0 devices? And new code is being added all the time. Maybe that's why the most recent commit as of right now is loaded with fixes of broken things: https://github.com/torvalds/linux/commit/08ad43d554bacb9769c6a69d5f771f02f5ba411c https://github.com/torvalds/linux/commit/08ad43d554bacb9769c... It's one thing to say that it's impossible to test effectively locally before release (which I'm not sure if that's true or not). But you're saying it's just not worth testing because it'll break in real life and that's even better, which I"m not sure I can agree with.
- sph 4y ago> Maybe that's why the most recent commit as of right now is loaded with fixes of broken things Are you saying that if Linux had tests, there would be no bugs?
- agent281 4y agoI think that's a bit of a strawman. I don't think anyone says tests prevent all bugs. They can help fix some parts of the contract down and give you a place to check against regressions.
- sph 4y agoGP pointed out to a bug fix commit as proof that the kernel would benefit from having tests, so it's not that much of a straw man.
- svenpeter 4y agoThat merge only has fixed because of how the development model works: there’s a merge window where new features are merged and which then becomes -rc1. After that only fixes are allowed until the final release of that version and then the merge window opens again.
- adwn 4y ago> Do you really need tests for the read(2) syscall when Linux is running on a billion devices, and that syscall is called some 10^12 times per second globally? This attitude blows my mind. The point of tests is to find bugs before you ship them to the user! Writing tests after you've already verified that some piece of code is working correctly (e.g., by executing it 10^15 or whatever times on different systems with different inputs) is useless, I give you that. But what if you want to make a non-trivial change to the code? How are you going to make sure you didn't introduce any bugs without tests? Automated tests are not the same as TDD.
- mberning 4y agoI think the OP took the strictest and dumbest definition of testing possible, used that as his straw man, and then lit it on fire. The fact that systems level programmers, generally speaking, have a disregard for testing automation is apparent. Over the years if somebody had the "pleasure" of running any type of linux distro, how many times would things break randomly for reasons that the end user could only imagine? Many of those issues probably could have been surfaced and fixed before the code shipped had there been any testing automation in place. But let's forget linux for a minute. Look at Windows. For the longest time the quality of windows was the biggest joke in software. And those engineers working on it, sorry to say, were cut from the same cloth as the Linux kernel folks and all other systems programmers. We don't need no stinking tests. Well eventually somebody that cared about the reputation of the company forced testing upon these teams and lo and behold things have gotten way better. Apple is the same way. To me it is honestly astounding that there is still such a strong anti-testing sentiment in the industry. When you have more than a handful of people working on a complex system I greatly prefer to know that there is a robust test suite looking at important functionality, seeing if performance degrades, checking for static analysis errors, etc. And when something does break, since I already have a testing solution in place, it is generally easy to add a test which covers the regression and ensures that it never comes back.
- kapp_in_life 4y ago>What is the external interface of the kernel? What is its surface area? A kernel is so central and massive the only way to test its complete contract with the user (space) is... just to run stuff and see if it breaks. Having never interacted with the kernel, I have to assume it isn't just one massive file right? It's broken into separate files and components? And if you ever want to modify or refactor one of those components its nice to have the confidence that your code changes are safe without having to rebuild the whole thing and run your integration/behavior/e2e tests. Especially if you aren't the one who originally wrote the component you're modifying and don't know what the intended behavior for edge cases was. Obviously if you're mocking everything then the tests might be pointless(there can still value in preserving in the test what you assumed the behavior was), but I feel like in some ways people have over-corrected on TDD and claiming unit tests are pointless.
- philjohn 4y agoYep. Only time I've written 100% branching coverage unit tests was when I needed to implement a spec, with many alternate representation formats, and a plethora of test cases I could run through as a litmus test of conformity.
- nu11ptr 4y agoAgreed, people tend to over complicate and make testing religious. In reality, testing is nothing more than a means to an end. Ask yourself these questions for your project: 1. Are there pieces that can be tested independently where verification is valuable? 2. What is the impact of a bug or regression? 3. What type of testing is most likely to uncover issues in my product given its architecture and domain? 4. Am I confident to refactor the code base without creating new bugs? etc. etc. Iow, testing isn't magic, it should be driven by business goals. For example, I use Rust which due its great type system makes some types of code "just work", however, Rust cannot prevent logic errors. This means when I'm writing 'algorithmic code' I tend to write heavy unit tests. When I write API driven code, I tend to use more integration/end-to-end style testing. Do what makes sense for your code and goals. Tests take time and need to be refactored, so they aren't free, but can be valuable.
- titzer 4y agoUnittests is not the same thing as TDD religion. At its base, it just means dividing things up into small units and testing their functionality as exhaustively as possible. It's a divide-and-conquer. The hard part in adding unittests is deciding what a unit is, and when a unit is important enough that it should have its own battery of tests. Choosing the wrong boundaries means a lot of wasted time and effort testing things that likely won't break or change so fast that put a drag on refactoring. I disagree that a kernel can't or shouldn't be unit tested. At the very least, it has a strong interface in the userspace system calls. Of course you should unit test the system calls. Especially because Linux's motto is not to break userspace. The other benefit from unit testing that is overlooked is that it accelerates optimization. If you have a good set of unit tests that test all observable behaviors of a system means you can optimize the logic inside and constantly rerun those tests to make sure it's not broken. This speeds up the experimentation process and clearly delineates the interface so that you can see when and how you can "cheat" on things that aren't observable. So, hard disagree. Test the bejesus out of the kernel. It will harden it and open up the ability hyperoptimize the units inside. edit to add: well-written (i.e. terse and readable) tests are excellent documentation on how a unit should behave.
- nu11ptr 4y agoI both agree and disagree. There are PARTS of the kernel that probably should be unit tested. As I specified in my answer, these tend to be algorithmic. I would also agree that an interface that cannot change ("don't break user space!") makes a good candidate possibly. However, I think there are parts that probably should NOT be unit tested, because unit testing isn't free and slows down change by getting very close to internals. As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver. As such, I retain my answer of: it depends what you are trying to accomplish. Different testing strategies for different types of code and domains.
- titzer 4y ago
- GaryPalmer 4y agoAt the company I am working 80% of the code needs to be unit tested or you won't be able to commit. This results in tons of useless unit tests and you learn to restructure classes in a way that makes writing the unit tests later easy instead of what would make the most sense, for example exposings fields that should be private.
- wpietri 4y agoI'm a huge proponent of good testing, mainly because it's so much easier to work sustainably on a code base with good tests. But I hate, hate, hate test coverage mandates. Every time I've seen them, it's as you say: some people write garbage tests to hit the metric. Before with those people you wouldn't have been able to trust their code. Now you have two problems, because you can't trust their tests either.
- GaryPalmer 4y agoSometimes you are forced to write a garbage test because there is not much to say about some simple code in terms of tests.
- wpietri 4y agoThat's an interesting phrasing. Who forces you?
- GaryPalmer 4y agoIt's integrated into the build process. The build fails if there are not enough lines covered by tests.
- renox 4y agoYou're lucky. It's 96% for me. I guesstimate that the tests are >2/3 of the development effort.
- makstaks 4y agoI was genuinely interested in OP's question and was hoping responses would not fallback on the much-already discussed "TDD or not" topic.
- thiht 4y ago> Do you really need tests for the read(2) syscall when Linux is running on a billion devices, and that syscall is called some 10^12 times per second globally? I’m not sure why you’re expecting the answer “no” here. The biggest value of automated tests is non-regression. So with billions of users, yes, you absolutely need reliable and reproducible tests. Maybe not unit tests (I agree they’re overrated, you want to test contracts and behaviors, not implementation details) but a strong integration test suite is a must have. To ensure you don’t break anything your billion users might rely on.
- sph 4y agoI do actually agree with you, what I am saying is that a big part of Linux testing is crowdsourced: you and I and hundreds of millions are testing if there's a regression every time we run Linux, or running buggy, incorrect software on machines in random levels of stability and soundness. Few unit tests in the kernel would be able to compete with so many chaos monkeys (a reference to Netflix's https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey).
- yjftsjthsd-h 4y ago> It is good to know that your base64 encoding function is tested for all corner cases, but integration and behaviour tests for the external interface/API are more important than exercising an internal implementation detail. Okay, but Linux has internal implementations of base64, assorted hashing algorithms, some compression, probably other algorithmic tidbits; wouldn't those be easy to test? Edit: And downthread people are saying that Linux does do that:)
- mmcnl 4y agoI agree, but how about regression testing?
- nyanpasu64 4y agoThe Linux Elantech V3 touchpad driver for an old 2013 laptop of mine (and most laptops of the similar era) has been broken since 2019 (reports half the size compared to actual coordinates, offsets vertical coordinates, so the entire right half is interpreted as edge scroll). I debugged and proposed a fix months ago, and there's been no motion towards merging it: https://lore.kernel.org/linux-input/20221014111533.908-1-nyanpasu256@gmail.com/T/ https://lore.kernel.org/linux-input/20221014111533.908-1-nya.... Evidently, the amount of "run stuff and see if it breaks" they're doing so far is insufficient for niche hardware. Additionally the in-tree Realtek R8169 driver fails to redetect Ethernet being plugged in after I unplug it for 5-10 seconds, and I had to resort to the out-of-tree R8168 driver as a workaround.
- clnq 4y agoUnit tests and TDD should be used as tools when necessary. Unfortunately, many team leads and higher-level engineers don't take the time to explain to juniors when to use these tools. Just that the tools are essential. And therefore, for many engineers, they have become a religion, or worse - a cargo cult. But really, most code does not need to be tested. The parts that are likely to break when others change them should be. Testing can mitigate risks for complex, esoteric, highly dependable, domain-specific, and technical code. In such cases, tests can save a lot of time that would be spent fixing defects later. In contrast, testing as a religion is entirely counter-productive. It's a time sink in an effort whose whole purpose is to save time. And what is worse, poorly written tests make the codebase less maintainable, more complex (to pass wrongly written tests), and degrade code quality (through a false sense of security). Functional/integration testing can suffer from the same problems. There's nothing more annoying than fixing a defect in a system only to uncover a mess in functional tests that relied on the bug. Of course, some of this is unavoidable. However, cargo cult-style testing is avoidable and entirely self-inflicted harm in many teams. In short: testing is a tool to save time and increase code quality; such a tool should be used where it saves time and improves code quality, and it should not be used where it just results in the opposite.
- xg15 4y agoOk, you're essentially saying that kernel code is too important to be tested. Honestly, that logic sounds like Effective Altruism levels of twisted to me. So if tests don't cut it, what does the Kernel community then do instead of testing to verify correctness? > Do you really need tests for the read(2) syscall when Linux is running on a billion devices, and that syscall is called some 10^12 times per second globally? That's exactly the kind of code that needs tests. What happens if someone changes some code that affects the read call and expects it to run on said billion of devices?