8 ms·

How to Survive Your Project's First 100k Lines

• slondr 3y ago
  Good advice in here. The focus on assertion-oriented programming is interesting. Erlang doesn't have assertions available in application code directly, but I think structure-enforced assignments are really similar in practice.
  • fredrikholm 3y ago
    Yes! The reasoning behind 'let it crash' is just this; assume happy path destructuring (effectively asserts) and let the supervisor(s) decide what happens to a process that crashes. Now you have very explicit expectations and contracts on data, yet the code is very straight forward and easy to reason about. Layering OTP on top means you can now decide how you react to one-off errors and when you're drowning in bad data. I love it.
• baby 3y ago
  About the first advice, if we’re talking about Rust, it might be a good idea to turn a number of assertions into debug_assert, so that it does not affect performance during prod. Of course it’s almost always not the case that you would want to not fail in prod, but sometimes you KNOW that an assert will HAVE TO be triggered in tests. Another way to think about this is that there’s three kind of errors: - recoverable (return Err) - unrecoverable (debug assert) - unrecoverable and tricky (assert)
  • slashdev 3y ago
    Assertions are often free or nearly so. On modern out of order processors, it’s a fully predictable branch and a cheap assertion that nothing depends on. So the pipeline is unaffected and there’s usually a cheap test that often is completely free because there’s spare capacity. If the assert tests something expensive, I will usually switch to a debug assert.
• stephendause 3y ago
  > Use a language with good compile speeds. This is an interesting suggestion to me. Honestly, I consider a lot of other things like type safety and ergonomics of the language before I think about compile speed if at all. However, that's not to say it's irrelevant. Perhaps I should be thinking about it more. I did notice that they didn't suggest any particular language with this comment. Compile speed seems like a difficult thing to compare between languages in general, since it would be difficult to make a genuine apples to apples comparison.
  • lionkor 3y ago
    I often spend hours of writing code without ever compiling - once you are familiar with the language and how the compiler behaves, that should be the norm. When I then compile, its okay if it takes 10 seconds or 10 minutes.
  • _xivi 3y ago
    > I consider a lot of other things like type safety and ergonomics of the language before I think about compile speed if at all I believe the good compile speed is hinting at having shorter feedback loops in your workflow, and how it can significantly boost your productivity It's something you'll do often and can have hidden costs. Especially with cases when it takes half an hour to build a software.
    • dimitrios1 3y ago
      I believe it's mostly just a direct jab at C++. Almost every other language has fast or fast enough compilation speeds.
      • intelVISA 3y ago
        Rust sends its regards.
        • rockwotj 3y ago
          So does kotlin
• deleted 3y ago
  [deleted]
• ohwellhere 3y ago
  Has anyone used Vale or have any thoughts on it? (That's my main curiosity after reading the post and clicking around. It looks neat but I'm just hearing of it.)
  • MuffinFlavored 3y ago
    https://vale.dev/ https://vale.dev/ for anybody else who might not have heard of it
• xiphias2 3y ago
  ,,Dependency injection (the pattern, not the kind of framework) Encapsulation Polymorphism'' These were just thrown in, but they both make the code less readable and less performant if they are not really needed. In my experience while having separation of concerns is critical for software development, the only way to find the right interface for that separation is trial and error. Iterating on the interface between layers is the most important part of software development.
  • verdagon 3y ago
    Those first two principles help promote simpler and more stable APIs, which is a must for large projects with many teams. If you iterate on the interface between large enough layers, it ends up anywhere between "very costly migration" and "breaking change". In a smaller project though, then yeah, it's better to just refactor away one's woes. Like you said, if it's not really needed, they can have their drawbacks.
• josephg 3y ago
  I'd add one more tip that I think far more software developers should do: Add unit-level fuzz testing throughout your projects. Fuzzy bois are like assertions on steroids. With large projects you often get modules which have an API boundary, complex internals and clear rules for what correct / incorrect look like. For example, data structures or some complex algorithms. (A-star, or whatever). Every time I have a system like this, I'm now in the habit of writing 3 pieces of code: 1. A function that checks the internal invariants are true. Eg, in a Vec, the allocated length should be >= the current length. In a sorted tree, if you iterate through the items, they're sorted. And children are always >= the internal nodes (or whatever the rules are for your tree). During development, I wrap my state mutators in check() calls. This means I know instantly if one of my mutating functions has broken something. (This is a godsend for debugging.) 2. A function which randomly exercises the code, in a loop. Eg, if you're writing a hash table, write a function which creates a hash table and randomly inserts and deletes items in a loop for awhile. If you've implemented a search algorithm, generate random data and run searches on it. Most complex algorithms and data structures have simple ways to tell if the return value of a query is correct. So check everything. For example, a sorted tree should contain the same items in the same order as a sorted list. Its just faster. So if you're writing a sorted tree, have your randomizer also maintain a sorted list and then periodically check that the sorted list contains the same items in the same order as your tree. If you're writing A-star, check that an inefficient flood fill search returns the same result. Your randomizer should always be explicitly seeded so when it finds problems you can easily and deterministically reproduce them. 3. A test which calls the randomizer over and over again, and checks all the invariants are correct. When this can run overnight with optimizations enabled, your code is probably ok. There's a bunch of delicate performance balances to strike here - its easy to spend too much CPU time checking your invariants. If you do that, you won't find rare bugs because your test won't run enough times. I often end up with something like this: loop (ideally on all cores) { generate random seed initialize a new Foo for i in 0..100 { randomly make foo more complicated (at first check invariants here) } (then later move invariants here) } Every piece of a large program should be tested like this. And if you can, test your whole program like this too. (Doable for most libraries, databases, compilers, etc. This is much harder for graphics engines or UI code.) I've been doing this for years and I can't remember a single time I set something like this up and didn't find bugs. I'm constantly humbled by how effective fuzzy bois are. This sounds complex, but code like this will usually be much smaller and easier to maintain than a thorough unit testing suite. Here's an example from a rope (complex string) library I maintain. The library lets you insert or delete characters in a string at arbitrary locations. The randomizer loop is here[1]. I make Rope and a String, then in a loop make random changes and then call check() to make sure the contents match and all the internal invariants hold. [1] https://github.com/josephg/jumprope-rs/blob/ae2a3f3c2bc7fc1fcf4b5e2f04031d785ade9a26/tests/test.rs#L357-L428 https://github.com/josephg/jumprope-rs/blob/ae2a3f3c2bc7fc1f... When I first ran this test, it found a handful of bugs in my code. I also ran this same code on a few rust rope libraries in cargo. About half of them fail this test.
• danybittel 3y ago
  Another tip that helped my compiler. A compiler usually has passes which are optional (optimization) or things that can be done one or another way. Like a pass that can come befor or after another pass. Inputs that can be processed from left to right or right to left etc.. It is quick and easy to make flags for all these variants. Than run the integration tests with different flags. Another tip, validation pass. If you have a (complex) data structure like an IR. Create a validation pass, which becomes the specification, of what exactly is legal in this data structure. This is like an assert on steroids. And.. have a bigger use case at hand. Integration tests are great, but also have a real world case that can be run.
  • JanisErdmanis 3y ago
    I like the idea of writing a validation pass, but I don’t want to pay a cost at runtime. This could be much to ask, but are there any programming languages which allow to write a validation pass which are enforced at compile time?
    • JonChesterfield 3y ago
      Move the invariants on the IR into the type system representing the IR. Then no validation pass involved. Requires an adequate type system.
• AdieuToLogic 3y ago
  Regarding assertions: The sections "If you think you're using enough assertions, you're wrong" and "Bonus: Sanity Checking" can be considered a subset of "Design by Contract"[0]. Regarding: However, comments can become out-of-date, and you might not be lucky enough to stumble across the right comment before you embark on a refactoring adventure. Not if comments are treated as first-class citizens, just as important as the code change(s) which make them out-dated. Better still is to ensure unit tests are "executable documentation"[1]. Regarding "Prefer end-to-end tests": In my experience, this leads to unit test atrophy, degrading system verification to rely solely on integration tests ("end-to-end tests"). This has often resulted in higher coupling and always resulted in interminable build times. 0 - https://en.wikipedia.org/wiki/Design_by_contract https://en.wikipedia.org/wiki/Design_by_contract 1 - https://softwareengineering.stackexchange.com/questions/154615/are-unit-tests-really-used-as-documentation https://softwareengineering.stackexchange.com/questions/1546...
  • forgotmypw17 3y ago
    A comment with first class citizen status is an assert?
    • sideshowb 3y ago
      I've never seen an assert explain why
      • tleb_ 3y ago
        The signature for static_assert(3) in C has a second argument that is `char *msg`. With C23 onwards, it becomes optional.
        • marginalia_nu 3y ago
          Java has this too. assert isSorted(array, start, end) : "The input array for binary search is expected to be sorted"; That said, I think in many cases asserts tend to be fairly self-evident, they typically just list the invariants, e.g. assert start <= end; assert (end - start) % stepSize == 0; // some logic assert pos - start >= 0; assert pos - end < 0; assert (pos-start) % stepSize == 0; return pos;
• hyperman1 3y ago
  There is some very good advice in here for when you reach medium size. I found myself nodding along, saying of course, happy to see someone write all this up so well, good reference... Until the fast turn around time item came around, and I knew exactly why my current work was bothering me. I gues sometimes you need to be whacked around the ears with a checklist of stuff you should know, before you realize something's off.
• fredrikholm 3y ago
  This is one of the better articles (of this type of content) that I've read in a while. Pushing implicit logic towards types, keeping non-determinism/impurities at the edges of your program by asserting around edges is a massive win. The only language that encourage this type of programming (as in it's idiomatic across the ecosystem) whilst still keeping compile times down is OCaml.
• BiteCode_dev 3y ago
  I'd love to do this in Python, but unfortunately we can't. assert is supposed to be cheap in production because you remove them there. In python, this is done python passing the "-oo" flag to the interpreted. Unfortunately, the community never got the memo: - if you put it in a lib, nobody is going to use "-oo" in prod, and they will think your lib is very slow - if you using it in prod, some of your 3rd party deps may very well be using "assert" to make required production sanity checks. So using "-oo" would suddenly break your system.
  • biorach 3y ago
    > I'd love to do this in Python, but unfortunately we can't. > assert is supposed to be cheap in production because you remove them there. well... Python is so slow that are you going to notice the overhead of assert calls? (Not being negative, I really like Python)
    • BiteCode_dev 3y ago
      Of course. Python is a great glue language, so you will do most of the expensive call outside of Python. However, things you typically put in assert are usually pure Python you call repeatedly, putting you in the worse of both worlds.
• DotaFan 3y ago
  I'm close to reaching 100k on current greenfield project I'm working on. We have the right tools in place, fast build times, CI/CD, conventions, documentation, code reviews. Most challenging thing I've found is to keep my teammates on the right path (we are 2 seniors and 1 mid level dev). Constant ignoring of the rules and going over same code review comments drives me nuts.
  • damethos 3y ago
    Perhaps you can use linting to force some (or all) of the rules instead of having the same discussion in code reviews.
  • jjice 3y ago
    > Constant ignoring of the rules I don't know your situation, but I'd be interested to know if these are agreed upon rules that people just get lazy about (really common, I've been there) or what. If it's forgetfulness or it's tedious to follow, I'd consider looking into setting up tooling to enforce the rules a bit better. Just throwing things out there, I have no idea what the situation looks like.
    • MrGilbert 3y ago
      > If it's forgetfulness or it's tedious to follow, I'd consider looking into setting up tooling to enforce the rules a bit better. It might also make sense to question "The Rules" from time to time. Sometimes, you have rules in place that are now grown old, and are really tedious to follow. If it becomes a chore to adhere to the rules in a project, it's fair to question the rules and drill down to the issue. Doesn't work for every situation, though - just an idea.
• jongjong 3y ago
  Non-determinism is possible if your system's inputs are entirely deterministic. If your system's inputs are deterministic and your initial state is deterministic, you don't even have to worry about state management or side effects (no need for functional programming). This is possible to achieve with certain fork-resistant blockchain systems. You can also achieve determinism across multiple blockchain systems with different consensus mechanisms so long as they are both fork-resistant.
• Narann 3y ago
  I'm happy to see more love to assertion. I worked on a 15K loc Python project. We didn't have tests because it was hard to isolate. We started to puts some asserts here and there on parts of the code we weren't sure about. After few months, we simply puts assert _everywhere_ and it quickly became a weird but efficient way to cover our backs and it help us to make refactoring fun. With assert, you can start to think if particular branching is relevant or should be placed on the caller side. We consider an AssertionError as a developer error and it makes a real difference in investigation time between other errors that could raise.