21 ms·

Learnings from 5 years of tech startup code audits

798 points by lordofmoria 4y ago

alexfoo 4y ago
Nit: "...or example because it’s so fast, it’s often used in automated testing to quickly generate a whole lot of sudo-random GUIDs." ITYM: "pseudo-random" Although I do like the mash-up concept of "sudo random"-ness.
- mtVessel 4y ago
  It's higher-privileged randomness. As in, all GUIDs are random, but some are more random than others.
zer01 4y ago
This is an interesting write up! The only question I have is around your point on monorepos - every monorepo I’ve seen has been a discoverability nightmare with bespoke configurations and archaic incantations (and sometimes installing Java!) necessary to even figure out what plugs in to what. How do you reason about a mono repo with new eyeballs? Do you get read in on things from an existing engineer? I struggle to understand how they’d make the job of auditing a software stack easier, except for maybe 3rd party dependency versions if they’re pulled in and shared.
- yardstick 4y ago
  Probably because: 1) It's easy to miss a repo, if you don't have a list of them all somewhere. 2) It's easy to get out of sync with what version of your software corresponds to what branch/tag in each repo.
  - yourapostasy 4y ago
    > 2) It's easy to get out of sync with what version of software corresponds to what branch/tag in each repo. I'd like to hear how others solve this. The way I've addressed this is I bake into the build pipeline some way to dump to a text file all the version control metadata I could ever want to re-build the software from scratch. Then this text file is further embedded into the software primary executable itself, in some platform-compatible manner. Then I make sure the support team has the tooling to identify it in a trivial manner, whether a token-auth curl call to retrieve it over a REST API, or what have you. This goes well beyond the version number the users see, and supports detailed per-client patching information for temporary client-specific branches until they can be merged back into main without exposing those hairy details into the version number. While this works for me and no support teams have come to me yet with problems using this approach, it strikes me as inelegant and I'm for some reason dissatisfied with "it ain't broke so don't fix it".
    - treis 4y ago
      We have a releases repo that takes in the git version SHA for each application and handles deploys. It's... ok I guess. Just another example of complexity to meet the growing complexity of the system.
    - yardstick 4y ago
      In our case we abandoned individual repos and went back to a monorepo to solve this issue. In theory the separation of code was nice, but in practice it was a real pain when a service added new APIs you wanted to update another service to use it. All of our services do also print out in their startup logs what version they are based on git branch name and commit. Monorepo or not this was useful.
  - cerved 4y ago
    find / -type d -name .git
    - yardstick 4y ago
      As an auditor you don't have anything checked out locally yet, so no .git will exist. If you ask an individual developer or randomly picked developers, they will only have their specific repos checked out. If you look at the server hosting the repos then yes you may get them all. Assuming they are all on one server...
  - ge96 4y ago
    The thing I'm working on has 5 main repos that all run (yarn start) for the app to be fully functional. I need to put that down somewhere the order/matching branches.
  - marcosdumay 4y ago
    > 2) It's easy to get out of sync with what version of your software corresponds to what branch/tag in each repo. That's what the `[dependencies] my-lib = "1.0"` was supposed to solve.
- pianoben 4y ago
  Monorepos do require upkeep beyond that of single-product repositories. You need some form of discipline for how code is organized (is it by Java package? by product? etc). You need to decide how ownership works. You need to decide on (and implement) a common way to set up local environments. Crucially, you need to reevaluate all these decisions periodically and make changes. On the other hand... this is all work you'd have to do anyways with multiple repositories. In the multi-repo scenario, it's even tougher to coordinate the dev environment, ownership, and organization principles - but the work isn't immediately obvious on checkout, so people don't always consider it. Regarding auditing, I have always found that having all the code in one place is tremendously useful in terms of discoverability! Want to know where that class comes from? Guaranteed if it's not third-party, you know where it is. Not to minimize the pain of poorly-managed monorepos - it's not a one-size-fits-all solution, and can definitely go sideways if left untended.
- deleted 4y ago
  [deleted]
yardstick 4y ago
> Custom fuzzing was surprisingly effective. A couple years into our code auditing, I started requiring all our code audits to include making a custom fuzzers to test product APIs, authentication, etc. Any recommendations for a good fuzzing tool for testing both web-based APIs and language specific APIs (C and Java in my case)?
- asicsp 4y ago
  These might help: * https://github.com/Endava/cats https://github.com/Endava/cats * https://github.com/google/oss-fuzz https://github.com/google/oss-fuzz
  - deptm 4y ago
    Paid but integrates with CI/CD - https://www.code-intelligence.com/ https://www.code-intelligence.com/
- agumonkey 4y ago
  I'm thinking about introducing fuzzing too. And property based testing. Manual testing only is too limited.
7speter 4y ago
Its kind of discouraging to see the part where he says almost no one gets web tokens right the first time. Working on projects as someone entering the industry, its pretty clear that security is the most important part of a web app, and its so seemingly easy to get woefully wrong, especially if you’re learning this stuff on your own and trying to build working crud projects
- yourapostasy 4y ago
  That's usually because security is a bolt-on instead of bake-in within the control and data structures themselves. Too many people interpret "Make It Work Make It Right Make It Fast" to mean security is implemented at the "Make It Right" stage, when it should be at the "Make It Work" stage. That's if they're the lucky ones who get security designed in from the beginning into the architecture. We're paying for the sins of that in Unix these days, the kernel attack surface is in-feasibly large to remediate to correctness anytime soon (if ever?).
  - makeitdouble 4y ago
    I think there is still more to it that just not taking it seriously or planning for it. JWT in particular has the weird quirks you need to know to prevent encryption swapping attacks, and I'm sure there's more traps I myself am not aware of. At this point I think security can be seen on the same plan as legal: assuming a random dev will be able to plan and navigate out all the issues by sheer common sense hasn't been a viable approach for long now.
    - yourapostasy 4y ago
      > At this point I think security can be seen on the same plan as legal:... Considering how Uber ignored legal ramifications of ride sharing intersecting with incumbent regulations until they were dragged into courts, that paints a potentially rather grim picture of the equivalent in software security. But your gist sounds more along the lines of, "include the experts along at the beginning of the ride". When I said security as a "bolt-on", I should have been more clear. Most of the time when I see it happening, it has been at the behest of the business stakeholders overriding the earnest developers trying to include the security teams from the beginning, but waved off with "it can be added later". The business stakeholders see in their real life housing contractors walk into finished houses, attach some doodads, pop in some batteries to wireless sensors and the central base station, and ta da!, they "have security"! And think, "just how hard can it be to do the same in software?", dismissing what their tech leads try to tell them. There is a large element of the principal-agent problem here as well. Shiny proofs of concepts and shallow implementations get immediate bonuses and promotions. Taking 1.1-2.0X as long to implement the right way, the result of which is no drama and no discernible difference to the casual business user, get no or even negative recognition. The incentives structure the choices. There are no incentives that structure payouts over the long-haul tying back to original historical choices, with an increasing gradient of the payout the longer the original choices prove sound. Naturally, since measuring that accurately would be impossible. The closest I've come to an analogy that works in these discussions but not as often as I'd like is this. I don't throw together four tilt-walls, top off with a roof, move in with a 20-ton safe, open the doors for business and call it a regional bank depository. There are bedrock anchors, sensors, inner reinforced concrete walls, SOP's, audits, man traps, insurance reviews, and on and on, that get designed in before the foundation is even poured. Clients who didn't find this convincing wave it off with a, "haha, this isn't that important lol". I want a better analogy.
peppermircat 4y ago
Interesting to see the JWT issue. I have recently found a vulnerability in a publicly traded CRM SaaS that was also about JWT claims validation. It’s also quite amazing that popular Auth SaaS rely so heavily on JWTs with 1 hour expiry times, making it impossible to log users as you can’t invalidate the token for the next hour.
- dodgerdan 4y ago
  https://www.howmanydayssinceajwtalgnonevuln.com/ https://www.howmanydayssinceajwtalgnonevuln.com/
- blenderdt 4y ago
  "...making it impossible to log users as you can’t invalidate the token for the next hour." I have no idea what you are talking about here, can you explain this? I work with systems that have a minute expire time. The only issue is that the clocks on all clients should be in sync with the auth server.
  - habosa 4y ago
    I believe they are referring to the fact that most JWT-based auth systems use one-hour token expiry and have no ability to remotely revoke tokens. You can only revoke the user's ability to get the next token. This often leaves a one hour window between when you want the user locked out of your system and when they are practically logged out. The only way I know of to implement instant revocation in a system like this is to keep a blocklist of users/tokens that is constantly checked, which can be slow and removes some of the benefits of JWTs in the first place (that they carry all the auth information you need).
    - blenderdt 4y ago
      Ah! Yes this is why we use an expiration of one minute. For us the extra load that the refreshes give is not a problem. Keeping a blocklist seems unnecessary to me, you can just lower the expiration time.
- 4y ago
neilv 4y ago
> All the really bad security vulnerabilities were obvious. All the really bad security vulnerabilities that were found were obvious? One is more likely to find things that are obvious?
- blenderdt 4y ago
  When a team is so focused on the todo list they sometimes forget the obvious mistakes they still needed to fix.
- lbriner 4y ago
  But the auditers were experts and used all the latest and greatest tools. I think they are implying that if they couldn't find it with code inspection then a hacker wouldn't find it by probing. Of course, they might not find zero-days but most hackers wouldn't find those either.
- itsdrewmiller 4y ago
  Yeah this was a great article overall but that stood out as sus. Also the “last few hours found the most stuff”. Seems like they could probably stop the audit once they found enough problems, which skewed hard to easy to find and or last time to look.
jillesvangurp 4y ago
Guilty as charged. I bootstrapped a our startup with just myself and 2 junior engineers in the last year during Covid. Junior in the sense that they are young. But actually they outperform many older engineers I've worked with. We are starting to close some pretty big deals and I'm dreading the moment where I have to turn this into a normal development team. In my experience velocity drops when you do that and you lose a lot of momentum. 3 people can do a lot. 6 people don't do that much more. I'm not so young myself and quite experienced. But I lean heavily on my team for doing the work. I'm the CTO, the CPO, and I need to worry about team management, sales, and a few other things. So, less than half of my time is spent coding. This it the reality of startups. You have to do all of it. I made some technology choices early on. We use docker but not Kubernetes. There is one server, it's a monolith. There is one language, it is Kotlin. And we even use it on our frontend (web only). The latter is not something I would do normally or recommend. But both my junior engineers only knew Kotlin and we just went with it and never actually ended up regretting this. This surprised me and at this point I don't feel React/typescript have anything that I need or want. We're doing a lot of asynchronous stuff, websockets, maps (via libremaps), etc. And it all runs smoothly and responsively. Kotlin is kind of awesome for this actually. Originally our frontend was Android only. We ditched that app in favor of a Kotlin-js based web app that started out as a proof of concept that just more than proved the concept and became the actual thing. At the time we had a demand for IOS and web and no IOS or web developers on the team. Hence Kotlin for the web. When this looked like it was workable we actually lost our Android developer. So the decision to forget about that app was pretty easy. At that point it was half working and full of bugs and technical debt. Fixing that would only half fix our problem because we'd still need IOS and Web. So we did web first. And we are packaging it up with Cordova for those people that want something from an app store. It's a good lesson on prototyping. If it works, do more of it. At the same time, I normally recommend minimizing risk and not building too many things in parallel. Like building 3 apps for 3 platforms instead of just a web app. Our server is Kotlin/Spring boot and we use a lot of Elasticsearch because that's what I've been using for the last decade. A little bit of Redis and I've so far found no execuse to use a relational database. But I'd probably end up with mysql if that ever comes up. Done right, Elasticsearch makes for a nice key value store without transactions but with optimistic locking on documents. If I get some time, I add a database for safety at some point. But less moving parts means less headaches. Having just one language means the distinction between backend and frontend is a bit blurry. We have an api client that we use in our spring tests that also compiles to kotlin-js. That library contains a lot of code that we use in our front-end. Model classes, caching layers, functions that call various of our APIs, etc. And it's all covered in tests. All the business logic basically. If we ever need to do native apps, we'll use that there as well. On the devops front I'm a combination of very pragmatic but also focused. We use stuff that works that doesn't distract us. So, no terraform for a setup we only create once; in 20 minutes. Not worth spending weeks automating but worth documenting. But we do have CI/CD via github actions. So we don't manually deploy anything. And we have lots of API integration tests. If it builds, it ships. No rollbacks; roll forward only. Keeps things simple. We use Google Cloud and keep our cost low. A couple of VMs, a loadbalancer, a managed redis, and a managed elastic cloud cluster. That's it. Nice and simple.
lifeisstillgood 4y ago
At what point is there something beyond a framework - a SaaS in a box perhaps, that just avoids many of these basic problems (oh and the HR, legal, etc problems) of starting a startup. Startups are not snowflakes apart from that one little core competency. In short most serial founders say the second one was easier, simply because they followed the template ground out in the first. Would it be easier to start with that template?
- alfonsodev 4y ago
  all that exists as SAAS products that target non-technical cofounders, and it is very hard to justify to co-founders, investors, advisors .. any investment in time in something that is not your core problem, and I think is for a good reason.
- gunnr15 4y ago
  There are some good open source options like https://getzero.dev/ https://getzero.dev/
- MathCodeLove 4y ago
  I've seen boilerplate applications for <insert tech stack> but the open-source ones tend not to be great, and the closed source ones could be great - but I'm not willing to pay $XXX for code I haven't seen.
- scottharveyco 4y ago
  In the Rails world we have https://bullettrain.co https://bullettrain.co and https://jumpstartrails.com https://jumpstartrails.com which both have open source templates for building SaaS services.
- czue 4y ago
  There are a ton of products like this out there that build on popular frameworks: Saas Pegasus (https://www.saaspegasus.com/ https://www.saaspegasus.com/) for Python/Django, Bullet Train (https://bullettrain.co/ https://bullettrain.co/) and JumpStart (https://jumpstartrails.com/ https://jumpstartrails.com/) for Rails, Spark (https://spark.laravel.com/ https://spark.laravel.com/) for Laravel, Gravity (https://usegravity.app/ https://usegravity.app/) for JS You can find an even bigger list here: https://github.com/smirnov-am/awesome-saas-boilerplates https://github.com/smirnov-am/awesome-saas-boilerplates though those are the market leaders (I make one of them and follow things closely)
MattPalmer1086 4y ago
Interesting that he feels the default state of software security has improved a lot in the last few years. Anecdotally I'd also agree with that. Certainly better defaults and more secure libraries is a major factor. I haven't noticed a huge increase in developer security awareness, although I'd say it is also better than 10 years ago.
- anticristi 4y ago
  Unfortunately, I get the feeling that that is compensated by increasing risk. Attackers have found clever ways to monetize their work beyond just "fun". Hence, I feel the overall "security damage" has kind of stayed constant.
  - MattPalmer1086 4y ago
    For sure, the threat level hasn't dropped. What is different is that attackers have to use different techniques, since the software isn't as easily exploitable as it used to be. Ten years ago, any pen test of a web application revealed loads of vulnerabilities. These days I rarely find anything really significant (although maybe I work at better places!). This is not to say that software isn't exploitable any more, only that the cost has been raised sufficiently to make cheaper attacks more attractive (e.g. phishing).
lifeisstillgood 4y ago
>>> Business logic flaws were rare, but when we found one they tended to be epically bad. oh yes ... I always bang on to my junior staff that their job was known as "analyst programmer" for a reason. The analyst part matters probably even more than the programmer part. In large companies just discovering what needs to be coded is 90% of the job, (the securely coding it in the constraints of the enterprise the other 90% while the final 90% is marketing your solution internally) Anyway .. yes
- zeristor 4y ago
  This goes with doing the first 90% of the work, then the second 90% of the work then the last 90% of the work. And engineers multiplying their initial estimate by 3, the project manager then multiplying that by 3 and rounding it up to be ten times more than the initial estimate.
- watwut 4y ago
  > In large companies just discovering what needs to be coded is 90% of the job Yes, but that is quite massive dysfunction of those companies. Meaning, we can yell at analysts-programmers as much as we want, what really needs to be fixed is the process that makes finding out requirements so ridiculously hard. And yes, I work in one of those companies, it very clearly is dysfunction.
  - carlmr 4y ago
    I think this can only change when we, as a society, expect code literacy from every person that finishes high school. I don't mean expert programmers, but at least being able to read basic pseudocode algorithms. It's hard to describe a problem if you don't even understand any language.
    - sicp-enjoyer 4y ago
      The "magic AI" has undone years of coaching management about software expectations.
    - 4y ago
lars512 4y ago
Ah, yes, I’ve felt the pain of an unnecessary microservices migration. It ate time for years and the core was still a mess
- anticristi 4y ago
  I think people really exaggerated with the microservices trend. Today, I recommend to keep code in the same executable unless there is a good reason not to. Good reasons include: - Stateful vs stateless: databases and message queues should be your first (hopefully off-the-shelf) "microservices". - Different lifecycles: API serving vs background task - Different security needs: Frontoffice vs Backoffice code - Different teams: But make sure to introduce a clear customer-vendor relationship.
BrissyCoder 4y ago
> All the really bad security vulnerabilities were obvious. Isn't that just tautological? They are bad because they are obvious?
davedx 4y ago
Great article. To expand a little on why “Keep It Simple” is so powerful: less code = less bugs and less security issues. Less code = easier to change.
- anticristi 4y ago
  Also cleverness is overrated. You might be able to be clever once, but mid-term you will struggle to keep up with "collective cleverness". Sure today you might implement a better authentication code than the one offered in your favorite framework, but will you keep up with the new cleverness that will pour into the framework tomorrow?
- muglug 4y ago
  I don't think less code = less security issues. Often using those secure-by-default frameworks require more code. The simplest example in PHP (highlighted in the article for its default-insecurity): echo '<h1>Hello ' . $_GET['name'] . '</h1>'; is vulnerable to XSS. echo '<h1>Hello ' . htmlentities($_GET['name']) . '</h1>'; is not vulnerable
- UK-Al05 4y ago
  Less code and simple are not often related.
- TheCondor 4y ago
  I’ll go much further… I think it creates severe cultural problems. It creates the belief that problems are more difficult than they might be, it creates the belief that a particular solution may be more valuable than it actually is, and then it biases future team expansion and retention. Perhaps more ultimately, if the complexity creeps in before the real challenge gets do, it radically affects the team’s ability to reason about it.
jacquesm 4y ago
That's a very interesting set of findings. What is important to realize when reading this that it is a case of survivorship bias. The startups that were audited were obviously still alive, and any that suffered from such flaws that they were fatal had most likely already left the pool. In 15 years of doing technical due diligence (200+ jobs) I have yet to come across a company where the tech was what eventually killed them. But business case problems, for instance being unaware of the true cost of fielding a product and losing money on every transaction are extremely common. Improper cost allocation, product market mismatch, wishful thinking, founder conflicts, founder-investor conflicts, relying on non-existent technology while faking it for the time being and so on have all killed quite a few companies. Tech can be fixed, and if everything else is working fine there will be budget to do so. These other issues usually can't be fixed, no matter what the budget.
- jackblemming 4y ago
  >relying on non-existent technology while faking it for the time being and so on have all killed quite a few companies This doesn’t count as a tech problem?
  - jacquesm 4y ago
    Obviously not, it's a problem to decide to fake non-existing tech but more of a management decision than a problem with the technology itself, there is an infinite number of things that don't exist, no matter how much you want them to exist and if you are not capable of coming up with a working solution and rely on the world around you to move fast enough to bail you out then I would say that is a psychological problem more than anything else. A common theme right now is 'AAI', using people to fake an AI that may not come into being at all, let alone before your runway (inevitably) runs out.
  - lozenge 4y ago
    I saw one "secretary AI" that schedules meetings over email in your calendar. Just cc it to start using it (once you signed up). The idea seemingly being, fake it with low cost outsourcing to prove there's a demand for this and then make it. The developers you'd hire to make it an actual AI and the developers you'd hire to make it a Mechanical Turk are very different skill sets.
lstamour 4y ago
> For example because it’s so fast, [MD5 is] often used in automated testing to quickly generate a whole lot of sudo-random GUIDs. Actually, it’s because programmers are lazy. GUIDs or UUIDs are 128-bits and MD5 produces 128-bits. A string like “not-valid” is not a valid UUID, but MD5(“not-valid”) is both possible to format like a UUID when output as hex (with dashes) but also self-descriptive - so you can name the token when generating it in a fixture function and know how to regenerate it later in a test, for example. All the normal ways of generating UUIDs, including v6 and v7, are about trying to make them unique and collision resistant. But that’s nonsense when you want deterministic, reproducible tests. Hard-coding 32 characters is too much work, ain’t nobody got time for that. Magic numbers? Pfft. Just MD5 and write your own text… Pro tip: have data model creator helper functions include a counter that resets every test (every time the database resets) and then assign a UUID like MD5(`InsertTableName-${counter}`) that way you have a unique ID that’s also easy to predict/regenerate. That said… I’ve always personally preferred simple database IDs to be generally preferable over using UUIDs. It’s easier to understand THING 20 as an ID than 32-odd characters. But UUIDs are an industry standard, so they end up in your test code everywhere anyway…
- jve 4y ago
  > I’ve always personally preferred simple database IDs to be generally preferable over using UUIDs. Unless you start migrating data between environments and want references to be alive. Anyway, if you need a hardcoded GUID for tests or what, paste this into PowerShell: [Guid]::NewGuid() Not arguing, just developing for a system that uses guids as primary IDs and writing tests for that system. I don't even need to hardcode GUID, as within test bootstrap I'm creating objects with generated IDs I can reference later for comparison.
  - lstamour 4y ago
    I’ve done that before too - but it’s always possible if you run tests often enough that you’ll get an ID collision that randomly fails a test and causes a developer some grief. Easier to not use random sources of data as a rule of thumb within your unit tests.
- 4y ago
mattbillenstein 4y ago
I've seen most of the architectural problems in consulting - it's amazing how a team of clever engineers can take a simple thing and make it sooo convoluted. Microservices, craptons of cloud-only dependencies, no way to easily create environments, ORMs and tooling that wraps databases creating crazy schemas... The list goes on and on; if you're early, you could do a lot worse than create a sensible monolith in a monorepo that uses just postgres, redis, and nginx and deploys using 'git pull' ...
- claytonjy 4y ago
  The worst architecture I ever saw came from consultants, who built the initial bits of a startup I was hired into. It was nice to have a no-longer-present scapegoat to shake fists at when frustrated, but over time I came to realize their most maddening choices were at the behest of one of our founders, who had no software experience.
  - teaearlgraycold 4y ago
    I saw the same thing. Founders asking the world of consultants who would try to deliver and then fail to be a responsible engineer. I started my previous job by telling the founders they were asking for the wrong things and the consultants work needed to be thrown out. Thankfully they listened and we ended up with a TypeScript monorepo monolith deployed to Heroku.
- commandlinefan 4y ago
  > Microservices, craptons of cloud-only dependencies, no way to easily create environments, ORMs and tooling that wraps databases So, Spring Boot you mean?
  - noisy_boy 4y ago
    A Spring Boot service doesn't have to Microservice - you can happily fatten it up into a monolith. Cloud-only dependencies would come into play for Spring cloud (or something that is using cloud specific features) - for a "vanilla" CRUD app, they are not needed. Creating virtual/physical environments is out of Spring Boot's scope and better left to external tools though it has support for separate environments via profiles. ORMs/tooling that wraps database doesn't have to be part of Spring Boot - using Hibernate/JPA isn't mandatory; plain JDBC Template with hand-written SQLs would work fine.
arethuza 4y ago
"All the really bad security vulnerabilities were obvious." I used to work for a company that did a lot of acquisitions and I often involved in working with teams at newly acquired companies - although it wasn't my main focus I did used to ask some simple security questions and it was remarkable what these uncovered. I literally had people run from the meeting to fix services after I had asked a simple question....
- spuz 4y ago
  Can you give some examples of some simple questions you would ask?
  - arethuza 4y ago
    By the nature of that particular domain a lot of systems delivered important documents (often containing data of rather extreme commercial sensitivity) to customer organisations. A standard question I always asked was "given a URL that links to a document how do you authorise access" i.e. what happens if someone who is logged in to the site in question gets a link to a document and passes it to a friend via instant messaging.
    - davedx 4y ago
      Ha recognisable. A very annoying problem to solve with web tech too - there’s no perfect solution to this problem (that I know of).
dzonga 4y ago
> Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering. I wrote about this before that as an industry, we have made writing software complex for complexity's sake. > imagine, in a world where there wasn't a concept of clean code but to write code as simply as possible. not having thousands of classes with indirection. what if your code logic was simple functions and a bunch of if statements. not clever right, but it would work. what if your hiring process was not optimizing for algorithm efficiency but that something simply works reliably. imagine a world where the tooling used by software engineers wasn't fragile but simple to use and learn. oh the world would be a wonderful place, but the thing is most people don't know how to craft software. but here we're building software on a house of cards [0] :[0] - https://news.ycombinator.com/item?id=30166677 https://news.ycombinator.com/item?id=30166677
- bacza2 4y ago
  The problem is simple means diffrent thing in a small codebase than in a big one. A bunch of if statements in a code that is small enough to understand everything is ok but when it become big it's hard to understand flow of data. I do favor simple code but some complexity/abstracion is needed to make it easier to understand
  - emn13 4y ago
    But picking the right abstractions that aren't leaky in any of the aspects you really care about is critical, hard to measure (leakiness isn't obvious, nor what kind of aspects you care about), hard to get right, and hard to maintain (because your abstraction may need to evolve, which is extra tricky). Obviously, getting that right makes subsequent developments much, much easier, but it's hardly a simple route to success.
    - ItsMonkk 4y ago
      I see tech debt and simplicity as a mixture between 'tyranny of small decisions' and each individual coders 'cleanliness' level. Each individual coder has a code cleanliness level, similar to how every friend's Mom growing up would always remark "Sorry the house is a mess", when it was spotless. If your used to 9/10 and it's a 7, that looks like a wreck. If you are used to 5 and it's a 7, that looks great. I urge other coders to increase their cleanliness level, and to look for others with high cleanliness for guidance. If you are coding next to people that 5 looks good to them, no matter how much they try to pay down technical debt, they never will. I think tyranny is ultimately showing us that the tooling that we currently have is making is much trickier than should be to evolve those abstractions. Partially this is because of bad abstractions that caused bad tooling and bad tooling that caused bad abstractions. Because it's so difficult, we don't do it. We take the small decision and work slightly harder in a slightly buggier environment to get the new thing done. But of course now the problem is bigger which means its even less likely for us to ever actually pay down that debt. > “I’m sorry I wrote you such a long letter. I didn’t have time to write you a short one.” – Blaise Pascal
huma 4y ago
> Generally, the major foot-gun that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs. Finally, someone said it
- suzzer99 4y ago
  This was 100% true for that startup I worked for as a side job. They would have been so much better off just building a standard java, PHP or .NET back end and calling it a day. The head engineer (who had known the guy funding the thing since childhood) had no clue how node, stateless architecture, or asynchronous code worked. He had somehow figured out how to get access to one particular worker of a node instance, through some internal ID or something, and used that to make stateful exchanges where one worker in some microservice was pinned to another. Which goes against everything node and microservices is about. I tried to talk some sense into them but they didn’t want to hear it. So for the last six months I just did my part and drained the big guy’s money like everyone else. I hate doing that - way more stressful than working your ass off.
- wuliwong 4y ago
  It is interesting, I've been at a company for a few years now and we've been slowing trying to break apart the two monoliths that most of the revenue generating traffic passes through. I am on board with the move to microservices. It's been a lot of fun but also a crazy amount of work and time and effort has been spent to do this. I've pondered both sides of this argument. On one hand if this move had been done earlier it might not have been as difficult a multi-year project. On the other hand, when I look at the Rails application in particular, it was coded SO poorly that I if it was just written better, initially, it wouldn't even need to be deconstructed at this point. Also, if the same engineers that wrote that Rails app tried to write a bunch of distributed, even-driven microservices instead of the Rails app, we would probably be in even worse shape. (ᵔ́∀ᵔ̀)
  - ravedave5 4y ago
    Are you me? o_0. Shockingly similar situation.
gusbremm 4y ago
Once I worked on a team that none of the engineers knew that jwt payload was readable on the frontend. They were in shock when I extracted the payload and started asking questions about the data structure.
- samhw 4y ago
  I mean, I'd be rather surprised too. What were you using JWTs for, if not asymmetric crypto? Presumably you weren't using it to sign the tokens, if they were surprised the client could access them? And I can't see many contexts where you would use it with a shared secret, where just sending JSON over HTTPS wouldn't suffice. (I'm assuming 'frontend' here denotes a client on the other side of the trust boundary.)
  - FlorianRappl 4y ago
    I'm not getting your comment. The payload is not encrypted. I think you refer to the signature. The payload can always be decoded. It's just JSON into base64.
    - samhw 4y ago
      Ah, sorry, that was what I was referring to when I said "Presumably you weren't using it to sign the tokens, if they were surprised the client could access them?". I classed that as too obvious for it to be what you meant.
  - chrisandchris 4y ago
    For SSO? The biggest advantage (besides being stateless) about a JWT is that it is signed with an asymetric key and the client can validate the authenticity of the content. You can encrypt the content of the token, but that does not make to much sense (because the client anyway needs to decrypt it).
- lmc 4y ago
  It's kinda baffling that JWTs are unencrypted by default, to be fair.
jseban 4y ago
If people would just admit, and adapt to, the fact that the browser won over native, and Oracle won over Sun, we would avoid this situation with armies of Java developers making rube goldberg variations of basic relational, sysadmin, and gui programming tasks. But then again, how would we employ all the people with these extreme productivity multipliers, as long as our politics and economic system is still pretending that we just had the industrial revolution, and need to man the assembly lines, then Parkinson's law will apply for tech work just like everything else.
- l33t2328 4y ago
  When you say browser won over native, are you referring to the fact that software is more commonly accessed via web instead of software actually downloaded and installed on a user’s machine?
lucideer 4y ago
This is a great list. One minor criticism on... > Monorepos are easier to audit. > Speaking from the perspective of security researcher ergonomics, it was easier to audit a monorepo than a series of services split up into different code bases. There was no need to write wrapper scripts around the various tools we had. It was easier to determine if a given piece of code was used elsewhere. And best of all, there was no need to worry about a common library version being different on another repo. This is much more dependent on the auditor's personal workflows (as well as the relative hygiene of any team's monorepos), rather than being universal. I've found the opposite to be true for e.g. the current orgs that I am auditing: individually split up repos tend to be idiomatically structured, and "just work" as expected more often than monorepos, which more often than not have a lot of custom glue or unusual monorepo-management init scripts. Comments on the other (generally very good) points in the list: > Writing secure software has gotten remarkably easier in the last 10 years. I don’t have statistically sound evidence to back this up I suspect compiling such statistical evidence would also be impossible as detection of security issues has also improved, so any data would never be comparable over time. > The counterargument to this is that heavily weighting discoverability perpetuates ”Security by Obscurity,” since it relies so heavily on guessing what an attacker can or should know. But again, personal experience strongly suggests that in practice, discoverability is a great predictor of actual exploitation. This is a tough circle to square because security by obscurity works. It's probably the best security measure you can have in place. But it's bad for two reasons: (1) The process of obscuring often (doesn't need to, but very often) obscures auditing, which means you end up relying upon obscurity solely. It's not a worthwhile trade-off. (2) In a simplistic marketing world, the idea of obscurity as a standalone measure is so tempting to non-technical decision makers that I believe it requires a bit of innocent dishonesty about it's effectiveness to dissuade. > (on auditing dependencies) Node and npm were absolutely terrifying in this regard—the dependency chains were just not auditable. I agree with the overarching bullet point this is said within, but I see this point about NPM said a lot, and I'm not sure how people are going about auditing or how many language ecosystems they're looking at. I have found Node/NPM to be the best / second best popular system for auditing dependency chains. I have significant experience in this area: the relative consistency of package management config across the JS/TS ecosystem is enormously helpful for software composition analysis - the only package manager configs I've found that may be slightly better is Composer, but the inconsistent usage of Composer by many PHP devs still makes it a little worse than NPM in practice. PIP /PyPi / setuptools is an inconsistent moving target of requirements.txt (is it a lockfile?), pipfile.lock, setup.cfg -vs- setup.py, pyproject.toml, and whatever else. Maven is a nightmare of multiple registry endpoints, and issues parsing custom <dependencyManagement> directives, extensions (without even starting on maven wrappers and pom.xml templating strings). Don't get me started on Gradle. Go's idea of package management is: just pull it from Git; good luck automating it if you've got private repos with any kind of secure ssh auth. I have less personal experience with Rust/Cargo. > for some reason, PHP developers love to serialize/deserialize objects instead of using JSON PHP serialize/deserialize predates the existence of the JSON spec., so that might have something to do with it. A lot of PHP code is old. > Almost no one got JWT tokens and webhooks right on the first try. Nor the second try...
bambax 4y ago
> the major foot-gun that got a lot of places in trouble was the premature move to microservices I sometimes wonder if the move to microservices isn't just a weird consequence of Conway's law in reverse: make a department of each developer, let them have their thing. (See also this amazing video about Conway's law: https://www.youtube.com/watch?v=5IUj1EZwpJY https://www.youtube.com/watch?v=5IUj1EZwpJY )
- mjdiloreto 4y ago
  This is absolutely what microservices are about. It's arguably their strongest strength, because (at least in theory) I can decouple my team from your team and we can _only_ communicate over a strict interface.
  - treis 4y ago
    You can do that without introducing a HTTP/RPC boundary.
    - mejutoco 4y ago
      You can, but it requires discipline and/or tooling. With microservices you are very incentivized (I would say forced).
      - mjdiloreto 4y ago
        Exactly! To be clear to parent commenter, I'm not endorsing microservices to solve this organizational problem, just pointing out it's part of the reason to choose microservices.
      - treis 4y ago
        >You can, but it requires discipline and/or tooling. Pretty much every language comes with a way of exposing a limited API to other parts of the application. Java, as an example, requires you to specifically export the parts of your module that other modules are allowed to consume. If you only export a public API then you've achieved the same benefit as a microservice except now it's type checked and doesn't encounter the pitfalls of a network call.
criddell 4y ago
Does learnings ever mean anything different than lessons? How did this enter corporate-speak?
- phren0logy 4y ago
  Ugh, seriously. Like utilize instead of use.
  - quesera 4y ago
    In earlier days, I thought it'd be fun to have two versions of my resume. They had parallel content, but one was fluffed up in corporate-speak, and the other was human English. I included links, e.g. resume-fluffy.html or resume-direct.html, and (somewhat seriously) suggested that hiring managers read the first and tech evaluators the second. It made for some light humor in discussions with hiring groups. And also some effectively-paralyzed recruiters, which added to the fun of the former.
- _pete_ 4y ago
  it's almost - almost - as bad as 'vinyls'
- fluctor 4y ago
  I came here for this comment.
- vemv 4y ago
  Learnings can be more easily interpreted as "something I learned', while lessons can come across as 'lessons for you'.
- quesera 4y ago
  It pains my prescriptivist instincts to say so, but FWIW I do interpret them differently, frequently as the complementary sides of a single event: A learning is a successfully learned thing. Or a received lesson. A lesson is a taught thing. When effective, this would be one path to a learning for the receiver.
deleted 4y ago
[deleted]
rco8786 4y ago
> the major foot-gun (which I talk about more in a previous post on foot-guns) that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs. Preach! Micro services are a solution to a problem that affects effectively 0 startup sized systems. That problem is scale. Micro services are hard. WAY harder than monoliths. They become necessary only once your physical hardware can no longer keep up in a monolithic fashion and parts of your system need dedicated compute and/or storage. And no, they are not automatically necessary once your engineering team reaches N size either. Introducing network boundaries as a way to scale your engineering organization is a bad idea.
lpolovets 4y ago
> Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering. My hunch is that this is related to the nature of product-market fit. If a company is very successful, there's a decent shot that market demand became overwhelming at some point early on. That demand, in turn, becomes a strong motivator to keep things simple and ship quickly, instead of writing code The Right Way. Facebook using PHP might be one of the best examples of this: if their user base didn't explode, maybe they would've taken the time to carefully rewrite their code in Java or Python. But the fact that they would've had time to do that would've made it less likely that they'd become $500b company today.
- debacle 4y ago
  You're inverting the relationship. Simple solutions (technologically) can approach product/market fit much faster. Businesses fail for reasons besides tech, but on the tech side when businesses fail (in my experience), it's usually either from unwillingness to serve the sales cycle, or creating a technological solution that is not malleable.
maupin 4y ago
I've been a part of 3 startups, 2 of which failed and are no longer around. What they all could have benefited from was a business audit.
JoeNr76 4y ago
yes. When I get called in as a senior consultant for some business app, it's always for the same reason: development speed has crawled to an almost stop. And it is always caused by unnecessary complexity. I blame the fact that design patterns and specific architectures are being taught to people who don't understand the problem those things are trying to solve and just apply them everywhere. Any senior dev or architect should always live by this maxim: make it as simple as possible.
- Tomis02 4y ago
  A lot of unnecessary complexity comes from the use of library-like objects instead of plain functions + data. A recurring theme is "refactoring" specific functionality away into a generic object, and the consequence is a disconnect between the problem you are solving and the problem the object is solving. I often see objects that handle every possible input, ignoring that the business is only concerned with a small subset of inputs. You end up with a lot of "if impossible_condition_if_you_actually_look_at_your_data { /*some_dead_code*/ }". Another side-effect can be similar/identical input validation done at different levels of the stack. If you have object A calling object B calling object C, you sometimes notice how each one of those does the same exact thing in isolation of the others. You end up with a lot of extra checks and error handling because developers insist on writing their code in complete isolation from the context, pretending they don't know how it will be used. Of course, everything I described can also be "achieved" with plain functions + data, but (anecdotally) they usually produce better results, perhaps because it helps the devs not think in terms of objects.
Jwarder 4y ago
At what point should I push for a code audit? I don't think any of the codebases I worked on ever had a "real" audit. Best case was reviews pre/post acquisitions. An external audit seems like a good thing, but I have no idea how to argue for such a thing.
r2sk5t 4y ago
Thanks for writing this down @Ken. You're another example that learning the failure modes is the main benefit of being a consultant for many clients. Since I'm sure you began each audit meeting with the CTO/VPE and possibly others like senior devs/architects, how much of what you ended up finding in the audits was predictable based on those meetings? (I'm guessing almost everything). My follow-up question is that once you heard about their snazzy microservices architecture, were you ever surprised by it being a good decision based on the product type and how well it was engineered?
- lordofmoria 4y ago
  Honestly, early on in our code auditing days, there were surprises - a lot of the more meta-lessons in here fomented in the last few years, looking back, and would NOT have been something I’d have thought early on. On the other hand, regarding micro-services question: no, not even one surprised us positively. Now keep in mind, we didn’t audit absolutely massive FANG companies where mice services are probably necessary for org reasons(though a few unicorns/near-unicorns).
  - r2sk5t 4y ago
    Tangentially, I'm also guessing you can learn a lot by asking if they have an API for partners/customers, and if their application developers use the API internally, and then by looking at the API to see how well it is architected. When we integrate with 3rd party systems it's pretty easy to detect the well engineered systems from the ones built with baling wire and duct tape.
jrochkind1 4y ago
> Surprisingly, sometimes the most impressive products with the broadest scope of features were built by the smaller teams. Would probably not be surprising to Fred Brooks author of the _The Mythical Man-Month_, but as much as we think that book is famous/impactful, it still surprises us!
ghiculescu 4y ago
I assume this isn't content marketing because PKC doesn't seem to exist anymore. But this post made me really want to get Ken to audit our code. Are there any vendors that do similar work that people here recommend?
mpcannabrava 4y ago
Although I strongly agree with it in principle, I'm growing seriously tired of the "simpler is better" argument. It hides all the nuance, hard work and, guess what, complexity, that goes into making something simple. Simplicity is different to each person. What seems like unnecessary abstractions with complex inner workings often exist to actually hide other complexity away. Know the in and outs of Kubernetes? Maybe it's easier (simpler) for you than directly provisioning different pieces of infra. Have a team of over 10 [1] working on the same monolithic codebase? Productivity while maintaining sane separation of concerns might increase going for a more domain-service-oriented architecture [2]. How can we teach what simplicity is instead of just calling it better or saying arrogant platitudes like KISS? [1] yes, the number is that low, and often lower [2] yes, "micro" services does seem like a mistake in most cases
- idunnoman 4y ago
  As developers, I've come to believe that complexity is the worst sin we commit. Everything we talk about can be traced back to this issue. This is largely due to paying attention to Rich Hickey and learning Clojure. https://www.youtube.com/watch?v=SxdOUGdseq4 https://www.youtube.com/watch?v=SxdOUGdseq4
- lordofmoria 4y ago
  (op here) I actually completely agree - you're right: "simple outperformed smart" doesn't point to a useful, nuanced solution. I wrote more in-depth here about slightly-more-specifically where there are problems, curious your thoughts, feel free to DM me or comment on the blog (this thread is kinda dead)! https://kenkantzer.com/5-software-engineering-foot-guns/ https://kenkantzer.com/5-software-engineering-foot-guns/.
AtNightWeCode 4y ago
Point 2. I think this is a common misunderstanding in what engineering is. To come up with good simple solutions to complex problems often takes a lot of experience and domain knowledge.

798 points by lordofmoria 4y ago

Nit: "...or example because it’s so fast, it’s often used in automated testing to quickly generate a whole lot of sudo-random GUIDs." ITYM: "pseudo-random" Although I do like the mash-up concept of "sudo random"-ness.

mtVessel 4y ago
It's higher-privileged randomness. As in, all GUIDs are random, but some are more random than others.

zer01 4y ago

This is an interesting write up! The only question I have is around your point on monorepos - every monorepo I’ve seen has been a discoverability nightmare with bespoke configurations and archaic incantations (and sometimes installing Java!) necessary to even figure out what plugs in to what. How do you reason about a mono repo with new eyeballs? Do you get read in on things from an existing engineer? I struggle to understand how they’d make the job of auditing a software stack easier, except for maybe 3rd party dependency versions if they’re pulled in and shared.

yardstick 4y ago
Probably because: 1) It's easy to miss a repo, if you don't have a list of them all somewhere. 2) It's easy to get out of sync with what version of your software corresponds to what branch/tag in each repo.
- yourapostasy 4y ago
  > 2) It's easy to get out of sync with what version of software corresponds to what branch/tag in each repo. I'd like to hear how others solve this. The way I've addressed this is I bake into the build pipeline some way to dump to a text file all the version control metadata I could ever want to re-build the software from scratch. Then this text file is further embedded into the software primary executable itself, in some platform-compatible manner. Then I make sure the support team has the tooling to identify it in a trivial manner, whether a token-auth curl call to retrieve it over a REST API, or what have you. This goes well beyond the version number the users see, and supports detailed per-client patching information for temporary client-specific branches until they can be merged back into main without exposing those hairy details into the version number. While this works for me and no support teams have come to me yet with problems using this approach, it strikes me as inelegant and I'm for some reason dissatisfied with "it ain't broke so don't fix it".
  - treis 4y ago
    We have a releases repo that takes in the git version SHA for each application and handles deploys. It's... ok I guess. Just another example of complexity to meet the growing complexity of the system.
  - yardstick 4y ago
    In our case we abandoned individual repos and went back to a monorepo to solve this issue. In theory the separation of code was nice, but in practice it was a real pain when a service added new APIs you wanted to update another service to use it. All of our services do also print out in their startup logs what version they are based on git branch name and commit. Monorepo or not this was useful.
- cerved 4y ago
  find / -type d -name .git
  - yardstick 4y ago
    As an auditor you don't have anything checked out locally yet, so no .git will exist. If you ask an individual developer or randomly picked developers, they will only have their specific repos checked out. If you look at the server hosting the repos then yes you may get them all. Assuming they are all on one server...
- ge96 4y ago
  The thing I'm working on has 5 main repos that all run (yarn start) for the app to be fully functional. I need to put that down somewhere the order/matching branches.
- marcosdumay 4y ago
  > 2) It's easy to get out of sync with what version of your software corresponds to what branch/tag in each repo. That's what the `[dependencies] my-lib = "1.0"` was supposed to solve.
pianoben 4y ago
Monorepos do require upkeep beyond that of single-product repositories. You need some form of discipline for how code is organized (is it by Java package? by product? etc). You need to decide how ownership works. You need to decide on (and implement) a common way to set up local environments. Crucially, you need to reevaluate all these decisions periodically and make changes. On the other hand... this is all work you'd have to do anyways with multiple repositories. In the multi-repo scenario, it's even tougher to coordinate the dev environment, ownership, and organization principles - but the work isn't immediately obvious on checkout, so people don't always consider it. Regarding auditing, I have always found that having all the code in one place is tremendously useful in terms of discoverability! Want to know where that class comes from? Guaranteed if it's not third-party, you know where it is. Not to minimize the pain of poorly-managed monorepos - it's not a one-size-fits-all solution, and can definitely go sideways if left untended.
deleted 4y ago
[deleted]

yardstick 4y ago

> Custom fuzzing was surprisingly effective. A couple years into our code auditing, I started requiring all our code audits to include making a custom fuzzers to test product APIs, authentication, etc. Any recommendations for a good fuzzing tool for testing both web-based APIs and language specific APIs (C and Java in my case)?

asicsp 4y ago
These might help: * https://github.com/Endava/cats https://github.com/Endava/cats * https://github.com/google/oss-fuzz https://github.com/google/oss-fuzz
- deptm 4y ago
  Paid but integrates with CI/CD - https://www.code-intelligence.com/ https://www.code-intelligence.com/
agumonkey 4y ago
I'm thinking about introducing fuzzing too. And property based testing. Manual testing only is too limited.

7speter 4y ago

Its kind of discouraging to see the part where he says almost no one gets web tokens right the first time. Working on projects as someone entering the industry, its pretty clear that security is the most important part of a web app, and its so seemingly easy to get woefully wrong, especially if you’re learning this stuff on your own and trying to build working crud projects

yourapostasy 4y ago
That's usually because security is a bolt-on instead of bake-in within the control and data structures themselves. Too many people interpret "Make It Work Make It Right Make It Fast" to mean security is implemented at the "Make It Right" stage, when it should be at the "Make It Work" stage. That's if they're the lucky ones who get security designed in from the beginning into the architecture. We're paying for the sins of that in Unix these days, the kernel attack surface is in-feasibly large to remediate to correctness anytime soon (if ever?).
- makeitdouble 4y ago
  I think there is still more to it that just not taking it seriously or planning for it. JWT in particular has the weird quirks you need to know to prevent encryption swapping attacks, and I'm sure there's more traps I myself am not aware of. At this point I think security can be seen on the same plan as legal: assuming a random dev will be able to plan and navigate out all the issues by sheer common sense hasn't been a viable approach for long now.
  - yourapostasy 4y ago
    > At this point I think security can be seen on the same plan as legal:... Considering how Uber ignored legal ramifications of ride sharing intersecting with incumbent regulations until they were dragged into courts, that paints a potentially rather grim picture of the equivalent in software security. But your gist sounds more along the lines of, "include the experts along at the beginning of the ride". When I said security as a "bolt-on", I should have been more clear. Most of the time when I see it happening, it has been at the behest of the business stakeholders overriding the earnest developers trying to include the security teams from the beginning, but waved off with "it can be added later". The business stakeholders see in their real life housing contractors walk into finished houses, attach some doodads, pop in some batteries to wireless sensors and the central base station, and ta da!, they "have security"! And think, "just how hard can it be to do the same in software?", dismissing what their tech leads try to tell them. There is a large element of the principal-agent problem here as well. Shiny proofs of concepts and shallow implementations get immediate bonuses and promotions. Taking 1.1-2.0X as long to implement the right way, the result of which is no drama and no discernible difference to the casual business user, get no or even negative recognition. The incentives structure the choices. There are no incentives that structure payouts over the long-haul tying back to original historical choices, with an increasing gradient of the payout the longer the original choices prove sound. Naturally, since measuring that accurately would be impossible. The closest I've come to an analogy that works in these discussions but not as often as I'd like is this. I don't throw together four tilt-walls, top off with a roof, move in with a 20-ton safe, open the doors for business and call it a regional bank depository. There are bedrock anchors, sensors, inner reinforced concrete walls, SOP's, audits, man traps, insurance reviews, and on and on, that get designed in before the foundation is even poured. Clients who didn't find this convincing wave it off with a, "haha, this isn't that important lol". I want a better analogy.

peppermircat 4y ago

Interesting to see the JWT issue. I have recently found a vulnerability in a publicly traded CRM SaaS that was also about JWT claims validation. It’s also quite amazing that popular Auth SaaS rely so heavily on JWTs with 1 hour expiry times, making it impossible to log users as you can’t invalidate the token for the next hour.

dodgerdan 4y ago
https://www.howmanydayssinceajwtalgnonevuln.com/ https://www.howmanydayssinceajwtalgnonevuln.com/
blenderdt 4y ago
"...making it impossible to log users as you can’t invalidate the token for the next hour." I have no idea what you are talking about here, can you explain this? I work with systems that have a minute expire time. The only issue is that the clocks on all clients should be in sync with the auth server.
- habosa 4y ago
  I believe they are referring to the fact that most JWT-based auth systems use one-hour token expiry and have no ability to remotely revoke tokens. You can only revoke the user's ability to get the next token. This often leaves a one hour window between when you want the user locked out of your system and when they are practically logged out. The only way I know of to implement instant revocation in a system like this is to keep a blocklist of users/tokens that is constantly checked, which can be slow and removes some of the benefits of JWTs in the first place (that they carry all the auth information you need).
  - blenderdt 4y ago
    Ah! Yes this is why we use an expiration of one minute. For us the extra load that the refreshes give is not a problem. Keeping a blocklist seems unnecessary to me, you can just lower the expiration time.
4y ago

neilv 4y ago

> All the really bad security vulnerabilities were obvious. All the really bad security vulnerabilities that were found were obvious? One is more likely to find things that are obvious?

blenderdt 4y ago
When a team is so focused on the todo list they sometimes forget the obvious mistakes they still needed to fix.
lbriner 4y ago
But the auditers were experts and used all the latest and greatest tools. I think they are implying that if they couldn't find it with code inspection then a hacker wouldn't find it by probing. Of course, they might not find zero-days but most hackers wouldn't find those either.
itsdrewmiller 4y ago
Yeah this was a great article overall but that stood out as sus. Also the “last few hours found the most stuff”. Seems like they could probably stop the audit once they found enough problems, which skewed hard to easy to find and or last time to look.

jillesvangurp 4y ago

Guilty as charged. I bootstrapped a our startup with just myself and 2 junior engineers in the last year during Covid. Junior in the sense that they are young. But actually they outperform many older engineers I've worked with. We are starting to close some pretty big deals and I'm dreading the moment where I have to turn this into a normal development team. In my experience velocity drops when you do that and you lose a lot of momentum. 3 people can do a lot. 6 people don't do that much more. I'm not so young myself and quite experienced. But I lean heavily on my team for doing the work. I'm the CTO, the CPO, and I need to worry about team management, sales, and a few other things. So, less than half of my time is spent coding. This it the reality of startups. You have to do all of it. I made some technology choices early on. We use docker but not Kubernetes. There is one server, it's a monolith. There is one language, it is Kotlin. And we even use it on our frontend (web only). The latter is not something I would do normally or recommend. But both my junior engineers only knew Kotlin and we just went with it and never actually ended up regretting this. This surprised me and at this point I don't feel React/typescript have anything that I need or want. We're doing a lot of asynchronous stuff, websockets, maps (via libremaps), etc. And it all runs smoothly and responsively. Kotlin is kind of awesome for this actually. Originally our frontend was Android only. We ditched that app in favor of a Kotlin-js based web app that started out as a proof of concept that just more than proved the concept and became the actual thing. At the time we had a demand for IOS and web and no IOS or web developers on the team. Hence Kotlin for the web. When this looked like it was workable we actually lost our Android developer. So the decision to forget about that app was pretty easy. At that point it was half working and full of bugs and technical debt. Fixing that would only half fix our problem because we'd still need IOS and Web. So we did web first. And we are packaging it up with Cordova for those people that want something from an app store. It's a good lesson on prototyping. If it works, do more of it. At the same time, I normally recommend minimizing risk and not building too many things in parallel. Like building 3 apps for 3 platforms instead of just a web app. Our server is Kotlin/Spring boot and we use a lot of Elasticsearch because that's what I've been using for the last decade. A little bit of Redis and I've so far found no execuse to use a relational database. But I'd probably end up with mysql if that ever comes up. Done right, Elasticsearch makes for a nice key value store without transactions but with optimistic locking on documents. If I get some time, I add a database for safety at some point. But less moving parts means less headaches. Having just one language means the distinction between backend and frontend is a bit blurry. We have an api client that we use in our spring tests that also compiles to kotlin-js. That library contains a lot of code that we use in our front-end. Model classes, caching layers, functions that call various of our APIs, etc. And it's all covered in tests. All the business logic basically. If we ever need to do native apps, we'll use that there as well. On the devops front I'm a combination of very pragmatic but also focused. We use stuff that works that doesn't distract us. So, no terraform for a setup we only create once; in 20 minutes. Not worth spending weeks automating but worth documenting. But we do have CI/CD via github actions. So we don't manually deploy anything. And we have lots of API integration tests. If it builds, it ships. No rollbacks; roll forward only. Keeps things simple. We use Google Cloud and keep our cost low. A couple of VMs, a loadbalancer, a managed redis, and a managed elastic cloud cluster. That's it. Nice and simple.

lifeisstillgood 4y ago

At what point is there something beyond a framework - a SaaS in a box perhaps, that just avoids many of these basic problems (oh and the HR, legal, etc problems) of starting a startup. Startups are not snowflakes apart from that one little core competency. In short most serial founders say the second one was easier, simply because they followed the template ground out in the first. Would it be easier to start with that template?

alfonsodev 4y ago
all that exists as SAAS products that target non-technical cofounders, and it is very hard to justify to co-founders, investors, advisors .. any investment in time in something that is not your core problem, and I think is for a good reason.
gunnr15 4y ago
There are some good open source options like https://getzero.dev/ https://getzero.dev/
MathCodeLove 4y ago
I've seen boilerplate applications for <insert tech stack> but the open-source ones tend not to be great, and the closed source ones could be great - but I'm not willing to pay $XXX for code I haven't seen.
scottharveyco 4y ago
In the Rails world we have https://bullettrain.co https://bullettrain.co and https://jumpstartrails.com https://jumpstartrails.com which both have open source templates for building SaaS services.
czue 4y ago
There are a ton of products like this out there that build on popular frameworks: Saas Pegasus (https://www.saaspegasus.com/ https://www.saaspegasus.com/) for Python/Django, Bullet Train (https://bullettrain.co/ https://bullettrain.co/) and JumpStart (https://jumpstartrails.com/ https://jumpstartrails.com/) for Rails, Spark (https://spark.laravel.com/ https://spark.laravel.com/) for Laravel, Gravity (https://usegravity.app/ https://usegravity.app/) for JS You can find an even bigger list here: https://github.com/smirnov-am/awesome-saas-boilerplates https://github.com/smirnov-am/awesome-saas-boilerplates though those are the market leaders (I make one of them and follow things closely)

MattPalmer1086 4y ago

Interesting that he feels the default state of software security has improved a lot in the last few years. Anecdotally I'd also agree with that. Certainly better defaults and more secure libraries is a major factor. I haven't noticed a huge increase in developer security awareness, although I'd say it is also better than 10 years ago.

anticristi 4y ago
Unfortunately, I get the feeling that that is compensated by increasing risk. Attackers have found clever ways to monetize their work beyond just "fun". Hence, I feel the overall "security damage" has kind of stayed constant.
- MattPalmer1086 4y ago
  For sure, the threat level hasn't dropped. What is different is that attackers have to use different techniques, since the software isn't as easily exploitable as it used to be. Ten years ago, any pen test of a web application revealed loads of vulnerabilities. These days I rarely find anything really significant (although maybe I work at better places!). This is not to say that software isn't exploitable any more, only that the cost has been raised sufficiently to make cheaper attacks more attractive (e.g. phishing).

lifeisstillgood 4y ago

>>> Business logic flaws were rare, but when we found one they tended to be epically bad. oh yes ... I always bang on to my junior staff that their job was known as "analyst programmer" for a reason. The analyst part matters probably even more than the programmer part. In large companies just discovering what needs to be coded is 90% of the job, (the securely coding it in the constraints of the enterprise the other 90% while the final 90% is marketing your solution internally) Anyway .. yes

zeristor 4y ago
This goes with doing the first 90% of the work, then the second 90% of the work then the last 90% of the work. And engineers multiplying their initial estimate by 3, the project manager then multiplying that by 3 and rounding it up to be ten times more than the initial estimate.
watwut 4y ago
> In large companies just discovering what needs to be coded is 90% of the job Yes, but that is quite massive dysfunction of those companies. Meaning, we can yell at analysts-programmers as much as we want, what really needs to be fixed is the process that makes finding out requirements so ridiculously hard. And yes, I work in one of those companies, it very clearly is dysfunction.
- carlmr 4y ago
  I think this can only change when we, as a society, expect code literacy from every person that finishes high school. I don't mean expert programmers, but at least being able to read basic pseudocode algorithms. It's hard to describe a problem if you don't even understand any language.
  - sicp-enjoyer 4y ago
    The "magic AI" has undone years of coaching management about software expectations.
  - 4y ago

lars512 4y ago

Ah, yes, I’ve felt the pain of an unnecessary microservices migration. It ate time for years and the core was still a mess

anticristi 4y ago
I think people really exaggerated with the microservices trend. Today, I recommend to keep code in the same executable unless there is a good reason not to. Good reasons include: - Stateful vs stateless: databases and message queues should be your first (hopefully off-the-shelf) "microservices". - Different lifecycles: API serving vs background task - Different security needs: Frontoffice vs Backoffice code - Different teams: But make sure to introduce a clear customer-vendor relationship.

BrissyCoder 4y ago

> All the really bad security vulnerabilities were obvious. Isn't that just tautological? They are bad because they are obvious?

davedx 4y ago

Great article. To expand a little on why “Keep It Simple” is so powerful: less code = less bugs and less security issues. Less code = easier to change.

anticristi 4y ago
Also cleverness is overrated. You might be able to be clever once, but mid-term you will struggle to keep up with "collective cleverness". Sure today you might implement a better authentication code than the one offered in your favorite framework, but will you keep up with the new cleverness that will pour into the framework tomorrow?
muglug 4y ago
I don't think less code = less security issues. Often using those secure-by-default frameworks require more code. The simplest example in PHP (highlighted in the article for its default-insecurity): echo '<h1>Hello ' . $_GET['name'] . '</h1>'; is vulnerable to XSS. echo '<h1>Hello ' . htmlentities($_GET['name']) . '</h1>'; is not vulnerable
UK-Al05 4y ago
Less code and simple are not often related.
TheCondor 4y ago
I’ll go much further… I think it creates severe cultural problems. It creates the belief that problems are more difficult than they might be, it creates the belief that a particular solution may be more valuable than it actually is, and then it biases future team expansion and retention. Perhaps more ultimately, if the complexity creeps in before the real challenge gets do, it radically affects the team’s ability to reason about it.

jacquesm 4y ago

That's a very interesting set of findings. What is important to realize when reading this that it is a case of survivorship bias. The startups that were audited were obviously still alive, and any that suffered from such flaws that they were fatal had most likely already left the pool. In 15 years of doing technical due diligence (200+ jobs) I have yet to come across a company where the tech was what eventually killed them. But business case problems, for instance being unaware of the true cost of fielding a product and losing money on every transaction are extremely common. Improper cost allocation, product market mismatch, wishful thinking, founder conflicts, founder-investor conflicts, relying on non-existent technology while faking it for the time being and so on have all killed quite a few companies. Tech can be fixed, and if everything else is working fine there will be budget to do so. These other issues usually can't be fixed, no matter what the budget.

jackblemming 4y ago
>relying on non-existent technology while faking it for the time being and so on have all killed quite a few companies This doesn’t count as a tech problem?
- jacquesm 4y ago
  Obviously not, it's a problem to decide to fake non-existing tech but more of a management decision than a problem with the technology itself, there is an infinite number of things that don't exist, no matter how much you want them to exist and if you are not capable of coming up with a working solution and rely on the world around you to move fast enough to bail you out then I would say that is a psychological problem more than anything else. A common theme right now is 'AAI', using people to fake an AI that may not come into being at all, let alone before your runway (inevitably) runs out.
- lozenge 4y ago
  I saw one "secretary AI" that schedules meetings over email in your calendar. Just cc it to start using it (once you signed up). The idea seemingly being, fake it with low cost outsourcing to prove there's a demand for this and then make it. The developers you'd hire to make it an actual AI and the developers you'd hire to make it a Mechanical Turk are very different skill sets.

lstamour 4y ago

> For example because it’s so fast, [MD5 is] often used in automated testing to quickly generate a whole lot of sudo-random GUIDs. Actually, it’s because programmers are lazy. GUIDs or UUIDs are 128-bits and MD5 produces 128-bits. A string like “not-valid” is not a valid UUID, but MD5(“not-valid”) is both possible to format like a UUID when output as hex (with dashes) but also self-descriptive - so you can name the token when generating it in a fixture function and know how to regenerate it later in a test, for example. All the normal ways of generating UUIDs, including v6 and v7, are about trying to make them unique and collision resistant. But that’s nonsense when you want deterministic, reproducible tests. Hard-coding 32 characters is too much work, ain’t nobody got time for that. Magic numbers? Pfft. Just MD5 and write your own text… Pro tip: have data model creator helper functions include a counter that resets every test (every time the database resets) and then assign a UUID like MD5(`InsertTableName-${counter}`) that way you have a unique ID that’s also easy to predict/regenerate. That said… I’ve always personally preferred simple database IDs to be generally preferable over using UUIDs. It’s easier to understand THING 20 as an ID than 32-odd characters. But UUIDs are an industry standard, so they end up in your test code everywhere anyway…

jve 4y ago
> I’ve always personally preferred simple database IDs to be generally preferable over using UUIDs. Unless you start migrating data between environments and want references to be alive. Anyway, if you need a hardcoded GUID for tests or what, paste this into PowerShell: [Guid]::NewGuid() Not arguing, just developing for a system that uses guids as primary IDs and writing tests for that system. I don't even need to hardcode GUID, as within test bootstrap I'm creating objects with generated IDs I can reference later for comparison.
- lstamour 4y ago
  I’ve done that before too - but it’s always possible if you run tests often enough that you’ll get an ID collision that randomly fails a test and causes a developer some grief. Easier to not use random sources of data as a rule of thumb within your unit tests.
4y ago

mattbillenstein 4y ago

I've seen most of the architectural problems in consulting - it's amazing how a team of clever engineers can take a simple thing and make it sooo convoluted. Microservices, craptons of cloud-only dependencies, no way to easily create environments, ORMs and tooling that wraps databases creating crazy schemas... The list goes on and on; if you're early, you could do a lot worse than create a sensible monolith in a monorepo that uses just postgres, redis, and nginx and deploys using 'git pull' ...

claytonjy 4y ago
The worst architecture I ever saw came from consultants, who built the initial bits of a startup I was hired into. It was nice to have a no-longer-present scapegoat to shake fists at when frustrated, but over time I came to realize their most maddening choices were at the behest of one of our founders, who had no software experience.
- teaearlgraycold 4y ago
  I saw the same thing. Founders asking the world of consultants who would try to deliver and then fail to be a responsible engineer. I started my previous job by telling the founders they were asking for the wrong things and the consultants work needed to be thrown out. Thankfully they listened and we ended up with a TypeScript monorepo monolith deployed to Heroku.
commandlinefan 4y ago
> Microservices, craptons of cloud-only dependencies, no way to easily create environments, ORMs and tooling that wraps databases So, Spring Boot you mean?
- noisy_boy 4y ago
  A Spring Boot service doesn't have to Microservice - you can happily fatten it up into a monolith. Cloud-only dependencies would come into play for Spring cloud (or something that is using cloud specific features) - for a "vanilla" CRUD app, they are not needed. Creating virtual/physical environments is out of Spring Boot's scope and better left to external tools though it has support for separate environments via profiles. ORMs/tooling that wraps database doesn't have to be part of Spring Boot - using Hibernate/JPA isn't mandatory; plain JDBC Template with hand-written SQLs would work fine.

arethuza 4y ago

"All the really bad security vulnerabilities were obvious." I used to work for a company that did a lot of acquisitions and I often involved in working with teams at newly acquired companies - although it wasn't my main focus I did used to ask some simple security questions and it was remarkable what these uncovered. I literally had people run from the meeting to fix services after I had asked a simple question....

spuz 4y ago
Can you give some examples of some simple questions you would ask?
- arethuza 4y ago
  By the nature of that particular domain a lot of systems delivered important documents (often containing data of rather extreme commercial sensitivity) to customer organisations. A standard question I always asked was "given a URL that links to a document how do you authorise access" i.e. what happens if someone who is logged in to the site in question gets a link to a document and passes it to a friend via instant messaging.
  - davedx 4y ago
    Ha recognisable. A very annoying problem to solve with web tech too - there’s no perfect solution to this problem (that I know of).

dzonga 4y ago

> Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering. I wrote about this before that as an industry, we have made writing software complex for complexity's sake. > imagine, in a world where there wasn't a concept of clean code but to write code as simply as possible. not having thousands of classes with indirection. what if your code logic was simple functions and a bunch of if statements. not clever right, but it would work. what if your hiring process was not optimizing for algorithm efficiency but that something simply works reliably. imagine a world where the tooling used by software engineers wasn't fragile but simple to use and learn. oh the world would be a wonderful place, but the thing is most people don't know how to craft software. but here we're building software on a house of cards [0] :[0] - https://news.ycombinator.com/item?id=30166677 https://news.ycombinator.com/item?id=30166677

bacza2 4y ago
The problem is simple means diffrent thing in a small codebase than in a big one. A bunch of if statements in a code that is small enough to understand everything is ok but when it become big it's hard to understand flow of data. I do favor simple code but some complexity/abstracion is needed to make it easier to understand
- emn13 4y ago
  But picking the right abstractions that aren't leaky in any of the aspects you really care about is critical, hard to measure (leakiness isn't obvious, nor what kind of aspects you care about), hard to get right, and hard to maintain (because your abstraction may need to evolve, which is extra tricky). Obviously, getting that right makes subsequent developments much, much easier, but it's hardly a simple route to success.
  - ItsMonkk 4y ago
    I see tech debt and simplicity as a mixture between 'tyranny of small decisions' and each individual coders 'cleanliness' level. Each individual coder has a code cleanliness level, similar to how every friend's Mom growing up would always remark "Sorry the house is a mess", when it was spotless. If your used to 9/10 and it's a 7, that looks like a wreck. If you are used to 5 and it's a 7, that looks great. I urge other coders to increase their cleanliness level, and to look for others with high cleanliness for guidance. If you are coding next to people that 5 looks good to them, no matter how much they try to pay down technical debt, they never will. I think tyranny is ultimately showing us that the tooling that we currently have is making is much trickier than should be to evolve those abstractions. Partially this is because of bad abstractions that caused bad tooling and bad tooling that caused bad abstractions. Because it's so difficult, we don't do it. We take the small decision and work slightly harder in a slightly buggier environment to get the new thing done. But of course now the problem is bigger which means its even less likely for us to ever actually pay down that debt. > “I’m sorry I wrote you such a long letter. I didn’t have time to write you a short one.” – Blaise Pascal

huma 4y ago

> Generally, the major foot-gun that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs. Finally, someone said it

suzzer99 4y ago
This was 100% true for that startup I worked for as a side job. They would have been so much better off just building a standard java, PHP or .NET back end and calling it a day. The head engineer (who had known the guy funding the thing since childhood) had no clue how node, stateless architecture, or asynchronous code worked. He had somehow figured out how to get access to one particular worker of a node instance, through some internal ID or something, and used that to make stateful exchanges where one worker in some microservice was pinned to another. Which goes against everything node and microservices is about. I tried to talk some sense into them but they didn’t want to hear it. So for the last six months I just did my part and drained the big guy’s money like everyone else. I hate doing that - way more stressful than working your ass off.
wuliwong 4y ago
It is interesting, I've been at a company for a few years now and we've been slowing trying to break apart the two monoliths that most of the revenue generating traffic passes through. I am on board with the move to microservices. It's been a lot of fun but also a crazy amount of work and time and effort has been spent to do this. I've pondered both sides of this argument. On one hand if this move had been done earlier it might not have been as difficult a multi-year project. On the other hand, when I look at the Rails application in particular, it was coded SO poorly that I if it was just written better, initially, it wouldn't even need to be deconstructed at this point. Also, if the same engineers that wrote that Rails app tried to write a bunch of distributed, even-driven microservices instead of the Rails app, we would probably be in even worse shape. (ᵔ́∀ᵔ̀)
- ravedave5 4y ago
  Are you me? o_0. Shockingly similar situation.

gusbremm 4y ago

Once I worked on a team that none of the engineers knew that jwt payload was readable on the frontend. They were in shock when I extracted the payload and started asking questions about the data structure.

samhw 4y ago
I mean, I'd be rather surprised too. What were you using JWTs for, if not asymmetric crypto? Presumably you weren't using it to sign the tokens, if they were surprised the client could access them? And I can't see many contexts where you would use it with a shared secret, where just sending JSON over HTTPS wouldn't suffice. (I'm assuming 'frontend' here denotes a client on the other side of the trust boundary.)
- FlorianRappl 4y ago
  I'm not getting your comment. The payload is not encrypted. I think you refer to the signature. The payload can always be decoded. It's just JSON into base64.
  - samhw 4y ago
    Ah, sorry, that was what I was referring to when I said "Presumably you weren't using it to sign the tokens, if they were surprised the client could access them?". I classed that as too obvious for it to be what you meant.
- chrisandchris 4y ago
  For SSO? The biggest advantage (besides being stateless) about a JWT is that it is signed with an asymetric key and the client can validate the authenticity of the content. You can encrypt the content of the token, but that does not make to much sense (because the client anyway needs to decrypt it).
lmc 4y ago
It's kinda baffling that JWTs are unencrypted by default, to be fair.

jseban 4y ago

If people would just admit, and adapt to, the fact that the browser won over native, and Oracle won over Sun, we would avoid this situation with armies of Java developers making rube goldberg variations of basic relational, sysadmin, and gui programming tasks. But then again, how would we employ all the people with these extreme productivity multipliers, as long as our politics and economic system is still pretending that we just had the industrial revolution, and need to man the assembly lines, then Parkinson's law will apply for tech work just like everything else.

l33t2328 4y ago
When you say browser won over native, are you referring to the fact that software is more commonly accessed via web instead of software actually downloaded and installed on a user’s machine?

lucideer 4y ago

This is a great list. One minor criticism on... > Monorepos are easier to audit. > Speaking from the perspective of security researcher ergonomics, it was easier to audit a monorepo than a series of services split up into different code bases. There was no need to write wrapper scripts around the various tools we had. It was easier to determine if a given piece of code was used elsewhere. And best of all, there was no need to worry about a common library version being different on another repo. This is much more dependent on the auditor's personal workflows (as well as the relative hygiene of any team's monorepos), rather than being universal. I've found the opposite to be true for e.g. the current orgs that I am auditing: individually split up repos tend to be idiomatically structured, and "just work" as expected more often than monorepos, which more often than not have a lot of custom glue or unusual monorepo-management init scripts. Comments on the other (generally very good) points in the list: > Writing secure software has gotten remarkably easier in the last 10 years. I don’t have statistically sound evidence to back this up I suspect compiling such statistical evidence would also be impossible as detection of security issues has also improved, so any data would never be comparable over time. > The counterargument to this is that heavily weighting discoverability perpetuates ”Security by Obscurity,” since it relies so heavily on guessing what an attacker can or should know. But again, personal experience strongly suggests that in practice, discoverability is a great predictor of actual exploitation. This is a tough circle to square because security by obscurity works. It's probably the best security measure you can have in place. But it's bad for two reasons: (1) The process of obscuring often (doesn't need to, but very often) obscures auditing, which means you end up relying upon obscurity solely. It's not a worthwhile trade-off. (2) In a simplistic marketing world, the idea of obscurity as a standalone measure is so tempting to non-technical decision makers that I believe it requires a bit of innocent dishonesty about it's effectiveness to dissuade. > (on auditing dependencies) Node and npm were absolutely terrifying in this regard—the dependency chains were just not auditable. I agree with the overarching bullet point this is said within, but I see this point about NPM said a lot, and I'm not sure how people are going about auditing or how many language ecosystems they're looking at. I have found Node/NPM to be the best / second best popular system for auditing dependency chains. I have significant experience in this area: the relative consistency of package management config across the JS/TS ecosystem is enormously helpful for software composition analysis - the only package manager configs I've found that may be slightly better is Composer, but the inconsistent usage of Composer by many PHP devs still makes it a little worse than NPM in practice. PIP /PyPi / setuptools is an inconsistent moving target of requirements.txt (is it a lockfile?), pipfile.lock, setup.cfg -vs- setup.py, pyproject.toml, and whatever else. Maven is a nightmare of multiple registry endpoints, and issues parsing custom <dependencyManagement> directives, extensions (without even starting on maven wrappers and pom.xml templating strings). Don't get me started on Gradle. Go's idea of package management is: just pull it from Git; good luck automating it if you've got private repos with any kind of secure ssh auth. I have less personal experience with Rust/Cargo. > for some reason, PHP developers love to serialize/deserialize objects instead of using JSON PHP serialize/deserialize predates the existence of the JSON spec., so that might have something to do with it. A lot of PHP code is old. > Almost no one got JWT tokens and webhooks right on the first try. Nor the second try...

bambax 4y ago

> the major foot-gun that got a lot of places in trouble was the premature move to microservices I sometimes wonder if the move to microservices isn't just a weird consequence of Conway's law in reverse: make a department of each developer, let them have their thing. (See also this amazing video about Conway's law: https://www.youtube.com/watch?v=5IUj1EZwpJY https://www.youtube.com/watch?v=5IUj1EZwpJY )

mjdiloreto 4y ago
This is absolutely what microservices are about. It's arguably their strongest strength, because (at least in theory) I can decouple my team from your team and we can _only_ communicate over a strict interface.
- treis 4y ago
  You can do that without introducing a HTTP/RPC boundary.
  - mejutoco 4y ago
    You can, but it requires discipline and/or tooling. With microservices you are very incentivized (I would say forced).
    - mjdiloreto 4y ago
      Exactly! To be clear to parent commenter, I'm not endorsing microservices to solve this organizational problem, just pointing out it's part of the reason to choose microservices.
    - treis 4y ago
      >You can, but it requires discipline and/or tooling. Pretty much every language comes with a way of exposing a limited API to other parts of the application. Java, as an example, requires you to specifically export the parts of your module that other modules are allowed to consume. If you only export a public API then you've achieved the same benefit as a microservice except now it's type checked and doesn't encounter the pitfalls of a network call.

criddell 4y ago

Does learnings ever mean anything different than lessons? How did this enter corporate-speak?

phren0logy 4y ago
Ugh, seriously. Like utilize instead of use.
- quesera 4y ago
  In earlier days, I thought it'd be fun to have two versions of my resume. They had parallel content, but one was fluffed up in corporate-speak, and the other was human English. I included links, e.g. resume-fluffy.html or resume-direct.html, and (somewhat seriously) suggested that hiring managers read the first and tech evaluators the second. It made for some light humor in discussions with hiring groups. And also some effectively-paralyzed recruiters, which added to the fun of the former.
_pete_ 4y ago
it's almost - almost - as bad as 'vinyls'
fluctor 4y ago
I came here for this comment.
vemv 4y ago
Learnings can be more easily interpreted as "something I learned', while lessons can come across as 'lessons for you'.
quesera 4y ago
It pains my prescriptivist instincts to say so, but FWIW I do interpret them differently, frequently as the complementary sides of a single event: A learning is a successfully learned thing. Or a received lesson. A lesson is a taught thing. When effective, this would be one path to a learning for the receiver.

deleted 4y ago

[deleted]

rco8786 4y ago

> the major foot-gun (which I talk about more in a previous post on foot-guns) that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs. Preach! Micro services are a solution to a problem that affects effectively 0 startup sized systems. That problem is scale. Micro services are hard. WAY harder than monoliths. They become necessary only once your physical hardware can no longer keep up in a monolithic fashion and parts of your system need dedicated compute and/or storage. And no, they are not automatically necessary once your engineering team reaches N size either. Introducing network boundaries as a way to scale your engineering organization is a bad idea.

lpolovets 4y ago

> Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering. My hunch is that this is related to the nature of product-market fit. If a company is very successful, there's a decent shot that market demand became overwhelming at some point early on. That demand, in turn, becomes a strong motivator to keep things simple and ship quickly, instead of writing code The Right Way. Facebook using PHP might be one of the best examples of this: if their user base didn't explode, maybe they would've taken the time to carefully rewrite their code in Java or Python. But the fact that they would've had time to do that would've made it less likely that they'd become $500b company today.

debacle 4y ago
You're inverting the relationship. Simple solutions (technologically) can approach product/market fit much faster. Businesses fail for reasons besides tech, but on the tech side when businesses fail (in my experience), it's usually either from unwillingness to serve the sales cycle, or creating a technological solution that is not malleable.

maupin 4y ago

I've been a part of 3 startups, 2 of which failed and are no longer around. What they all could have benefited from was a business audit.

JoeNr76 4y ago

yes. When I get called in as a senior consultant for some business app, it's always for the same reason: development speed has crawled to an almost stop. And it is always caused by unnecessary complexity. I blame the fact that design patterns and specific architectures are being taught to people who don't understand the problem those things are trying to solve and just apply them everywhere. Any senior dev or architect should always live by this maxim: make it as simple as possible.

Tomis02 4y ago
A lot of unnecessary complexity comes from the use of library-like objects instead of plain functions + data. A recurring theme is "refactoring" specific functionality away into a generic object, and the consequence is a disconnect between the problem you are solving and the problem the object is solving. I often see objects that handle every possible input, ignoring that the business is only concerned with a small subset of inputs. You end up with a lot of "if impossible_condition_if_you_actually_look_at_your_data { /*some_dead_code*/ }". Another side-effect can be similar/identical input validation done at different levels of the stack. If you have object A calling object B calling object C, you sometimes notice how each one of those does the same exact thing in isolation of the others. You end up with a lot of extra checks and error handling because developers insist on writing their code in complete isolation from the context, pretending they don't know how it will be used. Of course, everything I described can also be "achieved" with plain functions + data, but (anecdotally) they usually produce better results, perhaps because it helps the devs not think in terms of objects.

Jwarder 4y ago

At what point should I push for a code audit? I don't think any of the codebases I worked on ever had a "real" audit. Best case was reviews pre/post acquisitions. An external audit seems like a good thing, but I have no idea how to argue for such a thing.

r2sk5t 4y ago

Thanks for writing this down @Ken. You're another example that learning the failure modes is the main benefit of being a consultant for many clients. Since I'm sure you began each audit meeting with the CTO/VPE and possibly others like senior devs/architects, how much of what you ended up finding in the audits was predictable based on those meetings? (I'm guessing almost everything). My follow-up question is that once you heard about their snazzy microservices architecture, were you ever surprised by it being a good decision based on the product type and how well it was engineered?

lordofmoria 4y ago
Honestly, early on in our code auditing days, there were surprises - a lot of the more meta-lessons in here fomented in the last few years, looking back, and would NOT have been something I’d have thought early on. On the other hand, regarding micro-services question: no, not even one surprised us positively. Now keep in mind, we didn’t audit absolutely massive FANG companies where mice services are probably necessary for org reasons(though a few unicorns/near-unicorns).
- r2sk5t 4y ago
  Tangentially, I'm also guessing you can learn a lot by asking if they have an API for partners/customers, and if their application developers use the API internally, and then by looking at the API to see how well it is architected. When we integrate with 3rd party systems it's pretty easy to detect the well engineered systems from the ones built with baling wire and duct tape.

jrochkind1 4y ago

> Surprisingly, sometimes the most impressive products with the broadest scope of features were built by the smaller teams. Would probably not be surprising to Fred Brooks author of the _The Mythical Man-Month_, but as much as we think that book is famous/impactful, it still surprises us!

ghiculescu 4y ago

I assume this isn't content marketing because PKC doesn't seem to exist anymore. But this post made me really want to get Ken to audit our code. Are there any vendors that do similar work that people here recommend?

mpcannabrava 4y ago

Although I strongly agree with it in principle, I'm growing seriously tired of the "simpler is better" argument. It hides all the nuance, hard work and, guess what, complexity, that goes into making something simple. Simplicity is different to each person. What seems like unnecessary abstractions with complex inner workings often exist to actually hide other complexity away. Know the in and outs of Kubernetes? Maybe it's easier (simpler) for you than directly provisioning different pieces of infra. Have a team of over 10 [1] working on the same monolithic codebase? Productivity while maintaining sane separation of concerns might increase going for a more domain-service-oriented architecture [2]. How can we teach what simplicity is instead of just calling it better or saying arrogant platitudes like KISS? [1] yes, the number is that low, and often lower [2] yes, "micro" services does seem like a mistake in most cases

idunnoman 4y ago
As developers, I've come to believe that complexity is the worst sin we commit. Everything we talk about can be traced back to this issue. This is largely due to paying attention to Rich Hickey and learning Clojure. https://www.youtube.com/watch?v=SxdOUGdseq4 https://www.youtube.com/watch?v=SxdOUGdseq4
lordofmoria 4y ago
(op here) I actually completely agree - you're right: "simple outperformed smart" doesn't point to a useful, nuanced solution. I wrote more in-depth here about slightly-more-specifically where there are problems, curious your thoughts, feel free to DM me or comment on the blog (this thread is kinda dead)! https://kenkantzer.com/5-software-engineering-foot-guns/ https://kenkantzer.com/5-software-engineering-foot-guns/.

AtNightWeCode 4y ago

Point 2. I think this is a common misunderstanding in what engineering is. To come up with good simple solutions to complex problems often takes a lot of experience and domain knowledge.