6 ms·

Sage advice. So often major architectural decisions are made in haste because of YAGNI arguments or business pressure. But a high-level design that can cope wit

Sage advice. So often major architectural decisions are made in haste because of YAGNI arguments or business pressure. But a high-level design that can cope with stress and volatility is so cheap to do early, and so expensive to try to back up and do later with an existing implementation.

davnicwil 3y ago
I guess it's only cheap (or good value) if you get it right though. How do you know you're getting it right early on?
- nine_k 3y ago
  Certain things, like using a database instead of a flat file when search is needed, or going with a linear algorithm instead of quadratic, are hard to do wrong. Deep assumptions about the problem space are hard(er) to do right. This is why the architecture should remain somehow flexible.
- IanCal 3y ago
  Experience in getting it wrong a whole bunch in the past, though you'll probably get something wrong this time too. It helps to know how far you can get with various approaches. Knowing that sqlite actually manages just fine for things up to X or where the reasonable edges are for using (say) a db for tasks rather than rabbitmq. Then some experience swapping things over. Been on a project upgrading from sqlite to postgres? Postgres on a single node to something larger? You'll have a better idea of how easy or hard these things are. Sometimes it's small things, like going with sqlite but being aware of the differences so that later the migration path is easier. You can also sketch out architectures, and see what does or doesn't change if you swap out a component/step. I'm probably going to have something do a basic iteration for comparing vectors right now because 1. I know the scale I'm at means that it's going to be near instantaneous 2. For a larger problem I know I can keep the same interface (find closest X) and swap in various other tools I know exist 3. Adding those tools should be possible but due to some constraints probably at least some hassle So I have a simple quick solution, but know how to change it later. I also have identified why I'm not going for the "proper" solution straight away, what the benefit is. However, I'm going to keep the interface as "find closest to string X" not "find closest to vector X" because I've seen the issues that arise when multiple different things care about exactly how the vector is constructed. I've been through that kind of thing which dramatically slowed deployment of updated systems. Perhaps in those other cases it wasn't particularly avoidable but the problem feels like it has the same shape (hard to describe but problems and solutions feel like they have shapes to me, I hope that makes sense regardless of how you picture them). These are some pretty small examples I know, but hopefully useful as approaches that can be communicated without a large background in a specific project. My main point is where I started - experience making these mistakes. I am quite aware of deadlocks because there's nothing that teaches the lesson about lock ordering quite as well as seeing a robot freeze mid-task in a demo. When mentoring, people seem surprised when I solve some of their problems immediately, it's not because I have quickly solved it from first principles but because I remember making the exact same mistake 10 times.
  - sanderjd 3y ago
    I think this is implicit in your comment, but making it more explicit: The more similar the new thing you're doing is to something you've already done, the better this works.
    - IanCal 3y ago
      Thanks that's worth making more explicit. To add to this, a very important part of the process to me is to start by asking how this problem is similar or different to other problems we've faced (very useful for estimation, you'd be shocked at how often people think a thing will take say 1 week when it's taken over a month each previous time something similar was done). This also means you can reach out to others if you identify a good similar problem but you weren't the one that worked on it. It decomposes too, so the vector thing above is new to me but "align version numbers" isn't.
- fluid_ident 3y ago
  > How do you know you're getting it right early on? I think that's definitely the key question. If it's the sort of situation where you've been given all the requirements up front - which does happen in certain domains - you can take the full-blown requirements engineering approach, and validation of the product will be built into the development lifecycle; it should be be effectively impossible to deliver an invalid product. (You might well go broke or lose all your team on death-marches trying to get there, but if you do, you know it'll be formally validated and signed off by the customer). But of course, this is expensive, doesn't preclude delivery of a brittle product that can cope with change, and, moreover, just won't fly for situations where the requirements aren't fixed at the outset and the business is relying on dynamic interaction with the development team to evolve the product. So, for those situations, we have, what? Heritage, heuristics, trends. And the results are, to me...unsatisfyingly contingent. Why is the software shaped this way, and not some way? For my $0.02, my projects are typically in between - there are some requirements provided, maybe they are a bit vague, but we largely know what we are trying to build, and we do some design to figure out what the big pieces are and how they ought to fit together. I've found some real success in approaches that subject the design of the software up front to the stresses of potential change, like those given by Juval Lowy (IDesign, Righting Software) and Barry O'Reilly (Black Tulip, Residuality Theory). And to subject a design to stress, of course first you have to do some system design! If you're not at the stage of being able to put together a design, I think it's likely more a prototyping or an investigative programming kind of activity, which to me has a different goal.
sanderjd 3y ago
I don't think it's cheap to do early. The costs are in people writing, drawing, talking, and often debating, rather than doing iterations on a basic working solution, and the outcome usually doesn't really work and needs to be redone in a year or two when more information is available. Which is not to say it isn't necessarily worth the cost. I just think the cost shouldn't be downplayed; it should be an investment undertaken with open eyes.
- ilyt 3y ago
  > I don't think it's cheap to do early. The costs are in people writing, drawing, talking, and often debating, rather than doing iterations on a basic working solution, and the outcome usually doesn't really work and needs to be redone in a year or two when more information is available. It is, for vast majority of apps. All you need to do to make sure your app running on one server can be run on 10 i.e. using actual SQL transactions to manage data, not in-program-memory locks. Then you can just get fatter DB server and boom, you're 10x Above that, sure, the "bigger server" starts to be expensive so you might need to start thinking about sharding, or moving some data off to separate instance (separate search cluster for example), or adding cache here and there, or, eventually, rebuilding the core of it for performance but that's 100x territory
  - sanderjd 3y ago
    Maybe. I dunno. I never seem to be working on things recently that fit into this "vast majority of apps" frame, where the right answer to up front architecture design is "just build it like a web app with a transactional database that scales with number of users". You're probably right that this is what the vast majority of projects look like, but it isn't very useful information when you aren't working on this kind of project.

davnicwil 3y ago

I guess it's only cheap (or good value) if you get it right though. How do you know you're getting it right early on?

nine_k 3y ago
Certain things, like using a database instead of a flat file when search is needed, or going with a linear algorithm instead of quadratic, are hard to do wrong. Deep assumptions about the problem space are hard(er) to do right. This is why the architecture should remain somehow flexible.
IanCal 3y ago
Experience in getting it wrong a whole bunch in the past, though you'll probably get something wrong this time too. It helps to know how far you can get with various approaches. Knowing that sqlite actually manages just fine for things up to X or where the reasonable edges are for using (say) a db for tasks rather than rabbitmq. Then some experience swapping things over. Been on a project upgrading from sqlite to postgres? Postgres on a single node to something larger? You'll have a better idea of how easy or hard these things are. Sometimes it's small things, like going with sqlite but being aware of the differences so that later the migration path is easier. You can also sketch out architectures, and see what does or doesn't change if you swap out a component/step. I'm probably going to have something do a basic iteration for comparing vectors right now because 1. I know the scale I'm at means that it's going to be near instantaneous 2. For a larger problem I know I can keep the same interface (find closest X) and swap in various other tools I know exist 3. Adding those tools should be possible but due to some constraints probably at least some hassle So I have a simple quick solution, but know how to change it later. I also have identified why I'm not going for the "proper" solution straight away, what the benefit is. However, I'm going to keep the interface as "find closest to string X" not "find closest to vector X" because I've seen the issues that arise when multiple different things care about exactly how the vector is constructed. I've been through that kind of thing which dramatically slowed deployment of updated systems. Perhaps in those other cases it wasn't particularly avoidable but the problem feels like it has the same shape (hard to describe but problems and solutions feel like they have shapes to me, I hope that makes sense regardless of how you picture them). These are some pretty small examples I know, but hopefully useful as approaches that can be communicated without a large background in a specific project. My main point is where I started - experience making these mistakes. I am quite aware of deadlocks because there's nothing that teaches the lesson about lock ordering quite as well as seeing a robot freeze mid-task in a demo. When mentoring, people seem surprised when I solve some of their problems immediately, it's not because I have quickly solved it from first principles but because I remember making the exact same mistake 10 times.
- sanderjd 3y ago
  I think this is implicit in your comment, but making it more explicit: The more similar the new thing you're doing is to something you've already done, the better this works.
  - IanCal 3y ago
    Thanks that's worth making more explicit. To add to this, a very important part of the process to me is to start by asking how this problem is similar or different to other problems we've faced (very useful for estimation, you'd be shocked at how often people think a thing will take say 1 week when it's taken over a month each previous time something similar was done). This also means you can reach out to others if you identify a good similar problem but you weren't the one that worked on it. It decomposes too, so the vector thing above is new to me but "align version numbers" isn't.
fluid_ident 3y ago
> How do you know you're getting it right early on? I think that's definitely the key question. If it's the sort of situation where you've been given all the requirements up front - which does happen in certain domains - you can take the full-blown requirements engineering approach, and validation of the product will be built into the development lifecycle; it should be be effectively impossible to deliver an invalid product. (You might well go broke or lose all your team on death-marches trying to get there, but if you do, you know it'll be formally validated and signed off by the customer). But of course, this is expensive, doesn't preclude delivery of a brittle product that can cope with change, and, moreover, just won't fly for situations where the requirements aren't fixed at the outset and the business is relying on dynamic interaction with the development team to evolve the product. So, for those situations, we have, what? Heritage, heuristics, trends. And the results are, to me...unsatisfyingly contingent. Why is the software shaped this way, and not some way? For my $0.02, my projects are typically in between - there are some requirements provided, maybe they are a bit vague, but we largely know what we are trying to build, and we do some design to figure out what the big pieces are and how they ought to fit together. I've found some real success in approaches that subject the design of the software up front to the stresses of potential change, like those given by Juval Lowy (IDesign, Righting Software) and Barry O'Reilly (Black Tulip, Residuality Theory). And to subject a design to stress, of course first you have to do some system design! If you're not at the stage of being able to put together a design, I think it's likely more a prototyping or an investigative programming kind of activity, which to me has a different goal.

sanderjd 3y ago

I don't think it's cheap to do early. The costs are in people writing, drawing, talking, and often debating, rather than doing iterations on a basic working solution, and the outcome usually doesn't really work and needs to be redone in a year or two when more information is available. Which is not to say it isn't necessarily worth the cost. I just think the cost shouldn't be downplayed; it should be an investment undertaken with open eyes.

ilyt 3y ago
> I don't think it's cheap to do early. The costs are in people writing, drawing, talking, and often debating, rather than doing iterations on a basic working solution, and the outcome usually doesn't really work and needs to be redone in a year or two when more information is available. It is, for vast majority of apps. All you need to do to make sure your app running on one server can be run on 10 i.e. using actual SQL transactions to manage data, not in-program-memory locks. Then you can just get fatter DB server and boom, you're 10x Above that, sure, the "bigger server" starts to be expensive so you might need to start thinking about sharding, or moving some data off to separate instance (separate search cluster for example), or adding cache here and there, or, eventually, rebuilding the core of it for performance but that's 100x territory
- sanderjd 3y ago
  Maybe. I dunno. I never seem to be working on things recently that fit into this "vast majority of apps" frame, where the right answer to up front architecture design is "just build it like a web app with a transactional database that scales with number of users". You're probably right that this is what the vast majority of projects look like, but it isn't very useful information when you aren't working on this kind of project.