6 ms·

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is not

by blagie 2y ago

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it. That's not just architecture. To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later: How do we make this practical / sustainable today? How do we make sure it can grow later when we have millions of customers? A lot of this can be left as scaffolding (we'll swap in [X], but for now, we'll do [Y]). But the key lessons are correct: - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) - Most data is rarely queried, and most queries are tiny. The first step in most big data jobs I've done is taking terabytes of data and shrinking it down to the GB, MB, or oven KB-scale data I need. Caveat: I have no algorithm for predicting what I'll need in the future. - Cost of data is increasing with regulatory.

deleted 2y ago
[deleted]
brtkdotse 2y ago
> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success
- blagie 2y ago
  It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons. If your goal is 5000 paying customers at $100 per year and $500k in annual revenues, that can lead to a very decent life. However, it's an entire different ballgame: (1) Don't take in investment (2) You probably can't hire more than one person (3) You need a plan for break-even revenue before you need to quit your job / run out of savings. (4) You need much greater than the 1-in-10 odds of success. And it's very possible (and probably not even hard) to start a sustainable 1-5 person business with >>50% odds of success, especially late career: - Find a niche you're aware of from your job - Do ballpark numbers on revenues. These should land in the $500k-$10M range. Less, and you won't sustain. More, and there will be too much competition. - Do it better than the (likely incompetent or non-existent) people doing it now - Use your network of industry contacts to sell it That's not a big enough market you need to worry about a lot of competition, competitors with VC funding, etc. Especially ones with tall moats do well -- pick some unique skillset, technology, or market access, for example. However, IF you've e.g. taken in VC funding, then you do need to plan for growth, and part of that is planning for the small odds your customer base (and ergo, your data) does grow.
  - IneffablePigeon 2y ago
    If you’re in b2b 5000 customers can be a lot more revenue than that. 10-100x, depending hugely on industry and product.
  - pas 2y ago
    > if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons most of the business plans are clear. acquire users and/or paying customers, build out the product, raise money and eventually raise prices too. engineering side is even simpler, keep the lights on, work with product designers to find incremental steps, iterate. as others have mentioned upthread if a startup finds a great niche (and manages to make it a whole new market for example) they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.) of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?)
    - blagie 2y ago
      > they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.) > ... > of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?) Netscape. (Plenty of others too)
      - pas 2y ago
        Netscape's main revenue came from Navigator, right? A browser. Classic commodity. How did they fail due to engineering deficiencies? (Netscape was better than Mosaic, they invented JavaScript, there was plenty of chops there, and initially they had revenue, IPO went great ... at that point it's arguably not a startup anymore.) And even when it failed it was worth 4.2B to AOL. ... if we're talking about browsers, Netscape's reincarnation Mozilla is again in the same tough spot. They have zero business sense, absolutely no plan, nada. At that point it's completely irrelevant how long it takes them to fix bugs or implement Web APIs or make it faster on various platforms. And still, they funded Rust and Servo.
- davedx 2y ago
  It's definitely an echo chamber. Most companies definitely do not want to become "unicorns" - most SME's around the world don't even know what a "unicorn" is, let alone be in an industry/sector where it's possible. Does a mining company want to become a "unicorn"? A fish and chip shop? Even within tech there is an extremely large number of companies whose goals are to steadily increase profits and return them to shareholders. 37 Signals is the posterchild there. Maybe if you're a VC funded startup then yeah.
- threeseed 2y ago
  HN is the worst echo chamber around. Obsessed with this "you must use PostgreSQL for every use case" nonsense. And that anyone who actually has unique data needs is simply doing it for their resume or are over-engineering.
  - paulryanrogers 2y ago
    > Obsessed with this "you must use PostgreSQL for every use case" nonsense. Pg fans are certainly here asking "why not PG?". Yet so are fans of other DBs; like DuckDB, CouchDB, SQLite, etc.
    - internet101010 2y ago
      I don't see so much DuckDB and CouchDB proselytizing but the SQLite force always out strong. I tend to divide the Postgres vs. SQLite decision on if the data in question is self-contained. Like am I pulling data from elsewhere (Postgres) or am I creating data within the application that is only used for the functionality of said application (SQLite).
      - int_19h 2y ago
        SQLite, in addition to just being plain popular, is a fairly natural stepping stone - you get a lot of fundamental benefits of an SQL RDBMS (abstract high-level queries, ACID etc) without the overhead of maintaining a database server. Postgres is the next obvious stepping stone after that, and the one where the vast majority of actual real-world cases that are not hypotheticals end up fitting.
      - assmanreturns 2y ago
        I think differently. If multiple users need to query the database directly then postgres. If no one is ever going to look at the db except one person or one application then sqllite. It's an interesting question as to whether micro services should be sqllite. Postgres is recommended because it's free, easy to maintain, install and it's has excellent query features.
  - 2y ago
- babel_ 2y ago
  Many startups seem to aim for this, naturally it's difficult to put actual numbers to this, and I'm sure many pursue multiple aims in the hope one of them sticks. Since unicorns are really just describing private valuation, really it's the same as saying many aim to get stupendously wealthy. Can't put a number on that, but you can at least see it's a hope for many, though "goal" is probably making it seem like they've got actually achievable plans for it... That, at least, I'm not so convinced of. Startups are, however, atypical from new businesses, ergo the unicorn myth, meaning we see many attempts to follow such a path that likely stands in the way of many new businesses from actually achieving the more real goals of, well, being a business, succeeding in their venture to produce whatever it is and reach their customers. I describe it as a unicorn "myth" as it very much behaves in such a way, and is misinterpreted similarly to many myths we tell ourselves. Unicorns are rare and successful because they had the right mixture of novel business and the security of investment or buyouts. Startups purportedly are about new ways of doing business, however the reality is only a handful really explore such (e.g. if it's SaaS, it's probably not a startup), meaning the others are just regular businesses with known paths ahead (including, of course, following in the footsteps of prior startups, which really is self-refuting). With that in mind, many of the "real" unicorns are realistically just highly valued new businesses (that got lucky and had fallbacks), as they are often not actually developing new approaches to business, whereas the mythical unicorns that startups want to be are half-baked ideas of how they'll achieve that valuation and wealth without much idea of how they do business (or that it can be fluid, matching their nebulous conception of it), just that "it'll come", especially with "growth". There is no nominative determinism, and all that, so businesses may call themselves startups all they like, but if they follow the patterns of startups without the massive safety nets of support and circumstance many of the real unicorns had, then a failure to develop out the business proper means they do indeed suffer themselves by not appreciating 5000 paying customers and instead aim for "world domination", as it were, or acquisition (which they typically don't "survive" from, as an actual business venture). The studies have shown this really does contribute to the failure rate and instability of so-called startups, effectively due to not cutting it as businesses, far above the expected norm of new businesses... So that pet peeve really is indicative of a much more profound issue that, indeed, seems to be a bit of an echo chamber blind spot with HN. After all, if it ought to have worked all the time, reality would look very different from today. Just saying how many don't become unicorns (let alone the failure rate) doesn't address the dissonance from then concluding "but this time will be different". It also doesn't address the idea that you don't need to become a "unicorn", and maybe shouldn't want to either... but that's a line of thinking counter to the echo chamber, so I won't belabour it here.
notachatbot1234 2y ago
> - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) Nitpick but I cannot help myself: 8 bits are not even enough for a unique integer ID per person, that would require 8 bytes per person and then we are at 60GB already. I agree with pretty much anything else you said, just this stood out as wrong and Duty Calls.
- iraqmtpizza 2y ago
  meh. memory address is the ID
  - L-four 2y ago
    Airline booking numbers used to just be the sector number of your booking record on the mainframes HDD.
    - switch007 2y ago
      My jaw just hit the floor. What a fascinating fact!
    - rrr_oh_man 2y ago
      That’s why they were constantly recycled?
    - mauriciolange 2y ago
      source?
    - devsda 2y ago
      This is such a simple scheme. I wonder how they dealt with common storage issues like backups and disks having bad sectors.
      - giantrobot 2y ago
        They're likely record based formatting rather than file based. At the high level the code is just asking for a record number from a data set. The data set is managed including redundancy/ECC by the hardware of that storage device.
OJFord 2y ago
> 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). I think you're missing quite a few 9s!
davedx 2y ago
> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because people never just make plans, they usually implement too. Plan for the next 3 months and you'll be much more agile and productive. Your startup will never become a unicorn if you can't execute.
- MOARDONGZPLZ 2y ago
  In my experience the drag caused from the thinking to plan for scalability early has been so much greater than the effort to rearchitect things when and if the company becomes a unicorn that one is significantly more likely to become a unicorn if they simply focus on execution and very fast iteration and save the scalability until it’s actually needed (and they can hire a team of whomever to effect this change with their newly minted unicorn cachet).
  - tdeck 2y ago
    Not only that but the work to rearchitect for scalability is often appealing, energizing work that lots of people want to do. While the work of maintaining overly complex software full of affordances for scalability scenarios (or use cases) that never come up is rarely as productive or motivating.
- newaccount74 2y ago
  The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all. I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.
  - fuzzy2 2y ago
underwater 2y ago
> To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later I think that in practice that’s counterproductive. A startup has a limited runway. If your engineers are spending your money on something that doesn’t pay off for years then they’re increasing the chance you’ll fail before it matters.
- blagie 2y ago
  You're confusing planning with implementation. Planning is a weekend, or at most a few weeks.
threeseed 2y ago
> nothing Mongo does which postgresql doesn't do better a) It has a built-in and supported horizontal scalability / HA solution. b) For some use cases e.g. star schemas it has significantly better performance. > Big data solutions aren't nosql Almost all big data storage solutions are NoSQL.
- ozkatz 2y ago
  > Almost all big data storage solutions are NoSQL. I think it's important to distinguish between OLAP AND OLTP. For OLAP use cases (which is what this post is mostly about) it's almost 100% SQL. The biggest players being Databricks, Snowflake and BigQuery. Other tools may include AWS's tools (Glue, Athena), Trino, ClickHouse, etc. I bet there's a <1% market for "NoSQL" tools such as MongoDB's "Atlas Data Lake" and probably a bunch of MapReduce jobs still being used in production, but these are the exception, not the rule. For OLTP "big data", I'm assuming we're talking about "scale-out" distributed databases which are either SQL (e.g. cockroachdb, vitess, etc) SQL-like (Casandra's CQL, Elasticsearch's non-ANSI SQL, Influx' InfluxQL) or a purpose-built language/API (Redis, MongoDB). I wouldn't say OLTP is "almost all" NoSQL, but definitely a larger proportion compared to OLAP.
- zxxh 2y ago
  Anyone reading the above comment ask yourself. Do you actually think when interest rates dropped to zero they suddenly inverted a system that was better than SQL. “Horizontal scaling” I’m sorry I don’t speak marketing language, what is that? I’ve only been doing this for decades.
- blagie 2y ago
  > Almost all big data storage solutions are NoSQL. Most I've seen aren't. NoSQL means non-relational database. Most big data solutions I've seen will not use a database at all. An example is hadoop. Once you have a database, SQL makes a lot of sense. There are big data SQL solutions, mostly in the form of columnar read-optimized databases. On the above, a little bit of relational can make a huge performance difference, in the form of, for example, a big table with compact data with indexes into small data tables. That can be algorithmically a lot more performant than the same thing without relations.
Toine 2y ago
> To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later Sources ?
boxed 2y ago
I see people planning for success to the point of guaranteeing failure, much more than people who suddenly must try to handle success in panic. It's a second system syndrome + survivor bias thing I think: people who had to clean up the mess of a good MVP complaining about what wasn't done before. But the companies that DID do that planning and architecting before did not survive to be complained about.
- CuriouslyC 2y ago
  It's not either or. There are best practices that can be followed regardless with no time cost up front, and there is taking some time to think about how your product might evolve (which you really should be doing anyhow) then making choices with your software that don't make the evolution process harder than it needs to be. Layers of abstraction make code harder to reason about and work with, so it's a lose lose when trying to iterate quickly, but there's also the idea of architectural "mise en place" vs "just dump shit where it's most convenient right now and don't worry about later" which will result near immediate productivity losses due to system incoherence and disorganization.
  - boxed 2y ago
    I'm a big fan of "optimize for deletion" (aka leaf-heavy) code. It's good for reasoning when the system is big, and it's good for growing a code base. It's a bit annoying how the design of Django templates works against this by not allowing free functions...
nemo44x 2y ago
Mongo allows a developer to burn down a backlog faster than anything else. That’s why it’s so popular. The language drivers interface with the database which just says yes. And whatever happens later is someone else’s problem. Although it’s a far more stable thing today.
zemo 2y ago
> The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it. If you have a product gaining that much traction, it’s usually because of some compound effect based on the existence and needs of its userbase. If on the way up you stumble to add new users, the userbase that’s already there is unlikely to go back to the Old Thing or go somewhere else (because these events are actually rare). For a good while using Twitter meant seeing the fail whale every day. Most people didn’t just up and leave, and nothing else popped up that could scale better that people moved to. Making a product that experiences exponential growth in that way is pretty rare, and struggling to scale those cases and having a period of availability degradation is common. What products hit an exponential growth situation failed because they couldn’t scale?
anon84873628 2y ago
>The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. I think that was exactly their point. If new architectures were actually necessary, we would have seen a greater rise in Mongo and the like. But we didn't, because the existing systems were perfectly adequate.
tdeck 2y ago
> Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) I suppose there are only 256 unique types of people on the planet? :)

by blagie 2y ago

deleted 2y ago

[deleted]

brtkdotse 2y ago

> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success

blagie 2y ago
It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons. If your goal is 5000 paying customers at $100 per year and $500k in annual revenues, that can lead to a very decent life. However, it's an entire different ballgame: (1) Don't take in investment (2) You probably can't hire more than one person (3) You need a plan for break-even revenue before you need to quit your job / run out of savings. (4) You need much greater than the 1-in-10 odds of success. And it's very possible (and probably not even hard) to start a sustainable 1-5 person business with >>50% odds of success, especially late career: - Find a niche you're aware of from your job - Do ballpark numbers on revenues. These should land in the $500k-$10M range. Less, and you won't sustain. More, and there will be too much competition. - Do it better than the (likely incompetent or non-existent) people doing it now - Use your network of industry contacts to sell it That's not a big enough market you need to worry about a lot of competition, competitors with VC funding, etc. Especially ones with tall moats do well -- pick some unique skillset, technology, or market access, for example. However, IF you've e.g. taken in VC funding, then you do need to plan for growth, and part of that is planning for the small odds your customer base (and ergo, your data) does grow.
- IneffablePigeon 2y ago
  If you’re in b2b 5000 customers can be a lot more revenue than that. 10-100x, depending hugely on industry and product.
- pas 2y ago
  > if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons most of the business plans are clear. acquire users and/or paying customers, build out the product, raise money and eventually raise prices too. engineering side is even simpler, keep the lights on, work with product designers to find incremental steps, iterate. as others have mentioned upthread if a startup finds a great niche (and manages to make it a whole new market for example) they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.) of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?)
  - blagie 2y ago
    > they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.) > ... > of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?) Netscape. (Plenty of others too)
    - pas 2y ago
      Netscape's main revenue came from Navigator, right? A browser. Classic commodity. How did they fail due to engineering deficiencies? (Netscape was better than Mosaic, they invented JavaScript, there was plenty of chops there, and initially they had revenue, IPO went great ... at that point it's arguably not a startup anymore.) And even when it failed it was worth 4.2B to AOL. ... if we're talking about browsers, Netscape's reincarnation Mozilla is again in the same tough spot. They have zero business sense, absolutely no plan, nada. At that point it's completely irrelevant how long it takes them to fix bugs or implement Web APIs or make it faster on various platforms. And still, they funded Rust and Servo.
davedx 2y ago
It's definitely an echo chamber. Most companies definitely do not want to become "unicorns" - most SME's around the world don't even know what a "unicorn" is, let alone be in an industry/sector where it's possible. Does a mining company want to become a "unicorn"? A fish and chip shop? Even within tech there is an extremely large number of companies whose goals are to steadily increase profits and return them to shareholders. 37 Signals is the posterchild there. Maybe if you're a VC funded startup then yeah.
threeseed 2y ago
HN is the worst echo chamber around. Obsessed with this "you must use PostgreSQL for every use case" nonsense. And that anyone who actually has unique data needs is simply doing it for their resume or are over-engineering.
- paulryanrogers 2y ago
  > Obsessed with this "you must use PostgreSQL for every use case" nonsense. Pg fans are certainly here asking "why not PG?". Yet so are fans of other DBs; like DuckDB, CouchDB, SQLite, etc.
  - internet101010 2y ago
    I don't see so much DuckDB and CouchDB proselytizing but the SQLite force always out strong. I tend to divide the Postgres vs. SQLite decision on if the data in question is self-contained. Like am I pulling data from elsewhere (Postgres) or am I creating data within the application that is only used for the functionality of said application (SQLite).
    - int_19h 2y ago
      SQLite, in addition to just being plain popular, is a fairly natural stepping stone - you get a lot of fundamental benefits of an SQL RDBMS (abstract high-level queries, ACID etc) without the overhead of maintaining a database server. Postgres is the next obvious stepping stone after that, and the one where the vast majority of actual real-world cases that are not hypotheticals end up fitting.
    - assmanreturns 2y ago
      I think differently. If multiple users need to query the database directly then postgres. If no one is ever going to look at the db except one person or one application then sqllite. It's an interesting question as to whether micro services should be sqllite. Postgres is recommended because it's free, easy to maintain, install and it's has excellent query features.
- 2y ago
babel_ 2y ago
Many startups seem to aim for this, naturally it's difficult to put actual numbers to this, and I'm sure many pursue multiple aims in the hope one of them sticks. Since unicorns are really just describing private valuation, really it's the same as saying many aim to get stupendously wealthy. Can't put a number on that, but you can at least see it's a hope for many, though "goal" is probably making it seem like they've got actually achievable plans for it... That, at least, I'm not so convinced of. Startups are, however, atypical from new businesses, ergo the unicorn myth, meaning we see many attempts to follow such a path that likely stands in the way of many new businesses from actually achieving the more real goals of, well, being a business, succeeding in their venture to produce whatever it is and reach their customers. I describe it as a unicorn "myth" as it very much behaves in such a way, and is misinterpreted similarly to many myths we tell ourselves. Unicorns are rare and successful because they had the right mixture of novel business and the security of investment or buyouts. Startups purportedly are about new ways of doing business, however the reality is only a handful really explore such (e.g. if it's SaaS, it's probably not a startup), meaning the others are just regular businesses with known paths ahead (including, of course, following in the footsteps of prior startups, which really is self-refuting). With that in mind, many of the "real" unicorns are realistically just highly valued new businesses (that got lucky and had fallbacks), as they are often not actually developing new approaches to business, whereas the mythical unicorns that startups want to be are half-baked ideas of how they'll achieve that valuation and wealth without much idea of how they do business (or that it can be fluid, matching their nebulous conception of it), just that "it'll come", especially with "growth". There is no nominative determinism, and all that, so businesses may call themselves startups all they like, but if they follow the patterns of startups without the massive safety nets of support and circumstance many of the real unicorns had, then a failure to develop out the business proper means they do indeed suffer themselves by not appreciating 5000 paying customers and instead aim for "world domination", as it were, or acquisition (which they typically don't "survive" from, as an actual business venture). The studies have shown this really does contribute to the failure rate and instability of so-called startups, effectively due to not cutting it as businesses, far above the expected norm of new businesses... So that pet peeve really is indicative of a much more profound issue that, indeed, seems to be a bit of an echo chamber blind spot with HN. After all, if it ought to have worked all the time, reality would look very different from today. Just saying how many don't become unicorns (let alone the failure rate) doesn't address the dissonance from then concluding "but this time will be different". It also doesn't address the idea that you don't need to become a "unicorn", and maybe shouldn't want to either... but that's a line of thinking counter to the echo chamber, so I won't belabour it here.

notachatbot1234 2y ago

> - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) Nitpick but I cannot help myself: 8 bits are not even enough for a unique integer ID per person, that would require 8 bytes per person and then we are at 60GB already. I agree with pretty much anything else you said, just this stood out as wrong and Duty Calls.

iraqmtpizza 2y ago
meh. memory address is the ID
- L-four 2y ago
  Airline booking numbers used to just be the sector number of your booking record on the mainframes HDD.
  - switch007 2y ago
    My jaw just hit the floor. What a fascinating fact!
  - rrr_oh_man 2y ago
    That’s why they were constantly recycled?
  - mauriciolange 2y ago
    source?
  - devsda 2y ago
    This is such a simple scheme. I wonder how they dealt with common storage issues like backups and disks having bad sectors.
    - giantrobot 2y ago
      They're likely record based formatting rather than file based. At the high level the code is just asking for a record number from a data set. The data set is managed including redundancy/ECC by the hardware of that storage device.

OJFord 2y ago

> 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). I think you're missing quite a few 9s!

davedx 2y ago

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because people never just make plans, they usually implement too. Plan for the next 3 months and you'll be much more agile and productive. Your startup will never become a unicorn if you can't execute.

MOARDONGZPLZ 2y ago
In my experience the drag caused from the thinking to plan for scalability early has been so much greater than the effort to rearchitect things when and if the company becomes a unicorn that one is significantly more likely to become a unicorn if they simply focus on execution and very fast iteration and save the scalability until it’s actually needed (and they can hire a team of whomever to effect this change with their newly minted unicorn cachet).
- tdeck 2y ago
  Not only that but the work to rearchitect for scalability is often appealing, energizing work that lots of people want to do. While the work of maintaining overly complex software full of affordances for scalability scenarios (or use cases) that never come up is rarely as productive or motivating.
newaccount74 2y ago
The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all. I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.
- fuzzy2 2y ago

underwater 2y ago

> To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later I think that in practice that’s counterproductive. A startup has a limited runway. If your engineers are spending your money on something that doesn’t pay off for years then they’re increasing the chance you’ll fail before it matters.

blagie 2y ago
You're confusing planning with implementation. Planning is a weekend, or at most a few weeks.

threeseed 2y ago

> nothing Mongo does which postgresql doesn't do better a) It has a built-in and supported horizontal scalability / HA solution. b) For some use cases e.g. star schemas it has significantly better performance. > Big data solutions aren't nosql Almost all big data storage solutions are NoSQL.

ozkatz 2y ago
> Almost all big data storage solutions are NoSQL. I think it's important to distinguish between OLAP AND OLTP. For OLAP use cases (which is what this post is mostly about) it's almost 100% SQL. The biggest players being Databricks, Snowflake and BigQuery. Other tools may include AWS's tools (Glue, Athena), Trino, ClickHouse, etc. I bet there's a <1% market for "NoSQL" tools such as MongoDB's "Atlas Data Lake" and probably a bunch of MapReduce jobs still being used in production, but these are the exception, not the rule. For OLTP "big data", I'm assuming we're talking about "scale-out" distributed databases which are either SQL (e.g. cockroachdb, vitess, etc) SQL-like (Casandra's CQL, Elasticsearch's non-ANSI SQL, Influx' InfluxQL) or a purpose-built language/API (Redis, MongoDB). I wouldn't say OLTP is "almost all" NoSQL, but definitely a larger proportion compared to OLAP.
zxxh 2y ago
Anyone reading the above comment ask yourself. Do you actually think when interest rates dropped to zero they suddenly inverted a system that was better than SQL. “Horizontal scaling” I’m sorry I don’t speak marketing language, what is that? I’ve only been doing this for decades.
blagie 2y ago
> Almost all big data storage solutions are NoSQL. Most I've seen aren't. NoSQL means non-relational database. Most big data solutions I've seen will not use a database at all. An example is hadoop. Once you have a database, SQL makes a lot of sense. There are big data SQL solutions, mostly in the form of columnar read-optimized databases. On the above, a little bit of relational can make a huge performance difference, in the form of, for example, a big table with compact data with indexes into small data tables. That can be algorithmically a lot more performant than the same thing without relations.

Toine 2y ago

> To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later Sources ?

boxed 2y ago

I see people planning for success to the point of guaranteeing failure, much more than people who suddenly must try to handle success in panic. It's a second system syndrome + survivor bias thing I think: people who had to clean up the mess of a good MVP complaining about what wasn't done before. But the companies that DID do that planning and architecting before did not survive to be complained about.

CuriouslyC 2y ago
It's not either or. There are best practices that can be followed regardless with no time cost up front, and there is taking some time to think about how your product might evolve (which you really should be doing anyhow) then making choices with your software that don't make the evolution process harder than it needs to be. Layers of abstraction make code harder to reason about and work with, so it's a lose lose when trying to iterate quickly, but there's also the idea of architectural "mise en place" vs "just dump shit where it's most convenient right now and don't worry about later" which will result near immediate productivity losses due to system incoherence and disorganization.
- boxed 2y ago
  I'm a big fan of "optimize for deletion" (aka leaf-heavy) code. It's good for reasoning when the system is big, and it's good for growing a code base. It's a bit annoying how the design of Django templates works against this by not allowing free functions...

nemo44x 2y ago

Mongo allows a developer to burn down a backlog faster than anything else. That’s why it’s so popular. The language drivers interface with the database which just says yes. And whatever happens later is someone else’s problem. Although it’s a far more stable thing today.

zemo 2y ago

> The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it. If you have a product gaining that much traction, it’s usually because of some compound effect based on the existence and needs of its userbase. If on the way up you stumble to add new users, the userbase that’s already there is unlikely to go back to the Old Thing or go somewhere else (because these events are actually rare). For a good while using Twitter meant seeing the fail whale every day. Most people didn’t just up and leave, and nothing else popped up that could scale better that people moved to. Making a product that experiences exponential growth in that way is pretty rare, and struggling to scale those cases and having a period of availability degradation is common. What products hit an exponential growth situation failed because they couldn’t scale?

anon84873628 2y ago

>The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. I think that was exactly their point. If new architectures were actually necessary, we would have seen a greater rise in Mongo and the like. But we didn't, because the existing systems were perfectly adequate.

tdeck 2y ago

> Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) I suppose there are only 256 unique types of people on the planet? :)