3 ms·

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

dnautics 6y ago
Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.
- ethbr0 6y ago
  > Please don't use csv Could you elaborate? I'm interested in the specific reasons.
  - judge2020 6y ago
    A quick rundown https://sqlite.org/aff_short.html https://sqlite.org/aff_short.html
  - apta 6y ago
    Reliability, atomic updates, roll backs, proper backups, history management, proper schema, etc.
  - duskwuff 6y ago
    High-quality software which interacts with real transactional SQL databases is readily available, free, and easy to use. Building an inferior solution yourself, like a CSV-based database, doesn't make your product any better. (If anything, it will probably make it worse.)
    - vorticalbox 6y ago
      It was never suggested to use a csv database, it was suggested that for small amounts of read only data a csv or other format file is a better option and I agree.
  - roenxi 6y ago
    If you don't know how to use a database for whatever reason and are doing something as a test-of-concept then CSV is fine. But for anything serious - databases, particularly older featureful ones like postgres, have a lot of efficiency tricks; clever things like indexes and nobody has ever come up with anything that is decisively better organised than the relational model of data. If you use a relational database, the worst-case outcome is you hit tremendous scale and have to do something special later on. The likely case scenario is some wins from the many performance lessons databases have learned. Best case outcome is avoiding a very costly excursion relearning lessons the database community has known about since 1970 (like reinventing transactions). Managing data with a csv (without a great reason) is like programming a complex GUI in assembly without a reason - it isn't going to look like a good decision in hindsight. Most data is not special, and databases are ready for it.
- theamk 6y ago
  Do use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have complex data (punctuation, newlines, etc..), it you might not be able to get the same data back using a different software. So consider some other storage formats -- there are tons. Like TSV (tab-separated-values) if you want it simple; json if you want great tooling support; jsonlines if you want to use json tools with old-school Unix tools as well; protobufs if you like schemas and speed; numpy's npy if you have millions of fixed-width records; and so on... There is no need to bother with SQL if the app will immediately load every row into memory and work with native objects.
  - daniellarusso 6y ago
    I agree with the JSON suggestion, but what advantage is there to TSV versus CSV? I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.
    - junon 6y ago
      Tabs do not appear in common literature. They're easier to justify inputs not have them in order to avoid having a quotation or escaping mess. Commas are _way_ too common. CSV is an awful format anyway.
    - theamk 6y ago
      If you can make sure the data has no newlines or tabs, the TSV needs no quoting. It is just "split" function, which is present in every language and very easy to use. When I use it, I usually add a check to writer that there is no newlines or tabs in data, and assert if this is not the case. You use this tsv with Unix tools like "cut", "paste", and ad-hoc scripts. There is also "tsv" as defined by excel, which has quoting and stuff. It is basically a dialect of CSV (Python even uses the same module to read it), and has all the disadvantages of CSV. Avoid it.
- hibbelig 6y ago
  The way I understand sushsh‘s suggestion is that CSV is an alternative to an API gateway and message queue. I don’t think the suggestion was to replace a database with CSV.
emodendroket 6y ago
There is sense in not building more services than you need. But many folks end up finding it hard to break away from their monolith and then it becomes an albatross. Not sure how to account for that.
- sushshshsh 6y ago
  parallel rewrites before the current prod system ever hits performance problems :)
  - arthurcolle 6y ago
    Sounds expensive
    - dwohnitmok 6y ago
      But potentially within the budget of a business with a large base of proven customers. While annoying technically, for early stage startups, performance problems caused by an overly large number of users are almost always a good problem to have and are a far rarer sight than startups that have over-architected their technical solution without the concomitant proven actual users.
- stepbeek 6y ago
  If a team doesn’t have the engineering know how to split a monolith into distinct services in a straightforward fashion, then I’m not sure that team will have the chops to start with microservices.
  - Aeolun 6y ago
    What you end up with is a set of tiny monoliths.
  - nucleardog 6y ago
    Dealing with existing code and moving it forward in significant ways without taking down production is always much more challenging than writing new code, whatever form those new ways take. You can get by with one strong lead defining services and interfaces for a bunch of code monkeys that write what goes behind them. Given an existing monolithic codebase, you can’t specify what high level services should exist and expect juniors to not only make the correct decisions on where the code should land up but also develop a migration plan such that you can move forward with existing functionality and customer data rather than starting from zero.
vp8989 6y ago
"for most people a web server and csv will serve the first thousand customers" I hear this kind of thought-terminating cliche a lot on here and it makes absolutely no sense. If # of users is a rough approximate of a company's success and more successful companies tend to hire more engineers ... then actually the majority of engineers would not have the luxury of not needing to think about scalability. With engineering salaries being what they are, why would you think that "most people" are employed working on systems that only have 1000 users?
- sushshshsh 6y ago
  "what is an MVP even", the post!

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

dnautics 6y ago

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

ethbr0 6y ago
> Please don't use csv Could you elaborate? I'm interested in the specific reasons.
- judge2020 6y ago
  A quick rundown https://sqlite.org/aff_short.html https://sqlite.org/aff_short.html
- apta 6y ago
  Reliability, atomic updates, roll backs, proper backups, history management, proper schema, etc.
- duskwuff 6y ago
  High-quality software which interacts with real transactional SQL databases is readily available, free, and easy to use. Building an inferior solution yourself, like a CSV-based database, doesn't make your product any better. (If anything, it will probably make it worse.)
  - vorticalbox 6y ago
    It was never suggested to use a csv database, it was suggested that for small amounts of read only data a csv or other format file is a better option and I agree.
- roenxi 6y ago
  If you don't know how to use a database for whatever reason and are doing something as a test-of-concept then CSV is fine. But for anything serious - databases, particularly older featureful ones like postgres, have a lot of efficiency tricks; clever things like indexes and nobody has ever come up with anything that is decisively better organised than the relational model of data. If you use a relational database, the worst-case outcome is you hit tremendous scale and have to do something special later on. The likely case scenario is some wins from the many performance lessons databases have learned. Best case outcome is avoiding a very costly excursion relearning lessons the database community has known about since 1970 (like reinventing transactions). Managing data with a csv (without a great reason) is like programming a complex GUI in assembly without a reason - it isn't going to look like a good decision in hindsight. Most data is not special, and databases are ready for it.
theamk 6y ago
Do use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have complex data (punctuation, newlines, etc..), it you might not be able to get the same data back using a different software. So consider some other storage formats -- there are tons. Like TSV (tab-separated-values) if you want it simple; json if you want great tooling support; jsonlines if you want to use json tools with old-school Unix tools as well; protobufs if you like schemas and speed; numpy's npy if you have millions of fixed-width records; and so on... There is no need to bother with SQL if the app will immediately load every row into memory and work with native objects.
- daniellarusso 6y ago
  I agree with the JSON suggestion, but what advantage is there to TSV versus CSV? I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.
  - junon 6y ago
    Tabs do not appear in common literature. They're easier to justify inputs not have them in order to avoid having a quotation or escaping mess. Commas are _way_ too common. CSV is an awful format anyway.
  - theamk 6y ago
    If you can make sure the data has no newlines or tabs, the TSV needs no quoting. It is just "split" function, which is present in every language and very easy to use. When I use it, I usually add a check to writer that there is no newlines or tabs in data, and assert if this is not the case. You use this tsv with Unix tools like "cut", "paste", and ad-hoc scripts. There is also "tsv" as defined by excel, which has quoting and stuff. It is basically a dialect of CSV (Python even uses the same module to read it), and has all the disadvantages of CSV. Avoid it.
hibbelig 6y ago
The way I understand sushsh‘s suggestion is that CSV is an alternative to an API gateway and message queue. I don’t think the suggestion was to replace a database with CSV.

emodendroket 6y ago

There is sense in not building more services than you need. But many folks end up finding it hard to break away from their monolith and then it becomes an albatross. Not sure how to account for that.

sushshshsh 6y ago
parallel rewrites before the current prod system ever hits performance problems :)
- arthurcolle 6y ago
  Sounds expensive
  - dwohnitmok 6y ago
    But potentially within the budget of a business with a large base of proven customers. While annoying technically, for early stage startups, performance problems caused by an overly large number of users are almost always a good problem to have and are a far rarer sight than startups that have over-architected their technical solution without the concomitant proven actual users.
stepbeek 6y ago
If a team doesn’t have the engineering know how to split a monolith into distinct services in a straightforward fashion, then I’m not sure that team will have the chops to start with microservices.
- Aeolun 6y ago
  What you end up with is a set of tiny monoliths.
- nucleardog 6y ago
  Dealing with existing code and moving it forward in significant ways without taking down production is always much more challenging than writing new code, whatever form those new ways take. You can get by with one strong lead defining services and interfaces for a bunch of code monkeys that write what goes behind them. Given an existing monolithic codebase, you can’t specify what high level services should exist and expect juniors to not only make the correct decisions on where the code should land up but also develop a migration plan such that you can move forward with existing functionality and customer data rather than starting from zero.

vp8989 6y ago

"for most people a web server and csv will serve the first thousand customers" I hear this kind of thought-terminating cliche a lot on here and it makes absolutely no sense. If # of users is a rough approximate of a company's success and more successful companies tend to hire more engineers ... then actually the majority of engineers would not have the luxury of not needing to think about scalability. With engineering salaries being what they are, why would you think that "most people" are employed working on systems that only have 1000 users?

sushshshsh 6y ago
"what is an MVP even", the post!