6 ms·

After 20 years of programming, this is what comes to my mind when I hear "How do Design Programs": 1: Start with the data structure The tools to operate on th

by TekMol 4y ago

After 20 years of programming, this is what comes to my mind when I hear "How do Design Programs": 1: Start with the data structure The tools to operate on the data can be rewritten or improved without much pain. Changing the data structure is what hurts. If the data structure is elegant, the code will fall into place. If the data structure is messy, the code will grow into a giant hairy ball of madness. 2: Don't use a DB. Use human-readable text files It makes everything so much easier. You can see if your data structure is elegant. Because then the files will be easy to read and navigate. You will be able to start up your software 20 years from now without any installation and without having to go through the hell that is setting up the environment and updating dependencies. The website you are reading this on, Hacker News, stores all data in text files. 3: Build small tools to operate on the data Like Linus started git by writing small tools "git-add", "git-commit", "git-log" etc. On the web you can have one url to do one thing (/urser, /item, /edit here on HN) and map them to one code file on your disk. If you want to use a framework like Django, Laravel or Express, those will unfortunately make it a bit harder to do that. But you can still stick somewhat closely to this approach. 4: Let people use it right away If you are building it for yourself, you already have a user. Great. If not, let your target group use it right away. Otherwise, you will waste many years of your life building software features nobody understands or even wants.

austinwade 4y ago
> 2: Don't use a DB. Use human-readable text files This is interesting, can you elaborate? Do you mean like CSV files, or some JSON structure, etc?
- TekMol 4y ago
  Depends on what you want to build. I do use CSV and JSON for some types of data, yes. Sometimes I use YAML. Sometimes just free text. Sometimes I roll my own format.
- sudopluto 4y ago
  this also holds true for serialized vs plain text data (json, csv, or roll your lang + parser : ))
sudopluto 4y ago
funnily enough, points 1, 3, and 4 are all core parts of the book : )
jolux 4y ago
I would agree with everything except 2). Human-readable files are ok if you're building a small program mostly by yourself, but in larger programs, on a team with other people, it becomes a mess very quickly. And if you need to enforce data integrity, forget about it. You're going to find yourself replicating what a DB gives you for free in your own code, except you're going to get it wrong. So: Hacker News, tiny team (one person?), very simple application, doesn't change. An accounting system: rather complex, frequently changing requirements, often developed by larger teams, needs to be accessible to non-developers, has data integrity requirements. You've really got no choice except SQL. You should definitely consider every application's needs individually, but there's a line for everything where it makes more sense to use a DB, and not understanding where the line is will bring you a lot of pain. As for longevity: the SQLite3 format has been around since 2004 and is utterly ubiquitous. I think it's pretty safe.
- TekMol 4y ago
  Small teams? Here is to small teams! Vim, Git, Hacker News, Photopea - many of my favorite, most used programs are created by a team of one! Larger programs? Look at Linux and its "everything is a file" philosophy. Linux is one of the largest codebases out there. SQLite will be around for a while, but not as long as text files. Who knows how long you can easily access present day sqlite files without fiddling with SQLite version numbers. And the format does not offer the easy access to the data that text files do. I'm not saying nobody should ever use a DB. But files are often the superior approach.
  - avgcorrection 4y ago
    Torvalds created the initial version of Git. It was eventually handed over to Hamano and has been developed by many different people since then.
  - jolux 4y ago
    > Vim, Git, Hacker News, Photopea - many of my favorite, most used programs are created by a team of one! Git is actively maintained by a number of people: https://github.com/git/git/graphs/contributors https://github.com/git/git/graphs/contributors Most commercial software is developed by teams. > Larger programs? Look at Linux and its "everything is a file" philosophy. Linux is one of the largest codebases out there. "Everything is a file" is an abstraction that Unix presents to the user. It's certainly not the case that that's how everything works underneath — the file system itself isn't a file, for example. > I'm not saying nobody should ever use a DB. But files are often the superior approach. And I'm not saying nobody should ever use text files. But a database is often the superior approach.
jonathanstrange 4y ago
Regarding 2: I've had problems with this strategy. Even though I used the swap technique instead of overwriting, files easily get corrupted on common file systems, and if it's just the user force quitting the application during the operation. (I mostly use Ext4 for testing.) Whatever Sqlite does under the hood is way more resilient to such problems. I guess that's because the Sqlite developers have been testing ACID compliance extensively. In one of my Racket applications with high integrity demands I've resorted to writing the temp file, swapping it with the original, then opening it read-only and verifying all of its contents. That is very reliable but obviously slow. But in another application written in Go I've switched entirely to Sqlite even for simple settings files and had no problems since.
- TekMol 4y ago
  We would have to discuss the specific use case and how the data got corrupted. When interacting with a typical web application, the user cannot terminate it in the middle of a file write application.
- Spakman 4y ago
  Is the swap technique you mention writing to a tempfile and then performing an (atomic) rename to overwrite the existing file? I understand how that could cause data loss if the process exited before the rename but I don't understand how it might cause corruption. Are you able to expand on that? A related thing I discovered recently is the potential non-atomicity of writes using O_APPEND on Linux[1], although I couldn't get the attached test program to fail on my machine. I would love to find some kind of confirmation that this behaviour has been changed and that appends can be relied upon to be atomic. In the application I'm working on at the moment I've managed to get add more certainty around this stuff since I'm only ever appending to the file in 4096 byte chunks, so can perform a simple file size check to see if data was written completely or if the file needs to truncated to the nearest multiple of 4096. I'm fortunate that it prevents having to do something like open it read only to verify contents as you mentioned. 1. https://bugzilla.kernel.org/show_bug.cgi?id=55651 https://bugzilla.kernel.org/show_bug.cgi?id=55651
  - 4y ago
lopatin 4y ago
So people shouldn’t use databases because 20 years down the line it will save someone the hassle of installing it? What about when the data grows a little bit and you need an index? Or you need to do a simple join? Implement your own BTree? Write your own join? Reimplenting a worse version of SQLite is not my definition of everything being easier. I agree with the rest of the points btw.
- chii 4y ago
  you migrate into a database when you find you need an index, or a join. This migration is easy to go from a flat file into a database tbh. And you can do this migration in pieces - copy the flat file and perform the ETL as a separate thing, all while your existing app runs fine unchanged. You can then implement the needed joins or indexing on this copy in the DB, and then schedule some sort of job to re-import the DB from the flat file. Sure it's not real time, but it's enough to implement most features, with little effort from migrating the read/write code in your app from flat file to the DB. And nothing stops you from migrating the read/write api from flat files into the DB, after you find that it is really needed.
  - dmitriid 4y ago
    > you migrate into a database when you find you need an index, or a join. So, almost always.
marginalia_nu 4y ago
While I agree databases are widely overused in areas where they are extreme overkill, whether human readable files are recommendable really sort of depends on what sort of data you are working with. Even just seeking in a line- or delimiter-based format is slow if the file is sufficiently many gigabytes. There are many cases where human-readable files fall apart, and ACID solves a lot of those problems. Honestly, the software I wrote 10-20 years ago, if it doesn't run now it's not because I used a database system that's fallen out of fashion. If anything, systems built back then most likely would have leaned into storing everything into some cronenbergian XML-amoeba. It's unpleasant to work with, but it's not like it's somehow no longer viable.
simion314 4y ago
The problems I hit are caused by changing requirements and some assumption that were I had before implementing something are no longer valid, and now you need to change a lot of code to work with the new assumption. Usually if I have the time I might get a beautiful data structure/code structure at the third attempt.
quickthrower2 4y ago
For HN text files is probably fine but for most enterprise apps with changing requirements and people expecting to do any kind of query I think relational is a good default. It also gives you a lot of tooling out of the box. Yes it does mean a bit more to set up later as a dependency. SQLite might be good there as you can just bundle the whole thing in the same repo.
Banana699 4y ago
1,3,4 are all fine but 2 ? really ? As far back as the 1970s Codd was already screaming at anybody who would listen not to roll your own persistence. I can see the appeal: text files are forever! read and write as simple strings!. But what do you do when you want to search ? whip up a regex or a for loop ? can you see how this will get unimaginably nasty with any remotely complex query, you will end up implementing a buggy, slow and informal subset of SQL without realizing it. Or, what if the data has constraints in it, like uniqueness of a particular column value across all rows, or other complex constraints, or types. Text files can't help you. Hell, if you aren't careful with charsets text files produced on 1 OS turn out to be garbage on another. Maybe use text files as a conceptual\prototyping tool to imagine the persistent form of your data structures, or as a dump of the database for backup. That's reasonable.
- zelphirkalt 4y ago
  I think it depends on what ones actions performed will be. If I am rendering a blog from some markup files, I might not need to build my own persistence or things like that. The information flow is unidirectional. In such and other simple cases, a relational database is overkill. It is also not too difficult to switch from plain text files to a database, if one does this step in time, before implementing ones own persistence, joins and whatever on basis of files.
dobin 4y ago
+1 for #2. Keep it simple
ImPleadThe5th 4y ago
2. Wow I didn't know that HN did that! I have never really considered using text files as long term storage outside of some very questionable things I had to implement at financial institutions for some ancient FTP/UserManagement stuff. Do you store in plain text or do you simply use CSVs and treat them like a database? Do you find that the repository pattern helps mitigate the issue with long-term DB deprecation? You could backup your databases as CSV files, and then load those into any DB server later. Your repository layer should allow you to use whatever DB tooling is fancy at the time and the CSV backups would be around for longevity?
jonjacky 4y ago
2: Don't use a DB. Use human-readable text files For even faster prototyping -- maybe better called experiments -- skip the text files and do everything in memory. Just use the built-in collection types of the programming language. In Python for example, that would be lists, dictionaries, tuples etc, then you can simulate database queries with set comprehension expressions.
deterministic 4y ago
Agree. And I will even take it one step further: 95% of the code you write will be code that transforms one data structure to another. Think about it: even UI code is exactly that. It transformed data in one representation (internal data) to another representation (the UI state). And back again when the user presses “OK’ or whatever.

by TekMol 4y ago

austinwade 4y ago

> 2: Don't use a DB. Use human-readable text files This is interesting, can you elaborate? Do you mean like CSV files, or some JSON structure, etc?

TekMol 4y ago
Depends on what you want to build. I do use CSV and JSON for some types of data, yes. Sometimes I use YAML. Sometimes just free text. Sometimes I roll my own format.
sudopluto 4y ago
this also holds true for serialized vs plain text data (json, csv, or roll your lang + parser : ))

sudopluto 4y ago

funnily enough, points 1, 3, and 4 are all core parts of the book : )

jolux 4y ago

I would agree with everything except 2). Human-readable files are ok if you're building a small program mostly by yourself, but in larger programs, on a team with other people, it becomes a mess very quickly. And if you need to enforce data integrity, forget about it. You're going to find yourself replicating what a DB gives you for free in your own code, except you're going to get it wrong. So: Hacker News, tiny team (one person?), very simple application, doesn't change. An accounting system: rather complex, frequently changing requirements, often developed by larger teams, needs to be accessible to non-developers, has data integrity requirements. You've really got no choice except SQL. You should definitely consider every application's needs individually, but there's a line for everything where it makes more sense to use a DB, and not understanding where the line is will bring you a lot of pain. As for longevity: the SQLite3 format has been around since 2004 and is utterly ubiquitous. I think it's pretty safe.

TekMol 4y ago
Small teams? Here is to small teams! Vim, Git, Hacker News, Photopea - many of my favorite, most used programs are created by a team of one! Larger programs? Look at Linux and its "everything is a file" philosophy. Linux is one of the largest codebases out there. SQLite will be around for a while, but not as long as text files. Who knows how long you can easily access present day sqlite files without fiddling with SQLite version numbers. And the format does not offer the easy access to the data that text files do. I'm not saying nobody should ever use a DB. But files are often the superior approach.
- avgcorrection 4y ago
  Torvalds created the initial version of Git. It was eventually handed over to Hamano and has been developed by many different people since then.
- jolux 4y ago
  > Vim, Git, Hacker News, Photopea - many of my favorite, most used programs are created by a team of one! Git is actively maintained by a number of people: https://github.com/git/git/graphs/contributors https://github.com/git/git/graphs/contributors Most commercial software is developed by teams. > Larger programs? Look at Linux and its "everything is a file" philosophy. Linux is one of the largest codebases out there. "Everything is a file" is an abstraction that Unix presents to the user. It's certainly not the case that that's how everything works underneath — the file system itself isn't a file, for example. > I'm not saying nobody should ever use a DB. But files are often the superior approach. And I'm not saying nobody should ever use text files. But a database is often the superior approach.

jonathanstrange 4y ago

Regarding 2: I've had problems with this strategy. Even though I used the swap technique instead of overwriting, files easily get corrupted on common file systems, and if it's just the user force quitting the application during the operation. (I mostly use Ext4 for testing.) Whatever Sqlite does under the hood is way more resilient to such problems. I guess that's because the Sqlite developers have been testing ACID compliance extensively. In one of my Racket applications with high integrity demands I've resorted to writing the temp file, swapping it with the original, then opening it read-only and verifying all of its contents. That is very reliable but obviously slow. But in another application written in Go I've switched entirely to Sqlite even for simple settings files and had no problems since.

TekMol 4y ago
We would have to discuss the specific use case and how the data got corrupted. When interacting with a typical web application, the user cannot terminate it in the middle of a file write application.
Spakman 4y ago
Is the swap technique you mention writing to a tempfile and then performing an (atomic) rename to overwrite the existing file? I understand how that could cause data loss if the process exited before the rename but I don't understand how it might cause corruption. Are you able to expand on that? A related thing I discovered recently is the potential non-atomicity of writes using O_APPEND on Linux[1], although I couldn't get the attached test program to fail on my machine. I would love to find some kind of confirmation that this behaviour has been changed and that appends can be relied upon to be atomic. In the application I'm working on at the moment I've managed to get add more certainty around this stuff since I'm only ever appending to the file in 4096 byte chunks, so can perform a simple file size check to see if data was written completely or if the file needs to truncated to the nearest multiple of 4096. I'm fortunate that it prevents having to do something like open it read only to verify contents as you mentioned. 1. https://bugzilla.kernel.org/show_bug.cgi?id=55651 https://bugzilla.kernel.org/show_bug.cgi?id=55651
- 4y ago

lopatin 4y ago

So people shouldn’t use databases because 20 years down the line it will save someone the hassle of installing it? What about when the data grows a little bit and you need an index? Or you need to do a simple join? Implement your own BTree? Write your own join? Reimplenting a worse version of SQLite is not my definition of everything being easier. I agree with the rest of the points btw.

chii 4y ago
you migrate into a database when you find you need an index, or a join. This migration is easy to go from a flat file into a database tbh. And you can do this migration in pieces - copy the flat file and perform the ETL as a separate thing, all while your existing app runs fine unchanged. You can then implement the needed joins or indexing on this copy in the DB, and then schedule some sort of job to re-import the DB from the flat file. Sure it's not real time, but it's enough to implement most features, with little effort from migrating the read/write code in your app from flat file to the DB. And nothing stops you from migrating the read/write api from flat files into the DB, after you find that it is really needed.
- dmitriid 4y ago
  > you migrate into a database when you find you need an index, or a join. So, almost always.

marginalia_nu 4y ago

While I agree databases are widely overused in areas where they are extreme overkill, whether human readable files are recommendable really sort of depends on what sort of data you are working with. Even just seeking in a line- or delimiter-based format is slow if the file is sufficiently many gigabytes. There are many cases where human-readable files fall apart, and ACID solves a lot of those problems. Honestly, the software I wrote 10-20 years ago, if it doesn't run now it's not because I used a database system that's fallen out of fashion. If anything, systems built back then most likely would have leaned into storing everything into some cronenbergian XML-amoeba. It's unpleasant to work with, but it's not like it's somehow no longer viable.

simion314 4y ago

The problems I hit are caused by changing requirements and some assumption that were I had before implementing something are no longer valid, and now you need to change a lot of code to work with the new assumption. Usually if I have the time I might get a beautiful data structure/code structure at the third attempt.

quickthrower2 4y ago

For HN text files is probably fine but for most enterprise apps with changing requirements and people expecting to do any kind of query I think relational is a good default. It also gives you a lot of tooling out of the box. Yes it does mean a bit more to set up later as a dependency. SQLite might be good there as you can just bundle the whole thing in the same repo.

Banana699 4y ago

1,3,4 are all fine but 2 ? really ? As far back as the 1970s Codd was already screaming at anybody who would listen not to roll your own persistence. I can see the appeal: text files are forever! read and write as simple strings!. But what do you do when you want to search ? whip up a regex or a for loop ? can you see how this will get unimaginably nasty with any remotely complex query, you will end up implementing a buggy, slow and informal subset of SQL without realizing it. Or, what if the data has constraints in it, like uniqueness of a particular column value across all rows, or other complex constraints, or types. Text files can't help you. Hell, if you aren't careful with charsets text files produced on 1 OS turn out to be garbage on another. Maybe use text files as a conceptual\prototyping tool to imagine the persistent form of your data structures, or as a dump of the database for backup. That's reasonable.

zelphirkalt 4y ago
I think it depends on what ones actions performed will be. If I am rendering a blog from some markup files, I might not need to build my own persistence or things like that. The information flow is unidirectional. In such and other simple cases, a relational database is overkill. It is also not too difficult to switch from plain text files to a database, if one does this step in time, before implementing ones own persistence, joins and whatever on basis of files.

dobin 4y ago

+1 for #2. Keep it simple

ImPleadThe5th 4y ago

2. Wow I didn't know that HN did that! I have never really considered using text files as long term storage outside of some very questionable things I had to implement at financial institutions for some ancient FTP/UserManagement stuff. Do you store in plain text or do you simply use CSVs and treat them like a database? Do you find that the repository pattern helps mitigate the issue with long-term DB deprecation? You could backup your databases as CSV files, and then load those into any DB server later. Your repository layer should allow you to use whatever DB tooling is fancy at the time and the CSV backups would be around for longevity?

jonjacky 4y ago

2: Don't use a DB. Use human-readable text files For even faster prototyping -- maybe better called experiments -- skip the text files and do everything in memory. Just use the built-in collection types of the programming language. In Python for example, that would be lists, dictionaries, tuples etc, then you can simulate database queries with set comprehension expressions.

deterministic 4y ago

Agree. And I will even take it one step further: 95% of the code you write will be code that transforms one data structure to another. Think about it: even UI code is exactly that. It transformed data in one representation (internal data) to another representation (the UI state). And back again when the user presses “OK’ or whatever.