3 ms·

(beginner question) How do they store the data? is an SQL db on overkill for such a use case? what would be the alternative? an ad-hoc filesystem based solution

(beginner question) How do they store the data? is an SQL db on overkill for such a use case? what would be the alternative? an ad-hoc filesystem based solution? then how do the two servers share the db? and is there redundancy at the db level? is it replicated somehow?

krapp 4y ago
Data is stored in flat text files containing Arc Lisp tables, or in RAM. There is no 'database' per se, unless they've added one and not mentioned it. You can get the software and language HN is based on here: http://arclanguage.org http://arclanguage.org
- niclo 4y ago
  I think the link is broken, it's not HTTPS
  - krapp 4y ago
    Force of habit, I fixed it.
- russelg 4y ago
  I love the design similarity to HN.
  - CobaltFire 4y ago
    That’s because HN is just about the only thing written in Arc, and everything else you see is a fork of an earlier version of HN.
petercooper 4y ago
"ad-hoc filesystem based solution" is the closest of your definitions, I think. Last time I saw/heard, HN was built in Arc, a Lisp dialect, and use(s/d) a variant of this (mirrored) code: https://github.com/wting/hackernews https://github.com/wting/hackernews Check out around this area of the code to see how simple it is. All just files and directories: https://github.com/wting/hackernews/blob/master/news.arc#L166 https://github.com/wting/hackernews/blob/master/news.arc#L16... .. the beauty of this simple approach is a lack of moving parts, and it's easy to slap Redis on top if you need caching or something. There is a modern maintained variant at https://github.com/arclanguage/anarki/tree/master/apps/news https://github.com/arclanguage/anarki/tree/master/apps/news as well if you want to spin up your own HN-a-like and have the patience. File syncing between machines is pretty much an easily solved problem. I don't know how they do it, but it could be something like https://syncthing.net/ https://syncthing.net/ or even some scripting with `rsync`. Heck, a cronned `tar | gzip | scp` might even be enough for an app whose data isn't exactly mission critical.
- scottwick 4y ago
  Wow, I had no idea HN was built like that - I'm impressed. I really wish I could read the Arc code better though since I'd love to know more about the details of how data is represented on disk and when things move in and out of memory, etc. Does anyone know of other open source applications with similar architectures like this?
  - krapp 4y ago
    >Does anyone know of other open source applications with similar architectures like this? There's a good reason everyone else just uses a relational database, and it isn't because everyone else is addicted to unnecessary complexity.
- justsomehnguy 4y ago
  > and it's easy to slap Redis on top if you need caching With filesystem as the storage you don't even need Redis, OS would cache the most recent files anyway.

krapp 4y ago

Data is stored in flat text files containing Arc Lisp tables, or in RAM. There is no 'database' per se, unless they've added one and not mentioned it. You can get the software and language HN is based on here: http://arclanguage.org http://arclanguage.org

niclo 4y ago
I think the link is broken, it's not HTTPS
- krapp 4y ago
  Force of habit, I fixed it.
russelg 4y ago
I love the design similarity to HN.
- CobaltFire 4y ago
  That’s because HN is just about the only thing written in Arc, and everything else you see is a fork of an earlier version of HN.

petercooper 4y ago

"ad-hoc filesystem based solution" is the closest of your definitions, I think. Last time I saw/heard, HN was built in Arc, a Lisp dialect, and use(s/d) a variant of this (mirrored) code: https://github.com/wting/hackernews https://github.com/wting/hackernews Check out around this area of the code to see how simple it is. All just files and directories: https://github.com/wting/hackernews/blob/master/news.arc#L166 https://github.com/wting/hackernews/blob/master/news.arc#L16... .. the beauty of this simple approach is a lack of moving parts, and it's easy to slap Redis on top if you need caching or something. There is a modern maintained variant at https://github.com/arclanguage/anarki/tree/master/apps/news https://github.com/arclanguage/anarki/tree/master/apps/news as well if you want to spin up your own HN-a-like and have the patience. File syncing between machines is pretty much an easily solved problem. I don't know how they do it, but it could be something like https://syncthing.net/ https://syncthing.net/ or even some scripting with `rsync`. Heck, a cronned `tar | gzip | scp` might even be enough for an app whose data isn't exactly mission critical.

scottwick 4y ago
Wow, I had no idea HN was built like that - I'm impressed. I really wish I could read the Arc code better though since I'd love to know more about the details of how data is represented on disk and when things move in and out of memory, etc. Does anyone know of other open source applications with similar architectures like this?
- krapp 4y ago
  >Does anyone know of other open source applications with similar architectures like this? There's a good reason everyone else just uses a relational database, and it isn't because everyone else is addicted to unnecessary complexity.
justsomehnguy 4y ago
> and it's easy to slap Redis on top if you need caching With filesystem as the storage you don't even need Redis, OS would cache the most recent files anyway.