12 ms·

Absurd Success

629 points by asicsp 3y ago

aidenn0 3y ago
Great to read this! Lots of people treat optimization as some deep-black-magic thing[1], but most of the time, it's actually easier than fixing a typical bug; all you have to do is treat excessive resource usage identical to how you would treat a bug. I'm going to make an assertion: most bugs that you can easily reproduce don't require wizardry to fix. If you can poke at a bug, then you can usually categorize it. Even the rare bugs that reveal a design flaw tend to do so readily once you can reproduce it. Software that nobody has taken a critical eye to performance on is like software with 100s of easily reproducible bugs that nobody has ever debugged. You can chip away at them for quite a while until you run into anything that is hard. 1: I think this attitude is a bit of a hold-out from when people would do things like set their branch targets so that the drum head would reach the target at the same time the CPU wanted the instruction, and when resources were so constrained that everything was hand-written assembly with global memory-locations having different semantics depending on the stage the program was in. In that case, really smart people had already taken a critical eye to performance, so you need to find things they haven't found yet. This is rarely true of modern code.
- marginalia_nu 3y ago
  I agree in general, but I think bugs are a lot easier to track down with divide and conquer strategies. If you're able to reproduce the bug by sending request X to service Y, gradually shrink the test case down until you've found the culprit. Optimization is often an architectural problem. Sure there are cases where you're copying a thing where you could recycle a buffer, but you run out of those fairly quickly, and a profiler will tell you what you need to know. A lot of the big performance wins are in changing the entire data logistics, possibly eliminating significant portions of the flow until the code does what it needs to in as few steps as possible.
  - aidenn0 3y ago
    This is a good point; bugs are less often an architectural problem, and that fixing any architectural problems (bugs or otherwise) are more difficult than fixing localized problems.
alberth 3y ago
TL;DR - SQLite is amazing even at scale.
BLKNSLVR 3y ago
Totally useless commentary: It makes me deeply happy to hear success stories like this for a project that's moving in the correctly opposite direction to that of the rest of the world. Engildification. Of which there should be more! My soul was also satisfied by the Sleeping At Night post which, along with the recent "Lie Still in Bed" article, makes for very simple options to attempt to fix sleep (discipline) issues.
- brutusborn 3y ago
  Not useless at all, thanks for posting! I’ve been struggling with sleep this year and finding out what works for others is very useful. I wouldn’t have found it if not for your comment. Link for others interested: https://www.marginalia.nu/log/86-sleep/ https://www.marginalia.nu/log/86-sleep/
- noman-land 3y ago
  I like this term, engildification.
  - BLKNSLVR 3y ago
    I was aiming for as opposite as possible to the overused (although often unfortunately appropriate) enshittification.
    - vaylian 3y ago
      Did you come up with it just for this post? It would be good to document it for the sake of "first known use".
      - bytefactory 3y ago
        I'm unreasonably excited about possibly witnessing a new popular term coming into being!
        BLKNSLVR 3y ago
paulcole 3y ago
[flagged]
- catchnear4321 3y ago
  what does scrum have to do with anything?
  - fjfuvucucuc 3y ago
    I dunno, it was your idea. Seems like a waste of time to me.
    - deprecative 3y ago
      We should have a meeting to discuss having a meeting about it.
      - catchnear4321 3y ago
        let’s talk about it at the end of standup.
bomewish 3y ago
I enjoyed reading this but I also fundamentally don't get it at a basic level like... why re-implement stuff that has already been done by entire teams? There are so many bigger and productionised search and retrieval systems. Why invest the human capital in doing it all again yourself? I just don't get it.
- ftxbro 3y ago
  a lot of 'productionised' things keep getting enshittified and ruined so i don't blame anyone for remaking a wheel
- lbotos 3y ago
  How does one learn new things if not first by understanding them, and then looking to evolve them? Sure, we can shell out to libraries and to other people's work, but at some point, you will have to understand the thing that you've abstracted away if you want to evolve it. Or as the kids say, let him cook: https://knowyourmeme.com/memes/let-him-cook-let-that-boy-cook https://knowyourmeme.com/memes/let-him-cook-let-that-boy-coo...
  - ZephyrBlu 3y ago
    FYI "let him cook" is usually used in a derogatory way.
    - tanjtanjtanj 3y ago
      Not really. The original usage of it was completely positive and it has evolved into something more ironic but not necessarily negative. I think the person you were replying to used it absolutely the way most people use the phrase. It can be more or less translated to "I want to see the outcome of what they're doing" usually used after the person who is "cooking" is criticized for their methods.
- catchnear4321 3y ago
  someone climbed everest. others tried and didn’t make it. some didn’t make it back. people keep doing it. plenty of folks dabble in art, many of which quite poorly, when good and even great artwork can be purchased for not all that much. some paint the walls of their house rather than call a professional painter. there are countless reasons, including but not limited to the easiest, most flippant, and possibly the most human response to your question. why not?
38 3y ago
if I search encoding/json, I get some interesting stuff: https://search.marginalia.nu/search?query=encoding%2Fjson https://search.marginalia.nu/search?query=encoding%2Fjson but NOT what I am looking for. If I try again with Google: https://google.com/search?q=encoding%2Fjson https://google.com/search?q=encoding%2Fjson first result is exactly what I want.
- algas 3y ago
  Google still kills for useful applications. I use marginalia when I have either A) something deeply obscure I want to research or B) when I just want to read some entertaining long-form text posts about something in a category that interests me.
- Stephen_0xFF 3y ago
  “So it’s a search engine. It’s perhaps not the greatest at finding what you already knew was there. Instead it is designed to help you find some things you didn’t even know you were looking for.”[0] [0] https://www.marginalia.nu/marginalia-search/about/ https://www.marginalia.nu/marginalia-search/about/
  - joshxyz 3y ago
    So a feature not a bug I guess! All good!
- marginalia_nu 3y ago
  If all you want is the same results as Google I say you should just use Google. I'm not trying to compete with them, I give the results you won't find on the big G.
fouc 3y ago
Good example of how complexity often engenders complexity. The wrong abstraction might create 100x more work to support it.
- gary_0 3y ago
  I also like how they decided to mix in sqlite alongside the existing MariaDB database because it gets the job done, and "a foolish consistency is the hobgoblin of little minds".
nicbou 3y ago
I always love seeing marginalia.nu updates here. You are a cherished user on this website, and I hope that you keep posting.
- marginalia_nu 3y ago
  Aww shucks.
gnyman 3y ago
On a side note inspired by this blog post. I'm wondering if humans are mostly incapable of producing great things without (artifical) restrictions. In this case, marginalia is (ridiculously) efficient because Victor (the creator) is intentionally restricting what hardware it runs on and how much ram it has. If he just caved in and added another 32GiB it would work for a while, but the inefficient design would persist and the problem would just show it's head later and then there would be more complexity around that design and it might not be as easy to fix then. If the original thesis is correct, then I think it explains why most software is so bad (bloated, slow, buggy) nowadays. It's because very few individual pieces of software nowadays are hitting any limits (in isolation). So each individual piece is terribly inefficient but with the latest M2 Pro and GiB connection you can just keep ahead of the curve where it becomes a problem. Anyways, turned into a rant; but the conclusion might be to limit yourself, and you (and e everyone else) will be better off long term.
- jancsika 3y ago
  > I'm wondering if humans are mostly incapable of producing great things without (artifical) restrictions. I think the real issue is that there isn't a program language that produces a compiler error if the given code can exceed a maximum specified latency. Even working on a program with soft-realtime scheduling, I've had to constantly push back against patches that introduce some obscure convenience without having measured worst case latency. The problem is so bad I doubt most people realize it's there. I don't know what the answer is, but I have the feeling there's an intersection with timing attacks on software/hardware. Some kind of tooling that makes both worst case times and variance as visible as the computed CSS in devTools would probably help. Added to some kind of static analysis, perhaps devs to hack their way to decently responsive interfaces and services.
- MichaelZuo 3y ago
  American Airlines ran SABRE, a sizeable airline ticketing and reservation system, in the mid-1970s on two system/360 mainframes that could only process a few tens of millions of instructions per second. A raspberry pi 2 can do over 4 billion Dhrystone instructions per second, and a pi 4 over 10 billion per second. Of course by modern standards mid-1970s SABRE was pretty barebones for an airline's main system, but it's at least theoretically possible to run simplified systems for over a 100 airlines simultaneously on a single pi 2... So yes modern programs are very far from optimized. 1000x or 10 000x improvements are possible, less for math heavy stuff.
janvdberg 3y ago
I love this as another example where restriction breeds innovation. More often than not this is found not in abundance but in limits.
newman123 3y ago
Wonder why SQLite was chosen over a key value store. Seems he wanted reads by id and no other columns so a relational db seems unnecessary?
- marginalia_nu 3y ago
  Could have, yeah. Could have also just have created a temporary table in mariadb. SQLite has the benefit that it's a single file though, and you can do cool things with that. Such as copy it, share it, etc.
  - jaggederest 3y ago
    Maybe I'm old school but I would just stuff that kind of data in a struct. It should be real, real small as a packed binary tree. just mmap the file and be there. If you're never writing to it, or only write to it via filesystem full-file overwrites, you can lay it out most conveniently in memory. We did a similar thing with the sparse matrix data for the Netflix prize back in the day - in a database it was 6gb, in C structs it was ~600mb. Of course I do respect the encapsulation of the existing SQL query basis but there is sometimes a time for something even more compact.
    - marginalia_nu 3y ago
      Yeah tempting for sure. I tried to resist it on the basis that I already have a bunch of similar solutions in the system, and while they're fast and effective, they're also always a bit of a headache to maintain and debug; and this isn't something that is extremely space constrained.
  - pjc50 3y ago
    I'd just like to put an opposite vote on the scale and say that SQLite is such an excellent default choice for this and you were right to do it this way. I can't really see why people argue for databases which have the "relationality" feature deliberately removed.
flexagoon 3y ago
By the way, Kagi, the paid search engine you might've seen on HackerNews as well, uses Marginalia as one of its data sources https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm... If you use the "non-commercial" lens, those results, among with results from Kagi's own index and a few other independent sources will be prioritized.
donotsay 3y ago
I just tried "Russian sources about Ukraine war", and it returns ukranian sources. So I guess it is of limited use to avoid censorship.
- uoaei 3y ago
  Did you try searching in Russian? Is it reasonable to expect English-language sources?
  - donotsay 3y ago
    Of course it is. There are many Russian sources in English on official government sites, twitter, whatever. And they represent a point of view of at least tens of millions of people, but probably more, invisible to the western audience.
  - ricardo81 3y ago
    Good point. It'd be reasonable to find English sources though, what with 'fake news' and the motivations around that.
- marginalia_nu 3y ago
  This is a keyword based search engine. It shows you pages with the keywords you typed. Though I strongly disagree with Putin's illegal war of aggression, I don't censor Russian websites. You'll find Russian truth on there as well: https://search.marginalia.nu/site/english.pravda.ru https://search.marginalia.nu/site/english.pravda.ru
mananaysiempre 3y ago
> In brief, every time an SSD updates a single byte anywhere on disk, it needs to erase and re-write that entire page. Is that actually true for SSDs? For raw flash it’s not, provided you are overwriting “empty” all-ones values or otherwise only changing 1s to 0s. Writing is orders of magnitude slower than reading, but still a couple orders of magnitude faster than erasing (resetting back to “empty”), and only erases count against your wear budget. It sounds like an own goal for an SSD controller to not take advantage of that, although if the actual guts of it are log-structured then I could imagine it not being able to.
- marginalia_nu 3y ago
  In this scenario I was basically re-writing the entire hard-drive completely in a completely random order, which is the worst case scenario for an SSD. Normally the controller will use a whole bunch of tricks (e.g. overprovisioning, buffering and reordering of writes) to avoid this type of worst case pattern, but that only goes so far.
- Filligree 3y ago
  > Is that actually true for SSDs? It's completely false. Even the most primitive SSD controllers would make some attempt at mitigating this.
- lelanthran 3y ago
  >> In brief, every time an SSD updates a single byte anywhere on disk, it needs to erase and re-write that entire page. > Is that actually true for SSDs? For raw flash it’s not, provided you are overwriting “empty” all-ones values or otherwise only changing 1s to 0s. Maybe it depends. I wrote the driver for more than one popular flash chips (don't remember which ones now, but that employer had a policy of never using components that were not mainstream and available from multiple suppliers) and all the chips I dealt with did read and write exclusively via fixed-size pages. Since SSDs are collection of chips, I'd expect each chip on the SSD to only support fixed-size paged IO.
- mikehollinger 3y ago
  > Is that actually true for SSDs? Not precisely. The logical view of a page living at some address of flash is not the reality. Pages get moved around the physical device as writes happen. The drive itself maintains a map of what addresses are used for what purpose, their health and so on. It’s a sparse storage scheme. There’s even maintenance ops and garbage collection that happens occasionally or on command (like a TRIM). In reality a “write” to a non-full drive is: 1. Figure out which page the data goes to. 2. Figure out if there’s data there or not. Read / modify / write if needed. 3. Figure out where to write the data. 4. Write the data. It might not go back where it started. In fact it probably won’t because of wear leveling. You’re right that the controller does a far more complex set of steps for performance. That’s why an empty / new drive performs better for a while (page cache aside) then literally slows down compared to a “full” drive that’s old, with no spare pages. Source: I was chief engineer for a cache-coherent memory mapped flash accelerator. We let a user map the drive very very efficiently in user space Linux, but eventually caved to the “easier” programming model of just being another hard drive after a while.
csours 3y ago
I took a start script from 90 seconds to 30 seconds yesterday, by finding a poorly named timeout value. Now I'm working on a graceful fallback from itimer to alarm instead of outdated c directives.
anyfactor 3y ago
Oh thank you. I have been doing a hobby project on search engines, and I kept searching of variations of "Magnolia" for some reason. ""Marginalia"" at least for me is hard to remember. Currently, I am trying to figure my way around Searx. Does Marginalia support "time filters" for search like past day, past week etc? According the special keywords the only search params accepted is based on years. year>2005 (beta) The document was ostensibly published in or after 2005 year=2005 (beta) The document was ostensibly published in 2005 year<2005 (beta) The document was ostensibly published in or before 2005
- meithecatte 3y ago
  It's the search engine for the niche stuff. Marginal stuff, if you will. The name makes sense to me.
- marginalia_nu 3y ago
  The search index isn't updated more than once every month, so no such filters. The year-filter is pretty rough too. It's very hard to accurately date most webpages.
_madmax_ 3y ago
I'm happy for you !
jorgeleo 3y ago
This baffled me: "I wish I knew what happened, or how to replicate it. It’s involved more upfront design than I normally do, by necessity. I like feeling my way forward in general, but there are problems where the approach just doesn’t work" Yes, immediate (or soon enough) gratification feels good... To me, and maybe is because I am an old fart, this is the difference between programming and engineering.
deleted 3y ago
[deleted]
ricardo81 3y ago
Just a shout out to my boss at Mojeek who presumably has a very similar path to this (the post resonates a lot with past conversations). Mojeek started back in 2004 and for the most part has been a single developer who built the bones of it, and in that, pretty much all of the IR and infrastructure. Limitations of finance and hardware, making decisions about 32 vs 64 bit ids, sharding, speed of updating all sound very familiar. Reminds me of Google way back when and their 'Google dance' that updated results once a month, nowadays it's a daily flux. It's all an evolution, and great to see Marginalia offering another view point into the web beyond big tech.

629 points by asicsp 3y ago

Great to read this! Lots of people treat optimization as some deep-black-magic thing[1], but most of the time, it's actually easier than fixing a typical bug; all you have to do is treat excessive resource usage identical to how you would treat a bug. I'm going to make an assertion: most bugs that you can easily reproduce don't require wizardry to fix. If you can poke at a bug, then you can usually categorize it. Even the rare bugs that reveal a design flaw tend to do so readily once you can reproduce it. Software that nobody has taken a critical eye to performance on is like software with 100s of easily reproducible bugs that nobody has ever debugged. You can chip away at them for quite a while until you run into anything that is hard. 1: I think this attitude is a bit of a hold-out from when people would do things like set their branch targets so that the drum head would reach the target at the same time the CPU wanted the instruction, and when resources were so constrained that everything was hand-written assembly with global memory-locations having different semantics depending on the stage the program was in. In that case, really smart people had already taken a critical eye to performance, so you need to find things they haven't found yet. This is rarely true of modern code.

marginalia_nu 3y ago
I agree in general, but I think bugs are a lot easier to track down with divide and conquer strategies. If you're able to reproduce the bug by sending request X to service Y, gradually shrink the test case down until you've found the culprit. Optimization is often an architectural problem. Sure there are cases where you're copying a thing where you could recycle a buffer, but you run out of those fairly quickly, and a profiler will tell you what you need to know. A lot of the big performance wins are in changing the entire data logistics, possibly eliminating significant portions of the flow until the code does what it needs to in as few steps as possible.
- aidenn0 3y ago
  This is a good point; bugs are less often an architectural problem, and that fixing any architectural problems (bugs or otherwise) are more difficult than fixing localized problems.

alberth 3y ago

TL;DR - SQLite is amazing even at scale.

BLKNSLVR 3y ago

Totally useless commentary: It makes me deeply happy to hear success stories like this for a project that's moving in the correctly opposite direction to that of the rest of the world. Engildification. Of which there should be more! My soul was also satisfied by the Sleeping At Night post which, along with the recent "Lie Still in Bed" article, makes for very simple options to attempt to fix sleep (discipline) issues.

brutusborn 3y ago
Not useless at all, thanks for posting! I’ve been struggling with sleep this year and finding out what works for others is very useful. I wouldn’t have found it if not for your comment. Link for others interested: https://www.marginalia.nu/log/86-sleep/ https://www.marginalia.nu/log/86-sleep/
noman-land 3y ago
I like this term, engildification.
- BLKNSLVR 3y ago
  I was aiming for as opposite as possible to the overused (although often unfortunately appropriate) enshittification.
  - vaylian 3y ago
    Did you come up with it just for this post? It would be good to document it for the sake of "first known use".
    - bytefactory 3y ago
      I'm unreasonably excited about possibly witnessing a new popular term coming into being!
      - BLKNSLVR 3y ago

paulcole 3y ago

[flagged]

catchnear4321 3y ago
what does scrum have to do with anything?
- fjfuvucucuc 3y ago
  I dunno, it was your idea. Seems like a waste of time to me.
  - deprecative 3y ago
    We should have a meeting to discuss having a meeting about it.
    - catchnear4321 3y ago
      let’s talk about it at the end of standup.

bomewish 3y ago

I enjoyed reading this but I also fundamentally don't get it at a basic level like... why re-implement stuff that has already been done by entire teams? There are so many bigger and productionised search and retrieval systems. Why invest the human capital in doing it all again yourself? I just don't get it.

ftxbro 3y ago
a lot of 'productionised' things keep getting enshittified and ruined so i don't blame anyone for remaking a wheel
lbotos 3y ago
How does one learn new things if not first by understanding them, and then looking to evolve them? Sure, we can shell out to libraries and to other people's work, but at some point, you will have to understand the thing that you've abstracted away if you want to evolve it. Or as the kids say, let him cook: https://knowyourmeme.com/memes/let-him-cook-let-that-boy-cook https://knowyourmeme.com/memes/let-him-cook-let-that-boy-coo...
- ZephyrBlu 3y ago
  FYI "let him cook" is usually used in a derogatory way.
  - tanjtanjtanj 3y ago
    Not really. The original usage of it was completely positive and it has evolved into something more ironic but not necessarily negative. I think the person you were replying to used it absolutely the way most people use the phrase. It can be more or less translated to "I want to see the outcome of what they're doing" usually used after the person who is "cooking" is criticized for their methods.
catchnear4321 3y ago
someone climbed everest. others tried and didn’t make it. some didn’t make it back. people keep doing it. plenty of folks dabble in art, many of which quite poorly, when good and even great artwork can be purchased for not all that much. some paint the walls of their house rather than call a professional painter. there are countless reasons, including but not limited to the easiest, most flippant, and possibly the most human response to your question. why not?

38 3y ago

if I search encoding/json, I get some interesting stuff: https://search.marginalia.nu/search?query=encoding%2Fjson https://search.marginalia.nu/search?query=encoding%2Fjson but NOT what I am looking for. If I try again with Google: https://google.com/search?q=encoding%2Fjson https://google.com/search?q=encoding%2Fjson first result is exactly what I want.

algas 3y ago
Google still kills for useful applications. I use marginalia when I have either A) something deeply obscure I want to research or B) when I just want to read some entertaining long-form text posts about something in a category that interests me.
Stephen_0xFF 3y ago
“So it’s a search engine. It’s perhaps not the greatest at finding what you already knew was there. Instead it is designed to help you find some things you didn’t even know you were looking for.”[0] [0] https://www.marginalia.nu/marginalia-search/about/ https://www.marginalia.nu/marginalia-search/about/
- joshxyz 3y ago
  So a feature not a bug I guess! All good!
marginalia_nu 3y ago
If all you want is the same results as Google I say you should just use Google. I'm not trying to compete with them, I give the results you won't find on the big G.

fouc 3y ago

Good example of how complexity often engenders complexity. The wrong abstraction might create 100x more work to support it.

gary_0 3y ago
I also like how they decided to mix in sqlite alongside the existing MariaDB database because it gets the job done, and "a foolish consistency is the hobgoblin of little minds".

nicbou 3y ago

I always love seeing marginalia.nu updates here. You are a cherished user on this website, and I hope that you keep posting.

marginalia_nu 3y ago
Aww shucks.

gnyman 3y ago

On a side note inspired by this blog post. I'm wondering if humans are mostly incapable of producing great things without (artifical) restrictions. In this case, marginalia is (ridiculously) efficient because Victor (the creator) is intentionally restricting what hardware it runs on and how much ram it has. If he just caved in and added another 32GiB it would work for a while, but the inefficient design would persist and the problem would just show it's head later and then there would be more complexity around that design and it might not be as easy to fix then. If the original thesis is correct, then I think it explains why most software is so bad (bloated, slow, buggy) nowadays. It's because very few individual pieces of software nowadays are hitting any limits (in isolation). So each individual piece is terribly inefficient but with the latest M2 Pro and GiB connection you can just keep ahead of the curve where it becomes a problem. Anyways, turned into a rant; but the conclusion might be to limit yourself, and you (and e everyone else) will be better off long term.

jancsika 3y ago
> I'm wondering if humans are mostly incapable of producing great things without (artifical) restrictions. I think the real issue is that there isn't a program language that produces a compiler error if the given code can exceed a maximum specified latency. Even working on a program with soft-realtime scheduling, I've had to constantly push back against patches that introduce some obscure convenience without having measured worst case latency. The problem is so bad I doubt most people realize it's there. I don't know what the answer is, but I have the feeling there's an intersection with timing attacks on software/hardware. Some kind of tooling that makes both worst case times and variance as visible as the computed CSS in devTools would probably help. Added to some kind of static analysis, perhaps devs to hack their way to decently responsive interfaces and services.
MichaelZuo 3y ago
American Airlines ran SABRE, a sizeable airline ticketing and reservation system, in the mid-1970s on two system/360 mainframes that could only process a few tens of millions of instructions per second. A raspberry pi 2 can do over 4 billion Dhrystone instructions per second, and a pi 4 over 10 billion per second. Of course by modern standards mid-1970s SABRE was pretty barebones for an airline's main system, but it's at least theoretically possible to run simplified systems for over a 100 airlines simultaneously on a single pi 2... So yes modern programs are very far from optimized. 1000x or 10 000x improvements are possible, less for math heavy stuff.

janvdberg 3y ago

I love this as another example where restriction breeds innovation. More often than not this is found not in abundance but in limits.

newman123 3y ago

Wonder why SQLite was chosen over a key value store. Seems he wanted reads by id and no other columns so a relational db seems unnecessary?

marginalia_nu 3y ago
Could have, yeah. Could have also just have created a temporary table in mariadb. SQLite has the benefit that it's a single file though, and you can do cool things with that. Such as copy it, share it, etc.
- jaggederest 3y ago
  Maybe I'm old school but I would just stuff that kind of data in a struct. It should be real, real small as a packed binary tree. just mmap the file and be there. If you're never writing to it, or only write to it via filesystem full-file overwrites, you can lay it out most conveniently in memory. We did a similar thing with the sparse matrix data for the Netflix prize back in the day - in a database it was 6gb, in C structs it was ~600mb. Of course I do respect the encapsulation of the existing SQL query basis but there is sometimes a time for something even more compact.
  - marginalia_nu 3y ago
    Yeah tempting for sure. I tried to resist it on the basis that I already have a bunch of similar solutions in the system, and while they're fast and effective, they're also always a bit of a headache to maintain and debug; and this isn't something that is extremely space constrained.
- pjc50 3y ago
  I'd just like to put an opposite vote on the scale and say that SQLite is such an excellent default choice for this and you were right to do it this way. I can't really see why people argue for databases which have the "relationality" feature deliberately removed.

flexagoon 3y ago

By the way, Kagi, the paid search engine you might've seen on HackerNews as well, uses Marginalia as one of its data sources https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm... If you use the "non-commercial" lens, those results, among with results from Kagi's own index and a few other independent sources will be prioritized.

donotsay 3y ago

I just tried "Russian sources about Ukraine war", and it returns ukranian sources. So I guess it is of limited use to avoid censorship.

uoaei 3y ago
Did you try searching in Russian? Is it reasonable to expect English-language sources?
- donotsay 3y ago
  Of course it is. There are many Russian sources in English on official government sites, twitter, whatever. And they represent a point of view of at least tens of millions of people, but probably more, invisible to the western audience.
- ricardo81 3y ago
  Good point. It'd be reasonable to find English sources though, what with 'fake news' and the motivations around that.
marginalia_nu 3y ago
This is a keyword based search engine. It shows you pages with the keywords you typed. Though I strongly disagree with Putin's illegal war of aggression, I don't censor Russian websites. You'll find Russian truth on there as well: https://search.marginalia.nu/site/english.pravda.ru https://search.marginalia.nu/site/english.pravda.ru

mananaysiempre 3y ago

> In brief, every time an SSD updates a single byte anywhere on disk, it needs to erase and re-write that entire page. Is that actually true for SSDs? For raw flash it’s not, provided you are overwriting “empty” all-ones values or otherwise only changing 1s to 0s. Writing is orders of magnitude slower than reading, but still a couple orders of magnitude faster than erasing (resetting back to “empty”), and only erases count against your wear budget. It sounds like an own goal for an SSD controller to not take advantage of that, although if the actual guts of it are log-structured then I could imagine it not being able to.

marginalia_nu 3y ago
In this scenario I was basically re-writing the entire hard-drive completely in a completely random order, which is the worst case scenario for an SSD. Normally the controller will use a whole bunch of tricks (e.g. overprovisioning, buffering and reordering of writes) to avoid this type of worst case pattern, but that only goes so far.
Filligree 3y ago
> Is that actually true for SSDs? It's completely false. Even the most primitive SSD controllers would make some attempt at mitigating this.
lelanthran 3y ago
>> In brief, every time an SSD updates a single byte anywhere on disk, it needs to erase and re-write that entire page. > Is that actually true for SSDs? For raw flash it’s not, provided you are overwriting “empty” all-ones values or otherwise only changing 1s to 0s. Maybe it depends. I wrote the driver for more than one popular flash chips (don't remember which ones now, but that employer had a policy of never using components that were not mainstream and available from multiple suppliers) and all the chips I dealt with did read and write exclusively via fixed-size pages. Since SSDs are collection of chips, I'd expect each chip on the SSD to only support fixed-size paged IO.
mikehollinger 3y ago
> Is that actually true for SSDs? Not precisely. The logical view of a page living at some address of flash is not the reality. Pages get moved around the physical device as writes happen. The drive itself maintains a map of what addresses are used for what purpose, their health and so on. It’s a sparse storage scheme. There’s even maintenance ops and garbage collection that happens occasionally or on command (like a TRIM). In reality a “write” to a non-full drive is: 1. Figure out which page the data goes to. 2. Figure out if there’s data there or not. Read / modify / write if needed. 3. Figure out where to write the data. 4. Write the data. It might not go back where it started. In fact it probably won’t because of wear leveling. You’re right that the controller does a far more complex set of steps for performance. That’s why an empty / new drive performs better for a while (page cache aside) then literally slows down compared to a “full” drive that’s old, with no spare pages. Source: I was chief engineer for a cache-coherent memory mapped flash accelerator. We let a user map the drive very very efficiently in user space Linux, but eventually caved to the “easier” programming model of just being another hard drive after a while.

csours 3y ago

I took a start script from 90 seconds to 30 seconds yesterday, by finding a poorly named timeout value. Now I'm working on a graceful fallback from itimer to alarm instead of outdated c directives.

anyfactor 3y ago

Oh thank you. I have been doing a hobby project on search engines, and I kept searching of variations of "Magnolia" for some reason. ""Marginalia"" at least for me is hard to remember. Currently, I am trying to figure my way around Searx. Does Marginalia support "time filters" for search like past day, past week etc? According the special keywords the only search params accepted is based on years. year>2005 (beta) The document was ostensibly published in or after 2005 year=2005 (beta) The document was ostensibly published in 2005 year<2005 (beta) The document was ostensibly published in or before 2005

meithecatte 3y ago
It's the search engine for the niche stuff. Marginal stuff, if you will. The name makes sense to me.
marginalia_nu 3y ago
The search index isn't updated more than once every month, so no such filters. The year-filter is pretty rough too. It's very hard to accurately date most webpages.

_madmax_ 3y ago

I'm happy for you !

jorgeleo 3y ago

This baffled me: "I wish I knew what happened, or how to replicate it. It’s involved more upfront design than I normally do, by necessity. I like feeling my way forward in general, but there are problems where the approach just doesn’t work" Yes, immediate (or soon enough) gratification feels good... To me, and maybe is because I am an old fart, this is the difference between programming and engineering.

deleted 3y ago

[deleted]

ricardo81 3y ago

Just a shout out to my boss at Mojeek who presumably has a very similar path to this (the post resonates a lot with past conversations). Mojeek started back in 2004 and for the most part has been a single developer who built the bones of it, and in that, pretty much all of the IR and infrastructure. Limitations of finance and hardware, making decisions about 32 vs 64 bit ids, sharding, speed of updating all sound very familiar. Reminds me of Google way back when and their 'Google dance' that updated results once a month, nowadays it's a daily flux. It's all an evolution, and great to see Marginalia offering another view point into the web beyond big tech.