8 ms·

Full Time

932 points by kevincox 3y ago

paddw 3y ago
Congrats to the author! Marginalia is a great service. I hope they find a way to make it viable to keep going, either through donations or some other model.
samwillis 3y ago
That feeling when you walk out of an office for the last time, to work on your own thing is exhilarating. I had my moment like that back in 2014 and can still remember it. Congrats to Viktor and good luck! Going to go and try your search engine now. Previous discussion of the search engine a couple of months ago: https://news.ycombinator.com/item?id=35611923 https://news.ycombinator.com/item?id=35611923 (196 comments) Many other posts and blog posts over the last couple of years: https://news.ycombinator.com/from?site=marginalia.nu https://news.ycombinator.com/from?site=marginalia.nu
- deleted 3y ago
  [deleted]
- munchler 3y ago
  It was March 2006 for me, and I haven’t worked in an office since. What a great feeling.
  - vasili111 3y ago
    What you are doing since 2006?
  - throwaway290 3y ago
    I went the other way, in 15 year career never worked a day in the office until last week. So far I can WFH but dunno for how long. There are positives with an office but I kinda envy how everyone celebrates the opposite move
- OJFord 3y ago
  I would love to (though I'm a long way off it, with not much to walk away to) but I wonder what the equivalent feeling is if you already/previously work from home? Shipping the work machine back? Turning it off for the last time? Unplugging web cam and microphone?
  - cambaceres 3y ago
    Perhaps removing Slack. What a feeling to never have to be online constantly from 8-5 so no one will think I’m not working.
- wahnfrieden 3y ago
  One year ago for me. I’ll never work for a boss again. And won’t be the boss of anyone again either - just as important to me.
  - stolsvik 3y ago
    What do you do then? Live off the interests of some fortune?
    - wahnfrieden 3y ago
      I am currently working all the time. Independently designed/developed/operated/owned apps, and worker coop structures for working with others. I like consent based decision making rather than hierarchical or consensus. Command is unnecessary for us to do well for ourselves.
nhance 3y ago
Im curious what percentage of living expenses are covered by the project for the author. I have a few products generating over 50% of my yearly expenses and am feeling like going full time is almost a possibility now. A bit too nervous to pull the trigger just yet
- samwillis 3y ago
  I pulled that trigger later than I could have, I was earning 2x my salary from my side project before I quit. At 50% if you can see an upward trend, ~6 months savings, and have a plan that the time will give you to execute, got for it.
  - bombcar 3y ago
    The "main thing" is how hard it would be to get back in the business (i.e, get a job) if the whole thing explodes. Also if you're going to quit anyway, you might as well ask the company you currently work for if they'll let you go on sabbatical, or part time, or consulting. That can give you a bit of extra runway/feeling of security.
    - valval 3y ago
      I suppose you can go back to any of the previous points in your CV as a software professional in today’a day and age if you never burned any bridges. Especially so if you make it obvious in your current job that you’re only leaving because it’s time to try your own thing - if it doesn’t end up working, people are likely to be very understanding.
      - marginalia_nu 3y ago
        It's not like I'm doing nothing for these upcoming years. Dunno what you're doing wrong if you can't land a job with a built-from-scratch internet search engine on your resume.
        3y ago
lucideer 3y ago
I've been trialling Marginalia Search a little and one thing that's struck me is the latency. The only other site I use with similar latency is HN; Marginalia seems even lower despite being dynamic (HN has a much easier caching story). I wonder is it just down to having lower traffic. It's certainly a lot lower than many low-/zero-traffic blogs I've frequented though. I've had a look at the README[0] for the Java sourcecode, but it's highly focused on crawls, database & indexing (understandable for search); would be cool to see a front-end focused write-up. [0] https://github.com/MarginaliaSearch/MarginaliaSearch/blob/master/code/readme.md https://github.com/MarginaliaSearch/MarginaliaSearch/blob/ma...
- marginalia_nu 3y ago
  The blog is just hugo so it's 100% static files over nginx. The search engine is serverside-rendered mustache templates via handlebars[1], via served via spark[2]. It's basically all vanilla Java. I do raw SQL queries instead of ORM, which makes it quite a bit snappier than most Java applications. The sheer size of the database also mandates that basically every query is a primary key lookup. The code is written around that constraint. Although the search engine is a bit on the slow side since it's routed through cloudflare and I think I'm relatively far away from the closest datacenter so it adds like 100ms to the load times. [1] https://github.com/jknack/handlebars.java https://github.com/jknack/handlebars.java [2] https://sparkjava.com/ https://sparkjava.com/
  - lucideer 3y ago
    > The blog is just hugo Yeah the static stuff being fast is less surprising - it was mainly the search results page that astounded me. > via served via spark[2] Had not heard of this Spark (only the other Apache one). Will definitely take a look. > Although the search engine is a bit on the slow side since it's routed through cloudflare and I think I'm relatively far away from the closest datacenter so it adds like 100ms to the load times. I've hit the CF loading screen which introduces a big delay, but when I don't see that the loading is really instantaneous.
    - 3y ago
ASlave2Gravity 3y ago
Congrats! I find I'm using Marginalia more and more, it's especially great for researching for novel writing, and can’t wait to see what the future holds! Good luck!
mannycalavera42 3y ago
congrats! you should put the patreon link in more places IMHO (e.g. the main page and the various page footers)
jimbokun 3y ago
Does Marginalia Search have selling points over other search engines? Different features or philosophy? First I'm hearing of it.
- marginalia_nu 3y ago
  It's something like Google but what it kept working like it did in 2002 and then added a bunch of discovery features.
- MikeSchurman 3y ago
  From the about: "This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed. " I personally find it hard to put into words, but the old internet and old search engines had this feel to them that you never knew what you were going to get. Each site looked different. Each site had it's own philosophy of content and design. Everybody was winging it. It just felt more personal and interesting. At the risk of hyperbole, now it seems search engines give back mostly SEO blogspam that all looks the same. Marginalia feels more like the former internet.
  - HankB99 3y ago
    > now it seems search engines give back mostly SEO blogspam And with technical questions, too many results are not correct. It seems that Google search is really going down hill in this regard. I'd like a way to vote down results that are obviously SEO trash, but I'm pretty sure if that were provided, it would be gamed too.
trollied 3y ago
The search engine: https://search.marginalia.nu/ https://search.marginalia.nu/
uticus 3y ago
I'd love to support, but Patreon link [0] on "supporting" page [1] is 404. Is another support option in progress to replace? [0] https://www.marginalia.nu/marginalia-search/supporting/patreon https://www.marginalia.nu/marginalia-search/supporting/patre... [1] https://www.marginalia.nu/marginalia-search/supporting/ https://www.marginalia.nu/marginalia-search/supporting/
- asicsp 3y ago
  Seems like the link text is correct (https://www.patreon.com/marginalia_nu https://www.patreon.com/marginalia_nu) but the linking functionality is missing.
- marginalia_nu 3y ago
  The links were cropped so I changed them with just a word for the service. But it turns out I can't markdown today and I changed the URLs instead of the text. Fixed now.
  - sovietmudkipz 3y ago
    Question for the tester type software engineers on HN... I like to write browser (puppeteer) tests for user-facing software criteria like "patreon link must work." In the past I've written similar tests for small websites I've created where the purpose is to surface affiliate links for users to click on. My criteria is "from a money standpoint, is this the call to action I want my users to engage in?" I don't know what type of test this is -- can anyone disambiguate testing terminology for me? P.S. Browser based testing is brittle but since I often create websites and because I want to really ensure that I'm not 'lying' to myself in tests, using a browser is often the best (albeit slower) choice. These tests usually run in CI and I get notifications if they break. P.P.S. I wish we had a better mental model for the types of tests than the "testing pyramid." I find the testing pyramid lacking.
    - codazoda 3y ago
      I would probably call these integration tests. I came across this interesting tool for similar tests the other day. It lets you request websites or API’s and then search the return for a string. It’s more for checking uptime, so I dunno if it would be acceptable for this type of test, but it looks like a cool tool. https://onlineornot.com https://onlineornot.com
ilrwbwrkhv 3y ago
This is fantastic and commendable. Too many good hackers are tied up in a stable job at companies. Building something out of passion is just so different and the end result is so amazing that one cannot really fake that. Marginalia is one of my favorite sites. Wishing you all the best.
UpToTheSky 3y ago
Congrats! Going full time is the only way to go for a project you love and want to grow. What is the business model? How much visitors does Marginalia have?
mathgladiator 3y ago
It's a great feeling to leave and set off on your own journey. Beyond the feeling, it's also educational as you learn about your deficients quickly (or, in some cases, too slowly). I'm wrestling with this now as I'm building my platform and looking to pivot into something that produces revenue.
bombcar 3y ago
How do you submit a site? I know of a few good, small blogs and forums that don't come up. They're not on a VPS, either.
- marginalia_nu 3y ago
  https://github.com/MarginaliaSearch/submit-site-to-marginalia-search https://github.com/MarginaliaSearch/submit-site-to-marginali...
  - wakamoleguy 3y ago
    Perhaps related (or I'm just not sure how this works), what criteria goes into whether a crawled site is indexed? My personal blog has 31 pages "known", 32 crawled, and only 3 indexed.
    - marginalia_nu 3y ago
      Whether it appears to be in English, whether there appears to be enough text, things like that. If you post the domain name (or email me) I can take a closer look tomorrow.
retrocryptid 3y ago
Cool that you're living your best life. Every time I leave a company, I think about the ending of The Prisoner: https://youtube.com/clip/UgkxgJAzCqKOL5yMg39wmtZi52tw8LAXOErQ https://youtube.com/clip/UgkxgJAzCqKOL5yMg39wmtZi52tw8LAXOEr...
- mayormcmatt 3y ago
  Great pull. I've always wanted the courage to slam my resignation down on my boss' desk and yell at them, so a different Prisoner scene for me.
  - retrocryptid 3y ago
    [flagged]
rscrawfo 3y ago
Both of your top two projects are very interesting to me at the moment. Especially your Wikipedia mirror. Just today I realized how distracting too many hyperlinks can be. And Wikipedia is full of them! It feels so much easier to read an article without them. Now I just wish Wikipedia had more supporting graphics to help engage readers in a more productive manner.
inconceivable 3y ago
not knowing what comes in the next 12-24 months is exhilarating because you're actually living life with an acceptance of the nature of reality instead of deluding yourself or fighting it.
gnramires 3y ago
Congratulations marginalia ! :D
zo1 3y ago
Tiny bit of feedback, your encyclopedia favicon seems to 404: https://encyclopedia.marginalia.nu/favicon.ico https://encyclopedia.marginalia.nu/favicon.ico Otherwise - Great job on the peppy site and breath of fresh air to open the network tab and see 1 html get, and another for the CSS. And the 404 favicon that I guess the browser insists on ;)
elteto 3y ago
I had seen marginalia mentioned here in HN a couple of times but never got around to use it. I'm very impressed. Using it I get this old Internet vibe (which someone else also mentioned). Just used it to get some information on a random topic I recently tried to research with Google but failed due to all the SEO crap. It produced several hits of old pages (with the tiny font and the early 2000's graphics and design), but _full_ of information. Not all the results are good though, it was mostly hit and miss, but the hits were _good_. Will use it from now on.
bigfatfrock 3y ago
You have inspired me for today, I appreciate it. Congratulations on cutting loose, always a great feeling.
Solvency 3y ago
Given that this is written on Java and running on a single server with fixed hardware.... Is there and what is the "peak" amount of optimization feasible in Java for this search engine before one would need to turn to C/Rust/etc to get any more performance out of this on the given hardware?
- lolinder 3y ago
  Presumably there is a peak, but Java can be really, really fast. I recently rewrote a heavy algorithm from Java to Rust, thinking that I'd get faster performance pretty much automatically. It turned out to be significantly slower than my optimized Java algorithm, and I didn't have the experience to tune the Rust version, so I ended up sticking to Java for now. I'm sure someone who knows how could have tuned the Rust version to get better performance, but native code is not my specialty and the Java version was doing fine. A warmed up JVM is a lot faster than most people think, especially for a long-running app like a search engine.
- marginalia_nu 3y ago
  Java's main limitation is probably in access to lower level I/O APIs, as well as vectorization support that is somewhat lackluster. There's almost definitely performance left on the table. It's relying quite heavily on memory mapped I/O and doing some clever things to work around language limitations in how much you can memory map at a time. This permits surprisingly good but not optimal performance. A bigger drawback is that this type of low level programming in Java is a serious pain in the ass.
thewataccount 3y ago
Is there a reason why stackoverflow isn't in there at all? EDIT: Also cool project!
- marginalia_nu 3y ago
  I have had it indexed in the past, but I don't at this point. It needs a special treatment since it's not really feasible to crawl, and you have to load it from their xml dumps instead.
stinkytaco 3y ago
This is one of my favorite HN adjacent projects and I use it with some frequency. Glad to see you are committed to it for the long term. Good luck.
adultSwim 3y ago
I really appreciate the candor in this post.
tern 3y ago
Feedback: I've tried the search engine a couple times and just bounced because there were no results for my queries, and it's hard to think of new queries that actually matter to me when I'm just trying something Hoping this turns into something magical. I dearly miss the old web!
- marginalia_nu 3y ago
  What queries did you try?
  - tern 3y ago
    The one I just tried was "urbit key rollover" Upon closer inspection, I think there's something disconcerting about using this engine in that—I don't know the right words for this but—it filters 100% by all the keywords. So, "rollover" is kind of a strange word, and because you don't have any results with these three words, I see nothing. I'd prefer to instead see results for "urbit key," given the circumstances. I imagine the algorithms to do this well are complicated though. Another query that had no results: "multi-band compression maselec" A query that has surprisingly few results: "qmmf-4." It shows a single forum post from a forum that has probably hundreds of posts matching this query. Why just one?
    - bornfreddy 3y ago
      Wow, thank you for the write-up! For me this is an endorsement of Marginalia, it seems to work exactly like it should... Like Google before it got "smart", or like AltaVista. Love it!
    - 082349872349872 3y ago
      > it filters 100% by all the keywords. IOW how queries ought to work. I am perfectly capable of changing the precision of my query to affect recall; I prefer that no "algorithm" ruins that ability.
      - pcthrowaway 3y ago
kaladin_1 3y ago
Great! Wishing you good luck. What I find quite nice about Marginalia is for discoveries outside the most popular destinations for such topics. For example, looking for a weekend movie but do not want to see all the SEO websites talking about movies. Marginalia surprises you with some unknown websites in the first page :) I use it when I want to be surprised by the results :D
- empyrrhicist 3y ago
  Just gave it a shot and this seems really interesting!
wg0 3y ago
I don't know it could be my perception but page loads on the the search page feel almost instantaneous. Curious to know about tech stack and especially underlying infrastructure.
- bornfreddy 3y ago
  It's actually a bit disconcerting to me because it feels like it didn't do anything. :)
- marginalia_nu 3y ago
  It's vanilla Java on non-virtualized/non-containerized on-prem hardware, a single server (actually just a big PC with an enterprise MVMe SSD). The search index is bespoke, but I'm using mariadb for the link database. It's a simple macroservice architecture.
daniel-cussen 3y ago
[dead]
ya1sec 3y ago
Best of luck. Easily my favorite project. Emailed Viktor last year about using the marginalia API for my side project[1] and he responded almost immediately. I use the API to get marginalia's arcane search results for a given query and choose a random link from those results to redirect. Endless fun. Hope to see it continue to grow until the internet goes dark. [1] https://moonjump.app/ https://moonjump.app/
- marginalia_nu 3y ago
  That's fantastic! A sort of indie internet discovery ecosystem effect is one of the things I've really been hoping to accomplish with Marginalia.
pluijzer 3y ago
Congratulations with your courageous step. I will really root for it, might even check out if I can contribute. I think Marginalia can have an amazing impact to the web. Right now it is dying from the cancerous growth of SEO spam and informations silos ever increasing in size. I tried Marginalia and already get amazing and fast results. This will make the web fun, creative and interesting again. Just like, I think, fellow countryman proving the world wrong that browsers cannot be created from scratch with Ladybird I think you will succeed also. (At least with search engines the competition gets worse every day.)
difflens 3y ago
Congratulations! I identify with this post a lot. Good luck! Your actions are certainly an inspiration
uptownfunk 3y ago
Working on my own project has easily become one of the most fulfilling things I’ve ever done in my life
iamthefury 3y ago
I’m curious about what and how crawling is done. I did a search for my own site and didn’t find it (it’s a redirect to another site, which I’m sure doesn’t help). What’s being indexed right now (out of curiosity, not trying to game SEO - that’s why I’m not mentioning the site I searched for here.)
- Kiro 3y ago
  I don't think it's supposed to index all sites. If you search for Twitter, Facebook, Instagram or even Hacker News you will not get any official results. It's meant to only show obscure sites but I'm unsure of the actual criteria.
doctor_eval 3y ago
> I gave it a shot, for no other reason than not being able to quite figure out why this supposedly impossible thing was impossible. Doing the napkin math, it seemed very possible. I thought this too! So happy someone has tested the assertion. Good luck! I’ve had Marginalia bookmarked for some time but this story will remind me to try it.
silcoon 3y ago
My main question is how you kept focused and motivated to work for two years on this project, especially at the beginning when no one was aware of it. What steps did you take to make sure you kept motivated?
- marginalia_nu 3y ago
  It's just an interesting project. Not really struggling with motivation as a result. It's also huge in terms of scope and ambition, so I can always take a break and take a weekend and go build something else within the scope of the project and still move it forward.
thr0w__4w4y 3y ago
I left a very good job 7 years in (digital design) to go out on my own. That was more than 2 decades ago. I could write paragraphs of the rookie mistakes (business-wise) and the financial ups & downs, but one thing has never changed... The "temporal freedom" I have in my work (Gad Saad, if you don't know the name). I love being the master of my own day, of my own time. I don't sit in Zoom meetings or have daily standups. I can get up at 5am and work until 11am, and then go hike, play with my dog, get ice cream with my daughters, workout, etc. and then work again from 7pm until midnight or whatever. Having (almost literally) full control over my daily schedule, week-in, week-out, year after year, is invaluable to me. One disclosure: a few times a year I do very hard things where I have very little freedom, but they allow me to have lots of freedom the rest of the year. Not to be a jerk, but I won't be elaborating. And I realize this life isn't for everyone!
- intelVISA 3y ago
  Sounds perfect :) I wonder how many great works we'd have built if most weren't trapped elsewhere.
_madmax_ 3y ago
Congratulations, I hope the monetization won't kill it
- marginalia_nu 3y ago
  My hope is that by engineering the service to have an extremely low burn rate (basically my living expenses plus change), I don't need to make that much money off it that it's viable to keep it close to the way it is.
bbor 3y ago
Congratulations! If you or anyone else knows of communities of likeminded people, please share :) I.e. excited about doing their own thing that they’re passionate about, but not all about the VC rat race
GenericDev 3y ago
I'm so happy to hear this! @marginalia_nu if you're reading this, please know that you're an inspiration and that I crazy appreciate what you're doing. We need people like you in the world pushing to make interesting things that aren't necessarily profit driven, but instead seek to help add flavor and interest back into the world. Your search engine is the kind of technology that reminds me of the technology ethos from the 90s and it's so amazing to see you get the chance to actualize it! Don't waste this chance! No matter what, know that this is the right decision. You have fans, and we're rooting for you! Thank you for making cool stuff!
- marginalia_nu 3y ago
  Aww shucks, thanks!

932 points by kevincox 3y ago

Congrats to the author! Marginalia is a great service. I hope they find a way to make it viable to keep going, either through donations or some other model.

samwillis 3y ago

That feeling when you walk out of an office for the last time, to work on your own thing is exhilarating. I had my moment like that back in 2014 and can still remember it. Congrats to Viktor and good luck! Going to go and try your search engine now. Previous discussion of the search engine a couple of months ago: https://news.ycombinator.com/item?id=35611923 https://news.ycombinator.com/item?id=35611923 (196 comments) Many other posts and blog posts over the last couple of years: https://news.ycombinator.com/from?site=marginalia.nu https://news.ycombinator.com/from?site=marginalia.nu

deleted 3y ago
[deleted]
munchler 3y ago
It was March 2006 for me, and I haven’t worked in an office since. What a great feeling.
- vasili111 3y ago
  What you are doing since 2006?
- throwaway290 3y ago
  I went the other way, in 15 year career never worked a day in the office until last week. So far I can WFH but dunno for how long. There are positives with an office but I kinda envy how everyone celebrates the opposite move
OJFord 3y ago
I would love to (though I'm a long way off it, with not much to walk away to) but I wonder what the equivalent feeling is if you already/previously work from home? Shipping the work machine back? Turning it off for the last time? Unplugging web cam and microphone?
- cambaceres 3y ago
  Perhaps removing Slack. What a feeling to never have to be online constantly from 8-5 so no one will think I’m not working.
wahnfrieden 3y ago
One year ago for me. I’ll never work for a boss again. And won’t be the boss of anyone again either - just as important to me.
- stolsvik 3y ago
  What do you do then? Live off the interests of some fortune?
  - wahnfrieden 3y ago
    I am currently working all the time. Independently designed/developed/operated/owned apps, and worker coop structures for working with others. I like consent based decision making rather than hierarchical or consensus. Command is unnecessary for us to do well for ourselves.

nhance 3y ago

Im curious what percentage of living expenses are covered by the project for the author. I have a few products generating over 50% of my yearly expenses and am feeling like going full time is almost a possibility now. A bit too nervous to pull the trigger just yet

samwillis 3y ago
I pulled that trigger later than I could have, I was earning 2x my salary from my side project before I quit. At 50% if you can see an upward trend, ~6 months savings, and have a plan that the time will give you to execute, got for it.
- bombcar 3y ago
  The "main thing" is how hard it would be to get back in the business (i.e, get a job) if the whole thing explodes. Also if you're going to quit anyway, you might as well ask the company you currently work for if they'll let you go on sabbatical, or part time, or consulting. That can give you a bit of extra runway/feeling of security.
  - valval 3y ago
    I suppose you can go back to any of the previous points in your CV as a software professional in today’a day and age if you never burned any bridges. Especially so if you make it obvious in your current job that you’re only leaving because it’s time to try your own thing - if it doesn’t end up working, people are likely to be very understanding.
    - marginalia_nu 3y ago
      It's not like I'm doing nothing for these upcoming years. Dunno what you're doing wrong if you can't land a job with a built-from-scratch internet search engine on your resume.
      - 3y ago

lucideer 3y ago

I've been trialling Marginalia Search a little and one thing that's struck me is the latency. The only other site I use with similar latency is HN; Marginalia seems even lower despite being dynamic (HN has a much easier caching story). I wonder is it just down to having lower traffic. It's certainly a lot lower than many low-/zero-traffic blogs I've frequented though. I've had a look at the README[0] for the Java sourcecode, but it's highly focused on crawls, database & indexing (understandable for search); would be cool to see a front-end focused write-up. [0] https://github.com/MarginaliaSearch/MarginaliaSearch/blob/master/code/readme.md https://github.com/MarginaliaSearch/MarginaliaSearch/blob/ma...

marginalia_nu 3y ago
The blog is just hugo so it's 100% static files over nginx. The search engine is serverside-rendered mustache templates via handlebars[1], via served via spark[2]. It's basically all vanilla Java. I do raw SQL queries instead of ORM, which makes it quite a bit snappier than most Java applications. The sheer size of the database also mandates that basically every query is a primary key lookup. The code is written around that constraint. Although the search engine is a bit on the slow side since it's routed through cloudflare and I think I'm relatively far away from the closest datacenter so it adds like 100ms to the load times. [1] https://github.com/jknack/handlebars.java https://github.com/jknack/handlebars.java [2] https://sparkjava.com/ https://sparkjava.com/
- lucideer 3y ago
  > The blog is just hugo Yeah the static stuff being fast is less surprising - it was mainly the search results page that astounded me. > via served via spark[2] Had not heard of this Spark (only the other Apache one). Will definitely take a look. > Although the search engine is a bit on the slow side since it's routed through cloudflare and I think I'm relatively far away from the closest datacenter so it adds like 100ms to the load times. I've hit the CF loading screen which introduces a big delay, but when I don't see that the loading is really instantaneous.
  - 3y ago

ASlave2Gravity 3y ago

Congrats! I find I'm using Marginalia more and more, it's especially great for researching for novel writing, and can’t wait to see what the future holds! Good luck!

mannycalavera42 3y ago

congrats! you should put the patreon link in more places IMHO (e.g. the main page and the various page footers)

jimbokun 3y ago

Does Marginalia Search have selling points over other search engines? Different features or philosophy? First I'm hearing of it.

marginalia_nu 3y ago
It's something like Google but what it kept working like it did in 2002 and then added a bunch of discovery features.
MikeSchurman 3y ago
From the about: "This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed. " I personally find it hard to put into words, but the old internet and old search engines had this feel to them that you never knew what you were going to get. Each site looked different. Each site had it's own philosophy of content and design. Everybody was winging it. It just felt more personal and interesting. At the risk of hyperbole, now it seems search engines give back mostly SEO blogspam that all looks the same. Marginalia feels more like the former internet.
- HankB99 3y ago
  > now it seems search engines give back mostly SEO blogspam And with technical questions, too many results are not correct. It seems that Google search is really going down hill in this regard. I'd like a way to vote down results that are obviously SEO trash, but I'm pretty sure if that were provided, it would be gamed too.

trollied 3y ago

The search engine: https://search.marginalia.nu/ https://search.marginalia.nu/

uticus 3y ago

I'd love to support, but Patreon link [0] on "supporting" page [1] is 404. Is another support option in progress to replace? [0] https://www.marginalia.nu/marginalia-search/supporting/patreon https://www.marginalia.nu/marginalia-search/supporting/patre... [1] https://www.marginalia.nu/marginalia-search/supporting/ https://www.marginalia.nu/marginalia-search/supporting/

asicsp 3y ago
Seems like the link text is correct (https://www.patreon.com/marginalia_nu https://www.patreon.com/marginalia_nu) but the linking functionality is missing.
marginalia_nu 3y ago
The links were cropped so I changed them with just a word for the service. But it turns out I can't markdown today and I changed the URLs instead of the text. Fixed now.
- sovietmudkipz 3y ago
  Question for the tester type software engineers on HN... I like to write browser (puppeteer) tests for user-facing software criteria like "patreon link must work." In the past I've written similar tests for small websites I've created where the purpose is to surface affiliate links for users to click on. My criteria is "from a money standpoint, is this the call to action I want my users to engage in?" I don't know what type of test this is -- can anyone disambiguate testing terminology for me? P.S. Browser based testing is brittle but since I often create websites and because I want to really ensure that I'm not 'lying' to myself in tests, using a browser is often the best (albeit slower) choice. These tests usually run in CI and I get notifications if they break. P.P.S. I wish we had a better mental model for the types of tests than the "testing pyramid." I find the testing pyramid lacking.
  - codazoda 3y ago
    I would probably call these integration tests. I came across this interesting tool for similar tests the other day. It lets you request websites or API’s and then search the return for a string. It’s more for checking uptime, so I dunno if it would be acceptable for this type of test, but it looks like a cool tool. https://onlineornot.com https://onlineornot.com

ilrwbwrkhv 3y ago

This is fantastic and commendable. Too many good hackers are tied up in a stable job at companies. Building something out of passion is just so different and the end result is so amazing that one cannot really fake that. Marginalia is one of my favorite sites. Wishing you all the best.

UpToTheSky 3y ago

Congrats! Going full time is the only way to go for a project you love and want to grow. What is the business model? How much visitors does Marginalia have?

mathgladiator 3y ago

It's a great feeling to leave and set off on your own journey. Beyond the feeling, it's also educational as you learn about your deficients quickly (or, in some cases, too slowly). I'm wrestling with this now as I'm building my platform and looking to pivot into something that produces revenue.

bombcar 3y ago

How do you submit a site? I know of a few good, small blogs and forums that don't come up. They're not on a VPS, either.

marginalia_nu 3y ago
https://github.com/MarginaliaSearch/submit-site-to-marginalia-search https://github.com/MarginaliaSearch/submit-site-to-marginali...
- wakamoleguy 3y ago
  Perhaps related (or I'm just not sure how this works), what criteria goes into whether a crawled site is indexed? My personal blog has 31 pages "known", 32 crawled, and only 3 indexed.
  - marginalia_nu 3y ago
    Whether it appears to be in English, whether there appears to be enough text, things like that. If you post the domain name (or email me) I can take a closer look tomorrow.

retrocryptid 3y ago

Cool that you're living your best life. Every time I leave a company, I think about the ending of The Prisoner: https://youtube.com/clip/UgkxgJAzCqKOL5yMg39wmtZi52tw8LAXOErQ https://youtube.com/clip/UgkxgJAzCqKOL5yMg39wmtZi52tw8LAXOEr...

mayormcmatt 3y ago
Great pull. I've always wanted the courage to slam my resignation down on my boss' desk and yell at them, so a different Prisoner scene for me.
- retrocryptid 3y ago
  [flagged]

rscrawfo 3y ago

Both of your top two projects are very interesting to me at the moment. Especially your Wikipedia mirror. Just today I realized how distracting too many hyperlinks can be. And Wikipedia is full of them! It feels so much easier to read an article without them. Now I just wish Wikipedia had more supporting graphics to help engage readers in a more productive manner.

inconceivable 3y ago

not knowing what comes in the next 12-24 months is exhilarating because you're actually living life with an acceptance of the nature of reality instead of deluding yourself or fighting it.

gnramires 3y ago

Congratulations marginalia ! :D

zo1 3y ago

Tiny bit of feedback, your encyclopedia favicon seems to 404: https://encyclopedia.marginalia.nu/favicon.ico https://encyclopedia.marginalia.nu/favicon.ico Otherwise - Great job on the peppy site and breath of fresh air to open the network tab and see 1 html get, and another for the CSS. And the 404 favicon that I guess the browser insists on ;)

elteto 3y ago

I had seen marginalia mentioned here in HN a couple of times but never got around to use it. I'm very impressed. Using it I get this old Internet vibe (which someone else also mentioned). Just used it to get some information on a random topic I recently tried to research with Google but failed due to all the SEO crap. It produced several hits of old pages (with the tiny font and the early 2000's graphics and design), but _full_ of information. Not all the results are good though, it was mostly hit and miss, but the hits were _good_. Will use it from now on.

bigfatfrock 3y ago

You have inspired me for today, I appreciate it. Congratulations on cutting loose, always a great feeling.

Solvency 3y ago

Given that this is written on Java and running on a single server with fixed hardware.... Is there and what is the "peak" amount of optimization feasible in Java for this search engine before one would need to turn to C/Rust/etc to get any more performance out of this on the given hardware?

lolinder 3y ago
Presumably there is a peak, but Java can be really, really fast. I recently rewrote a heavy algorithm from Java to Rust, thinking that I'd get faster performance pretty much automatically. It turned out to be significantly slower than my optimized Java algorithm, and I didn't have the experience to tune the Rust version, so I ended up sticking to Java for now. I'm sure someone who knows how could have tuned the Rust version to get better performance, but native code is not my specialty and the Java version was doing fine. A warmed up JVM is a lot faster than most people think, especially for a long-running app like a search engine.
marginalia_nu 3y ago
Java's main limitation is probably in access to lower level I/O APIs, as well as vectorization support that is somewhat lackluster. There's almost definitely performance left on the table. It's relying quite heavily on memory mapped I/O and doing some clever things to work around language limitations in how much you can memory map at a time. This permits surprisingly good but not optimal performance. A bigger drawback is that this type of low level programming in Java is a serious pain in the ass.

thewataccount 3y ago

Is there a reason why stackoverflow isn't in there at all? EDIT: Also cool project!

marginalia_nu 3y ago
I have had it indexed in the past, but I don't at this point. It needs a special treatment since it's not really feasible to crawl, and you have to load it from their xml dumps instead.

stinkytaco 3y ago

This is one of my favorite HN adjacent projects and I use it with some frequency. Glad to see you are committed to it for the long term. Good luck.

adultSwim 3y ago

I really appreciate the candor in this post.

tern 3y ago

Feedback: I've tried the search engine a couple times and just bounced because there were no results for my queries, and it's hard to think of new queries that actually matter to me when I'm just trying something Hoping this turns into something magical. I dearly miss the old web!

marginalia_nu 3y ago
What queries did you try?
- tern 3y ago
  The one I just tried was "urbit key rollover" Upon closer inspection, I think there's something disconcerting about using this engine in that—I don't know the right words for this but—it filters 100% by all the keywords. So, "rollover" is kind of a strange word, and because you don't have any results with these three words, I see nothing. I'd prefer to instead see results for "urbit key," given the circumstances. I imagine the algorithms to do this well are complicated though. Another query that had no results: "multi-band compression maselec" A query that has surprisingly few results: "qmmf-4." It shows a single forum post from a forum that has probably hundreds of posts matching this query. Why just one?
  - bornfreddy 3y ago
    Wow, thank you for the write-up! For me this is an endorsement of Marginalia, it seems to work exactly like it should... Like Google before it got "smart", or like AltaVista. Love it!
  - 082349872349872 3y ago
    > it filters 100% by all the keywords. IOW how queries ought to work. I am perfectly capable of changing the precision of my query to affect recall; I prefer that no "algorithm" ruins that ability.
    - pcthrowaway 3y ago

kaladin_1 3y ago

Great! Wishing you good luck. What I find quite nice about Marginalia is for discoveries outside the most popular destinations for such topics. For example, looking for a weekend movie but do not want to see all the SEO websites talking about movies. Marginalia surprises you with some unknown websites in the first page :) I use it when I want to be surprised by the results :D

empyrrhicist 3y ago
Just gave it a shot and this seems really interesting!

wg0 3y ago

I don't know it could be my perception but page loads on the the search page feel almost instantaneous. Curious to know about tech stack and especially underlying infrastructure.

bornfreddy 3y ago
It's actually a bit disconcerting to me because it feels like it didn't do anything. :)
marginalia_nu 3y ago
It's vanilla Java on non-virtualized/non-containerized on-prem hardware, a single server (actually just a big PC with an enterprise MVMe SSD). The search index is bespoke, but I'm using mariadb for the link database. It's a simple macroservice architecture.

daniel-cussen 3y ago

[dead]

ya1sec 3y ago

Best of luck. Easily my favorite project. Emailed Viktor last year about using the marginalia API for my side project[1] and he responded almost immediately. I use the API to get marginalia's arcane search results for a given query and choose a random link from those results to redirect. Endless fun. Hope to see it continue to grow until the internet goes dark. [1] https://moonjump.app/ https://moonjump.app/

marginalia_nu 3y ago
That's fantastic! A sort of indie internet discovery ecosystem effect is one of the things I've really been hoping to accomplish with Marginalia.

pluijzer 3y ago

Congratulations with your courageous step. I will really root for it, might even check out if I can contribute. I think Marginalia can have an amazing impact to the web. Right now it is dying from the cancerous growth of SEO spam and informations silos ever increasing in size. I tried Marginalia and already get amazing and fast results. This will make the web fun, creative and interesting again. Just like, I think, fellow countryman proving the world wrong that browsers cannot be created from scratch with Ladybird I think you will succeed also. (At least with search engines the competition gets worse every day.)

difflens 3y ago

Congratulations! I identify with this post a lot. Good luck! Your actions are certainly an inspiration

uptownfunk 3y ago

Working on my own project has easily become one of the most fulfilling things I’ve ever done in my life

iamthefury 3y ago

I’m curious about what and how crawling is done. I did a search for my own site and didn’t find it (it’s a redirect to another site, which I’m sure doesn’t help). What’s being indexed right now (out of curiosity, not trying to game SEO - that’s why I’m not mentioning the site I searched for here.)

Kiro 3y ago
I don't think it's supposed to index all sites. If you search for Twitter, Facebook, Instagram or even Hacker News you will not get any official results. It's meant to only show obscure sites but I'm unsure of the actual criteria.

doctor_eval 3y ago

> I gave it a shot, for no other reason than not being able to quite figure out why this supposedly impossible thing was impossible. Doing the napkin math, it seemed very possible. I thought this too! So happy someone has tested the assertion. Good luck! I’ve had Marginalia bookmarked for some time but this story will remind me to try it.

silcoon 3y ago

My main question is how you kept focused and motivated to work for two years on this project, especially at the beginning when no one was aware of it. What steps did you take to make sure you kept motivated?

marginalia_nu 3y ago
It's just an interesting project. Not really struggling with motivation as a result. It's also huge in terms of scope and ambition, so I can always take a break and take a weekend and go build something else within the scope of the project and still move it forward.

thr0w__4w4y 3y ago

I left a very good job 7 years in (digital design) to go out on my own. That was more than 2 decades ago. I could write paragraphs of the rookie mistakes (business-wise) and the financial ups & downs, but one thing has never changed... The "temporal freedom" I have in my work (Gad Saad, if you don't know the name). I love being the master of my own day, of my own time. I don't sit in Zoom meetings or have daily standups. I can get up at 5am and work until 11am, and then go hike, play with my dog, get ice cream with my daughters, workout, etc. and then work again from 7pm until midnight or whatever. Having (almost literally) full control over my daily schedule, week-in, week-out, year after year, is invaluable to me. One disclosure: a few times a year I do very hard things where I have very little freedom, but they allow me to have lots of freedom the rest of the year. Not to be a jerk, but I won't be elaborating. And I realize this life isn't for everyone!

intelVISA 3y ago
Sounds perfect :) I wonder how many great works we'd have built if most weren't trapped elsewhere.

_madmax_ 3y ago

Congratulations, I hope the monetization won't kill it

marginalia_nu 3y ago
My hope is that by engineering the service to have an extremely low burn rate (basically my living expenses plus change), I don't need to make that much money off it that it's viable to keep it close to the way it is.

bbor 3y ago

Congratulations! If you or anyone else knows of communities of likeminded people, please share :) I.e. excited about doing their own thing that they’re passionate about, but not all about the VC rat race

GenericDev 3y ago

I'm so happy to hear this! @marginalia_nu if you're reading this, please know that you're an inspiration and that I crazy appreciate what you're doing. We need people like you in the world pushing to make interesting things that aren't necessarily profit driven, but instead seek to help add flavor and interest back into the world. Your search engine is the kind of technology that reminds me of the technology ethos from the 90s and it's so amazing to see you get the chance to actualize it! Don't waste this chance! No matter what, know that this is the right decision. You have fans, and we're rooting for you! Thank you for making cool stuff!

marginalia_nu 3y ago
Aww shucks, thanks!