9 ms·
Marginalia Goes Open Source
- lowwave 4y agoThank god a search engine that try to sell you stuff. Glad it is open sourced. Are they from Scandinavia?
- marginalia_nu 4y agoI'm based in Sweden, yeah.
- SemanticStrengh 4y agoThat's one small step for man, but one giant leap for mankind.
- nonrandomstring 4y agoWell done and thank you. Not just for the work, but for the attitude and courage of your convictions that come across clearly in your comments here on HN. Search is presently a neighbourhood of technology that's fallen into a downtrodden ghetto. Maybe you've sowed some seeds that can grow between the cracks. But I think there are gangs and pushers roaming the streets that would be happy for it remain derelict. Thinking about the dynamics of open source, with respect to the many different interpretations of search that create tension within such a project: It occurs to me that whenever a somewhat mature project enters the ecosystem a robust strategy is to immediately clone-fork multiple identical copies of a project. This is like a "backup strategy". From there, form several loosely coordinated project groups that can spread risk, share threat intelligence, but each work with different models of crawling, filtering, funding, presentation and so on. Keep the original as an upstream parent. One of the children may survive.
- josefcullhed 4y agoGreat job with Marginalia. Do you plan to open source your data as well or only the code?
- marginalia_nu 4y agoIf I can solve the logistics, maybe. I don't have the needed bandwidth and off-site storage at this point.
- SahAssar 4y agoWhat sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.
- marginalia_nu 4y agoI'm probably producing around 250-500 Gb data/month at this point.
- SahAssar 4y agoWhat's the cumulative size for the index to date? I'm not rich by any measure, but if it's within reach I'd probably fund the storage costs.
- justinclift 4y agoAny idea how compressible that is? If it's something that compresses really well (eg text data in a database), then live compression filesystems (eg ZFS, likely others) could potentially help make that workable.
- marginalia_nu 4y agoThe data is either already compressed or dense binary soup, so no luck.
- indrora 4y agoReach out to Jason Scott -- textfiles.com -- and see if he knows anyone who would be interested. He might know some folks.
- deleted 4y ago
- hestefisk 4y agoVery cool! And brave! Good on you.
- hestefisk 4y agoDo you have any high level architecture docs? Would love to hack on it.
- marginalia_nu 4y agoI'm working on it. Still in the process of setting up the project, so it will be a while before any sort of useful level of documentation is in place.
- firedev 4y agoWhat is Marginalia? Feels like a links database.
- marginalia_nu 4y agoIt's a grab-bag of projects geared toward internet discovery and information access. It's got an internet search engine, a lo-fi wikipedia mirror, this thing <https://search.marginalia.nu/explore/random https://search.marginalia.nu/explore/random>, and more.
- marginalia_nu 4y agoActual announcement: https://memex.marginalia.nu/log/58-marginalia-open-source.gmi https://memex.marginalia.nu/log/58-marginalia-open-source.gm... My gitea instance is on a poor Raspberry Pi, probably won't survive HN front page :P
- deleted 4y ago[deleted]
- slowbdotro 4y agoIf you want, come mirror it on my gitea instance: gitea.slowb.ro. I'd be happy to take the load ^-^
- tejtm 4y agoI am not surprised you seem compelled to do the right thing
- pabs3 4y agoThanks for choosing the GNU AGPLv3 license! That is the best license for services like search engines, especially if they intend to cement the license by having a diverse set of copyright holders that ensure the license doesn't change, rather than going the open core route and requiring copyright assignment via a CLA and selling proprietary versions. What is your plan for sustaining the project in terms of contributions, funding and hardware?
- marginalia_nu 4y agoYeah, that was my analysis as well. > What is your plan for sustaining the project in terms of contributions, funding and hardware? Still much to figure out. The project is still fairly immature in general. As far as hardware and funding goes, it's sort of sustainable now through not demanding very much in either, but more of both would no doubt be necessary to grow much farther. It's a bit claustrophobic not really having a proper CI machine or test instance for example.
- pabs3 4y ago
- opan 4y agoAGPLv3+, nice!
- formerkrogemp 4y agoHaven't there been estimates that running a globally indexed search engine product costs a billion dollars per year? As always, thank you for marginalia!
- Aeolun 4y agoI think the helpful thing is that if you don’t intend to keep all blogspam, you are in a relatively good position.
- marginalia_nu 4y agoThis only does a fraction of what the big search engines do, though. Still intresting to find out what is possible given those constraints.
- thrtythreeforty 4y agoOpen source has a wonderful property: it only ever gets better as time goes on. Thank you for permanently improving the FOSS search state of the art.
- mikewarot 4y agoTLDR; There are lots of interesting tangents here, even if you don't want to run a web search engine. I'm in search of kindred spirits seeking to flush out the full power of the memex. I found the notes it quite interesting[1] I've gone through quite a bit of the related content, up to watching a bit about the Gemini protocol, which is interesting. 1 - https://memex.marginalia.nu/log/08-whatever-happened-to-the-memex.gmi https://memex.marginalia.nu/log/08-whatever-happened-to-the-...
- jwilk 4y agoThe search engine discussed on HN: https://news.ycombinator.com/item?id=28550764 https://news.ycombinator.com/item?id=28550764 (September 2021, 725 comments)
- ternaryoperator 4y ago> I feel GitHub has taken an incredibly toxic turn with its emphasis on social features This the first time I've seen this view of GitHub, and I'm struggling to figure out what specifically he's referring to.
- pelagicAustral 4y agoSneedacity debacle.
- baisq 4y agoFunny you'd mention this. I installed Audacity a few hours ago and much to my dismay the application informed me upon starting it for the first time that update checks were enabled by default - in other words, Audacity had already called home and I couldn't have done anything to stop it from doing so. So yes, they deserved the abuse.
- ntauthority 4y agoFirst off, nobody deserves any sort of abuse. Secondly, the Sneedacity incident was abuse targeted at a fork of the Audacity project, not at the original 'hostile takeover' project itself. The fork was intended to remove said abuse you feel is deserving of more abuse.
- Xeoncross 4y agoThis is a great gift to the community, but I really wish it would be hosted where the community is active. It seems like a lot of people will miss out because they won't be able to discover this. I find a lot of projects because I follow people on github that star or fork really neat projects. I download projects from bitbucket, gitlab, and other self-hosted instances. I don't know that I've ever contributed a PR back as it's just a more friction-prone process. However, I've opened hundreds of PR's and Issues on github because it's already a part of my daily process. Thanks again
- robertlagrant 4y ago
- throwaway83242 4y agoThanks for making this open source. What about the index? Are you open sourcing that also?
- marginalia_nu 4y agoIf I can solve the logistics of publishing that data, then sure. In its most compressed form it's still of order 100 Gb. The intermediate goal is to have some standardized testing dataset of a couple of hundred megabytes to a gigabyte or so.
- Jarwain 4y agoLike another commenter suggested, torrents might be a good solution once it's seeded
- throwaway83242 4y agoCool. Looking forward to see the intermediate dataset. I think you should post a ToDo list on the git repo. People can then contribute their skills.
- marginalia_nu 4y agoYeah, that's a good idea. I'm looking at a bunch of ideas for reducing the friction to contributing, still a bit of work that needs doing in that area.
- sdoering 4y ago> I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions. I really don't have the bandwidth to serve it myself. It's only a couple of hundred megabytes so it'll probably be solvable somehow. I have a bit of (rented) space available. Most of it on Uberspace. What are the expected traffic requirements? Contact me if you like. Will update info in profile immediately.
- sva_ 4y agoI'm not sure if it is appropriate for this use case, but wouldn't it be possible to host it on Backblaze, distribute through Cloudflare for free traffic? Or even just a torrent.
- remram 4y agoGitHub Releases can host files up to 2 GB, for free.
- closedloop129 4y agoWouldn't this be a good match for BitTorrent distribution? The data will be updated regularly and then, all participants will be online and able to share their bandwidth while they download the new frequencies.
- trenchgun 4y agoAlright, this seems great. Going to test it in daily use. I am ready to start paying money for search engine. So fed up with google SEO spam.
- marginalia_nu 4y agoProbably not going to replace Google, it's at best aiming to pick up the slack where it's struggling. But dunno, maybe down the line it will have grown into a real open source alternative.