9 ms·
Lobste.rs is now running on SQLite
- tetris11 3mo agoOh wow. Off-topic but quite refreshing to see a site not absolutely plastered in AI posts
- grebc 3mo agoHN is increasingly bot central, I kind of assume 80% of the respondents are artificial. Lobsters still got a pulse, largely.
- rickcarlino 3mo agoInvitation-only is the future of online communities. Lobsters still has a pulse because you need to know someone to get in and it’s a bad look if you invite annoying people.
- OutOfHere 3mo agoThat's nonsense wrt invitation-only being the future. I can equally say that free speech is the future. For comparison, Reddit does very well without being invitation only.
- Xeronate 3mo agoreddit does okay from a profit standpoint. reddit does terribly from a bot or even a good community standpoint.
- inigyou 3mo agoReddit is literally the dead internet right now. It's also not open access. You have to run an invisible tightrope so its algorithms won't automatically shadowban you and then you have to run a barely visible tightrope to get enough karma to not have your posts automatically removed from every subreddit.
- inigyou 3mo agoAlso, Reddit now requires you to verify your ID with Persona.
- tverbeure 3mo agoI wish lobste.rs would be a tiny bit easier to get access to. Some of my blog posts have been discussed there and every time I try to get access, I can’t get past their IRC stuff. As in: I just don’t understand how IRC works…
- Xeronate 3mo agoim sorry but maybe connecting to irc is a bit of a filter on the users they expect. especially with ai it should be almost no effort.
- devilsdata 3mo agoI've been waiting for my lobste.rs invite for years. I joined the IRC for a few days, but being in Australia, rarely saw too much activity. I could chase someone down for an invite, but at that point I just don't care that much.
- abetusk 3mo agoI was given my invite from a decent person who, I believe, basically has an "open door" policy. I'm happy to pay it forward and give you an invite if you want it. abetusk@mechaelephant.com
- cpach 3mo agoFWIW, you might want to consider publishing your e-mail address in your HN profile or on your website. Otherwise it’s hard for people to get in touch with you.
- pudgywalsh 3mo agoYou should overlay a banner on your blog when they send you an HTTP Referer explaining their nonsense. Maybe then they'll get the message. Then again, maybe they like the ability to critique your posts without you being able to rebut their opinions. It's an effective tactic in online echo chambers.
- accelbred 3mo ago
- cdrini 3mo ago> Invitation-only is the future of online communities. This feels antithetical/dystopian/elitist compared to what online communities are, for me. I think it's ok for some communities to be like this, but I don't think it's the future of online communities.
- qsera 3mo agoI think it is better to have some elitist communities, than to drag everyone down to some least common denomination... I think that was one of the big mistakes/oversight that came with the Internet..
- cdrini 3mo agoI think some are fine, but if I had to choose one I'd choose open access, so I think it's the more important of the two. And in fact it's because the web is open, "no permission needed" by default that allows elitist groups to exist on the web. So I don't think I see it as a mistake. The alternative would be... You can only create a website if you're invited to the internet by someone who already has access? You have to get approval for your website from some authority before it goes public? Those were all potential futures we imo thankfully and carefully avoided!
- inigyou 3mo agoThat's actually how it already works, but there are large commercial access providers known as "ISPs". Before there were ISPs, you literally could only get on the internet with a direct connection to someone on the internet, usually a large university.
- qsera 3mo ago>Those were all potential futures we imo thankfully and carefully avoided! I am not so thankful. But it is coming from my view of the current world. I think open Internet for business is good. But open Internet for communication was bad. I cannot think of one good thing that have resulted from an open internet for communication/discussions. It has turned out to result in a collective, perpetual state of seizure for all humanity. I am someone who had to learn stuff from random encounters with books in the 90s. Even then, I would not have it any other way...
- OutOfHere 3mo agoThat's an unsubstantiated and false accusation wrt the 80%. If you believe it, why are you even here? Are you one of them?
- grebc 3mo agoBad bot.
- synqvest 3mo ago[dead]
- inigyou 3mo agoHN has always been "YC bootlickers only"
- gspr 3mo agoIt's a wonderful place. It feels like HN, but with the Silicon Valley attitude dialed way way way down, and with a much narrower scope (politics, economics and entrepreneurship are all off-topic, for example).
- christophilus 3mo agoHuh. I’ve largely avoided it, but that does sound pretty nice.
- dgellow 3mo ago[flagged]
- dprkh 3mo agoIf still relevant, you can email me and I will send you an invite. My email is dprkh at fastmail dot com
- NetOpWibby 3mo agoThe SV attitude isn't there but there's active assholes who wield their moderation weirdly. I deactivated my account before deactivations were added to the logs and I don't remember what made me quit. I've largely avoided Lobsters since. I joined their IRC channel last week and it's been rather chill. I do like the invite-only nature of the site though, keeps quality high...while making some folks tyrannical.
- cromka 3mo agoEvery such "club" inevitably attracts a specific sort that badly want to join just for the sense of exclusivity and a chance to virtue-signal their member status. It's same thing with some hard-to-get-into torrent trackers — once in, people become poster-children for Stockholm syndrome and the mods really get off on their perceived power. It's also the same with expensive new sport car models that manufacturers sell to collectors. My intuition says it's to do with Status Signaling theory .
- 3mo ago
- Aurornis 3mo agoLobste.rs is very anti-AI. They force any LLM topic to be tagged with “vibecoding” even when the majority of LLM posts are not about vibecoding. AI posts are usually the most commented on. Linus Torvalds’ comments saying that LLMs are actually useful is still on the front page, tagged as vibecoding, and has a lot of comments from people mostly disagreeing with him. Lobste.rs is more of a monoculture than Hacker News. If you get downvoted enough they stick a banner on the top of the page inviting you to delete your account. It has a nasty side effect of driving away good contributors who don’t align perfectly with the hive mind. It’s a site where you learn to keep your mouth shut if your input doesn’t agree with what the core users want to hear.
- dgellow 3mo agoI honestly don’t find any of this concerning
- mh- 3mo agoThe concern usually only comes when you fall on the "outside" of the prevailing opinion within the group of people voting on comments. edit: my response isn't about lobste.rs. I don't have an account there.
- xtracto 3mo agoI dont remember why I was kind of barred from lobsters , but I applaud all this behavior you mentioned. In the end, that's how normal clickes of friends/interest groups form (or formed 30 years ago in real life.) If you got into my group of death metal people, and suddenly started talking about Oasis... we would invite you to get out haha. Same with patriots.win . They may be a group of right wing biggots, but hey, they did their platform and are happy talking there.
- OutOfHere 3mo ago[flagged]
- 3mo ago
- cromka 3mo agoWait, no edge cache, just one node with SQLite db?
- thomasdziedzic 3mo agoCaddy does cache a portion of the traffic, and what it can't cache makes its way to rails.
- packetlost 3mo agoFor a relatively low-traffic site like Lobster's, yeah I'm not surprised they don't need anything crazy. SQLite is really efficient and modern hardware is really fast.
- booi 3mo agoI don't think SQLite is particularly optimized for modern hardware. It's just crazy efficient with a regression test suite many projects wish they could have.
- packetlost 3mo agoModern hardware mostly benefits from the optimizations made for older hardware too, so I'm not really sure what your point is.
- homebrewer 3mo agoIt's been fairly unstable recently, pages sometimes render for several seconds which I've never seen under MariaDB. Used to be instantaneous, always. Sometimes (maybe 5% or less) the request won't render at all, and you get a browser error page. Today they ran into this bug, lost a bunch of voting data, and went into read-only mode for several hours: https://github.com/rails/rails/pull/57128 https://github.com/rails/rails/pull/57128 I wonder how much of this is usual bugs which crop up during major database migrations, and how much is caused by choice of SQLite.
- srcreigh 3mo agoPages (especially threads while logged in) taking seconds to load definitely happened to me pre SQLite migration.
- eterm 3mo agoSQLite definitely seems like a poor choice for dealing with many concurrent requests. Maybe it's improved since I last used it, but to my knowledge SQLite essentially forces all writes to be serialised, at risk of data corruption otherwise. There are tricks for improving the performance such as WALs, but that is merely a performance boost rather than genuine concurrency with things like row-level locks that you might find in other databases. I guess if the whole thing is architected with a write-through cache that handles concurrent writes and deals with serialising all the writes, then it can be a single writer streaming changes through to the database, but then you still have a point of serialisation, it just will manifest itself slightly differently. And SQLite is something that will give you constraints you will always have to consider. Whereas running mariadb or postgres on the same machine would deliver similar benefits without a risky migration. If your DB is small enough to run as a SQLite database, then it probably ought to have never been on a different machine in the first place. There is a very happy medium between SQLite and a database on a different machine, one I am continually surprised to see people ignore.
- veqq 3mo agoedit: misread requests as writes
- ethagnawl 3mo ago[flagged]
- sgt 3mo agoWhat's lobster.rs, some kind of Temu HN? Or is it the real deal?
- written-beyond 3mo agoIt's superior and invite only.
- xgbi 3mo agoWay more technical, way smaller. HN is a lot more wide in terms of audience and subjects, Lobsters is mainly programming. I love it.
- veqq 3mo ago[flagged]
- ethagnawl 3mo agoThank you!
- Kuyawa 3mo agoNo intention to antagonize but... why not postgre? Even my side projects run on postgresql with no overhead at all
- adenta 3mo agoWhere are you hosting postgres
- dewey 3mo agoOn the same machine where they are currently storing the sqlite file maybe? It's not that hard to run. A sqlite backup cron is also not that much more work than a pg_dump cron.
- inigyou 3mo agoif they have only one server, then on that server. If they have a load-balanced front end and central database, then on the database.
- gnulinux 3mo ago> run on postgresql with no overhead at all Citation needed. I love postgres as much as the next person, but it does have more overhead than Sqlite which is in an in-process db linked through compiled C code, it doesn't run as a separate server. Very significant difference that when you use Sqlite db, there is typically no db process other than your application logic, unless you implement the server yourself. If you don't want your application to have multiple processes (say, as a toy example) then it totally makes sense to prefer Sqlite just for this reason. Sqlite and postgres are different tools, they serve different purposes.
- christophilus 3mo agoIf you run it on your one app server, it’s not much more work than SQLite. SQLite requires some special care when you’re dealing with multiple processes, and backups are about the same amount of effort.
- 3mo ago
- cromka 3mo agoIronically, this post resurfaced on HN with my comment here shown as if made 15 minutes ago, which I in reality did last week?
- stymaar 3mo agoSecond chance pool, most likely.
- mh- 3mo agoI didn't realize that re-timestamped comments. That's a pretty unexpected behavior, if true. I wouldn't like if that happened to a comment I left - a lot can happen in a week.
- inigyou 3mo agoEvery time it happens there's a sub-thread asking about the weird timestamps and dang replies to it saying changing the timestamps is done so that there won't be a sub-thread about the weird timestamps.
- tayo42 3mo agoI'm pretty sure this comment chain is in every popular 2nd chance pool post though anyway
- stymaar 3mo agoTrue. I don't understand why they overwrite the date at all actually, for either the post or the comments, that sounds like the hackiest way of implementing the “second chance pool” feature.
- dredmorbius 3mo ago<https://news.ycombinator.com/item?id=48740177 https://news.ycombinator.com/item?id=48740177> (dang, on 2CP & timestamps). More: <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20timestamps&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
- pstuart 3mo ago[flagged]
- evanelias 3mo agoSome of the cited reasons for moving off MariaDB [1] seem misguided, in my opinion. Especially the part about "K1 are very enterprise-focused, so the database is likely to focus its work on features that are not relevant to us. There's increased risk they drop the free/open source version we use" K1 acquired the commercial entity behind MariaDB Enterprise, but that's separate from the non-profit MariaDB Foundation. And there's literally zero risk of the MariaDB server suddenly going closed-source; as a fork of MySQL (which is GPL), this is not even legally possible! [1] https://github.com/lobsters/lobsters/issues/539#issuecomment-2605214056 https://github.com/lobsters/lobsters/issues/539#issuecomment...
- edoceo 3mo agoAnd even if it did go closed-souce, your existing install can run for ages because the vendor cannot force upgrade. One of my favorite features of FOSS.
- vasco 3mo agoThis makes little sense but I guess they get little traffic so pretty much any database will work for them. If traffic grows they will find problems, then limitations, then move to postgres. And if not it's fine. Story old as time.
- saila 3mo agoThe GPLv2 doesn't prevent a company from distributing binaries and making the source available on request while development is done behind closed doors. This wouldn't really be "open source" as we typically understand it and might be cause to exclude it from some Linux distros, for example. I'm not saying that K1 would do this (I have no idea who they are or what they do), but I also think it's a reasonable risk to consider.
- kenforthewin 3mo agoThe site is performing poorly for me. For example, the login page took 9 seconds to load, same with the home page.
- cenazoic 3mo agoMay I have an invite, if anyone has one to spare?
- abetusk 3mo agoProvide an email or email me (abetusk@mechaelephant.com) so I can get yours for the invite.
- etbusch 3mo ago[flagged]
- abofh 3mo agoI hope it's profitable, because this feels like engineering for engineering sake and not actually improving the product in a way that draws customers.
- jdw64 3mo ago[flagged]
- rfmoz 3mo agoAnother SQLite user here. My site gets constantly hammered by crawlers, and SQLite just keeps going on hit miss. Anyway, the real magic is on the ram and the ssd perf.
- cdrini 3mo ago> keeps going on hit miss (Is this a typo?)
- varun_ch 3mo agoDoes HN still use simple files for storing its data?
- luciana1u 3mo ago[flagged]
- oandrew 3mo agoI'd appreciate a lobste.rs invite if anyone has one. Email is in my profile. Thanks!
- Imustaskforhelp 3mo ago> I'd appreciate a lobste.rs invite if anyone has one. Email is in my profile. Thanks! I'd like one too if someone wishes to invite me as well, Thanks and have a nice day!
- uproarchat 3mo agoSuccessfully e-mailed invitation to anonplusplus@proton.me
- Imustaskforhelp 3mo agoI have signed up, thanks! You are awesome mate!, have a good day :-D
- uproarchat 3mo agoTake it easy :)
- mixmastamyk 3mo agoPlease don’t post other’s addresses, after a bit are unable to be removed. If they decide to later it’s too late.
- deleted 3mo ago[deleted]
- uproarchat 3mo agoSuccessfully e-mailed invitation to hn@oandrew.dev.
- osinix 3mo ago[flagged]
- amazingamazing 3mo agoSoftware “engineering”. Thank goodness bridges aren’t built with this level of rigor. In any case I will take an invite if one has one because the community is solid.
- abetusk 3mo agoGive your email or send me a message (abetusk@mechaelephant.com) so I can send the invite.
- amazingamazing 3mo agoDone. Just added to profile, thank u!
- Peanuts99 3mo agoI've got a few planks over a ditch in my garden. The bridge only needs to be designed for the load and then some. In this case, this seems to work just fine.
- chasil 3mo agoSome observations: 1. Stats are used by "Cost-Based Optimizers." I think this is the most appropriate wiki: https://en.wikipedia.org/wiki/Query_optimization https://en.wikipedia.org/wiki/Query_optimization On my Oracle databases, I run the dbms_stats.gather_database_stats package procedure once a week. SQL Server also has stats for sure, and I would be surprised by a relational database that did not. SQLite will quietly gather stats with each database connection. Running an explicit analyze devotes full attention to an otherwise piecemeal group effort. 2. I don't think rows are physically removed when deleted, instead they are marked. Removal will definitely happen at vacuum. 3. There is a backup command in the CLI that will make a transaction consistent copy of the database file, and there are lower-level C functions behind it that can be used with object code. 4. It is important to remember the many restrictions of WAL mode, including the impact on ATTACHing multiple database files: ACID consistency is lost. https://sqlite.org/lang_attach.html https://sqlite.org/lang_attach.html 'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL.'
- aboardRat4 3mo ago[dead]
- blobbers 3mo agoDo you feel like lobste.rs is a better version of HackerNews?
- veqq 3mo agoThey're very distinct. Both are good in their own ways like elephants and dolphins. Though HN suffers more under AI spam, it permits wider topics and has a wider userbase.
- xboxnolifes 3mo agoI only lurk occasionally, but lobsters feels like a much smaller, less jaded community then HN. The content is very similar. Hopefully not too many people who get invites from this thread bring over the jadedness.
- synqvest 3mo ago[dead]
- arjie 3mo agoPretty cool. I do default to SQLite these days simply because I can just put it on the host, set up a PVC to it, and run the app in Kube pointing at a persistent SQLite. Absent some other need (it's a fork of psql- or mysql-based software) I prefer using it for simplicity. I do have to say that I've found lobste.rs generally a pretty slow site to render and navigate and I wonder whether that's the Rails backend, or underpowered hardware, or lack of caching. It's more feature-ful than a HN or Reddit though with much lower usage so I wonder what it is.
- dmix 3mo agoNice to see Lobsters is made with ruby DHH has made a big deal about using SQLite in rails
- kgeist 3mo agoThey use WAL in SQLite. If I continuously perform reads/writes so that they overlap with no gaps, I can make their VM go down because SQLite will not have time to initiate a checkpoint to trim the WAL file. SQLite waits for a time window without any active reads/writes before starting a WAL checkpoint. If there isn't one, the WAL will grow indefinitely, eating up all the disk space on the VM. It's in SQLite's documentation, and almost no one switching to Sqlite seems to be aware of it because no one discusses it in blog posts like these. I guess most projects switching to SQLite have very low traffic and no malicious users (yet)
- abetusk 3mo agoCould you go into more detail about this issue and provide some links to the documentation? I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]: > 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of. and from their "checkpointing" documentation [1]: > By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages. I'm only skimming but I see no mention of waiting for an idle time window. Have you been successful in DoS'ing lobste.rs? [0] https://sqlite.org/wal.html https://sqlite.org/wal.html [1] https://sqlite.org/wal.html#ckpt https://sqlite.org/wal.html#ckpt
- kgeist 3mo agoOn the page you linked: >However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound. >This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times. Dunno, maybe Rails has a built-in workaround for this. My workaround was to run a separate thread that monitored the WAL size on disk every second. If it went above the target size of 8 MB, my framework would enter "slow down" mode, where all reads and writes were artificially delayed by calling "sleep()", starting at 16 ms and gradually increasing the sleep time based on a few heuristics. This allowed the application to have short gaps with no reads or writes, so the checkpointer could actually proceed.
- rsync 3mo agosqlite3_rsync exists: https://sqlite.org/rsync.html https://sqlite.org/rsync.html ... and is installed and available: # sqlite3_rsync -vv ./test.db user@rsync.net:test.db "Ordinary rsync does not understand SQLite transactions. Rsync will make a copy of ORIGIN into REPLICA, however the copy might not be consistent. Parts of the copy might be from one transaction, while other parts might be from a different transaction. The database copy might be corrupt."
- dcmatt 3mo agoWow, lobsters seems refreshing! Invite only to contribute must be why they can run on sqlite so effectively. I've run into so many issues with WAL (real time market data) unless I'm painfully deliberate with reads and writes. Don't feel like jumping on the "ohhh invite me" bandwagon, but definitely subscribing to their feed.
- synqvest 3mo ago[flagged]
- hnarn 3mo agoMoving from MariaDB to SQLite and not Postgres for a web application of this scale is insane. I wonder what went into the discussion behind this.
- HackerThemAll 3mo agoHave you ever contemplated the true performance of SQLite? Expensify reached 4M point database queries per second, and an artificial (more like parser-testing, not database-touching) 160M qps. If your backend is on a single machine of predictable performance, and you properly index your queries, and you don't require remote DB connections, you don't need anything else but SQLite for 95% of use cases. https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a-single-server https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...
- pushcx 3mo agoThe web app typically sees 10 rps and most of those don't write.
- MitziMoto 3mo agoAlthough the site is very fast for me as of this writing, it looks like they're running into an issue: https://lobste.rs/s/ko1ji1/lobste_rs_is_now_running_on_sqlite#c_gkmzsf https://lobste.rs/s/ko1ji1/lobste_rs_is_now_running_on_sqlit... I hope they get it sorted soon, I'm rooting for sqlite.
- p2hari 3mo agoEverytime Lobste.rs get posted, I get to think should I dive in. This time maybe. Could somebody be kind enough to give me an invite. Email in my profile. Thanks.
- abetusk 3mo agoInvite sent. Let me know if you didn't get it.
- wewewedxfgdf 3mo agoI would visit lobste.rs more if it filtered out stories that are on Hacker News. I'm not interested in a rehash of Hacker News but I am interested in more tech news. Anyhow on topic: I find it puzzling the whole "use sqlite in production!" thing. - you're stuck on a single machine now. - you don't have remote database access. - you have to live with the inflexibility of sqlite's schema change capability. - all writes go through a single writer. - fewer data types - fewer backup/security/high availability options - foreign keys and strict disabled by default. - you're single core in a multi core world. It's not a terrible decision but when you can choose Postgres, why would you go with sqlite?
- simonsan 3mo agoA single page answers that question: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
- Levitating 3mo agoLobste.rs seems cool. Would someone invite me?
- dzonga 3mo agomy take to benefit from SQLite: make sure you're not doing a bunch of migrations. that's the SQLite weak point that I know of & have experienced. use SQLite - as if you're using a KV or better yet maybe a CQRS approach.