10 ms·
The Winamp Skin Museum is powered by a SQLite3 database with 1.2GB of metadata
- keewee7 4y agoI remember reading about a company that upgraded their production database from a JSON file to SQLite. We really need this counter culture in software development that emphasis simplicity over the "Start with Kafka-on-k8s" madness.
- inportb 4y agoTailscale https://tailscale.com/blog/database-for-2022/ https://tailscale.com/blog/database-for-2022/
- Xeoncross 4y agoBut tailscale is a distributed, multi-region SaaS provider. It's easy for them to use SQLite. What about a real-world workload? For example, I have 10 users a day on my new next.js app so I clearly need a RDS cluster for burst traffic.
- bigdubs 4y agoTailscale has some server components (account management etc.) that are powered with SQLite. Can read more here: https://tailscale.com/blog/database-for-2022/ https://tailscale.com/blog/database-for-2022/
- deleted 4y ago[deleted]
- batch12 4y agoOne of my websites has been using SQLite3 as a backend database since 2018ish and works perfectly fine.
- lillecarl 4y agoI mean, aws makes it quite easy to run postgres so I don't see why not either, you don't need a dba more or less than with SQLite + rqlite. If you wanna run onprem you can usually do with one container for quite awhile too. I wouldn't call postgres a "complex dependency" anymore.
- samb1729 4y agoThe existing replies to this comment of yours make it apparent why “/s” is still necessary :)
- Illniyar 4y agoI find it hard to believe that someone would mistake the comment as not sarcastic if they read the second paragraph. So if people comment based only on the first paragraph, would they even get to the /s in the end ?
- bazmattaz 4y agoI thought that SQLlite databases are not suitable for large multi user websites. Something got to do with only being able to handles one transaction at a time. Isn’t that right?
- teirce 4y agoI think this is kind of their point. If you're starting out, you may not need to meet the same considerations as a "large multi user website". So you don't get stuck trying to do that - you start with what you need (appropriately) and adjust down the line. It is hard to do right.
- Gigachad 4y agoThe problem is you have SQLite which works some of the time and Postgres which works all of the time. Postgres is a great option for websites of all sizes. There is vanishingly few reasons to ever pick SQLite for a website. When the only real advantage is not having to run a second process which only really matters for local apps.
- themerone 4y agoSQLlite can do handle concurrent reads, writes are a problem. A site like this which is mostly a read-only archive is a perfect use case for SQLite.
- colechristensen 4y agoSure, but you can have raised millions with large numbers of customers before this is an issue. Having multiple customers using your platform at the same time to the extent that their transactions actually end up having to wait a meaningful amount of time requires a lot of usage that you often don’t reach for a long long time.
- derefr 4y agoOr just having any long OLAP queries mixed into your OLTP workload. A readers-writer lock can often end up with tons of writers waiting on one last long read to drain.
- vsroy 4y agoContext: I see a bunch of people here recommending SQLite. I have a suggestion, try out LMDB. It's kind of like noSQL SQLite. It's a simple in-process K/V store with a few features like compound keys that allow you to model relational data well enough. I recently used lmdb for webhighlighter.com (specifically the wrapper: https://www.npmjs.com/package/node-lmdb https://www.npmjs.com/package/node-lmdb), and it was a fantastic decision. A lot of people here say "use SQLite for small projects". But even using SQLite can be significant over-engineering. Running migrations? Writing SQL? That's too much effort for me. For example: In my application, people can leave comments on a (what is effectively) a post. A SQL-native solution might have a table for comments, with foreign keys to post IDs. That's 10x more engineering then I want to do for an MVP. I just store all the comments as an array on a post. This means reads read all the comments and writes require reading all the comments, appending, and then re-writing. That's totally fine, and will probably scale me to 100x my current traffic. LMDB-JS is great. It allows you to serialize arbitrary JS objects to LMDB using the message pack encoding system. This makes for some super concise code. Here's my entire data layer: - Interface: https://github.com/vedantroy/grape-juice/blob/main/site/app/db/types.server.ts https://github.com/vedantroy/grape-juice/blob/main/site/app/... - Implementation: https://github.com/vedantroy/grape-juice/blob/main/site/app/db/lmdb.server.ts https://github.com/vedantroy/grape-juice/blob/main/site/app/... TL;DR -- I won't use SQLite, for, I don't know, my first 10K users?
- pfraze 4y agoFlashback to when leveldb was hot in the node ecosystem KV stores are fun but you end up writing a lot of queries and indexes manually
- ascar 4y ago> But even using SQLite can be significant over-engineering. Running migrations? Writing SQL? That's too much effort for me. Key-value stores are not a replacement for SQL, they solve a simpler problem. If you actually can do with a key value store I would argue a SQL table with two columns key and value with key being a primary key will do just fine. And using it needs simple select and 'insert on duplicate key update' statements. The effort for migrations won't be higher than whatever you had to configure for your key-value store. However, if you can't actually do with a key-value store you will end up writing a lot of the features SQL provides in application code. Indices, joins, grouping, ordering, etc. You might not notice it at first, but you will blow up complexity in your business logic reinventing existing SQL features and chances are high you're doing it worse and less performant than what e.g. Postresql offers out of the box. SQL is just such a powerful tool that is at the same time incredibly easy to use for simple scenarios. Added benefit of using SQLite, you can migrate to a more powerful SQL database later without having to reengineer your whole data layer.
- ukyrgf 4y agoIt is very cool that this is archived. I was the application skin admin over at deviantArt and Deskmod way back in the day (wild times, pre-Web 2.0 so everybody was just making it up as they go) but sadly none of these names ring a bell. I think Deskmod just piddled out and dA got bought and the original crew all left. I lived in the middle of nowhere so being exposed to that level of programming as well as passion for art was extremely inspiring. I even made a punk music spinoff called I Hate Music that introduced me to lifelong friends all over the world.
- duxup 4y agoAny reason / use for all that metadata?
- captbaritone 4y agoContains all the text content extracted from each skin. Readme text, config files, as well as metadata about how the skins performed on social media (likes/retweets from the @winampskins Twitter bot) and also NSFW stats (as reported by users and confirmed by me) and if the skin is approved by users in Discord to be tweeted by the bot. All of these are used for search (text is indexed into Algolia) and ranking: Most likes/retweets are shown first, then approved, then rejected. NSFW skins are deprioritized as well.
- captbaritone 4y agoExtracted config files are also useful for quickly testing/validating the various parsers that I had to write as part of the Webamp project.
- walrus01 4y agohow many llamas and whips can you fit in 1.2GB?
- 8n4vidtmkvmk 4y agoall of them apparently
- hackernewds 4y agoit really does kick the llama's *ss
- sedatk 4y agoI love the webamp website. But, in the world of multigig RAM and multigbps I/O speeds, 1.2GB isn't that much. Not sure if that's something novel for SQLite3 though.
- walrus01 4y agounlike a webserver setup maybe 20 years ago now it's trivially easy to hold even a 30GB database entirely in RAM and never touch disk, for a vast performance increase. I think I saw a used x86-64 server with 512GB of RAM for sale for $1700 recently.
- metadat 4y ago> I think I saw a used x86-64 server with 512GB of RAM for sale for $1700 recently. Where? Sounds like a terrific bargain, my old supermicro only has 384GB.
- walrus01 4y agoebay, was a dell r630 with two somewhat older 14-core xeons, 28 cores
- tiernano 4y agoYou could pick up an R620 with 768GB Ram and 2 8 core Xeons for around 1400GBP (https://www.bargainhardware.co.uk/dell-poweredge-r620-10-sff-bays-configure-to-order https://www.bargainhardware.co.uk/dell-poweredge-r620-10-sff...) without disks... DDR4 is a bit more expensive... but DDR3 is still fast for a lot of jobs...
- walrus01 4y agoR620 is a problem for anything that needs high bandwidth networking, however, because the pci-e slots are only 2.0. ruling out the use of many 10GbE interfaces or things like two port 100GbE
- midislack 4y ago
- tuxie_ 4y agoI don't have a Twitter account, so I can only read so much until Twitter hides the content behind a login/signup screen. Maybe it's just me but I find it ironic that a writeup about preserving content is given away to a corporation that blocks it in this manner.
- captbaritone 4y agoTweet author here (not op). I'll repost here: The Winamp Skin Museum is powered by a sqlite3 database containing 1.2gb of metadata about 86,000 Winamp skins. It's all exposed in this explorable GraphQL endpoint https://api.webamp.org/graphql https://api.webamp.org/graphql A bit about the data... It includes: * Original filenames and md5 hashes of each skins * Names/metadata of all files compressed WITHIN the skins (file size, date, filename) * Text content of all text files found within the skins * URL/likes/retweets if the skin was share by @winampskins (or on Instagram) * Full metadata/info about each skin's @internetarchive page * Info about manual reviews (good to tweet? NSFW?) * URLs to download skin files or screenshots Kind [of] fun data to comb though (if you're like me). If anyone is interested in getting the raw DB to play with, or has ideas for extra stuff to expose in the graph, get in touch.
- deleted 4y ago[deleted]
- tuxie_ 4y agoThanks! That's really nice of you :)
- abofh 4y agoI want to know more about the bowling pin man - I mean, there's probably not much more to be said about him, but he raises so many questions!
- captbaritone 4y agoMe too!
- 4y ago
- captn3m0 4y agoI have a lot of smaller datasets I am hoping to expose similarly once CloudFlare D1 becomes available. Fly.io does a decent job, but hoping it gets cheaper/simpler with D1.
- simonw 4y agoFly.io will give you an always-on 256MB of RAM instance for $1.94/month - and they offer 2,340 hours per month in their free tier, which is enough to run three of those for the whole month without spending any money at all.
- password4321 4y agoIf we're spamming free DB's I always mention Supabase, 2*500MB PostgreSQL.
- DustinBrett 4y agoWebamp is quite amazing. I use it on my desktop environment website. I recently added playlist support (m3u/pls/asx) and integrated lazy(er) Butterchurn so it can run as the wallpaper. Winamp/Milkdrop forever! https://dustinbrett.com/ https://dustinbrett.com/
- nighthawk454 4y agothis is insanely cool! it's hard to imagine wasm stuff has come this far
- captbaritone 4y agoSpeaking specifically about Webamp, it is a manual reimplementation in JavaScript. But we did use Wasm as a tool for the music visualizer: https://jordaneldredge.com/blog/speeding-up-winamps-music-visualizer-with-webassembly/ https://jordaneldredge.com/blog/speeding-up-winamps-music-vi...
- nighthawk454 4y agothat's awesome, I'll check it out. hopefully the WebAudioAPI has gotten better to work with over the years
- captbaritone 4y agoThe web audio api code is mostly in this directory: https://github.com/captbaritone/webamp/tree/master/packages/webamp/js/media https://github.com/captbaritone/webamp/tree/master/packages/...
- DustinBrett 4y agoAmazing stuff! Especially the https://github.com/captbaritone/eel-wasm https://github.com/captbaritone/eel-wasm part. It's great to be able to salvage all this internet history and keep it alive and going.
- airstrike 4y ago
- TheChaplain 4y agoCool, so much good memories. There's a bug in webamp though, choosing "Options -> Double Size" only enlarges the player and equalizer. The playlist remains the same size.
- ChickeNES 4y agoI don't believe that's a bug, that's how Winamp behaved
- captbaritone 4y agoCorrect
- midislack 4y agoBe so much better if it was just normal files on disk.
- CharlesW 4y agoSQLite files are normal files. It's a "Recommended Storage Format" by the U.S. Library of Congress. https://www.sqlite.org/locrsf.html https://www.sqlite.org/locrsf.html
- Vladimof 4y agoI should go back to Winamp if its still the same (best music player I ever used)... I used to to make it small and always on top and drag it on my current window's title bar next to the minimize button...
- b0ner_t0ner 4y agoI use WACUP, no issues at all.
- deleted 4y ago[deleted]
- johtso 4y agoSeems like a great use case for static sqlite hosting https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs
- zelphirkalt 4y agoIs there such a thing as a Winamp VS Aimp2 war? Once I discovered Aimp back then, when I still used Windows on a daily basis, I never switched back to Winamp. Aimp seemed so much more refined and organized. Wondering, if most people, who reminisce about Winamp, even know about Aimp.
- lysecret 4y agoThe love affair of Hackernews and SQLite continues ;)
- mekster 4y agoThought it was TB but then was just GB.
- programmerdad 4y agoI have built a graph-based knowledge management system (https://github.com/brettkromkamp/contextualise https://github.com/brettkromkamp/contextualise) on top of SQLite. It runs great. Also, from a management point of view (e.g., deployments, backups) its ease of use is second to none. I migrated the application from PostgreSQL (which is also a great RDBMS) to SQLite and I haven’t looked back.
- joshspankit 4y agoI’d love to have this kind of database for everything! Music, themes, packages, applications. All of it! Great work