19 ms·
If anyone is interested in the technical details, the database itself is a 4GB SQLite file which we are hosting with Datasette running on Fly. More details in
by simonw 4y ago
If anyone is interested in the technical details, the database itself is a 4GB SQLite file which we are hosting with Datasette running on Fly.
More details in our repo: https://github.com/simonw/laion-aesthetic-datasette https://github.com/simonw/laion-aesthetic-datasette
Search is provided by SQLite FTS5.
- sdwr 4y agoThe search speed is amazing!! Do you have to do a lot of pre-indexing to get it so fast?
- simonw 4y agoIt's SQLite's built in FTS index, nothing special on top of it. I built the index by running: sqlite-utils enable-fts data.db images text https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search https://sqlite-utils.datasette.io/en/stable/cli.html#configu...
- sdwr 4y agoAre you running on anything special compute-wise? I have a budget node running Mongo, it takes almost a second to fetch a single 1MB document. Writing it out I realize it's not indexed by the attribute I'm retrieving by...
- isoprophlex 4y agoUse the index, Luke!
- snovv_crash 4y agoBut MongoDB is webscale!
- simonw 4y agoI started this on a Fly instance with 256MB of RAM and a shared CPU. This worked great when it was just a couple of people testing it. Once it started getting traffic it started running a bit slow, so I bumped it up to a 2 CPU instance with 4GB of RAM and it's been fine since then. The database file is nearly 4GB and almost all memory is being used, so I guess it all got loaded into RAM by SQLite. I'll scale it back down again in a few days time, once interest in it wanes a bit.
- yojo 4y agoI notice a surprising number of duplicates. E.g. if I sort by aesthetic, there’s the same 500x500 Tuscan village painting multiple times on the first page of results. Presumably it wouldn’t be so hard to hash the images and filter out repeats. Is the idea to keep the duplicates to preserve the description mappings?
- bambax 4y agoI noticed this too, with the same description every time. How does this work in the model, does this give repeated images a bigger weight? It's surprising that these weren't filtered out, and it would be interesting to know the number of unique images. (When it is mentioned that a model was trained on 10 billion images for example, obviously if each image is repeated 5 times then the actual number of images is 2 billions, not 10.)
- jadbox 4y agoIs there plans to expand the model to be even larger?