5 ms·
> saved image in the DB Whoa now... you're putting images in the database?
by pkulak 5y ago
> saved image in the DB
Whoa now... you're putting images in the database?
- stavros 5y agoYes, it's a database. That's where you're meant to put DATA.
- debarshri 5y agoHow about filesystem or object storage? That where you meant to put a FILE which is what the user uploaded.
- dsr_ 5y agoIf IMGZ was a file sharing service it would be called FILZ or something. Obviously this is not that.
- noobgrammer 5y agoAnd if it were a hotel it could just be called ZZZ.
- jasode 5y ago>Yes, it's a database. That's where you're meant to put DATA. Most (all?) large-scale high-volume image storage architectures I'm aware of do not store billions of images as blobs in a database. Instead, they typically store the images as files. The database only holding metadata info and a pointer to the image filename location (or url if using AWS S3 or CDN). E.g. Facebook is one of the largest MySQL sites (if not the largest) and they don't store photos directly in MySQL.
- stavros 5y agoI'm joking, but the volume for IMGZ is way too low (and always will be) for data storage to matter.
- jamesjyu 5y agoStill low even after HN surge?
- stavros 5y agoLet me check... TWO more images were added today! At this rate, I'm going to need to buy a whole datacenter by next week. https://imgz.org/blog/2020/12/23/haha-suckers/ https://imgz.org/blog/2020/12/23/haha-suckers/
- gauravjain13 5y agoDon't knock exponential growth!
- ryan-allen 5y agoI like you (and I like storing images in the database). EDIT: I uploaded an image into your database https://imgz.org/i3joC5jC/ https://imgz.org/i3joC5jC/
- toomuchtodo 5y agoMajestic Taco.
- stavros 5y agoThat is an excellent image, worthy of gracing my database.
- matkoniecz 5y agoBTW, https://imgz.org/i9xyk2H2.png https://imgz.org/i9xyk2H2.png is broken and included prominently in https://imgz.org/blog/2020/12/23/haha-suckers/ https://imgz.org/blog/2020/12/23/haha-suckers/ Not sure is it some intentional irony or an actual bug.
- stavros 5y agohttps://imgz.org/i9xyk2H2/ https://imgz.org/i9xyk2H2/
- jrochkind1 5y agoI understand that postgres is pretty decent at storing large binary data though? I've been meaning to investigate trying storing images in postgres on a hobby web project where it could be convenient; I'd still want to make sure I was streaming bytes from postgres to the client, not loading the whole image into memory and only once fully loaded sending it to the client. Looking at the pg API's for my language/platform of choice, it looked at least plausible to do this with a BLOB.
- Arnt 5y agoPostgres is indeed good at that. I know from experience. If you want to commit ten things every second all week, don't transfer so much data in a single transaction that you hold any important locks for half a minute. And if you use replication, test while replicating. If the blobs you store are small enough to not disturb your commit rate, then pg's drawbacks are IMO smaller than those of the alternatives, particularly if you want some sort of commit that returns when both image data and something else have been committed.
- jrochkind1 5y agoThanks for info! So you've stored images in the db in production and had it work out fine? I don't want to commit ten things a second. It would be a pretty read-heavy write-light load. Sounds like that would avoid one path of riskiness at least.
- Arnt 5y agoYes, exactly. Many people answer a question like this: "How good is postgres at storing big blobs, relative to small things?" Postgres is indeed worse at storing large things. However, I think the most interesting question is something along these lines: "What's best, storing everything in Postgres and accepting that inefficiency for the big blobs, or implementing transactions, replication and/or backup with a part of the data in Postgres and the big blobs elsewhere?" I'm awfully fond of having working backups, and postgres' performance problems with the blobs haven't been big enough that I've really noticed.
- Arnt 5y agoI too do that, in a different context, and it's fine. Databases have problems with big items, but images aren't big in that sense nowadays. The problems relate mostly to backups/copies/reloads, that kind of thing, and copying an image doesn't cause anything to hiccup. A transaction that transferred 1MB over a modem line was a problem, that could block other commits for a long time, on today's networks 1MB doesn't take long.
- dijit 5y ago... depending on the type of data. Binary data in databases is usually not a great idea, but tolerable in low amounts. I can't imagine storing images in a database, you're probably better off with a document store. Relational Databases (well, the ones that I know the internals of) tend to have soft limits on row sizes, in PostgreSQL this is 8kb (which is very small for an image); to overcome this limitation Postgres uses what they call a TOAST table, which forces compression by default, which absolutely crumbles on binary data. https://www.postgresql.org/docs/13/storage-toast.html https://www.postgresql.org/docs/13/storage-toast.html But, if it works, I'm not complaining- and I'm very certain that other such sites have worse architectural flaws in the beginning- this one is pretty easy to overcome if it becomes an issue and can be alleviated with aggressive caching of CDNs/varnish to buy you time if there's immediate issues. :)
- sverhagen 5y agoIf you were to go with PostgreSQL, wouldn't you store this in a large object, which is effectively stored outside of the actual table row? So, if you can store enough rows of meta data AND you provision enough storage, why wouldn't the database be able to do this. I don't imagine you want to put a primary key on the actual image data column (not possible with large objects, since the column just contains an "oid"). I'm not arguing for storing images in the database, I've seen it done either way, and done it myself either way, but I believe that it's an ongoing source of heated debate between proponents and opponents. I.e. not settled out of hand.
- dijit 5y agoLarge objects are _mostly_ an access method, the underlying implementation is staggeringly similar to TOAST. > The large object implementation breaks large objects up into “chunks” and stores the chunks in rows in the database. A B-tree index guarantees fast searches for the correct chunk number when doing random access reads and writes. You might try Foreign Data Wrapping. In which case I still wouldn't because the database is going to be spending more time in doing round trips, better my application server does that, there's usually more of them.
- 5y ago
- deleted 5y ago[deleted]
- jagged-chisel 5y agoAdvice these days is to rely on the database to do the right thing with your uploaded files. I think they all handle this rather well.
- pkulak 5y agoThat sounds right. But even if it works great, it's gonna be the most expensive place to put your data. Those image blobs would be fine on spinning rust. But yes, there's probably time for that optimization later, if the service takes off.
- LegitShady 5y agosay hello to my friend BLOB
- emfax 5y agoDoesn't iCloud use FoundationDB for storage of files?