8 ms·
Show HN: S3mini – Tiny and fast S3-compatible client, no-deps, edge-ready
- hsbauauvhabzb 1y agoI found the words used to describe this jarring - to me it makes sense to have an s3 client on my computer, but less so client side on a webapp. On further reading, it makes sense, but highlighting what problem this package solves in the first few lines of the readme would be valuable for people like me at least
- willwade 1y agoI have a good suspicion this has been written with help from an LLM. The heavy use of emojis and strong hyper confident language is the giveaway. Proof: my own repos look like this after they’ve had the touch of cursor / windsurf etc. still doesn’t take away if the code is useful or good.
- gchamonlive 1y ago> to me it makes sense to have an s3 client on my computer, but less so client side on a webapp What do you mean with a webapp?
- neon_me 1y agohe expected to be s3 client on desktop/local machhine
- deleted 1y ago[deleted]
- gchamonlive 1y agoIt's a typescript client it seems. While you can bundle it in a webapp, typescript application goes beyond just web applications, this is why I was confused.
- neon_me 1y agotbh - english is not my mother-language so i do help myself with copy and typos ... but, if it feels uncomfy please feel free to open PR - I want it to be as reasonable as possible
- JimDabell 1y agoI think “for node and edge platforms” and “No browser support!” makes this pretty clear? Those are in the title and first paragraph.
- hsbauauvhabzb 1y agoI think if you asked the average IT person what those buzzwords mean, you’ll find the answer unclear…
- JimDabell 1y agoI was responding to this: > to me it makes sense to have an s3 client on my computer, but less so client side on a webapp The relevant audience in this situation is not the average IT person, but a person who might mistake this for client-side web app functionality. If you think that something might run in the browser, then “no browser support!” is not complicated jargon that you won’t understand.
- dev_l1x_be 1y agofor Node. These are nice projects. I had a few rounds with Rust S3 libraries and having a simple low or no dep client is much needed. The problem is that you start to support certain features (async, http2, etc.) and your nice nodep project is starting to grow.
- terhechte 1y agoI had the same issue recently and used https://crates.io/crates/rusty-s3 https://crates.io/crates/rusty-s3
- maxmcd 1y agoalso: https://crates.io/crates/object_store https://crates.io/crates/object_store
- pier25 1y agofor JS > It runs on Node, Bun, Cloudflare Workers, and other edge platforms
- everfrustrated 1y agoPresumably smaller and quicker because it's not doing any checksumming
- neon_me 1y agodoes it make sense or should that be optional?
- tom1337 1y agochecksumming does make sense because it ensures that the file you've transferred is complete and what was expected. if the checksum of the file you've downloaded differs from the server gave you, you should not process the file further and throw an error (worst case would probably be a man in the middle attack, not so worse cases being packet loss i guess)
- neon_me 1y agoyes, you are right! On the other hand S3 uses checksums only to verify expected upload (on the write from client -> server) ... and suprisingly you can do that in paralel after the upload - by checking the MD5 hash of blob to ETag (*with some caveats)
- supriyo-biswas 1y ago> checksumming does make sense because it ensures that the file you've transferred is complete and what was expected. TCP has a checksum for packet loss, and TLS protects against MITM. I've always found this aspect of S3's design questionable. Sending both a content-md5 AND a x-amz-content-sha256 header and taking up gobs of compute in the process, sheesh... It's also part of the reason why running minio in its single node single drive mode is a resource hog.
- dboreham 1y agoWell known (apparently not?) that applications can't rely on TCP checksums.
- nodesocket 1y agoSomewhat related, I just came across s5cmd[1] which is mainly focused on performance and fast upload/download and sync of s3 buckets. > 32x faster than s3cmd and 12x faster than aws-cli. For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd and aws-cli can only reach 85 MB/s and 375 MB/s respectively. [1] https://github.com/peak/s5cmd https://github.com/peak/s5cmd
- rsync 1y agos5cmd is built into the rsync.net platform. See: https://news.ycombinator.com/item?id=44248372 https://news.ycombinator.com/item?id=44248372
- uncircle 1y agoI prefer s5cmd as well because it has a better CLI interface than s3cmd, especially if you need to talk with non-AWS S3-compatible servers. It does few things and does them well, whereas s3cmd is a tool with a billion options, configuration files, badly documented env variables, and its default mode of operation assumes you are talking with AWS.
- bradknowles 1y agoFor the time I worked at AWS, pretty much everyone inside the company used s5cmd for its speed. I think that speaks pretty highly of it.
- tommoor 1y agoInteresting project, though it's a little amusing that you announced this before actually confirming it works with AWS?
- neon_me 1y agoPersonally, I don't like AWS that much. I tried to set it up, but found it "terribly tedious" and drop the idea and instead focus on other platforms. Right now, I am testing/configuring Ceph ... but its open-source! Every talented weirdo with free time is welcomed to contribute!
- leansensei 1y agoAlso try out Garage.
- zikani_03 1y agoGood to see this mentioned. We are considering running it for some things internally, along with Harbor. The fact that the resource footprint is advertised as small enough is compelling. What's your experience running it?
- yard2010 1y agoTangibly related: Bun has a built-in S3-compatible client. Bun is a gift, if you're using npm consider making the switch.
- neon_me 1y agois there a way to wrap their s3 client for use in HonoJS/CF workers?
- oakesm9 1y agoNo. It's implemented in native code (Zig) inside bun itself and just exposed to developers as a JavaScript API. Source code: https://github.com/oven-sh/bun/tree/6ebad50543bf2c4107d4b4c2ecc062a15a94b71e/src/s3 https://github.com/oven-sh/bun/tree/6ebad50543bf2c4107d4b4c2...
- neon_me 1y ago10/10 Loving it (and how fast it is!) - its just not the use-case that fits my needs. I want maximum ability to "move" my projects among services/vendors/providers
- throawayonthe 1y agoi assume you can just use it[0] in your project and then build (using bun) for cf workers[1] [0] https://bun.sh/docs/api/s3 https://bun.sh/docs/api/s3 [1] https://hono.dev/docs/getting-started/cloudflare-workers https://hono.dev/docs/getting-started/cloudflare-workers \ (https://bun.sh/docs/api/workers https://bun.sh/docs/api/workers ?)
- ChocolateGod 1y agoI tried to go this route of using Bun for everything (Bun.serve, Bun.s3 etc), but was forced back to switch back to NodeJS proper and Express/aws-sdk due to Bun not fully implementing Nodes APIs.
- biorach 1y ago
- akouri 1y agoThis is awesome! Been waiting for something like this to replace the bloated SDK Amazon provides. Important question— is there a pathway to getting signed URLs?
- neon_me 1y agoFor now, unfortunately, no - no signed URLs are supported. It wasn't my focus (use case), but if you find a simple/minimalistic way to implement it, I can help you with that to integrate it. From my helicopter perspective, it adds extra complexity and size, which could maybe be ideal for a separate fork/project?
- mannyv 1y agoSigned URLs are great because it allows you to allow third parties access to a file without them having to authenticate against AWS. Our primary use case is browser-based uploads. You don't want people uploading anything and everything, like the wordpress upload folder. And it's timed, so you don't have to worry about someone recycling the URL.
- jmogly 1y agoI use presigned urls as part of a federation layer on top of an s3 bucket. Users make authenticated requests to my api which checks their permissions (if they have access to read/write to the specified slice of the s3 bucket), my api sends a presigned url back to allow read/write/delete to that specific portion of the bucket.
- ecshafer 1y agoYou can just use s3 vis rest calls if you dont like their sdk.
- nikeee 1y agoI've built an S3 client with similar goals like TFA, but supports pre-signing: https://github.com/nikeee/lean-s3 https://github.com/nikeee/lean-s3 Pre-signing is about 30 times faster than the AWS SDK and is not async. You can read about why it looks like it does here: https://github.com/nikeee/lean-s3/blob/main/DESIGN_DECISIONS.md https://github.com/nikeee/lean-s3/blob/main/DESIGN_DECISIONS...
- shortformblog 1y agoThis is good to have. A few months ago I was testing a S3 alternative but running into issues getting it to work. Turned out it was because AWS made changes to the tool that had the effect of blocking non-first-party clients. Just sheer chance on my end, but I imagine that was infuriating for folks who have to rely on that client. There is an obvious need for a compatible client like this that AWS doesn’t manage.
- kinton 1y ago[dead]
- _1 1y agoSame as this https://github.com/minio/minio https://github.com/minio/minio ?
- carlio 1y agominio is an S3-compatable object store, the linked s3mini is just a client for s3-compatable stores.
- arbll 1y agoNo this is an S3-compatible client, minio is an S3-compatible backend
- prmoustache 1y agoThe minio project provides both.
- deleted 1y ago[deleted]
- EGreg 1y agoYou know what would be really awesome? Making a fuse-based drop-in replacement for mapping a folder to a bucket, like goofys. Maybe a node.js process can watch files for instance and backup, or even better it can back the folder and not actually take up space on the local machine (except for a cache). https://github.com/kahing/goofys https://github.com/kahing/goofys
- arbll 1y agoThis seem completely unrelated to the goal of OP's library ?
- EGreg 1y agoIt seems to be related to what a lot of people want and is low hanging fruit now that he has this library!
- TuningYourCode 1y agoYou mean like https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse ? It‘s so old that even debian has precompiled packages ;)
- EGreg 1y agoI was talking about goofys because it is not POSIX compliant, so it's much faster than s3fs-fuse But either one can only work with s3. His library works with many other backends. Get it? I'm saying he should consider integrating with goofys!
- cosmotic 1y ago> https://raw.githubusercontent.com/good-lly/s3mini/dev/performance-screenshot.png https://raw.githubusercontent.com/good-lly/s3mini/dev/perfor... It gets slower as the instance gets faster? I'm looking at ops/sec and time/op. How am I misreading this?
- xrendan 1y agoI read that as the size of file it's transferring so each operation would be bigger and therefore slower
- math-ias 1y agoIt measures PutObject[0] performance across different object sizes (1, 8, 100MiB)[1]. Seems to be an odd screenshot of text in the terminal. [0] https://github.com/good-lly/s3mini/blob/30a751cc866855f783a1f50ccff04c451c0dc058/tests/perf/performance.test.js#L133 https://github.com/good-lly/s3mini/blob/30a751cc866855f783a1... [1] https://github.com/good-lly/s3mini/blob/30a751cc866855f783a1f50ccff04c451c0dc058/tests/perf/performance.test.js#L35-L37 https://github.com/good-lly/s3mini/blob/30a751cc866855f783a1...
- cosmotic 1y agoOh, I see my mistake. Those are payload sizes not intance sizes in the heading for each table.
- arianvanp 1y agolibcurl also has AWS auth with --aws-sigv4 which gives you a fully compatible S3 cliënt without installing anything! (You probably already have curl installed)
- busymom0 1y agoDoes this allow generating signed URLs for uploads with size limit and name check?
- brendanashworth 1y agoHow does this compare to obstore? [1] [1] https://developmentseed.org/obstore/latest/ https://developmentseed.org/obstore/latest/
- glub103011 1y ago[dead]
- linotype 1y agoThis looks slick. What I would also love to see is a simple, single binary S3 server alternative to Minio. Maybe a small built in UI similar to DuckDB UI.
- koito17 1y ago> What I would also love to see is a simple, single binary S3 server alternative to Minio Garage[1] lacks a web UI but I believe it meets your requirements. It's an S3 implementation that compiles to a single static binary, and it's specifically designed for use cases where nodes do not necessarily have identical hardware (i.e. different CPUs, different RAM, different storage sizes, etc.). Overall, Garage is my go-to solution for object storage at "home server scale" and for quickly setting up a real S3 server. There seems to be an unofficial Web UI[2] for Garage, but you're no longer running a single binary if you use this. Not as convenient as a built-in web UI. [1] https://garagehq.deuxfleurs.fr/ https://garagehq.deuxfleurs.fr/ [2] https://github.com/khairul169/garage-webui https://github.com/khairul169/garage-webui
- dzonga 1y agothis looks dope. but has anyone done a price comparison of edge-computing vs say your boring hetzner vps ?