8 ms·
Spiral
- deleted 1y ago[deleted]
- all2 1y agoSpelling error "sttill" > P.S. If you're sttill managing data in spreadsheets, this post isn't for you. Yet. --- Since I discovered the ECS pattern, I've been curious about backing it with a database. One of the big issues seems to be IO on the database side. I wonder if Spiral might solve this issue.
- lordnacho 1y agoIf the ECS data is grid-like, perhaps you could use a columnar database for time series? Then you could save every single state change and scroll back and forth. But I'm not sure if you were looking for that.
- all2 1y agoThis would make netcode state rollback easy. But I'm not sure the DB is the right place for that.
- harwoodr 1y agoHave a look at something like spacetimeDB - caveat, I've only read about it and not directly used it: https://github.com/ClockworkLabs/SpacetimeDB https://github.com/ClockworkLabs/SpacetimeDB
- all2 1y agoThis was part of my inspiration. They do some interesting things with their systems, like hot reloads, that makes the dev loop a lot tighter.
- SomeHacker44 1y ago"100KiB images"... This is odd. Most of my images are 2.5-4 MB. My raw images are 3-10x larger.
- turnsout 1y agoI bet this refers to some common training use case that leverages 512px or 1024px images. Or it’s just Palantir scanning security camera frames.
- pauldix 1y agoI've been following this team's work for a while and what they're doing is super interesting. The file format they created and put into the LF, Vortex, is very welcome innovation in the space: https://github.com/vortex-data/vortex https://github.com/vortex-data/vortex I'm excited to start doing some experimentation with Vortex to see how it can improve our products. Great stuff, congrats to Will and team!
- dist-epoch 1y agohttps://vortex.dev https://vortex.dev doesn't work in my Firefox: Application error: a client-side exception has occurred while loading vortex.dev (see the browser console for more information). Console: unable to create webgl context
- arusahni 1y agoWorks for me. Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0
- miloignis 1y agoPresumably you don't have WebGL enabled or supported - the main page is just a cute 3D landing page. You may be interested in https://github.com/vortex-data/vortex https://github.com/vortex-data/vortex which of course has an overview and links to their docs and benchmark pages.
- brunohaid 1y agoIf anyone ever writes a post of why that error keeps happening with browsers that should support it, I'd be incredibly grateful. Keep seeing it in our (unrelated to OP company) Sentry logs and zero chance to reproduce them.
- bflesch 1y agoI assume it's just people who do not have a graphics card
- 1y ago
- deleted 1y ago[deleted]
- djfobbz 1y agoSo this Vortex engine is a combination of OLTP and OLAP on steroids?
- reactordev 1y agoAnyone that can improve upon the parquet hell that is my life is gladly welcomed...
- riku_iki 1y agowhy you don't like parquet?
- indoordin0saur 1y agoParquet seems easy and straight-forward. The only issue I see people having with it is if they aren't used to non-human-readable formats and have to use special tools to look at it (as opposed to something like CSV). In that case this new file format will absolutely be worse.
- reactordev 1y agoNot my issue at all. My issue is someone dumping 4gb of data into a parquet file thinking it’s fine…
- riku_iki 1y agoI operate xxxGB files. What do you think is wrong with this?..
- indoordin0saur 1y agoIs there a better file format for that?
- paxys 1y agoWasn't "3.0" supposed to be crypto? Is it AI now? It's had to keep track.
- ionwake 1y agoI think AI is 4.0 EDIT> Maybe its how some poeple call the 4th dimension time when there is infact a 4th spatial dimension. So I guess if this is the 3rd Data dimension like what is the 4th one?
- jppope 1y agoI think some of the crypto companies tried to get cute and leapfrog 3.0 going straight to 4.0, so that would put us at either 5.0, 4.0, 3.1, 2.2, or 2.1 depending on how you feel about the crypto space, and which groups you were validating
- bee_rider 1y agoNo, Web 3.0 was the Semantic Web. Thankfully, the silly idea of having major-number versions for the entire internet died when that it happen. Now we can safely ignore anybody who tries to do it.
- adfm 1y agoYou’re conflating concepts. FWIW, Web3 is snake oil or wishful thinking at best. As much as people like to bang on the old Web 2.0, it still holds up conceptually. And if you only know it as a buzz word, I suggest you go back and familiarize yourself with it if you’re looking for incremental change. Who knows, maybe a Web 3.1 will deliver us from Enshitification.
- stronglikedan 1y agoI think we're in a new era, so I consider this version of the web to be "AAI 1", and next year it will be "AAI 2", and so on. This era will be hereafter referred to as "in the year of the AI overlord", or "Anno Domini Artificialis Intellegentiae Artificialis" (according to google translate).
- holoduke 1y agoSo basically this is a file system that runs on your gpu?
- cryptonector 1y agoI can't tell what this is about.
- dkdbejwi383 1y agoDo you remember the days of “mongodb is web-scale”? It’s that but “spiral is ai-scale”
- nwhnwh 1y agoSo it will be irrelevant after a few years?
- zzzeek 1y agomaybe just a few months, AI scale is much faster than web scale of course
- steve_adams_86 1y agoMongo is still very relevant For better or worse
- znort_ 1y ago"I've been building data systems for long enough to be skeptical of “revolutionary” claims, and I’m uncomfortable with grandiose statements like “Built for the AI Era”. Nevertheless, ... ... i'm gonna make revolutionary claims and grandiose statements like "built for the ai era".
- bee_rider 1y agoProbably either overcoming giant robots with the power of friendship and a giant drill, or a cursed village with an obsession-inducing whirlpool.
- riku_iki 1y agomy reading that it will be some hyper-performant db thanks to some very low level optimization utilizing recent hw advancements and formats/pipelines unification and simplification.
- 4ndrewl 1y agoThe three eras of database systems starts with a client-server Postgres, but missed the daddy of the generation before that - xBase (ie dBase, FoxPro etc).
- khaledh 1y agoIt goes way before that. It starts with IDS (Integrated Data Store) from GE (1964), which was a network database system. Next was IBM's hierarchical database system IMS (Information Management System, 1966), still in use today. Then the CODASYL model (late 1960s), which was an effort to standardize the network model. And then Codd came up with the relational model in the early 70s, upon which an explosion of database systems were built (first is IBM System R, SQL, Oracle, DB2, Ingres). Then came the PC-based database systems you mentioned.
- 4ndrewl 1y agoOh for sure. To suggest we're only on generation 3 of "databases" is way off the mark.
- spankalee 1y agoI'm curious... I'm not a database or AI engineer. The last time I did GPU work was over a decade ago. What is the point of the "saturate an H100" metric? I would think that a GPU isn't just sitting there waiting on a process that's in turn waiting for one query to finish to start the next query, but that a bunch of parallel queries and scans would be running, fed from many DB and object store servers, keeping the GPUs as utilized as possible. Given how expensive GPUs are, it would seem like a good trade to buy more servers to keep them fed, even if you do want to make the servers and DB/object store reads faster.
- vouwfietsman 1y agoMy guess is that just the raw data size, combined with the physical limitations of your RU, makes it hard for the GPU to be fully utilized. Instead you will always be stuck on CPU (decompressing/interpreting/uploading parquet) or bandwidth (transfer from s3) being the bottleneck. Seems that they are targeting a low-to-no overhead path from s3 bucket to GPU, by targeting: same compression/faster random access, streamed encoding from S3 while in flight, zero copy to GPU. Not 100% clear on the details, but I doubt that they can actually saturate the cpu/gpu bus, but rather just saturate the GPU utilization, which is itself dependent on multiple possible bottlenecks but generally not on bus bandwidth. That's not criticism: it literally means you can't do better unless you improve the GPU utilization of your AI model.
- otterley 1y agoThe idea is that in a pipeline of work, throughput is limited by the slowest component. H100 GPUs have a lot of memory bandwidth. The question then becomes how to eliminate any bottlenecks between the data store and the GPU's memory. First is the storage bottleneck. Network-attached storage is usually a bottleneck for uncached data. Then there is CPU work decoding data. Spiral claims that their table format is ready to load by the GPU so they can bypass various CPU-bound decoding stages. Once you eliminate storage and CPU bottlenecks, the remaining bottleneck is usually the PCI bus that sits between the host memory and the GPU, and they can't solve that themselves. (And no amount of parallelization can help when the bus is saturated.) What they can do is use the network, the host bus, and the GPU more efficiently by compressing and packing data with greater mechanical sympathy. They've left unanswered how they're going to commercialize it, but my guess is that they're going to use a proprietary fork of Vortex that provides extra performance or features, or perhaps they'll offer commercial services or integrations that make it easier to use. The open-source release gives its customers a Reason to Believe, in marketing parlance.
- curtisszmania 1y ago[dead]
- deleted 1y ago[deleted]
- zzzeek 1y agoThis links to a super long winded blog post that sounds more like a toast at a wedding, so I went to the main page to try to see what their product is, and you just get a blitz of fancy animations of table diagrams and things and lots of very cheap sounding slogans pushed out like "Works with any data! Fully XYZ 2.0 compliant! Ties your shoes!" basically im not sure where the product is hiding under all of this bluster but this doesnt feel very "hacker"-Y
- vouwfietsman 1y agoAlthough I welcome a parquet successor, I am not particularly interested in a more complicated format. Random access time improvements are nice, but really what I would like just storing multiple tables in a single parquet file. When I read "possible extension through embedded wasm encoders" I can already imagine the c++ linker hell required to get this thing included in my project. I also don't think a lot of people need "ai scale".
- alfalfasprout 1y agoalso what does "ai scale" even mean?
- aakkaakk 1y agoIt’s obvious a jab at mongo’s ”web scale”. https://youtube.com/watch?v=b2F-DItXtZs https://youtube.com/watch?v=b2F-DItXtZs
- vouwfietsman 1y agoI think its a bit markety, but they explain it rather well: because of AI your data needs to be consumed by machines on an unprecedented scale, which requires new solutions to problems. Historically we mostly did large input -> small output, now we're doing large input -> large output. The existing tools are (supposedly) not ready.
- alfalfasprout 1y agono, I read that. It doesn't really add any more practical detail.
- drdaeman 1y agoStoring multiple tables in a single file would be trivially solvable by storing multiple Parquet files in a most basic plain uncompressed tarball (to retain ability to access any part of any file without downloading the whole thing). Or maybe ar or cpio - tar has too many features (such as support for links) that are unnecessary here. Basically, anything well-standardized that implements a very basic directory structure, with a simple index located at a predictable offset. If any tools would've supported that.
- mellosouls 1y agoThis is a pretty website but doesn't actually give us anything to actually look at, its just blurb. For anybody confused, the "Vortex" stuff is the underlying data format used but isn't the database/whatever this website (by the creators of Vortex) is pushing.
- kmoser 1y ago> Spiral is our database built on Vortex [...] No surprise there's nothing to look at, since it's basically a press release posted on their blog.
- mlhpdx 1y agoI stopped reading at “new era”. At this point in time with the deluge of content, start with a problem and solution in a concise statement if you want my attention. I’m not reading your opinion piece.
- bflesch 1y agoBig ick from my side. Manifest-style marketing blog post talking about revolutionary things but it seems their main metric is in the image above the post: "hey, we've raised $22M in funding". Landing pages of both spiral and vortex are GPU-hugging animations and void of any technical information. Empty nothing-statements like "machine scale". They claim 100x improvements but don't link any metrics. Maybe this is a "don't hate the player, hate the game" situation but somehow the collective of likeminded AI engineers decided to upvote this post to #1 on HN.
- indoordin0saur 1y ago> Vortex is designed to support decoding data directly from S3 to GPU, skipping the CPU bottleneck entirely. If this is true I'm inclined to believe their claims.
- bflesch 1y agoMY PERSONAL BOTTLENECK between S3 and GPU is my credit card and not some new cargo module by some already-rich AI engineer and a fancy marketing website that must've cost a couple hundred grand. And if this module provides a benefit I'm sure it will find its way into our stack, just like PostgreSQL did. And PostgreSQL never had $22M to begin with - no shiny marketing, just technological skills. The whole "donated by spiral" on the vortex.dev website also gives big tax write-off vibes. IMO best case is that this will be a mongodb scenario, but with the current track record of tech grifters enshittifying everything they might find a creative new way.
- xyzzy_plugh 1y ago> The whole "donated by spiral" on the vortex.dev website also gives big tax write-off vibes. I've never heard of this sort of OSS work being used as a tax write-off. Could someone please either clarify, or enlighten me?
- bflesch 1y agoLook at the website linked in the vortex website footer: https://lfprojects.org/ https://lfprojects.org/ It has all the bells and whistles of using an expensive law firm. I have no idea who exactly is behind this, but to me it does definitely not seem like a no-name open source genius, I assume it is some lucky AI grifter. They have two nicely designed, expensive marketing websites. They have all the legal documents for the parent LLC in Delaware. The delaware corp "donates" the multi-million-worth tech to linux foundation, and uses it as tax write-off to offset gains from some other lucky AI grifter play the person did. Just the chuzpe to self-compare yourself to something like PostgreSQL is what gets me. Why can't they just be rich and leave people doing actual work for the benefit of our common good be. No, they must make big blog posts claiming they are the next big thing after PostgreSQL. So many red flags..
- raziel2p 1y ago> Vortex is designed to support decoding data directly from S3 to GPU, skipping the CPU bottleneck entirely. how is this significant? surely either the network or the GPU calculations is the bottleneck here?
- dwb 1y agoIf you don't clearly detail what your new tech product or system is bad at, as well as what it's good at, I'm not interested. So much of engineering is about navigating the inevitable tradeoffs. Marketing should have no place in engineering.
- derekhecksher 1y agoThe AnyBlox paper from the folks at TUM, and linked to in the post, is a bit more interesting, imo, since it looks to solve the data systems x storage format problem in composable data architectures - https://gienieczko.com/anyblox-paper https://gienieczko.com/anyblox-paper
- aduffy 1y agoThe AnyBlox paper has some very cool ideas, and the authors are friends. In the paper you'll notice a large portion of it analyzes Vortex, both standalone and embedded. Definitely worth a read.
- whalesalad 1y agoThe hot new aesthetic these days is either "receipt printer" or "liquid glass". I dig it, tbh.
- skywhopper 1y agoMan, they are really proud of that initial seed round funding aren’t they? Forgive me, but $22 million does not sound like enough to truly revolutionize data processing technology. The gist seems to be that they can overcome network latency issues when dealing with huge numbers of smallish objects in S3-like storage systems that need to be fed into GPUs? Yeah, those formats and systems were not designed to feed that type of processor. You’re doing it wrong if this is your problem. After a lot of nonsense, it sounds like they just reformat the data into something more efficient instead. But they forget about the network latency and blame CPUs for slowing things down? And what was that sidetrack about S3 permissions? I wouldn’t jump right onto this… well, it’s not clear what this even is exactly. But you can probably wait it out.
- bflesch 1y agoEven though Linux Foundation is only a 501(c)(6) nonprofit and not a 501(c)(3) nonprofit I wonder if all this legalese about "donation" might have very positive tax consequences for the donor.
- datadrivenangel 1y ago$22 million gets you a potentially commercially viable narrow featured database and some time to get enough sales to validate the VCs throwing another $50m at you to flush it out and build out the sales arm seriously.
- redwood 1y agoSo it's for low change rate data that needs to be bulk processed during ML model training. Cool. But hardly the same thing as what you need for powering live AI applications... which is what I assumed this was upon reading the intro and the mention of Postgres.. Postgres (and MongoDB) are the king and prince of data due to their transactional capabilities.
- rubenvanwyk 1y agoInteresting that Joran from Tigerbeetle posted this? So must be legit.
- jorangreef 1y agoI’ve known Will for some time while they’ve been in stealth.
- rubenvanwyk 1y agoHow does Vortex compare to Lance? I imagine Lance is already a good solution for AI on CPUs.
- donperignon 1y ago“ We work in person at our offices in London and New York. Face to face is better: if uncertain, the answer is “yes, get on the plane”. On Wednesdays, we wear pink.” No comments.
- sys13 1y agoI wonder how much we need this vs implementing it as part of Delta Lake or Iceberg
- mritchie712 1y agoyeah, I think you could get much of the benefits if Iceberg (or ducklake) added support for `.vortex` (e.g. ducklake only supports `.parquet` right now)
- dwroberts 1y ago> Remember that uncanny valley between 1KB and 25MB? The problem isn't the sizes—it's that Second Age systems force you to choose between two bad options: inline the data (killing performance) or store pointers (breaking governance). Spiral eliminates this false choice. We store 10KB embeddings directly in Vortex for microsecond access, intelligently batch 10MB blocks of images for optimal S3 throughput, and externalize 4GB videos without copying a single byte. One system, no compromises. No compromises but isn’t ‘externalising’ a large video the equivalent of storing a pointer in the first example? Can’t really see any other way to understand what that means (it goes to an external system and you store where it is)