6 ms·
ArcticDB: Why a Hedge Fund Built Its Own Database
- bdjsiqoocwk 2y agoIsn't it constrained to minutely timestamps or something like that.
- dnadler 2y agoNope, it’s fairly generic. I worked at Man for a number of years and with Arctic a lot during that time. In my role I was mostly working with daily data, but also higher and lower frequency data, and never had issues.
- quickvi 2y agoanyone knows how it compares to TileDB? Seems like TileDB is just a better ArticDB
- sgarland 2y agoNo, but you’re the first person I’ve ever seen mention TileDB in the wild. I applied there a few years ago; didn’t go anywhere, but I’ve been keeping an eye on them ever since, because I think it’s an interesting idea.
- dang 2y agoRelated: ArcticDB: A high-performance, serverless Pandas DataFrame database - https://news.ycombinator.com/item?id=35198131 https://news.ycombinator.com/item?id=35198131 - March 2023 (1 comment) Introducing ArcticDB: Powering data science at Man Group - https://news.ycombinator.com/item?id=35181870 https://news.ycombinator.com/item?id=35181870 - March 2023 (1 comment)
- Nelkins 2y agoI don’t think the last link is related. Different database.
- silisili 2y agoCorrect. They renamed FrostDB, here is the announcement - https://www.polarsignals.com/blog/posts/2022/06/16/arcticdb-is-now-frostdb https://www.polarsignals.com/blog/posts/2022/06/16/arcticdb-...
- dang 2y agoThanks for the correction! I'll take it out from the list above and put it here instead: Introducing ArcticDB: A Database for Observability - https://news.ycombinator.com/item?id=31260597 https://news.ycombinator.com/item?id=31260597 - May 2022 (31 comments)
- OutOfHere 2y agohttps://github.com/man-group/arcticDB https://github.com/man-group/arcticDB
- stackskipton 2y agoRead the presentation. Answer was what I expected. We had unique problem and because we make oil drums amount of cash, dipping a bucket and taking that cash to solve the problem was easy justification. These are really smart people solving problems they have but many companies don't have buckets of cash to hire really smart people to solve those problems. Also, the questions after presentation pointed out the data isn't always analyzed in their database so it's more like storage system then database. >Participant 1: What's the optimization happening on the pandas DataFrames, which we obviously know are not very good at scaling up to billions of rows? How are you doing that? On the pandas DataFrames, what kind of optimizations are you running under the hood? Are you doing some Spark? >Munro: The general pattern we have internally and the users have, is that your returning pandas DataFrames are usable. They're fitting in memory. You're doing the querying, so it's like, limit your results to that. Then, once people have got their DataFrame back, they might choose another technology like Polars, DuckDB to do their analytics, depending on if they don't like pandas or they think it's too slow.
- datahack 2y agoThis comment is underrated comedy gold. You clearly have worked with big data.
- primitivesuave 2y agoI skipped to the "why build a database" section and then skipped another two minutes of his tangential thoughts - seems like the answer is "because Moore's law"?
- tda 2y agoI know there are tons of problems that are solved in excel while they really shouldn't. Instead of getting the expert business analyst to use a better tool (like pandas), money is spent to "fix" excel. Apparently there is also a class of problems that outgrow pandas. And instead of the business side switching to more suitable tools, some really smart people are hired to build crutches around pandas. Oh well, they probably had fun doing it. Maybe they get to work on nogil python next
- laweijfmvo 2y agoi’m so sorry someone built something that you don’t agree with
- beckingz 2y agoThere's value in 'backwards compatibility' from a process/skills perspective. I agree that companies usually pay too high a premium on that, but there is value.
- smabie 2y agoAll quants know pandas. beyond that their technical skills are often quite limited. It makes sense to invest resources into something everyone knows how to use, even if it's suboptimal.
- paretoer 2y agoThe old "can't make money with numpy, its all produced in already"
- chirau 2y agoTwo Sigma did a similar thing a few years back. It's called Smooth Storage. https://www.twosigma.com/articles/smooth-storage-a-distributed-storage-system-for-managing-structured-time-series-data-at-two-sigma/ https://www.twosigma.com/articles/smooth-storage-a-distribut...
- gadders 2y agoI believe when a bunch of ex-Goldman's were at BofA (they move from bank to bank, trying to re-implement SecDB [1]), they were creating their own time series database as well. [1] https://www.wsj.com/articles/understanding-secdb-goldman-sachss-most-valued-trading-weapon-1473242401 https://www.wsj.com/articles/understanding-secdb-goldman-sac...
- dnadler 2y agoIf it wasn’t clear from the article, this is open source and available on Man’s GitHub page: https://github.com/man-group/arcticDB https://github.com/man-group/arcticDB I used to work at man, so take this with a grain of salt, but I really liked this and have spun it up at home for side projects over the years. I’m not aware of other specialized storage options for dataframes, but would be curious if anyone knows of any.
- cgio 2y agoNeed a licence apparently, source available.
- camkego 2y agoThe license is the Business Source License, which is not open source: https://en.wikipedia.org/wiki/Business_Source_License https://en.wikipedia.org/wiki/Business_Source_License The current license says the terms shall change to Apache after two years a release has been available. Not for me, but some might find it interesting.
- diggan 2y agoNot sure if it's standard practice when using Business Source License, but seems to have additional terms too, like: > BSL features are free to use and the source code is available, but users may not use ArcticDB for production use or for a Database Service, without agreement with Man Group Operations Limited. [...] Use of ArcticDB in production or for a Database Service requires a paid for license from Man Group Operations Limited So not just "source available, go ahead and use it" but basically "free for personal use only".
- wdb 2y agoFeels more feel free to use it as long you aren't a competitor
- diggan 2y ago> Free as long as you aren't a competitor, or you use ArcticDB for anything we could consider "in production" This is how I understand their text, and I wouldn't be super excited about trying to figure out exactly what "in production" means with their lawyers in court.
- andrewstuart 2y agoBlog post from same company in two years: "How we switched from a custom database to Postgres".
- Loughla 2y agoI don't understand your cynicism here. Should people/businesses not try new things?
- andrewstuart 2y ago>> Should people/businesses not try new things? Well yes, in the right context, like hobby/personal programming. But things like "we built our own database" tend to be really hard to justify and mostly represent technical people playing toys when they actually have an obligation to the business that is spending the money to not play toys but to spend the money wisely and build a wise architecture, especially for business critical systems. It's indulgent and irresponsible to do otherwise if standard technology will get you there. The other point is that there are few few applications in 2024 that would have data storage requirements that cannot be met by Postgres or some other common database and if not then perhaps the architecture should be changed to do things in a way that is compatible with existing data storage systems. Databases in particular as the core of a business system have not only the get "put data in/get data out" requirement but hundreds of sub requirements that relate to deployment and operations and a million other things. You build a database for that core set/get requirement and before you know it you're wondering how to fulfill that vast array of other requirements. This happens everywhere in corporate development that some CTO who has a personal like for some technology makes the business use that technology when in fact the business needs nothing more than ordinary technology, avoiding all sorts of issues such as recruiting. The CTO moves on, leaving behind the project built with the technology flavor of the month and a project that either needs to struggle to maintain it into the future, or the need to replace it with a more ordinary way of doing things. Likely even the CTO has now lost interest in their playtime technology fad that the industry toyed with and decided isn't a great idea. So I stand by my comment - they are likely to replace this within a few years with something normal, likely Postgres.
- faizshah 2y agoI still didn’t get why they built this, there’s a better explanation of the feature set in the FAQ comparison with parquet: https://docs.arcticdb.io/latest/faq/ https://docs.arcticdb.io/latest/faq/ > How does ArcticDB differ from Apache Parquet?¶ > Both ArcticDB and Parquet enable the storage of columnar data without requiring additional infrastructure. > ArcticDB however uses a custom storage format that means it offers the following functionality over Parquet: > Versioned modifications ("time travel") - ArcticDB is bitemporal. > Timeseries indexes. ArcticDB is a timeseries database and as such is optimised for slicing and dicing timeseries data containing billions of rows. > Data discovery - ArcticDB is built for teams. Data is structured into libraries and symbols rather than raw filepaths. > Support for streaming data. ArcticDB is a fully functional streaming/tick database, enabling the storage of both batch and streaming data. > Support for "dynamic schemas" - ArcticDB supports datasets with changing schemas (column sets) over time. > Support for automatic data deduplication. The other answer I was looking for was why not kdb since this is a hedge fund.
- mianos 2y agoI think people are getting a little tired of being held ransom to kdb/q and kdb consultants. Even if you have 'oil barrels' of money, eventually it is annoying enough to look elsewhere.
- blitzar 2y agoYou don't get 'oil barrels' of money if you are in the habit of giving big chunks of it to other people. Handing over seed capital levels of cash to external vendors inevitably gives someone the idea to seed a competitor instead.
- mianos 2y agoThe idea they had was to just make something up, like pandas and a series/columnar database, themselves. There are other competitors to kdb/q, but they are not as entrenched and maybe not a perfect fit. These guys cooked up a closer fit for their own systems, than, say clickhouse and other tools. It had to be a very close fit to what they do, as kdb/q is pretty damned good at some things in finance. Maybe there is not enough money in the highly specialised areas it does very well at, for other people to come in with something new. It would be a huge mistake to think sql is a replacement for Q.
- jmakov 2y agoIs there any reason to use that instead of Delta lake?
- jjmunro 2y agoHi. I'm the presenter. Thanks for the interest. Opinions here are my own. I'll put in a TLDR as the presentation is quite long. The other thing I'd like to say was that QCon London impressed me, the organisers spent time ensuring a good quality of presentation. The other talks that I saw were great. Many conferences I've been to recently are just happy to get someone, or can choose and go with well known quantities. I first attended QCon London early in my career, so it was interesting coming back after over a decade to present. TLDR: Why did we build our own database? In effort terms, successful quantative trading is more about good ideas well executed than it is about production trading technology (apart from perhaps HFT). We needed something that helped the quants be the most productive with data. We needed something that was: - Easy to use (I mean really easy for beginner/moderate programmers). We talk about day 1 productivity for new starters. Python is a tool for Quants not a career. - Cost effective to run (no large DB infra, easy to maintain, cheap storage, low licensing) - Performant (traditional SQL DBs don't compare here, we're in the Parquet, Clickhouse, KBD, etc space) - Scalable (large data-science jobs 10K+ cores, on-demand) A much shorter 3 min intro from PyQuantNews: https://www.youtube.com/watch?v=5_AjD7aVEEM https://www.youtube.com/watch?v=5_AjD7aVEEM GitHub repo (Source-available/BSL): https://github.com/man-group/ArcticDB https://github.com/man-group/ArcticDB
- dan-robertson 2y agoThis sort of general architecture (store parquet-like files somewhere like s3 and build a metadata database on top) seems reasonably common and gives obvious advantages for storing lots of data, scaling horizontally, and scaling storage and compute separately. I wonder where you feel your advantages are compared to similar systems? Eg is it certain API choices/affordances like the ‘time travel’ feature, or having in-house expertise or some combination of features that don’t usually come together? A slightly more technical question is what your time series indexes are? Is it about optimising storage, or doing fast random-access lookups, or more for better as-of joins?
- joewood1972 2y agoFor example, Apache Iceberg is exactly this. Complete with bitemporal query support.