3 ms·

Lots of respect for Arthur, KDB+ was transformative when it arrived but I think now we need to move beyond that. Will take one or two decades for banks to migra

Lots of respect for Arthur, KDB+ was transformative when it arrived but I think now we need to move beyond that. Will take one or two decades for banks to migrate though lol.

steveBK123 4y ago
It's also a misunderstood product from the outside. Unfortunately banks had a tendency to pay a bunch of senior engineers to do a build out for 3 years and then turn it over to cheap nearshore consultants to operate for 10 years. The core of most big bank code you see was written before 2010. Where I have seen bakeoffs and largely failed attempts to replace, the challenge is the number of vendor & open solutions you need to cobble together to replace all the use cases that KDB supports. Every year it was a slightly different combination that didn't really cover all the bases. To replace KDB in many of these large use cases, you need - a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations, an event bus, a stream processor, and a programming language to write more complex applications close to the data. A lot of the newer cloud-centric offerings are more about massive parallelism/scale than about pure single thread/individual request-response performance. This is great for the vast majority of CRUD apps with millions/billions of users. It's even great for some financial "backtesting" type use cases where you can kick off large It generally isn't sufficient for many of the real-time financial use cases of doing ad-hoc analytics on billions/day datasets with expected responses in ms. The biggest complaints are its expensive, and the devs are expensive. On the other hand you usually don't need as much hardware or dev staff to support it.
- timeserious 4y ago
  Hi Steve. See my reply in above thread to Shin which calls out alot of the operational improvements we've made. Just an FYI, there's also been significant changes in how we price so it's now alot easier to start small, get started quickly and decrease the time to value with more mainstream skillsets. With the new product strategy we're seeing alot more buy side clients (hedge funds/asset managers) use kdb+/q for their workloads with KX Insights.
- zX41ZdbW 4y ago
  > a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations Sounds like ClickHouse. And we see kdb slowly start being replaced...
timeserious 4y ago
Hi Shin - KX employee here (the creators of q/kdb+). We launched a new product to take our time-series tech to new markets and usecases with high performance analytic needs. Over the past 2-3 years we've listened closely to the community and have made huge improvements to make kdb+ easier to develop with using options like ANSI-SQL/Python/Stream Processing DSL/microservices/Open API etc. The banks are adopting all of the new kit quickly and leverage the cloud native capabilities (like kubernetes operators/object storage) to help scale and take their deployments to new heights without deep technical kdb+ expertise. Thought it was worthwhile calling out the adoption and product strategy here at KX.. lots more to come! https://code.kx.com/insights/ https://code.kx.com/insights/
- shin_lao 4y ago
  Hey there thanks for your answer, know about this product, think you need to do better :-) Best of luck!

Lots of respect for Arthur, KDB+ was transformative when it arrived but I think now we need to move beyond that. Will take one or two decades for banks to migrate though lol.

steveBK123 4y ago

It's also a misunderstood product from the outside. Unfortunately banks had a tendency to pay a bunch of senior engineers to do a build out for 3 years and then turn it over to cheap nearshore consultants to operate for 10 years. The core of most big bank code you see was written before 2010. Where I have seen bakeoffs and largely failed attempts to replace, the challenge is the number of vendor & open solutions you need to cobble together to replace all the use cases that KDB supports. Every year it was a slightly different combination that didn't really cover all the bases. To replace KDB in many of these large use cases, you need - a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations, an event bus, a stream processor, and a programming language to write more complex applications close to the data. A lot of the newer cloud-centric offerings are more about massive parallelism/scale than about pure single thread/individual request-response performance. This is great for the vast majority of CRUD apps with millions/billions of users. It's even great for some financial "backtesting" type use cases where you can kick off large It generally isn't sufficient for many of the real-time financial use cases of doing ad-hoc analytics on billions/day datasets with expected responses in ms. The biggest complaints are its expensive, and the devs are expensive. On the other hand you usually don't need as much hardware or dev staff to support it.

timeserious 4y ago
Hi Steve. See my reply in above thread to Shin which calls out alot of the operational improvements we've made. Just an FYI, there's also been significant changes in how we price so it's now alot easier to start small, get started quickly and decrease the time to value with more mainstream skillsets. With the new product strategy we're seeing alot more buy side clients (hedge funds/asset managers) use kdb+/q for their workloads with KX Insights.
zX41ZdbW 4y ago
> a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations Sounds like ClickHouse. And we see kdb slowly start being replaced...

timeserious 4y ago

Hi Shin - KX employee here (the creators of q/kdb+). We launched a new product to take our time-series tech to new markets and usecases with high performance analytic needs. Over the past 2-3 years we've listened closely to the community and have made huge improvements to make kdb+ easier to develop with using options like ANSI-SQL/Python/Stream Processing DSL/microservices/Open API etc. The banks are adopting all of the new kit quickly and leverage the cloud native capabilities (like kubernetes operators/object storage) to help scale and take their deployments to new heights without deep technical kdb+ expertise. Thought it was worthwhile calling out the adoption and product strategy here at KX.. lots more to come! https://code.kx.com/insights/ https://code.kx.com/insights/

shin_lao 4y ago
Hey there thanks for your answer, know about this product, think you need to do better :-) Best of luck!