6 ms·
Shakti, the new data platform from Arthur Whitney
- kthielen 7y agoMorgan Stanley in recent years has used this where they’d previously used kdb: https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes From the article, it sounds like the Kx folks are working on the issues that led to the development of hobbes in the first place. I’d be surprised if they’ve changed their opinions about type systems though (or bullying people with lawyers — those guys get nasty when their ideas are challenged).
- anonu 7y agoInteresting. So they're taking the "app engine" concept that Kdb provides and promoting that bit. Anyone have experience using Hobbes in the wild?
- scottlocklin 7y agoScrolling through the thing: I'm pretty sure not many (possibly not any) people are going to have experience using this thing in the wild.
- kthielen 7y agoExcept at Morgan Stanley (where the same could be said for kdb 20 years ago).
- scottlocklin 7y agoWell, do you know anyone at Morgan who is using it? I must say that I do not.
- nuclearnice1 7y agoFive of his last eight posts contain links to it. So I surmise he is the primary author of it.
- kthielen 7y agoYes I do.
- scottlocklin 7y agoGreat! Glancing at it, I suspected it wasn't actually used; looked too "Buck Rogers," but maybe I am old and set in my ways. Do you have a pointer to setting something like a ticker plant up like in the old K3 manuals? Maybe over here? https://hobbes.readthedocs.io/en/latest/logging/hog.html#on-the-receiving-host https://hobbes.readthedocs.io/en/latest/logging/hog.html#on-...
- nuclearnice1 7y agoSomething about the language itself, the presentation and documentation, or the dependency environment? BTW, it would be a treat to see an open sourcing of kerf at some point! I am an admirer.
- beagle3 7y agoDoes hobbes have any kind of db or persistence functionality? As a programming language, k/kdb is nice (and I personally prefer it to APL and J), but without the db part I suspect that it wouldn't have been as popular (in the niches that it is popular). Seems like a mostly eager haskell; am I correct? How does it compare performance-wise to ghc? (It obviously does much better on embedding and interfacing C++)
- kthielen 7y agoYes there are persisted data structures and databases built on top of them. The whole type system applies to persisted data structures so you can make your own data structures and fill out a few type class instances to query them with plain list comprehensions (e.g. btree indexes are defined and queried this way in some internal tools). Eager Haskell with a structural type system, deterministic garbage collection, and no overloading penalty, that would be a fair summary. It'd be interesting to see how GHC would do in the same area, there would be some significant challenges. I haven't tried to use GHC there so I can't say.
- SifJar 7y agoI might be misinterpreting your message, but if not - Shakti is not made by Kx, which AW has been bought out of. Shakti is a new company (although AFAIK everyone there is ex-Kx)
- gbronner 7y agoThey use hobbes mostly for high speed trade capture and limited querying. KDB is still used extensively for less structured /performant examples.
- __s 7y agoTen trillion rows, zero information, nice
- deleted 7y ago[deleted]
- dang 7y agoThe submitted URL was https://shakti.com/database-software-history/ https://shakti.com/database-software-history/. Their press page links to https://news.efinancialcareers.com/uk-en/3002752/shakti-arthur-whitney https://news.efinancialcareers.com/uk-en/3002752/shakti-arth..., which reads like PR but arguably contains a bit more information (e.g. that it was released last month), so I guess we'll link to that instead.
- anon1m0us 7y agoI'd like to see some real metrics. All the words are subjective and the technical ones are common to other databases as well. How many results can it return in how many seconds from a table with how many rows? Can it do full text indexing? Can it aggregate? It feels like something from wallstreet, the language is almost pump and dump "I have this really great stock tip from this guy who made millions once, he's a legend. He says buy, so you should." We need more tangible and quantifiable information about this database. Until then it's not even hype it's vaporware.
- kick 7y agoCalling anything from Arthur "vaporware" is really laughable. This article isn't great, but it's from a finance site, not a technical site: of course it's going to be low on technical details. You would think that people who want technical details would know how to use a search engine. Further, kdb/kdb+ (which Shakti is the successor of) is a columnar-based database. "How many results can it return [...] from a table with how many rows?" is completely irrelevant. There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled.
- deleted 7y ago[deleted]
- mhd 7y ago> Calling anything from Arthur "vaporware" is really laughable. kOS?
- shrubble 7y agoIt sounds like they are using the AVX primitives to accelerate certain operations, when they talk about 256 and 512 bit wide instructions.
- 3xblah 7y agoMaybe this is why shakti fails to start when I have tried it with 1.6 GHz Intel Atom and Celeron CPUs. When I tried on Ubuntu with Celeron I got "Illegal instruction (core dumped)"
- dantiberian 7y agoI doubt there would be a hard requirement on any AVX instructions, just that it can take advantage of them if available.
- deleted 7y ago[deleted]
- plorkyeran 7y agoA hard requirement on AVX instructions is exactly the sort of thing I would expect out of Arthur Whitney. His software tends to be very opinionated, and that's one of the things that lets him do so much in so little code.
- tluyben2 7y agoThe version I downloaded from the site does (did?) depend on AVX instructions and would die on startup. There are other ('nightly builds') that fix this. One of the goals of the current team seems to be portability.
- scottlocklin 7y agoFWIIW J is moving in this direction as well; the last release version of it (807) needed a special build for older intel hardware (they subsequently made separate binaries for older hardware). AVX has been around since 2011/2012, so it doesn't seem like that big a deal. Admittedly I myself have older hardware kicking around I see no real reason to upgrade, but AVX is such a big win on array processing you might as well make it a dependency for the kinds of use cases Shakti or J get used for in business.
- LittlePeter 7y agoIt seems like Shakti is just the k language version 7 as per README at https://github.com/kevinlawler/kona https://github.com/kevinlawler/kona I remember reading an interview with Arthur and he said that he writes each version of k from scratch. Most likely this version has also been written from scratch.
- chrisaycock 7y agoAn evaluation copy is available for installation via Anaconda: https://anaconda.org/shaktidb/shakti https://anaconda.org/shaktidb/shakti The tutorial mentions streaming and distributed capabilities, but then doesn't cover them: https://shakti.com/tutorial/ https://shakti.com/tutorial/ Having used q/kdb+ a bunch in the beginning of my quantitative finance career, I hold a special place for Arthur's languages. I look forward to what Shakti develops into.
- kragen 7y agoHow widely used is kdb+ today?
- burtonator 7y agoI'm super skeptical of these articles about programmers with superhuman powers.
- steego 7y agoMaybe they're not superhuman? Maybe they applied their time, practice and attention to developing skills that have yielded very big returns?
- jshaqaw 7y agoI suggest reading the J interpreter Whitney wrote in a weekend and reconsidering! https://code.jsoftware.com/wiki/Essays/Incunabulum https://code.jsoftware.com/wiki/Essays/Incunabulum
- kuroguro 7y agoWriting in that style is not that hard, I do something similar if I need to make a prototype of something. The short variable and function names really save time as long as you remember what is what. Reading or debugging it even a week later is torture tho. Writing the interpreter in an afternoon is impressive but most likely he had done something similar before and had most of the algorithm in his head before he even began. You don't need to be superhuman to do that.
- beagle3 7y ago"Superhuman" hyperbole aside, some people are demonstrably orders of magnitude more productive than others, and that's true for almost any field of work (depending on your metric, of course - but that's true for some of the metrics that actually matter to most people), and programming is no different. Arthur Whitney, Fabrice Bellard, Linus Torvalds, Mike Pall are well known examples. It obviously requires a lot of skill, and I agree with [0] that it also requires the skill to be selective - some problems require a lot of work for which there are no "better ways". E.g. Mike Pall, alone and in short time, produced a JIT that runs way faster than V8 (which probably already had 100x more man-months[1] of really really smart people at that point). This amazing feat is a result of Pall's amazing talent - but it was only possible (IIRC, he said so himself), because Lua, although it is essentially as dynamic as JavaScript, made a few choices that made it much more amenable to effective JITting. [0] https://yosefk.com/blog/10x-more-selective.html https://yosefk.com/blog/10x-more-selective.html [1] I use it properly here as a unit of effort/cost, not as a unit of productivity.
- chrispsn 7y agoA group of k enthusiasts working under the 'kparc' brand has been writing docs for this new in-development k: https://kcc.kparc.io/ https://kcc.kparc.io/ https://ref.kparc.io/ https://ref.kparc.io/
- pinewurst 7y agoWhen this was first announced, they used the word “blockchain” a lot. It’s funny that this is now deleted, as they’ve learned the word is credibility poison.
- emanuensis 7y agoAs a k email list reader i noted in https://news.ycombinator.com/item?id=21660005 https://news.ycombinator.com/item?id=21660005 that shakti(k7) MAY become FOSS...
- SifJar 7y agohighly unlikely it'll go that route IMO, the messages in google group are just wishful thinking
- osrec 7y agoNitpick from the article: it suggests kdb underpins most algorithmic trading systems. This is simply not true. I've worked in a number of banks and hedge funds, and most have their own home-brew storage and analysis engines (I've helped build a few myself).
- smabie 7y agoq is an interesting language and am using it for the advent of code. While the language superficially seems simple, it features a lot of hidden complexity and dubious design decisions. Every operator is massively overloaded on rank and type. There are functions that accept functions or lists or atoms as arguments: supposedly there’s supposed to be mathematical commonality, but coming from the statistically typed FP world, this kind of polymorphism makes me extremely uncomfortable. The dynamic typing bugs me as well, it would really help to have an editor be able to tell ahead of time whether an expression is valid. Speaking of valid code, the error messages are absolutely terrible and stink of arrogance. I’m talking about ed level error messages: no words, no pointing out the error, just some one letter message that is supposed to indicate a type or function argument error. A lot of the culture of the language is built in Arthur Whitney’s own image: error messages are bad, comments are for stupid people, white space is for “Qbies” Speaking of cultural, the jargon of the language is didactic and annoying. All over the documentation references to monadic or niladic or diadic functions. Also calling things verbs or adverbs or nouns is annoying and unhelpful. Even in the book “Q for mortals” the author prizes using different terminology than literally anyone else and strikes me as trying to be deliberately obtuse. He even spends an entire chapter talking about the mathematical nature of natural numbers, which strikes me as both unhelpful and narcissistic. Going back to the language, not only are functions overloaded to a ridiculous level, but the syntax of the language itself is inconsistent. Multiple ways to call a function, all in the name of reducing unnecessary typing. Also the syntax and name of some functions strikes me as very add: ‘/‘ is not used for division, but instead for both comments and folds. ‘,’ is used for appending lists, while ‘;’ is used to separate function parameters. Instead of parents, Q uses ‘[]’ for function parameters. Here’s an example of finding the square of the hypotenuse in idiomatic q: pyth_sq:{x^2+y^2} One can use xyz without explicitly declaring them as parameters, which is both nice and arbitrary. I would much prefer to have something like Scala’s underscore or maybe something like _1,_2,... Here’s the function written with explicit parameters: pyth_sq:{[x;y]x^2+y^2} Personally I think a syntax (and spacing) like: pyth_sq = { (x, y) => x^2 + y^2 } would be nicer. While the naming of the functions and syntax is arbitrary, I beginning to understand the rational and design of the language. For Whitney, code is the root of all evil: less code, less bugs. Also less code, the more you can fit in your head. This is the reason why everything is overloaded I think: because Whitney ran out of ASCII symbols and didn’t want to use names (more information to remember). Regardless, the real reason people use q isn’t because of the language, it’s often despite of it. The kdb+ database and q sharing the same memory space is the real advantage. The performance of the system is so exceptional that a single server can outperform even a large Hadoop cluster: the code is out of this world optimized. I’ve began to think that q is a natural and easier to use extension of Chuck Moore and Forth’s philosophy and language. The languages both share the same tenets of performance, terseness, and minimal syntax. While forth works on a fixed number of stacks, q works on a arbitrary number of n-dimensional vectors, a natural extension to the one dimensional stack. Anyways, sorry for the rant. I’m still learning q so take what I say with a grain of salt. I can imagine an expert q developer becoming extremely productive and comfortable: I’m personally just not there yet. When I finish advent of code, I’ll have a better perspective.