18 ms·
Ruins of forgotten empires: APL Languages (2013)
- Detrus 11y agoWould be interesting to see a more detailed explanation of how Q compares to Redis at least. The low level bits, performance, RAM usage, usage comparison of the same tasks etc.
- geocar 11y agoHow detailed do you want to get? If you're using Python or C or something else to use redis as a server, it's probably just as fast as Q/KDB, and some informal benchmarking supports this: redis-server & redis-benchmark -t set -n 1000000 -c 1 -P 1000000 0.00% <= 1368 milliseconds v. q)h:neg hopen`:localhost:1234 q)\t 1000000 h/"a:69" 1013 However I don't think this necessarily a good way to build your system, because if you're going to do a million of (read something write) every second, and then do a million of (read store), then you might as well write it in KDB and just make it (read something store) and save yourself 40% on your heating bill.
- joebo 11y agoIt seems KDB excels at queries, joins, and aggregates on large datasets. I have limited experience with redis, but if that type of code would need to be written in c/python with redis then KDB may have a performance and productivity lift. I would like to see an example like that. Another benchmark is here: http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt
- geocar 11y agoLike what? A program in KDB instead of a program in Python and Redis? Here's an implementation of a multiuser group chat in KDB: https://github.com/srpeck/kchat/ https://github.com/srpeck/kchat/ and here's one in Python and Redis: http://programeveryday.com/post/create-a-simple-chat-room-with-redis-pubsub/ http://programeveryday.com/post/create-a-simple-chat-room-wi...
- woah 11y agoSomeone should tell this guy about JS's Array.map()
- dang 11y agoThis is the kind of shallow, dismissive comment we need less of on HN. Please don't post them.
- pjonesdotca 11y agoThis.
- white-flame 11y agoMuch of his thrust seems to boil down to being really offended by the cost of safety of allowing concurrent writes while reads are going on. When dealing with analysis workloads, his line of thinking can yield fast runtimes and doesn't bring that much of a programmer burden. However, it's killingly complex when it comes to live mutating business databases that need to scale. Even many job-based analysis workloads have a lot of concurrent mutation in trying to efficiently figure out what's been done and what needs to be done next when the actual workflow is not a straight pipeline. I do agree that 'for' loops should be replaced by broader declarations of what needs to be done, instead of these explicit instructions on how to iterate through the data.
- brudgers 11y agoI take the author's viewpoint as: when the data is actually big, big data operations are for all practical purposes going to be run in batch mode - it's just hard to squeeze enough new data through the pipeline in ten minutes to change the statistical analysis of many terabytes of big data...if the data is so unpredictable that a snapshot isn't going to be meaningful then slowing operations down to allow concurrent writes isn't going to provide better analysis and what is really needed is not big data but some sort of near real-time stream processing. All this comes with the assumption that much of what people consider big data isn't really big data.
- wyager 11y agoCould you expand on what you mean about for loops? In the functional world, there are all sorts of precise constructs to operate on lists (maps, folds, unfolds, scans, etc.), and many versions of these that support concurrency. That's what I imagine when you say "declarations of what needs to be done".
- white-flame 11y agoThe standard "for" structure is necessarily ordered. Either an index is incremented/decremented, or an iterator object's next() comes in. Since there are side effects that can affect either the iterator or the index, and dependent side effects between loop body runs, obeying a "for" loop means strictly ordered (unless the iterator interface abstracts something else), serialized execution of the loop body. Even maps, folds, etc are often defined in terms of what order they process data. This is useful for early exit, and mutable state dependency between iterations. But again, this demands serialized, specifically ordered processing of elements. Lists themselves are ordered structures. Their generalized interface does not assume it's just a bag of items easily thrown around in sub-parts, especially if the linked list nodes are to be modified. Of course, the above applies to impure functional languages. Purely functional of course has far more freedoms, but even the Lisps and such ultimately demand sequential processing for most of their fundamental list operations.
- brudgers 11y agoPLaSM is an APL like language for describing building architecture (based on IBM's FL, a predecessor to J). http://www.plasm.net/docs/ http://www.plasm.net/docs/
- frozenport 11y ago>>Why have multiple threads tripping all over each other when a query is inherently one job? Because these systems are limited by IO latency, this way the threads respond immediatly when data is available. You dont have the 'stepping' on each other problem unless they are actually running. Further, threads on the same CPU socket, even share resources like memory access, even they can step on each other. You shouldnt be worried about threads and cpu matching if your computation takes 10ms, but data access is 150ms.
- vezzy-fnord 11y agoOr you can multiplex on a file descriptor to asynchronously handle events? Not sure what exactly you intended to say, or if there's any corner cases in job processing.
- GFK_of_xmaspast 11y agoNever read the comments. ("Computer science worked better historically in part because humorless totalitarian nincompoopery hadn’t been invented yet. People were more concerned with solving actual problems than paying attention to idiots who feel a need to police productive people’s language for feminist ideological correctness. You may now go fuck yourself with a carrot scraper in whatever gender-free orifice you have available. Use a for loop while you’re at it. ")
- rbanffy 11y agoThere can be hope. https://github.com/filipekiss/sanity https://github.com/filipekiss/sanity
- nickpsecurity 11y agoAnother person realizing my meme that IT regularly forgets and (when lucky) re-invents the past. Often in an inferior way. K and J were great tools for datamining. Well-implemented, too. Hadoop-style efforts could've saved a lot of time and gained plenty efficiency starting out with such a design. Any suggestions on best way to solve this? A Wikibooks-like, categorized, searchable site of all kinds of solutions to various problems, perhaps? Fortunately we have nice counter-examples with groups such as FoundationDB doing Hadoop scale with strong consistency. Building on the right, proven principles can get one ahead.
- MrBuddyCasino 11y agoYeah, a resource of canonical answers to questions. Often arguments are repeated again and again in discussions, often poorly. If you could just point people at this, and pros and cons would become better over time, I think that would be a real win.
- Blackthorn 11y agoI don't know. Seems like the sort of thing that would be useful for a good few years and get popular, but then new questions and resources would start getting closed and locked for being off-topic or irrelevant...Wink, wink.
- nickpsecurity 11y agoThat's exactly what I'd try to avoid. You could say all topics are shallow there and we'd be the deep thinkers. ;)
- rbanffy 11y agoWhat I find slightly disturbing is that people remember history but forget we got to where we are precisely because of it. 64-bit processors and more than 4 gigs of RAM are relatively new things. Mmap is useless for datasets larger than the disk you can attach to the CPU mmapping it or the size of its virtual address space. In 1991 it would be OK to call a couple billion entries "big data" (although we were yet to invent the term) but not anymore. Having said that, the heat dissipation of instantiating objects from their on-disk representation is probably a relevant part of the refrigeration costs of any data center. We just have unimaginable computing power available and, sometimes, we do things the easy way, not the most efficient one.
- trhway 11y ago>Comparing, say, Kx systems Q/KDB (80s technology which still sells for upwards of $100k a CPU, and is worth every penny) to Hive or Reddis is an exercise in high comedy. Q does what Hive does. It does what Reddis does. It does both, several other impressive things modern “big data” types haven’t thought of yet, and it does them better, using only a few pages of tight C code, and a few more pages of tight K code. it is exactly exercise in comedy. Hive and Reddis are free. They do for regular persons and tasks what only $100K/CPU systems of yesterday were doing. It is like comparing high-professional $100K film cameras of yesterday with today's consumer 5-10M cameras. A $100K thing which i would never had access to vs. real cheap/free gigantic expansion of my capabilities. The choice is obvious to me.
- taeric 11y agoThere is a difference. The new cameras are based on more advanced tech. The new storage machines are not necessarily based on new ideas in algorithmic execution. I'm not sure I agree, but the assertion of the article is that much of today's pain is a refusal to acknowledge the progress of the past. For myself, there is a lot to this. Many of my peers and the younger crowd are convinced that the problems they are solving have never been considered in the past. To the point that some of my coworkers are proud that they are "weak" in algorithms.
- ah- 11y agoKDB is actually free nowadays if you can live with the 32bit limitation.
- branchless 11y agoBanks have moved to linux from paid o/s yet they continue to shell out for KDB (and the dev are not cheap either). Why did they move a whole o/s platform but not from kdb?
- joebo 11y agoI assume to keep licensing current to support legacy code. I would be surprised if much revenue comes from new installs
- ah- 11y agoThe great thing about K/Q is it's simplicity and minimalism. It's fast because it doesn't do very much and doesn't have layers upon layers of abstractions. I would love to see a derivate of it with proper async support and more useful error messages.
- dkhenry 11y agoI think he came to the wrong conclusion. We shouldn't be looking at APL or J to see what they did right ( surprisingly little ) we should be focusing on how they failed and making sure we don't fail the same way. Yeah an APL master might be able to crunch numbers much better then the average Java programmer, but in reality I can get more number crunching implemented by spinning up an R based development team then I would ever get done trying to find, train, and pay an APL based team.
- eggy 11y agoYour statement validates his statement. Just because R-based development teams are in number, does not make quality, and vice versa: APL or J programmers may be lower in numbers, but you may be getting better quality. The catapult was invented, and then forgotten, and had to be re-invented. In today's age with the internet, and the relatively short span of computer science, I must say I agree that the short-sightedness seems worse. I play with APL/J and I like Julia and Haskell. To me J (APL) are close to the succinctness and expressiveness of mathematical formula, and they work naturally with arrays, no batteries needed.
- dkhenry 11y agoIt sounds like you both have this ideal of quality that theoretical quality counts for something, even expressed quality doesn't really matter that much if you can't bring _quantity_ to bear.
- rdc12 11y agoWhat did APL and co get wrong, except for becoming popular (otherwise it is just an argument from popularity). The ultra terse syntax, will have lead to an extreme learning curve. But if you replaced it with more typical function names, I doubt it would have made history different/
- dkhenry 11y agoAPL and co got becoming popular wrong. As great as a singular developer might be they will never make the comunalative progress that a much larger number of acceptable developers will make. The software ecosystem evoloved so rapidly around APL that it didn't have time to really show off anything it was good at. All the problems that it solved were solved 100 times over by others, and they gained the market share to drive the state of the art forward. Sure Kdb might be great, but there is a reason why Oracle makes a few more orders of magnitude more money then they do, even the claim that they can replicate Hadoop and other "Big Data" solutions is nonense as even a modest Hadoop consulting firm does more buisness then the 10m that Kx brings in. The failure is one I have seen a lot. It doesn't matter what you do, if you are living on an island as a professional of any kind your contributions will be shortly forgotten and replaced by those who had the ability to share their ideas with the larger population.
- panjaro 11y ago>> " Modern code monkeys don’t even recognize mastery; mastery is measured in dollars or number of users, which is a poor substitute for distinguishing between what is good and what is dumb. Lady Gaga made more money than Beethoven, but, like, so what? " Spot On !
- pjmlp 11y agoHence why valley hipsters jump into things like Go. http://cowlark.com/2009-11-15-go/ http://cowlark.com/2009-11-15-go/
- leoc 11y agoI'm just going to leave this here: http://www-dev.ccs.neu.edu/home/pete/pub/esop-2014.pdf http://www-dev.ccs.neu.edu/home/pete/pub/esop-2014.pdf
- rebootthesystem 11y agoI used APL professionally for about ten years. I even have a picture with Ken Iverson, the language's inventor, when I was 19. While I still programmed in assembly, Forth and C, I was one of the "APL guys" at work. The language changes the way you solve problems computationally. It changes your focus away from the mechanics programming to inmerse your mind in the problem domain. It is hard to understand this without working with APL for a year or more. There's another element here that is often ignored: Notation. APL uses a unique set of symbols and it is these symbols that give you expressive power not found in text-based approaches. A paper by Ken Iverson himself, titled "Notation as a tool for thought", covers this: http://www.eecg.toronto.edu/~jzhu/csc326/readings/iverson.pdf http://www.eecg.toronto.edu/~jzhu/csc326/readings/iverson.pd... The value of notation has been largely forgotten today due to the fact that APL suffered a popularity gap of probably twenty years due to the language being way ahead of technoloy at the time. The language didn't fail. The hardware of the time did. This, BTW, is the reason I think J is an absolute abomination. Yes, Ken Iverson was behind J. The motivation behind going for ASCII and losing the very symbols that made APL notation so powerful was to attempt to deal with the limitations of computers at the time. For example, on some systems you had to manually reprogram the graphic card's character ROM just to be able to display the symbols. Iverson sought to make the APL concept more accessible by transliterating to ASCII when creating J. In doing so he utterly destroyed the very notation he understood to be critically valuable as a tool for thought. For more than one reason it would be wrong to attempt to bring APL back today. The concepts, the approach and a form of notation for the expression of computational concepts could and should come back.
- bane 11y agoThere's lots of good reasons for using text for programming. But I've long thought that, as a medium for expression, it's pretty terrible. I'm reminded of this constructive review of TempleOS [1] and thinking YES I'd like to have some of those code-writing features. I'm also reminded of Perl, a language that grows multi-character operator glyphs at a rate only slightly slower than teenagers invent emoticons. Imagine if Perl could just be programmed with proper alternative symbol support like APL. Having proper symbols might also stop the minor endless war of deciding to use "=" as both assignment and evaluation. 1 - https://news.ycombinator.com/item?id=9681501 https://news.ycombinator.com/item?id=9681501
- transfire 11y agoOTOH, http://cs.ecs.baylor.edu/~maurer/SieveE/apl.htm http://cs.ecs.baylor.edu/~maurer/SieveE/apl.htm