7 ms·
Jd
- ee99ee 5y agoI am fairly confident the world needs neither J nor Jd. Sorry.
- stefan_ 5y agoAn example J file because this link doesn't say much: https://github.com/jsoftware/data_jd/blob/master/csv/csv.ijs https://github.com/jsoftware/data_jd/blob/master/csv/csv.ijs
- diarrhea 5y agos=. 0 2}.each _3 _1{<;._2 (;i{ccfiles),'/' That is... not pretty.
- bryanrasmussen 5y agoI would say it is not knowledge leaking, most languages leak knowledge so that if you are not familiar with the language but you do know some other programming languages you can sort of figure out what they do. But some languages do not leak knowledge in this way. There is the concept of beauty in programming languages that the expression of an idea should be succinct. This J code might be beautiful, but unsure.
- 0des 5y agoYou're going too meta. Does it make you happy to write it? Does it fulfill its purpose? If yes, don't worry how it looks, it is fine.
- razetime 5y agoJ isn't really made to be pretty. It's made to be terse and simple to read once given enough learning effort, and it's made to be a consistent keyboard typable notation.
- jollybean 5y agoIt's meant to be efficient, not pretty. It's catching on within the AI community because the syntax matches well to the kinds of matrix operations common in that field. I think Nvidia's next chip is going to have a compiler for jlang. Also, I think they are starting to use it as an 'entry level' language for kids, you know, like grade school.
- forgotpwd16 5y agoTbh that file says even less. It's like in discussion about pandas giving a link to read_csv.py in pandas source.
- jenny91 5y agoIt tells me everything I need to know about this language!
- mlochbaum 5y agoHere's one of the more central files that ties into how a Jd database is laid out: https://github.com/jsoftware/data_jd/blob/master/base/common.ijs https://github.com/jsoftware/data_jd/blob/master/base/common... Not that I claim anyone in particular can read it of course. Jd uses a hierarchy of folder, database, table, column that's handled with an object system to share code between them. A folder is just a place to put databases and hardly needs to add anything, while the other levels have a lot of extra functionality. As an inverted database, Jd stores each column in a file, and accesses it using memory mapping. https://github.com/jsoftware/data_jd/blob/master/base/folder.ijs https://github.com/jsoftware/data_jd/blob/master/base/folder... https://github.com/jsoftware/data_jd/blob/master/base/table.ijs https://github.com/jsoftware/data_jd/blob/master/base/table.... (I designed this system when I did some of the early work to turn JDB into Jd as a summer intern)
- legalcorrection 5y agoI'm intrigued but skeptical of this bit: Jd is a columnar (column oriented) RDBMS. Most RDBMS systems are row oriented. Ages ago they fell into the trap of thinking of tables as rows (records). You can see how this happened. The end user wants the record that has a first name, last name, license, make, model, color, and date. So a row was the unit of information and rows were stored sequentially on disk. Row orientation works for small amounts of data. But think about what happens when there are lots of rows and the user wants all rows where the license starts with 123 and the color is blue or black. In a naive system the application has to read every single byte of data from the disk. There are lots of bytes and reading from disk is, by orders of magnitude, the slowest part of the performance equation. To answer this simple question all the data had to be read from disk. This is a performance disaster and that is where decades of adding bandages and kludges started. Jd is columnar so the data is 'fully inverted'. This means all of the license numbers are stored together and sequentially on disk. The same for all the other columns. Think about the earlier query for license and color. Jd gets the license numbers from disk (a tiny fraction of the database) and generates a boolean mask of rows that match. It then gets the color column from disk (another small fraction of the data) and generates a boolean mask of matches and ANDS that with the other mask. It can now directly read just the rows from just the columns that are required in the result. Only a small fraction of the data is read. In J, columns used in queries are likely already in memory and the query runs at ram speed, not the sad and slow disk speed. Both scenarios above are simplified, but the point is strong and valid. The end user thinks in records, but the work to get those records is best organized by columns. Row oriented is slavishly tied to the design ideas of filing cabinets and manila folders. Column oriented embraces computers. A table column is a mapped file. What's the other side of this argument?
- 0des 5y ago> Early adopters of Jd are assumed to have a J background and documentation and tutorials depend on that background. All 12 of us are jumping up and down saying "it's our time, finally the day has come"
- avg_dev 5y agoHahahah as someone who just took a course using SML I am with you buddy.
- jdshupe 5y agoSeeing J at the top was indeed a jump up and down moment.
- 0des 5y agoWe should have a secret handshake or some type of insignia to better signal to our peers. I've tried draping a J colored kerchief out of my back pocket but the results so far are not great, it appears there is more anti-J sentiment than I'd imagined, as I get harassed unduely in certain areas of town. May have to switch to maybe a hand gesture based signaling that can be done on the fly to signal allegiance.
- recuter 5y agohttps://www.atlasobscura.com/articles/hobo-code https://www.atlasobscura.com/articles/hobo-code
- deleted 5y ago[deleted]
- marcodiego 5y ago"Jd source is largely J code and that code is open and available to licensed users." License?
- anonu 5y agoIt's sort of misleading because J is closed source
- moonchild 5y agoJ is fully opensource: https://github.com/jsoftware/jsource https://github.com/jsoftware/jsource Most of jd's source is publicly available: https://github.com/jsoftware/data_jd https://github.com/jsoftware/data_jd
- SparkyMcUnicorn 5y agoIs it? https://github.com/jsoftware/jsource/blob/master/license.txt https://github.com/jsoftware/jsource/blob/master/license.txt
- moonchild 5y ago> J SOURCE can be used under a commercial license from Jsoftware, in which case the terms and conditions of that license apply. > OR > J Source can be used under GNU General Public License version 3, in which case the terms and conditions of that license apply. Seems pretty clear to me.
- Shared404 5y agoAs a side note, I really love the choice to dual license, and wish it were offered more often.
- jenny91 5y agoIt's extremely common: license it under GPL/AGPL or some other very copyleft license; get contributors to sign a CLA, then offer the library with hefty license fees for non-FOSS projects.
- anonu 5y agoJd has been around for a while. Buy is it production ready? I'm still looking for an open source replacement to kdb, that matches kdb's speed and featureset.
- yiyus 5y agoJd is not open source.
- kokizzu2 5y agoclickhouse '__')
- swasheck 5y agoi really dislike clickhouse for anything less than rudimentary analysis, but appreciate that it’s fast for that.
- nimrody 5y agoCan you give some tips on what do you mean by "less than rudimentary analysis"? Considering adopting Clickhouse and wondering whether we will encounter problems down the road.
- swasheck 5y agothe biggie for me was that analytic window functions are either non-existent or experimental and must be achieved with array function hacks. it does have nice built-in skew and kurtosis functions, though.
- pritambaral 5y agoClickHouse recently landed Window Functions in GA.
- ZeroCool2u 5y agoYeah, KDB is... Not super fun. I've been looking at TimeScaleDB recently, because it's just a PostgreSQL plug-in it seems nice and simple, but I haven't actually compared them directly yet.
- jve 5y agoI know nothing about J and JSoftware, but this reads like an Aprils fools joke. Is it? > In a naive system the application has to read every single byte of data from the disk. ... To answer this simple question all the data had to be read from disk. This is a performance disaster and that is where decades of adding bandages and kludges started. .... Think about the earlier query for license and color. Jd gets the license numbers from disk (a tiny fraction of the database) Ofcourse that data has to be read from disk. Well, for simple or aggregate queries he may gain performance. Moreover, as other commenter has commented, you can organize data in columns in MSSQL too for aggregations: https://docs.microsoft.com/en-us/sql/relational-databases/indexes/columnstore-indexes-overview?view=sql-server-ver15 https://docs.microsoft.com/en-us/sql/relational-databases/in... > columns used in queries are likely already in memory and the query runs at ram speed, not the sad and slow disk speed. ... Jd performance is affected primarily by ram. Lots of ram allows lots of rows Any other RDBMS can have sensible indexes that satisfy your queries. And, surprise, your data also lives in RAM once you read it. > You can backup a database or a table with standard host shell scripts, file copy commands, and tools such as tar/gzip/zip.... If you understand backing up file folders, then you pretty much understand backing up Jd databases. And... throw data consistency out of the window? I'm reading and I'm "not getting" the selling point - why is this better? Okay, I read that things are files. SQLite is also a file if physical format is a concern.
- OskarS 5y ago> Ofcourse that data has to be read from disk. Well, for simple or aggregate queries he may gain performance. Lets say you want to access 2 columns out of 100 in a particular table. In a row-oriented database, you have to read the full rows off the disk, which means that you have to read 98 pieces of data off the disk that you have no use for, a total waste of I/O. In a columnar database, you don't have to do that, you just read off the relevant columns. This is VERY similar to the "array of structs"/"struct of arrays" argument in gamedev (and related high performance fields), it's the same kind of tradeoff: slightly more complicated data layout traded in for much more efficient reads. In addition: if you have a columnar database, you can employ compression in a much more efficient manner. If you have 10 million rows with the same (or very similar) data in a column, you can compress that to a fraction of the size. This messes with indexes, but it's often worth it because it VASTLY speeds up aggregate calculations. Row-based and column-based databases have different tradeoffs and advantages, and it's not quite as clear-cut as the article makes it seem. But it's certainly no April fools joke: columnar databases (for many tasks, particularly aggregates) can vastly outperform row-oriented databases. This is why Google BigQuery is columnar, for instance. Another good example is kdb+ (which this is clearly based off of), which is widely used in places which value quick time-series aggregates (Wall Street, being the obvious example). The article is a bit over the top and one-sided, but it doesn't say anything that is particularly controversial. You might wanna read up on columnar database systems: https://en.wikipedia.org/wiki/Column-oriented_DBMS https://en.wikipedia.org/wiki/Column-oriented_DBMS
- kokizzu2 5y agoso any benchmark against clickhouse?
- mlochbaum 5y agoThe environment around Jd has changed a bit since it was young! Jsoftware[0] announced it in 2012, and this particular page has been effectively the same since it was created in 2017 (I suspect this was a page move, and the content is somewhat older). In these early days the column-oriented database was quickly gaining popularity but still obscure, which is why there's this "Columnar" section that goes to so much trouble to explain the concept. Now the idea is well known among database users and there are lots of other options[1]. The history goes back further, because column-oriented is the natural way to build a database in an array language (making a performant row-oriented DBMS would be basically impossible). This is because a column can be seen as a vector where every element has the same type. A row groups values of different types, and array languages don't have anything like C structs to handle this. In J, Jd comes from Chris Burke's JDB proof-of-concept (announced[2] 2008, looks like), and the linked page mentions kdb+ (K) and Vstar (APL). KDB, first released in 1993, is somewhat famous and gets a mention on Wikipedia's history of column-oriented databases[3]. [0] Company history: https://aplwiki.com/wiki/Jsoftware https://aplwiki.com/wiki/Jsoftware [1] https://en.wikipedia.org/wiki/List_of_column-oriented_DBMSes https://en.wikipedia.org/wiki/List_of_column-oriented_DBMSes [2] https://code.jsoftware.com/wiki/JDB/Announcement https://code.jsoftware.com/wiki/JDB/Announcement [3] https://en.wikipedia.org/wiki/Column-oriented_DBMS#History https://en.wikipedia.org/wiki/Column-oriented_DBMS#History
- bla3 5y agoMeta comment: I tried to click the link but since the title is so short and my mousing not very precise, I accidentally clicked the upvote arrow. I then clicked "unvote" and tried again, but the same thing happened. The third time round, I managed to click the link. Takeaway: Very short titles might get you some upvotes from clumsy users :)
- raphaelj 5y agoThe same happened with me...
- version_five 5y agoYes! I came here to post the same thing - the link didn't work for me, and I inadvertently upvoted as well. I assume the article has merits of it's own, but I do notice a huge ratio of votes to comments, I guess some are inadvertent. It would be interesting to search through other very short titles and look at the ratio of comments to votes vs others above some vote threshold... (I'm on my phone or I'd try) - edit, is there a regex search for HN anywhere? It looks like algolia doesn't support them
- rkalla 5y agolol same and would have never thought to comment on it because obviously user-error, then I saw this and thought twice.
- nathan_compton 5y agoI've programmed in J professionally (admittedly not for all that long) as a data scientist and, coincidentally, have just completed a small analysis using J as part of an internal workshop about data analysis I am planning. I typically work in R and Python and I have to say that at this stage there is almost no reason I would pick up J to do any work. Unless code-golf level conciseness is your only goal, these other platforms offer superior performance, clarity, ease of use, access to libraries and are, as programming languages, substantially better designed. I say this as a great lover of function-level programming and as a J enthusiast. I would say I am quite familiar with J's programming paradigm and conceptual widgets and doodads (I know the verbs, nouns, adverbs and conjunctions and can use them appropriately). I even remembered a pretty good portion of the Nuvoc. But doing even the simplest analysis in J was _excrutiatingly_ slow and inconvenient compared to using R and the tidyverse (in particular, I missed dplyr and ggplot). The tidyverse CSV readers are, for example, much faster and smarter and more convenient and informative than anything you'll get from the J universe. I love vector languages but at this point J can't compete with the major platforms for data analysis. Its less convenient, often _slower_, much more low level, strange, and its library situation is anemic at best. I recommend learning J because it will expand your mind, but I can't imagine picking it up for real work.
- VHRanger 5y agoThe fact there are vector languages in subsets of python (numpy, pandas, etc.) and R. And these already have great large columnar dataset support (eg. Apache Arrow) And an open source community intent on developing and maintaining the ecosystem.
- nathan_compton 5y agoOne of the nicest thing about J is the notion of verb rank. For non-J-programmers, you can apply a rank to a verb and this effects how the verb operates on its vector operands. A rank of zero means "operate on the entire object" whereas a rank of 1 means "operate on the (1) elements of the operands. Other ranks change the meaning of what counts as "an element." However, like most things in J, support for this excellent idea (which eliminates the need for most looping constructs and can be very performant) is irregular: it is limited to monadic and dyadic verbs. Nothing about verb rank forbids functions which accept more than two arguments, but the idea of a function which accepts more than 2 arguments is poorly supported in J (the idiom is to pass a boxed array to a monad, but the boxing of the items to be passed makes supporting rank behavior for the "arguments" impossible or absurdly complicated. Other beefs with J: J doesn't have first class functions as such. While you can represent functions as "nouns" in a few ways, you cannot have (for example) an anonymous reference to a function as a thing unto itself (you may denote a verb tacitly in a context where you need a verb, however, but this is not the same thing). If you want to pass around verbs in a way familiar to you as a contemporary programmer you have to use "adverbs" and "conjunctions" which are just higher order functions which (more or less) return verbs. But adverbs and conjunctions have their own peculiarities and restrictions (not the least of which is that they are not themselves verbs or nouns and thus cannot be passed around either). In contemporary programming languages the verb/adverb/conjunction space would just be represented by "functions" and to great effect. As a functional programmer and Lisp guy, I find the limitations on "verbs" very frustrating in J. J's error messages are also bad, never more than a few words. There are some great ideas in the language, but it feels very old-fashioned and out of touch. What I would like to see is a "array scheme." A lexically scoped Scheme-like language where every object is an array and function argument slots can be independently "ranked" to support the elimination of loops over array arguments. I'm too busy to put this together, but it would be great to have if you wanted to fiddle with arrays for some reason but could do without any library support for actually doing data analysis.
- michaelmcmillan 5y agoThis section was interesting! Somehow I've never realized that row oriented storage is orthogonal to how disks work... Jd is a columnar (column oriented) RDBMS. Most RDBMS systems are row oriented. Ages ago they fell into the trap of thinking of tables as rows (records). You can see how this happened. The end user wants the record that has a first name, last name, license, make, model, color, and date. So a row was the unit of information and rows were stored sequentially on disk. Row orientation works for small amounts of data. But think about what happens when there are lots of rows and the user wants all rows where the license starts with 123 and the color is blue or black. In a naive system the application has to read every single byte of data from the disk. There are lots of bytes and reading from disk is, by orders of magnitude, the slowest part of the performance equation. To answer this simple question all the data had to be read from disk. This is a performance disaster and that is where decades of adding bandages and kludges started. Jd is columnar so the data is 'fully inverted'. This means all of the license numbers are stored together and sequentially on disk. The same for all the other columns. Think about the earlier query for license and color. Jd gets the license numbers from disk (a tiny fraction of the database) and generates a boolean mask of rows that match. It then gets the color column from disk (another small fraction of the data) and generates a boolean mask of matches and ANDS that with the other mask. It can now directly read just the rows from just the columns that are required in the result. Only a small fraction of the data is read. In J, columns used in queries are likely already in memory and the query runs at ram speed, not the sad and slow disk speed. Both scenarios above are simplified, but the point is strong and valid. The end user thinks in records, but the work to get those records is best organized by columns. Row oriented is slavishly tied to the design ideas of filing cabinets and manila folders. Column oriented embraces computers. A table column is a mapped file.
- thethimble 5y agoIf you’re interested in this thought, check out Martin Kleppman’s book DDIA where he explains storage concepts like this and many more. One of the best architecture books out there!
- derefr 5y agoI would not that this query behavior (sorted data columns bitmasked together) is further orthogonal to primary-data storage representation. For example, Postgres can give you this same behavior if you declare a multi-column GIN index across the columns you want to be searchable.
- plibither8 5y ago763 (and counting) votes, no. 1 on the frontpage, ...and only 83 comments? This is one of the most skewed ratios I've seen on HN.
- upwardbound 5y agobla3 figured out that it's because the link text "Jd" is so short, and people are clicking the upvote button by mistake. https://news.ycombinator.com/item?id=30906989 https://news.ycombinator.com/item?id=30906989
- simonpure 5y agoThere's a wonderful podcast about array languages - https://www.arraycast.com/ https://www.arraycast.com/ Lots of great stories about software engineering besides talking about the different dialects of array languages.
- pastaking 5y agoSerious question: Why does this have so many upvotes? As a layman I have never heard of J or Jd before, someone please provide some context?
- deleted 5y ago[deleted]
- guidoism 5y agoJ is an APL language. APL is the coolest language you’ve never heard of. It’s mind blowing in the same way people talk about Lisp, but more so since the concepts are so alien to most programmers.
- upwardbound 5y agobla3 figured out that it's because the link text "Jd" is so short, and people are clicking the upvote button by mistake. https://news.ycombinator.com/item?id=30906989 https://news.ycombinator.com/item?id=30906989
- jpf0 5y agoI did some work to compare Jd to data.tables and found that it was more performant in some instances such as on derived columns, and approximately equally performant on aggregations and queries. Jd is currently single-threaded, whereas multiple threads are important on some types of queries. I tried to further compare with Julia DB at the same time (maybe a year ago) and found that was incorrectly benchmarked by the authors and far slower than both; that might be different now. Jd is more equivalent to data.tables on disk; Clickhouse is far better at being a large-scale database. Rules of thumb on memory usage: Python/Pandas (not memory-mapped): "In Pandas, the rule of thumb is needing 5x-10x the memory for the size of your data.” R (not memory-mapped): "A rough rule of thumb is that your RAM should be three times the size of your data set.” Jd: "In general, performance will be good if available ram is more than 2 times the space required by the cols typically used in a query.” Re CSV reading, Jd has a fast CSV reader whereas J itself does not. I have written an Arrow integration to enable J to get to that fast CSV reader and read Parquet.
- hnrj95 5y agoare there benchmarks against kdb+ and/or shakti?
- jollybean 5y agoI wonder when Java or Swift will finally get around to adopting 'self effacing references'. It's 2022. [1] https://code.jsoftware.com/wiki/Vocabulary/SpecialCombinations#Self-Effacing_References https://code.jsoftware.com/wiki/Vocabulary/SpecialCombinatio...
- yawaramin 5y ago'Open source' Is Jd licensed under an open source license? Because otherwise I don't think they should claim to be 'open source'.
- beyzakaratepe 5y agoOıhh
- hestefisk 5y agoIs this somehow similar to Adabas / Natural?