5 ms·

There is no shortage of excellent work being done today, too. With the large systems that are being created today I would take Python over APL or J any day, can

by stiff 13y ago

There is no shortage of excellent work being done today, too. With the large systems that are being created today I would take Python over APL or J any day, can you imagine building a company like Google or Facebook on top of APL, even assuming you would manage to find a team of good APL programmers? It is not a language fit for cooperation nor for large systems, it is a problem-solving language, back from the day where people used computers for relatively small tasks. Plenty of the good things in APL are now available in other forms anyway. NumPy with it's powerful set of linear algebra primitives and the huge set of implemented algorithms is just as great at data crunching. Want mmapped files? No problem: http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html http://docs.scipy.org/doc/numpy/reference/generated/numpy.me... That's not to say APL is not worth studying, it is built on powerful ideas. But to study the masters is to criticize the masters, not only blindly admire them. And history is not all romantic.

hat_matrix 13y ago
And R also has mmap, which is the the relevant package to reference rather than bigmemory in the original blog post?
- scottlocklin 13y ago
  bigmemory is a better package.
yummyfajitas 13y ago
There are big companies built on top of APL like languages. One example is Morgan Stanley (K). You are correct that not everything is built on K - typically there are gigantic concurrent enterprisey systems build in java/c++/scala which shell out to a computational core built in K. You are also right that Python/Numpy is getting to be a good alternative to K, giving you less performance but better cooperation. That's why BankAm is building their SecDB clone on top of it.
- oddthink 13y ago
  I definitely miss having the Morgan Stanley site license for K at my disposal. I can't quite justify the money to get a license, just for me, so I've been slowly assembling a poor-man's substitute built on numpy, pandas, and hdf5. K itself isn't as consistent as I'd like (odd corners like the auto-casting arrays, especially dict arrays to tables, and skipping NaNs in sums), but it's a great place to start. J really needs a table or dictionary data type, or at least the tools to make one yourself.
  - stiff 13y ago
    Mind to share some of the background of yours? How did you get to know it, what were you doing with it, how did you learn it, what did you like about it?
    - oddthink 13y ago
      Sure. At MS, I used it while working on mortgage prepayment and default models. Before we got the K license, I had been using SAS and R. There, I had been dumping aggregated (or sometimes not) CSV files from the database (Sybase) and running my model fits based on those. I had a bit of exposure to A+ then, trying to debug bits of the interest rate model subsystem. I hated it. We were mostly using A+ as the wire protocol at that point, and I had endless problems getting the APL fonts set up, and so on. Non-ASCII was a huge black mark. Once we got the license and expanded the team working on the models, we decided to switch over to using kdb+ for the basic data store. That switch was like going from night to day. I could operate over the entire dataset, rather than just on a subset. I could query things on the fly with real aggregations. Things that were a huge pain (and slow) using SQL were suddenly fast and easy, like "calculate the average default rate grouped by FICO in 25-point buckets, for loans issued after 2005" or "backfill missing data from the first non-missing observation." The first example can be done in SQL, but it takes a whole lot more typing and is (IMHO) much more error-prone. The last, I'm sure you can do it, but it's a nuisance. I first got comfortable with kdb+ as a database query language. I read "q for mortals" by Jeff Borror, and we actually had Jeff around for a while for questions. After that, it we started actually using q to estimate the models. It was great at the data aggregation and was a pretty natural fit. It didn't have anything built-in, but so we wrote a few functions for multinomial logistic regression, and that was good enough. That part didn't have a huge advantage over using something like R. However, the interaction with the large data store was far superior, better enough that it was worth it to write our own routines to do the estimation. We even used q to run the models in production. That was more of a reach, but it worked pretty well. So, I liked kdb+'s ability to handle large data sets in a speedy way. It makes a fantastic query language. I liked q's conciseness. It doesn't matter much for actual in-production code, but if you're doing research, every character counts. It seems shallow, but I think it's a real effect. Writing a half-line of q to do a paragraph of SQL is a huge win. I liked q's array operations, but by then I'd already been using Numeric and numpy and such for a decade, so those weren't really new. It was nice to be able to write them easily, as opposed to something like "np.concatenate((foo,bar))", but it wasn't really novel. Basically, it was kdb+ that sold me on q.
Joeri 13y ago
Maybe we've got it backwards. The only true problem in software development is miscommunication, and the bigger teams get the worse they perform per-programmer, with the most productivity loss experienced by the most senior team members. On large teams, the most experienced team members are often handicapped by having to operate in a context that remains comprehensible for junior team members, investing so much time in communication that they spend not luch of it on building things. that in turn produces a sort of lowest common denominator effect, where designs are deliberatey kept overly simplistic so they will be easy to communicate. The more expertise is applied to a solution, the less likely it is that people can cooperate on that codebase. The languages and frameworks of today try to minimize the need for expertise to get things done, because expertise doesn't blend well with communicating about code. I think part of the ideas in the article is that by running away from expertise we have often arrived at inefficient solutions. We scale out, with huge codebases that are like cities, sprawling and vast yet with each component isolated and easy to comprehend. Learning from the masters means understanding complex ideas which are based on scaling up, on building a project-specific grammar that encapsulates the problem domain so elegantly that it allows for one line features, at the expense of being incomprehensible to someone new to the codebase or the field.
- jacques_chester 13y ago
  This actually sorta rhymes with the design goals of Java -- a broadly featuresome language, with lots of safety nets, that could be used relatively effectively by the median programmer.
  - jdbernard 13y ago
    Having worked in Java for several years this is both a great strength and weakness in my opinion. On the one hand it keeps everyone speaking the same language as opposed to some languages where it seems each team is speaking it's own dialect. On the other hand it can shackle the truly proficient, makes it easier for complacent programmers to remain so, and masks (temporarily) the incompetence of people who should not be writing production systems.
jingo 13y ago
"To study today's programmers is to criticize them, not blindly admire them. And the present is not all romantic."

by stiff 13y ago

hat_matrix 13y ago

And R also has mmap, which is the the relevant package to reference rather than bigmemory in the original blog post?

scottlocklin 13y ago
bigmemory is a better package.

yummyfajitas 13y ago

There are big companies built on top of APL like languages. One example is Morgan Stanley (K). You are correct that not everything is built on K - typically there are gigantic concurrent enterprisey systems build in java/c++/scala which shell out to a computational core built in K. You are also right that Python/Numpy is getting to be a good alternative to K, giving you less performance but better cooperation. That's why BankAm is building their SecDB clone on top of it.

oddthink 13y ago
I definitely miss having the Morgan Stanley site license for K at my disposal. I can't quite justify the money to get a license, just for me, so I've been slowly assembling a poor-man's substitute built on numpy, pandas, and hdf5. K itself isn't as consistent as I'd like (odd corners like the auto-casting arrays, especially dict arrays to tables, and skipping NaNs in sums), but it's a great place to start. J really needs a table or dictionary data type, or at least the tools to make one yourself.
- stiff 13y ago
  Mind to share some of the background of yours? How did you get to know it, what were you doing with it, how did you learn it, what did you like about it?
  - oddthink 13y ago
    Sure. At MS, I used it while working on mortgage prepayment and default models. Before we got the K license, I had been using SAS and R. There, I had been dumping aggregated (or sometimes not) CSV files from the database (Sybase) and running my model fits based on those. I had a bit of exposure to A+ then, trying to debug bits of the interest rate model subsystem. I hated it. We were mostly using A+ as the wire protocol at that point, and I had endless problems getting the APL fonts set up, and so on. Non-ASCII was a huge black mark. Once we got the license and expanded the team working on the models, we decided to switch over to using kdb+ for the basic data store. That switch was like going from night to day. I could operate over the entire dataset, rather than just on a subset. I could query things on the fly with real aggregations. Things that were a huge pain (and slow) using SQL were suddenly fast and easy, like "calculate the average default rate grouped by FICO in 25-point buckets, for loans issued after 2005" or "backfill missing data from the first non-missing observation." The first example can be done in SQL, but it takes a whole lot more typing and is (IMHO) much more error-prone. The last, I'm sure you can do it, but it's a nuisance. I first got comfortable with kdb+ as a database query language. I read "q for mortals" by Jeff Borror, and we actually had Jeff around for a while for questions. After that, it we started actually using q to estimate the models. It was great at the data aggregation and was a pretty natural fit. It didn't have anything built-in, but so we wrote a few functions for multinomial logistic regression, and that was good enough. That part didn't have a huge advantage over using something like R. However, the interaction with the large data store was far superior, better enough that it was worth it to write our own routines to do the estimation. We even used q to run the models in production. That was more of a reach, but it worked pretty well. So, I liked kdb+'s ability to handle large data sets in a speedy way. It makes a fantastic query language. I liked q's conciseness. It doesn't matter much for actual in-production code, but if you're doing research, every character counts. It seems shallow, but I think it's a real effect. Writing a half-line of q to do a paragraph of SQL is a huge win. I liked q's array operations, but by then I'd already been using Numeric and numpy and such for a decade, so those weren't really new. It was nice to be able to write them easily, as opposed to something like "np.concatenate((foo,bar))", but it wasn't really novel. Basically, it was kdb+ that sold me on q.

Joeri 13y ago

Maybe we've got it backwards. The only true problem in software development is miscommunication, and the bigger teams get the worse they perform per-programmer, with the most productivity loss experienced by the most senior team members. On large teams, the most experienced team members are often handicapped by having to operate in a context that remains comprehensible for junior team members, investing so much time in communication that they spend not luch of it on building things. that in turn produces a sort of lowest common denominator effect, where designs are deliberatey kept overly simplistic so they will be easy to communicate. The more expertise is applied to a solution, the less likely it is that people can cooperate on that codebase. The languages and frameworks of today try to minimize the need for expertise to get things done, because expertise doesn't blend well with communicating about code. I think part of the ideas in the article is that by running away from expertise we have often arrived at inefficient solutions. We scale out, with huge codebases that are like cities, sprawling and vast yet with each component isolated and easy to comprehend. Learning from the masters means understanding complex ideas which are based on scaling up, on building a project-specific grammar that encapsulates the problem domain so elegantly that it allows for one line features, at the expense of being incomprehensible to someone new to the codebase or the field.

jacques_chester 13y ago
This actually sorta rhymes with the design goals of Java -- a broadly featuresome language, with lots of safety nets, that could be used relatively effectively by the median programmer.
- jdbernard 13y ago
  Having worked in Java for several years this is both a great strength and weakness in my opinion. On the one hand it keeps everyone speaking the same language as opposed to some languages where it seems each team is speaking it's own dialect. On the other hand it can shackle the truly proficient, makes it easier for complacent programmers to remain so, and masks (temporarily) the incompetence of people who should not be writing production systems.

jingo 13y ago

"To study today's programmers is to criticize them, not blindly admire them. And the present is not all romantic."