5 ms·
The most common "perform operation on bulk data simultaneously" language in wide use today is SQL. Joe Celko coined a phrase for this: think in sets. Not in lo
by jacques_chester 13y ago
The most common "perform operation on bulk data simultaneously" language in wide use today is SQL.
Joe Celko coined a phrase for this: think in sets. Not in loops, not in linear instructions -- but in sets, being transformed into other sets. It's a very powerful way of thinking. Most of the time, you don't need ordered evaluation and so loops are imposing unnecessary guarantees on you.
On the more general problem of our collective historical ignorance ... yes. That is us.
Well not entirely. That decades of the relevant literature is locked up in Fort ACM and IEEE Castle has a lot to do with it.
- jamii 13y agoThe BOOM guys have also shown that thinking in terms of unordered collections translates much better to distributed systems, where guaranteeing order is expensive: http://boom.cs.berkeley.edu/ http://boom.cs.berkeley.edu/
- gtani 13y agoThe other day i thought i had found a textbook chapter on stream algo's/online learning, but Chapter 17 has yet to be written http://ciml.info/ http://ciml.info/
- stass 13y agoBOOM project came up with a very impressive ideas. Their half page long paxos implementation in Overlog blew my mind. I don't think they new Bloom language is as expressive, however.
- beagle3 13y agoI'm going to read about BOOM later, but can you give an executive summary about why that is? People who subscribe to the Relational Database dogma often claim that sets are superior to vectors, and yet despite asking a lot of them I never received an example -- whereas I have numerous real life examples in which vectors are superior.
- jacques_chester 13y agoThe main thing is that order isn't always necessary, but maintaining order isn't free. So if it isn't necessary, why pay the cost? Moreover, the order you want will change from case to case. Today I care about the alphabetical order of names, tomorrow about the order of dates, next week I am most interested in the top 100 transactions by value. Imposing a single order imposes an access pattern which may or may not hold true. With sets I delay that binding until the last possible moment, when I understand it best of all.
- beagle3 13y ago> The main thing is that order isn't always necessary, but maintaining order isn't free. > So if it isn't necessary, why pay the cost? But if you use the APL/K approach, you don't pay the cost. There is in order to every result, but you can only rely on it if you know it (or force it, by some sort). Specifically, my question is this: The order of results of "select ... where" without an order-by clause in SQL is never guaranteed. The order of a ksql/q result is sometimes guaranteed, and can be used. Where's the cost of just giving every result row a running index, which some operations can rely on / change, and most ignore?
- jacques_chester 13y agoThat the order isn't guaranteed is a feature, not a bug. It frees the implementation to collect records in the most efficient fashion and from having to maintain ordering. > Where's the cost of just giving every result row a running index, which some operations can rely on / change, and most ignore? The cost is that you need to keep an index updated. A trite observation, but there it is. Sometimes that cost is worth paying, sometimes it isn't. SQL doesn't forcefully extract that cost, it lets the database programmer or administrator decide, based on local conditions, what is best.
- beagle3 13y ago> That the order isn't guaranteed is a feature, not a bug. It frees the implementation to collect records in the most efficient fashion and from having to maintain ordering. Yes, I understand the theoretical argument. I'm looking for real world examples where that actually helps anything, or theoretical examples where it is superior (in the sense that "there is no way to assign order which would not move the problem to a higher complexity class"). > The cost is that you need to keep an index updated. A trite observation, but there it is. Sometimes that cost is worth paying, sometimes it isn't. SQL doesn't forcefully extract that cost, it lets the database programmer or administrator decide, based on local conditions, what is best. My point is that ksql does the same -- except that in most cases, the cost is 0, so you get the order "for free"; when the cost is not zero, ksql doesn't make guarantees, and you'll need to ask for ordering, just like in sql. Every sql engine already assigns order in one way or another (ROWID, etc.), except it is unpredictable, non-standardized, and doesn't provide most of the benefits even though it is there. And the "no order" default often does have a cost; e.g., if you are trying to store e.g. tax brackets in your database and find out which tax bracket a salary falls into: Either you store both lower and upper bound of bracket in a record, which means you can't normalize (and it is hard to express the integrity requirement of "no gaps" as a constraint) - or you do normalize, and then every query is a self join or correlated query that many optimizers won't handle properly. Whereas in ksql, you store it normalized (just start-of-bracket), order by start-of-bracket, and then get the bin that corresponds to your question.
- seanmcdirmid 13y agoArray programming languages predate relational or even most functional languages (sans Lisp). It seems like relational and functional languages were heavily inspired by array languages, if you go back and read some of the old papers.
- jacques_chester 13y agoI think that's a reasonable conclusion to draw. I guess the insight of array languages was applying over collections, and the relational's insight is that you're performing operations on sets (in a mathematical sense).
- hamzamu 13y agoIt definitely is a powerful way of thinking. And yet, implementing transformations used for event sequence/path/time series/complex nested data structure analysis in SQL is like using tweezers for the job. Set thinking is powerful, SQL has limitations.
- jacques_chester 13y agoYes, SQL's biggest weakness is time and temporality. That's been an ongoing problem since the beginning and it leads to a lot of stress and trouble. I'm against "complex nested data structures", though. In my experience they are abused and there is always a normalised alternative.
- hamzamu 13y agoAgreed on the nested data structures. Sometimes you have to work with them though. I finished a project where I had to integrate/analyse data from a factory system originally developed in in the 60s (cobol). Even normalizing 'complicated nested data structures' != fun.
- dajohnson89 13y agoShortly after Aaron Schwartz's death, there was a lot of resentment for the more traditional information-distributors, such as Elsevier and other big publishing companies. But you make a great point about ACM and IEEE. As a student (and now an engineer in industry) in the CS and applied math fields, I can't count how many times during research where I see a useful article behind an IEEE/ACM paywall. Are you aware of any efforts/campaigns to demand more open access from these two behemoths, who pretend to be on our side?
- dasil003 13y agoI presume you mean Aaron Swartz?
- voidlogic 13y agoMaybe I am old school and blunt but IMHO: >As a student (and now an engineer in industry) in the CS and applied math fields, I can't count how many times during research where I see a useful article behind an IEEE/ACM paywall. I have never heard of a college that doesn't pay for this access for their students... >Are you aware of any efforts/campaigns to demand more open access from these two behemoths, who pretend to be on our side? If you are a computer science related professional you should be an ACM or IEEE member and thus have access to their digital libraries. If you want to have access to guild resources, join the guild.
- jasonwatkinspdx 13y ago> I have never heard of a college that doesn't pay for this access for their students... The vast majority of educational institutions are not in the US and do not have anywhere near the funding to pay rent to the ACM/IEEE. Every dollar they spend on something like pay-wall access would be better spent investing in their students. > If you want to have access to guild resources, join the guild. Guilds are cartels. That's why we don't have them anymore except in the more neutered form of trade unions. But also, the content ACM et all charge for is not their original contribution. Other people write the papers and give them to the ACM for free. Other people review the papers for the ACM for free. The ACM takes in about 60mm each year but only spends 40mm on it's various activities. At one time something like the ACM was necessary purely for pragmatic reasons. But now we have the Web, and there is no reason someone should be earning a 20mm profit for artificially restricting the flow of knowledge in the discipline. By comparison the total budget of arXive is around 400k. The ACM could make the digital library free not make a dent in funding for any of the actually useful stuff they do. It's also unconscionable that the ACM attempts to police 3rd parties linking to copies of papers published on author's homepages. This issue specifically is why I canceled my membership (after writing a letter of complaint that received the most self-aggrandizing response you can imagine) I'm more charitable to the IEEE as they do a lot of standards work, which does benefit from having a centralized bureaucratic organization.