6 ms·
100x better approach to software?
- thedeuce 15y agowhy do people continue to spam this nonsense blog! this guy is the definition of drivel.
- PaulHoule 15y ago100x better is a bit much. 10x better, maybe.
- skarayan 15y agoIn the large projects that I have seen, when I think about how much of the work being done is the core product vs frameworky stuff and/or integration, I think 100-1000x is more accurate. The pure business logic constitutes a very small part in comparison, but the type of environment/company also matters. Startups tend to have less fluff than large enterprises.
- tgflynn 15y agoIt seems to me that a huge amount of developer effort tends to be expended in mapping data between different data structures as required by different algorithms or API's. If a way could be found to automatically discover a set of feasible and approximately optimal data transforms to meet the requirements of the operations that need to be performed it seems like very large productivity gains might be obtained.
- gruseom 15y agoa huge amount of developer effort tends to be expended in mapping data between different data structures as required by different algorithms or API's This is indeed one of the biggest sources of complexity. A programmer I used to work with called it "meat grinding".
- mtraven 15y agoI call that "plumbing", since it's basically just getting different sizes and shapes of pipes to connect. But "meat grinding" is good too.
- gruseom 15y agoWhen I hear people say "plumbing" it seems to mean not data conversion but the internal workings of a system, especially anything infrastructurey. So "meat grinding" seems more specific. But I like it because it's appropriately disdainful. Who wants to grind meat all day? Dostoevsky said that if you want to drive a man insane all you need to do is sentence him to hard labor moving a pile of dirt from one side of a prison yard to another – and then back again.
- tptacek 15y agoFitting data in whatever form it has taken at point X in a program into the form that API Y expects is often more like meat ungrinding.
- AdrianRossouw 15y agoadding more programmers doesnt make it faster to develop things. it just means more time in meetings.
- me_again 15y agoSome software systems are too large for one person, however talented, to write. There's a reason not every line in the Linux kernel is written by Mr Torvalds.
- johnpmayer 15y agoBut that's maybe not the best example; wasn't it originally just him? I think GNU+Linux might better demonstrate what you're trying to say.
- me_again 15y agoIt was originally, but I don't think it's controversial to say it wouldn't be in its current state as a 1-man project.
- aaronblohowiak 15y agoI believe Linus has said that lately, he doesn't write code so much as review and discuss patches.
- DennisP 15y agoHaven't watch Kay's lecture yet, but I'll mention that he and some colleagues are about five years into a project to write an entire computer system, including OS, gui, networking, and programming tools, in 20K lines of code, using some very innovative techniques. Compare to the millions of LOC in other systems and it's easily a two orders of magnitude improvement. One example: a TCP stack is about 20K LOC in C. In their system it's 160 LOC: http://www.itsoftwarecommunity.com/author.asp?section_id=1624&doc_id=240682 http://www.itsoftwarecommunity.com/author.asp?section_id=162... Their language allows easy syntax modification within scopes, so they defined a syntax that matches the IETF ascii diagrams, and simply pasted in the diagrams from the standards. You can read their papers here: http://www.vpri.org/html/writings.php http://www.vpri.org/html/writings.php
- udp 15y agoIs any of their source published? I'm interested to see this.
- DennisP 15y agoLinks to sourcecode here: http://www.vpri.org/vp_wiki/index.php/Main_Page http://www.vpri.org/vp_wiki/index.php/Main_Page OMeta is the configurable syntax project, and there's a lot of other stuff.
- tgflynn 15y agoBut how many lines of code does it take to implement the language that allows them to implement TCP in 160 LOC ?
- mbrubeck 15y agoThat language is OMeta. There are several implementations; the latest JavaScript compiler is just a few hundred lines. http://www.tinlizzie.org/ometa/ http://www.tinlizzie.org/ometa/ The compiler is self-hosting; much of that JS code is actually compiled from OMeta source. For another example, the Common Lisp version includes fewer than 200 lines of OMeta source and fewer than 300 lines of Lisp source. OMeta was a very eye-opening language for me because of the great expressiveness it gets from its simple but novel approach. (I've written code in Haskell, Prolog, Scheme, OCaml, and plenty of other languages, but OMeta is not like any of them.) If you are interesting in programming languages and/or parsing, Warth's thesis is a great read.
- alexchamberlain 15y agoPhilosophical question: Is time in meetings wasted?
- scott_w 15y agoShort answer: no Long answer: it depends Some meetings are a complete waste of time, but they're one way of bringing people's thinking closer together. Meetings probably don't get everyone on the same page, but if they can get people reading from the same book, it was probably worthwhile. As an example, in our organisation, we use meetings as a way of getting information from development to our sales/support teams and vice-versa (what features are selling, where are the pain-points). Whoever is chairing the meeting will make efforts to remove people who don't need to be there e.g. if a meeting moves into technical discussions, we may ask the sales representatives if they want to go to avoid being bored.
- Aqua_Geek 15y ago> If the same team of developers could rewrite the system from scratch using what they learned from the original system? We're doing this right now at work - completely rewriting one of our apps from the ground up. It's been an awesome experience to look at what we did wrong in the old code base and discuss how to fix that with the new one. I definitely don't advocate doing this on every project, but unless there's been due diligence (in refactoring, consolidating duplicated code, etc) I think oftentimes we reach a point where this becomes somewhat inevitable for progress to be made. A forest fire now and again is a good thing to clear underbrush and replenish nutrients and whatnot.
- hxa7241 15y agoKay's assertion seems incredible -- he really has to prove it. It does not square with at least some research and what Brooks most perspicaciously noted in 'No Silver Bullet' -- that there is essential complexity in software, and it is the dominant factor. That really seems very opposed to anything anywhere near 100-fold general improvements. Looking at the VPRI Steps report of 2011-04, prima facie, overall, they seem to be producing a 'second-draft' of current practice, not a revolutionary general technique. People regularly say they translated their system from language A to language B and got 10-100 times code-size reductions. This seems essentially what STEPS is doing: when you look at something again, you can see how to improve it and do a much better job. But the improvement comes not from the general aspect -- the language B -- but from understanding better the particular project because you already had a first-draft. This can still be valuable work though, since it is about general things we all use. Second-drafts are valuable, and ones of important things even more so. But we should probably not infer that they are coming up with a revolutionary general way to write new software. Substantially because large scale software cannot be designed: it must be evolved. We only know what to focus a second-draft around because we muddled together the first draft and it evolved and survived. You cannot write a second-draft first. Having said that, if they find particular new abstractions, they could be very valuable. If, for example, VPRI came up with a neat way to express implicit concurrency, maybe, that might possibly be comparable to inventing/developing gargage-collection in common programming languages -- certainly useful and valuable.
- tgflynn 15y agoBut if given some set of tasks and a program A which performs them one can show that there exists a program B which performs the same tasks and is 100x smaller than program A (taking into account language/framework implementations) doesn't that show that at least 99% of the complexity of program A was not essential ?
- gruseom 15y agoKay's assertion seems incredible -- he really has to prove it. What assertion, exactly? That the personal computing stack can be implemented in 20K LOC? They've already achieved the bulk of it, so that claim no longer qualifies as incredible. Pooh-poohing this staggering achievement because they didn't reinvent personal computing while they were at it really takes the cake for moving the bar. If someone had gotten all the world's recorded music on a thumb drive in 1970, would you have complained that they didn't write the music themselves?
- mickeyp 15y agoThe issue, as always, is that you trade your "verbose" (by comparison) imperative programming language for something that let's you specify or model the behaviour of a system in very few lines of code. Impressive, indeed, until you realise that knowing how to specify what it is you want is as much a black art as it is a science. Tools like that are hard to make understandable enough to a large enough subset of developers that the investment in learning it fully will pay off. Back in University we had to so specify a minicomputer (PDP-11) in a metaprogramming language called "Maude" using its complex, Prolog-like rewriting functionality. Needless to say, the entire computer (CPU, ram bank, store) only took up maybe a hundred or so lines but boy oh boy did it take me ages of fiddling to get it just right. Languages like that are just too difficult to work with for a lot of things, and that's setting aside the inductive and provable nature of something as simple as building a minicomputer out of logic.
- its_so_on 15y agoActually there used to be a 100x better approach to software than this blog post. Yes, that meant 10000x better software. All you needed to do is break down what you were doing into steps, and pipe one step to the other with this character: | Unfortunately, one thing led to another and, well, we are 10,000 times slower now than then. whats your email address | confirm your email | get a link | pay with my payment processor | get my service Hahaha, setting this up today takes 8 hours. Approximately 10,000 times slower than it should be. Actually, someone who can set up a complete billing solution in a day is considered a hero. Amazon is the only company that even comes close to doing for the web what we had twenty years ago for local sys admin tasks. Only one lets you manage loads of complicated files (kind of cool) and the other lets you provide a service to hundreds or thousands of people (kind of awesome). where did we go wrong.
- its_so_on 15y agoI wish someone would reply instead of just downvoting, so I can at least see if I've made myself clear. It used to be that for most tasks a superuser, a real ace of an admin, used to do, they would NOT write new programs: instead they would stitch together old ones. I wish I could find it but am having trouble just now, but there was an experiment done among various programming groups completing the same task. You had people doing it in various scripting languages, C++, Java, whatever. The way I recall (again, having trouble finding this), the team or person/approach that won handily was the one using Unix from the standard commandline interface (e.g. bash) - no scripting or programming at all! Where instead of writing a program to do it all, the person or group simply used standard Unix tools, piping them together etc, until the problem was solved. This approach was by far the fastest. I'm saying, these days on the web we don't really have the same thing when you develop a new application. We don't have a "Unix of the web" - though, again, AWS and Amazon's ambitions on e.g. payment, database, etc, seem to be vaguely in that direction - which is far more productive than writing a piece of software. No matter how productive -- HOW PRODUCTIVE! - you are at writing a script to bill a user ten dollars, you can never -- NEVER! -- be as fast as typing "| bill 10.00" where "bill" takes an email address on standard input. That Unix program does not exist. The way the web is developing, it does not look like it will exist. This was my point. I guarantee you that typing "| bill 10.00" is nearly ten thousand times as fast as writing any program in any programming language that does that. Unix works because someone took the time to write programs that can be stitched together at the command prompt (or from a script). The Internet just doesn't work that way. The blog post I'm referring to ends with: "Is that just the way things must be, that geniuses are in short supply and ordinary intelligence doesn’t scale efficiently? How much better could we do with available talent if took a better approach to developing software?" I say, the problem is that the geniuses are no longer creating the "Unix programs" of the web. They are writing software, i.e. lines of code, they are not writing web "utilities". If all the geniuses got together and gave me the top thousand things you need an API for, and instead worked on making them a set of small, modular Unix commands, then I would be literally ten thousand times more productive than now. I could literally do in 20 seconds of typing what I can do in two days.
- mistermann 15y agoIn my relatively short time (~20 years) in the software industry (in the "enterprise software" corporate development world on the MS platform), it is my personal perception that it now takes nearing 10 times the LOC &/or classes &/or complexity in general to implement the same functionality now, using modern "improved" software architecture and design patterns, than it used to writing code the "old" way we did things when I started out. I'm deliberately very conscious of keeping an open mind and trying to discover just what it is that I don't "get" about the way things are done these days. I look at some of the incredibly complex software implementations of really very simple functionality these days, and for the life of me I just can't understand why people write things to be so complex. Now and then I just can't restrain myself and I'll ask "why didn't you just do it this way" (often with a proof of concept implementation) and the normal response is blank stares, and a changing of the subject shortly thereafter. This is not just with individuals, this happens in groups, and no one (so it seems) feels anything is out of order. Not surprisingly, implementing new features takes far longer than things used to, at least according to my perception. So, unless I am out of my mind, a 10x improvement could easily be realized (at least in the world of "enterprise" software), just by unadopting some of the practices adopted in the last decade or so.
- slewis 15y agoCan you provide a concrete examples?
- mistermann 15y agoOk, here's one extremely simple example.... Say you have a screen where you have to show sales for an individual customer. One way of doing this is, within the screen that requires this data, querying sales by that customer using the dynamic sql feature provided by the ORM you are already using. One line of code, and it exists in the source code exactly where it is consumed at runtime. Another way of doing this is writing a stored procedure that returns the same data and accepts customer name as an argument. Now, this SP has to be exposed through <n> application layers before the UI can ultimately call it. So now, you have a very specific (and obscure) stored procedure, that has to be exposed in <n> layers of application tiers. Now imagine you need sales for an individual customer, but now only for products from a particular manufacturer, and imagine the two ways that could be implemented. Rinse and repeat for every other distinct data retrieval use case you have. This is just one type of an implementation style that can be made excessively complex. Now throw some new design patterns that you just read about into the mix. Now, on top of your <n> app tiers, you now get several additional wrapper classes on top of all this, so a simple select from your database ends up with a call stack 12 levels deep with 6 levels of indirection. And if you suggest perhaps there's an alternate way, you're an outcast. At the very least, one would hope you could have an open-minded discussion about the relative merits of the two approaches. But this is just my experience, YMMV.
- InclinedPlane 15y agoMost coding tends to add a significant amount of accidental complexity. Take that into account across every layer of development from the requirements and design phase onward and you get an exponential explosion of complexity, and thus an explosion of code size, of bugs, etc. More so when you consider that the typical method of dealing with poor or leaky abstractions is to add another layer of abstraction.
- ced 15y agoFrom Alan Kay's interview: The problem with the Cs, as you probably know if you’ve fooled around in detail with them, is that they’re not quite kosher as far as their arithmetic is concerned. They are supposed to be, but they’re not quite up to the IEEE standards. Does anyone know what he's referring to?
- wmf 15y agoI wouldn't be surprised if he's talking about integer overflow, which has bitten people in surprising ways for decades.