10 ms·

Domain Knowledge or a Lack Thereof

203 points by ajhai 14y ago

kayoone 14y ago
I am currently working on a web software project in the medical/health space. Of course its really good to know how the people that use the software work and what they expect and i am eager to learn those things, but i will not dive into a sector i am not really interested and absorb any and all information i can find, since i probably wont need it for the next project and it personally does not interest me a lot. For technical domain knowledge i agree, but it also happens that this is often of personal interest to me anyway.
kevingadd 14y ago
The lack of this kind of domain knowledge - either from being unwilling to go find it, or not being given the time and the resources to do so - is a big problem for tools development in games. People doing the scheduling and planning tend to assume that if you put enough smart programmers in a room and give them detailed specifications, they'll be able to solve all the problems faced by some artists and writers in another room, even though they don't know anything about art or writing. Sometimes this results in the kind of obvious mistakes that you might otherwise assume only happen when you're writing software for the government. In a sense it's kind of sad: In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work, or ask them questions on a regular basis to understand their issues - you're separated from them by a sales process and probably some automated support tickets and a PR guy who won't let you have open discussion in public. People building tools for game development can have their customers sitting a few desks away!
- calpaterson 14y ago
  > In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work One of the nicest things about the Agile fashion at the moment is that my customer (or a reasonable representative of him) is a couple of desks away and I really can go and ask him about what he wants. It's incredible how much time is saved by avoiding doing things he doesn't care about and how much happier he is with the end result.
- jeltz 14y ago
  > Sometimes this results in the kind of obvious mistakes that you might otherwise assume only happen when you're writing software for the government. My personal guess is that the reason there too is lack of domain knowledge. That many government projects are ran by handing out specs to the people tasked with implementing them.
- Jare 14y ago
  I always asked my tools programmers (and do myself) to spend some time regularly sitting by an artist or designers while she works, and observe. HUGE difference.
kenkam 14y ago
This particularly resonates with me. I have been working in a bank as a graduate developer for the last year and a half, getting by without much care about how instruments are priced, traded, recorded, audited, etc. To me, I was moving a button around and colouring things using WPF. To me, the "Yield" column was just another column with numbers that were supposed to be 2 decimal places and turns red if negative. All the while I was dreaming I wished I worked at Google on 'technology'. Which, in hindsight, was a little misguided. There is so much value in learning and understanding the domain in which we are working in because it makes us so much more effective. The realisation came when I became frustrated in meetings where we would discuss ideas on financial models and I could not add any value to the discussion. I then imagined working at a company like Google, let's say, doing stuff I wouldn't normally be interested in. Would I bother learning the domain knowledge? I couldn't say yes with 100% conviction. I love programming but I also want to be effective in what I do. To me now, understanding the problems we are solving is probably just as important as writing maintainable code.
gcv 14y ago
Ah yes, the tragedy of studying computing to the exclusion of everything else. Graduates with computer science degrees have an in-depth understanding of... text editors. And maybe compilers, databases, and operating systems — all items of zero value until they are applied to solving some real problems in some other field. Problems which the computer science graduate knows nothing about. Then these graduates go into the working world, and they typically can't help solve problems in economics, physics, applied mathematics, or medicine except by coding up some system to someone else's specs. The more dedicated ones make an effort to understand what they're doing. They exceedingly rare, gifted, and motivated ones might learn enough about an industry to make real contributions. I hope that our field eventually evolves to the point that programming skill becomes an accessory. People will then learn to program just like they learn to write, and use the craft to help them solve their actual problems. The profession of programmers who only know how to program should disappear like the profession of scribes who only know how to write.
- GFischer 14y ago
  I hope we'll have specialized careers someday (Medical CS, Economics CS, etc...). In many careers, it's the other way round - most physics I know taught themselves how to program, for example, but they're lacking some tools that would probably help them a lot (and physics are very close to CS). I've seen accountants do a lot of programming in Excel macros and Access too. Doctors and Economists might have less interest in learning, but academy-minded ones will really appreciate the modeling they can do with the use of computers (I think I saw a medical paper where they reinvented some basic CS concept :) ). Some of the domain training comes from practical experience - I started out working for the credit report sector, and then moved on to the insurance sector, so I've picked along some domain knowledge in those areas. Other colleagues have worked in logistics, and now know about their problems, etc...
  - eru 14y ago
    > (I think I saw a medical paper where they reinvented some basic CS concept :) Perhaps more shocking: https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/ https://fliptomato.wordpress.com/2007/03/19/medical-research...
alemhnan 14y ago
My take is that one of the main improvements in programming languages will be to reduce the gap between domain experts and programming experts. Now we should face three cases: 1) the programmer needs to learn the domain; 2) the domain expert needs to know how to develop; 3) something in between. Most of developing is done in 1). There are some brilliant cases of 2). I once talked with a biologist doing some multi sequence alignment that told me that some routine in C was quite slow and was far better rewrite that routine from scratch in assembly. I was quite speechless. In order to create better programming languages we could: 1) express domains in a better way. Actually, just be able to express domains would be really cool (and not so easy). 2) provide higher levels of abstraction.
- jacquesm 14y ago
  Every programming language from the dawn of time up to the present has made that claim at some point. This time it would be different. I think it is just a people problem. Programmers tend to seclude themselves from their customers. Whole IT departments tend to seclude themselves from their users. When you are looking for underwater welders for the off-shore industry, what is better, teaching a welder to dive or teaching a diver to weld? (that's a serious question with a real answer). It is much easier for a programmer to acquire the required domain knowledge than it is (usually) for a domain specialist to learn how to program. But you can only do so if you're prepared to 'dive in'.
  - unwind 14y ago
    My guess would be the first, teaching a welder to dive. I base that on the fact that the core value, the welds, will be higher-quality and that's more important than the diving execution. The diving might be off, but it shouldn't be that hard to learn well enough to survive and be effective. Of course, I'm neither a diver nor a welder. Did I win?
    - jacquesm 14y ago
      > Did I win? Yep. But a drowned welder is also not much ;)
Nursie 14y ago
Actually I think domain knowledge is overrated. A good hacker can pick up the domain knowledge needed to contribute to a project very quickly. Now, if you're designing a product to solve a problem in a particular domain you need the knowledge ahead of time, sure. But if you're a shit-hot coder and a fast learner, brought in to make things work, who cares?
- rcfox 14y ago
  Because, as the article clearly mentions, there's a difference between making something work and making something work right.
  - Nursie 14y ago
    I disagree that it's important unless you're the person designing the solution. If you're not (and in many/most teams you won't be) it's far less important.
    - rcfox 14y ago
      Sure, if the problem is solved, then you probably don't need to know anything. But implementing other people's solutions is only satisfying for so long.
      - Nursie 14y ago
        I don't disagree with that, but there are plenty of people that make a career out of it. In the case of ex-colleagues who find a corner at a big multinational and get told what to do, someone else translates domain knowledge for them, and they come up with tech solutions to the presented requirements. And then there are people like me who get called in because we're good at picking up large, existing, broken codebases quickly and somebody's in the sort of trouble that's best summed up as "Oh $&%£ we've got three months to deliver, the product's less than half-done and it's full of weird intermittent bugs". I'm finding it satisfying enough for now :)
        14y ago
qxf2 14y ago
The line I found most insightful: >>"Software is much more intertwined with the institution commissioning it and the people using it ..." Given that, I'm surprised at the emphasis on 'reading' as the primary source of gaining domain knowledge. There are just 2-3 mentions of speaking/collaborating with the domain experts that are driving you to build the software itself. If you are lucky enough to work with acknowledged domain experts, spend time talking to them. Pick their brain. Two techniques I use: a) forward interesting articles to them with a question or an analogous concept in my domain b) resist the urge to show them a 'better' way. E.g.: In the HealthCare domain, I learnt people don't really want quicker or lesser clicks. They would prefer a set, stable workflow. If you cater to senior-citizens, keyboard shortcuts are less important than instructions in large font. I still suck at engaging domain experts. So, I'd love to learn more techniques to hold an interesting conversation with domain experts. Ideas?
- jacques_chester 14y ago
  I can read a book faster than I can speak or hear words. In addition, books can usually be found, on almost any topic, which are intended to bring a novice up to a basic level of conversancy. Reading one (or two) such books will make discussions with the experts much more productive. Plus: it's fun to learn about new things.
  - qxf2 14y ago
    Oh, no argument on the fact that reading is extremely useful. I read too. In expressing the balance between reading and collaborating, I felt that your article emphasized too much on reading - but I fully recognize that you do collaborate. >>"I can read a book faster than I can speak or hear words." I disagree that this is an argument in favor of reading. Reading is easier but I think its not always as high in quality as collaborating. >> "Reading one (or two) such books will make discussions with the experts much more productive." Totally agree. I try my best to read as much with the end goal being picking the brains of my colleagues.
    - jacques_chester 14y ago
adrianbg 14y ago
Programmers should learn more about other domains and domain experts should learn how to program. Programming is becoming nearly as basic a skill as writing.
mathattack 14y ago
This is a very true post, but I don't think the solution is at the university, at least for undergrads. Schools barely have enough time in 4 years to create entry level programmers. Lopping off half the coursework in exchange for domain knowledge leaves grads unprepared. My observation is if you can learn to program well, you can learn any field of knowledge. A programmer can learn consumer products easier than a brand manager can learn Python. Similarly a CS major can learn bond math easier than a Finance major can learn database design. (Yes, these are generalizations) What's this mean? That bridging the Tech/Client divide is on the shoulder of the technologist. In addition to learning the latest frameworks, we have to learn our customers business. In addition to tech credentials, we should add theirs too.
- jacques_chester 14y ago
  The degree gives you a tour of the foundations of the solution domain. Mastery of the solution domain and knowledge of various problem domains cannot be taught in a 3 or 4 year degree.
  - mathattack 14y ago
    Agree completely.
marpalmin 14y ago
Perhaps we should all work in compilers ;)
dschiptsov 14y ago
Another metaphor for necessity of Domain Knowledge is translation. Give an average translator a technological text full of idioms from this technology domain and you will laugh your ass off. I remember reading reports about some SAP installation process (written by idiots) full of stupid special terms such as "System Landscape" and "User Privileges" translated by an English teacher.) Why we have this as a top news? Isn't it obvious that you must know the slang before you're trying to speak, let alone translate? Well, most of coders use a slightly different strategy - jump right into IDE and pile it up, using all that containers and design patterns they have heard of (without understanding, why bother?) until it satisfies the manager (compiles) who know absolutely nothing (this is why everything must be statically typed). Then test it like a black-box - correct input produces correct output (never try incorrect or unexpected) and then ship and forget.)
euroclydon 14y ago
I've held programming jobs in: Water Treatment Process Control, Workflow Management, Clinical Trials Management, Electrical Power Monitoring and Simulation, and Medical Imaging. In each case my natural curiosity led me to learn as much as I could about the field. I always enjoyed talking to the domain experts at each company and asking them as many questions as they could handle. There are usually one or two people who relish answering these questions and I especially like the ones who talk fast, mock astonishment at my ignorance, and proceed to try to fix it. In water treatment, I learned about oxic/anoxic bacteria and how they're grown and recycled in the process to treat waste, how to write chemical metering logic in a PLC because I was the only one left on-site to do it, how motors are controlled with variable frequency drives, how proportional-integral-derivatives are used to prevent oscillation in feedback based controls. In clinical trials management, I learned how double-blind studies are made, the process of randomization and dispensing drugs, medical coding and the life cycle of the trial. In power systems simulation, I learned about three-phase power transmission, resistive and capacitive loads, active and reactive power, the challenges of hooking up a lot of solar panels our existing grids, how data centers are build and cooled and powered, microgrids, power meters, intelligent breakers and more. I feel privileged that my skills at software development have exposed me to a variety of fields and that I know how several important processes in our word work. And I got to take home a nice salary while gaining that exposure. I do lament that, unless I pick a field and stick with it, I'll never get to work on the difficult programming problems that require a decade or more of experience and knowledge.
danboarder 14y ago
This article is talking about what I learned as classic Systems Analysis and Design. It is an interdisciplinary intentional practice of seeking to fully understand the business processes involved. See http://en.m.wikipedia.org/wiki/Systems_analysis_and_design http://en.m.wikipedia.org/wiki/Systems_analysis_and_design as a starting point.
- treerock 14y ago
  So have you ever tried what the articles suggest? Ever spent a week working on the 'shop floor'? In my experience of public sector software projects, the programmers seem to be kept as far away from the end users as possible.
  - danboarder 14y ago
    As a independent contractor / systems analyst, yes I have. However I agree, most solutions providers don't bother.
markbao 14y ago
Turning the tables around: perhaps the software (or even physical products) that we use where we think, "this is stupid! why would anyone build it this way?" were similarly built by people with no domain expertise in the product. Introduce a software engineer with an interest in the subject matter into that environment, and provided it isn't too big (like government), it might be self-correcting.
smoyer 14y ago
I believe this is actually the major problem faced by outsourcing projects. When I was in the cable industry, I saw several projects fail when the "Systems Engineers" (who were in charge of writing the specifications) were unable to convey enough information to the developers. There's a real limit to the amount of "truth" you can put into a specification, and at some point you have to trust the developer to "do the right thing". If they don't have domain knowledge in the industry they're writing for, they don't stand a chance ... and so your project fails even when everyone works hard and has good intentions.
martinced 14y ago
Sadly in some domain it's not possible. I've had cases where to understand a particularly complex business logic case it turned out nobody had any idea how that case should be resolved. Lawyers / jurists, employees, managers, etc. We tried to go up the chain up to someone, anyone, who could explain how that case should be resolved but nobody could tell. Why? Legalese. And just like most legalese documents: there's more than one interpretation possible because the very concept of legalese (based on thousands years old tradition) is inherently flawed and open to interpretation. So, somewhere, some politicians, probably out of bikeshedding fashion, decides to vote an amendment exceedingly badly written and exceedingly complex. Then it becomes law. And law has to be followed. So that case has to be taken into account by the software but isn't (yet). Everything is all and well until that specific case appear. The day it appears all bets are of. It's not working and nobody knows why. Nobody knows what is not working and why it's not working, nor how it should work. The law is bogus and the business case cannot be resolved without contradicting with the law(s). Trying to determine this ended up with the programming team receiving a PDF with tens and tens of pages of legalese text that nobody up the chain (including lawyers) could explain to us. Probably that the politicians who voted these changes couldn't even themselves explain it. We simply couldn't make the changes. How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse. And there came the technopriests: fixing the production DB...
- raverbashing 14y ago
  > How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse. This is usually how it ends up. I've been close to similar situations (but while creating something, not changing something that already exists), but usually, if nobody understands how this is to be done, the fact that you can't code it is second in importance.
jedbrown 14y ago
In science and engineering, it's common to hear of "Computational X" where X is "Plasma Physics", "Geodynamics", "Ocean Science", "Electrical Engineering", "Aerodynamics", or any of a myriad of other disciplines. In these fields, the primary task is formulating Science/Engineering questions that can be effectively answered using computation, then writing the software to do so. There is also the more generic field "Computational Science and Engineering" (CS&E) [1], which is primarily the development of methods and software that is applied in many specific disciplines. I work primarily on the library development side of CS&E, working with many applications to understand their needs, often finding ways to satisfy those needs with better software reuse and better ultimate flexibility, then implementing any components that belong in libraries. Additionally, we do basic research by recognizing unexploited common themes and designing new algorithms that will benefit existing and potential users. There is currently a great deal of demand for Computational X, for CS&E through the postdoc level, and for CS&E in industry and national labs. It is recognized that a versatile practitioner of CS&E can effectively assist Computational X for many values of X, since there really is a lot in common. Once you've put in 10000 hours of CS&E, plus 1000 hours in a handful of disciplines, you should be able to get orientedy quickly in new applications. At the faculty level, support for CS&E is highly variable. At some universities, X departments have the view that all their faculty should "X first, computation second", while the mathematics department wants to see theorems and isn't overly concerned with following through to see the method applied to obtain a new science or engineering result. When hiring committees say "we need CS&E, but some other department should have the CS&E faculty", it drives a lot of talent away from teaching roles (e.g., into industry), perpetuating the relative lack of early education in CS&E fundamentals. On the other hand, the trend is for universities to have more direct interest in CS&E (sometimes recognizing it with a dedicated department or inistitute). It helps that computation stimulates many interdisciplinary collaborations, leading to more diverse funding possibilities. [1] We have a big meeting next week: http://www.siam.org/meetings/cse13/ http://www.siam.org/meetings/cse13/
- pmahoney 14y ago
  What sort of backgrounds do people working in this field have? More specifically, I have a bachelor's degree in materials science and currently work doing run-of-the-mill web development (were the typical background of my peers is a bachelor's in CS). What steps might I take to move into computational engineering?
hawleyal 14y ago
OP might be right if they choose to write software without collaboration with clients or subject matter experts. IMHO, with collaboration, it's more a matter of providing a hyper-rational process by which the purpose and priorities of the target audience(s) are focused and honed, and then producing the software as feedback, automation, and a usable compliment to their needs.
keithpeter 14y ago
Does anyone here find the work of Lave and Wenger ('situated learning') of any use in their attempts to understand a problem area for a customer? http://www.infed.org/biblio/communities_of_practice.htm http://www.infed.org/biblio/communities_of_practice.htm Basically, some organisations make their processes easily visible to most employees, and some don't. The ones that do can train people more quickly and improve their processes more quickly. Organisations that make 'peripheral participation' easy might be better customers for developers than those that don't. You might not have to make many sails or use a height gauge too often to 'see' what the issues are, just interview knowledgeable people carefully and cross reference the results with observation. As a concrete example, the original author invested a considerable amount of (quite expensive) time in learning about sail making and lathe operating. In some of the 'business applications' I have to use daily, it is pretty obvious that the customer was a manager higher up and that the interface we have to use to populate the database that generates the wonderful reports has not really had much time spent on it!
DirtyMonkey 14y ago
I agree, but this is something I've found resistance to in the past. For example, last year I was building an automated shipping system for a client and I wanted to spend a few days in the shipping department - this was flat out refused as the management didn't see it how it would be useful, since the process was fully documented. Turns out, much had to be rewritten because the shipping staff were (predictably) doing 'undocumented' things to speed up their workflow, the new shipping system messed this up and made them slower.
mark_l_watson 14y ago
Jacques is right-on. For a lot of the work in my life, most of the effort was coming up to speed on a domain rather than just technology. Much of my work has been in 'artificial intelligence' (for some definition; never liked that term because it is too general) so domain knowledge is central if you are trying to get human level or better performance in some narrow domain. Having to spin up on new domains also keeps work and life interesting.
bwooceli 14y ago
So I don't work in a scientific field, and I'm not a "programmer" or "software engineer". I am a business analyst. I am a business analyst who CAN code. My code isn't always pretty, and I recycle a crap ton of boilerplate and hack the hell out of django apps. And I read the heck out of HN and have spent 5 years slowly refining my design chops. But most importantly, I know top-to-bottom the ins and outs of my business. So when our call center shopped around for a software solution to provide a call center full of agents realtime access to their stats, we were given a 3 month time frame and quoted $100k+ for a solution. So I wrote and deployed a web based results dashboard of realtime stats over two weekends. And we were just quoted a price on a quality management tool for $225k. Guess what's next for me? I've been an agent on the phones. I've sold in a retail environment. I've been the corporate trainer dealing with the LMS. Domain knowledge plus tech chops has proven a powerful combination for me.
jiggy2011 14y ago
I agree about the importance of gathering as much domain knowledge as possible. One thing that I have always had issues with when gathering requirements around something for an unfamiliar industry is that people who are in that industry will often neglect to mention things that are both important and non-obvious/counter-intuitive to someone unfamiliar with the industry. This is probably because they are so familiar with the quirks of the way that the particular industry works that such things seem so obvious as to not need mentioning (and they probably don't see how it could possibly be any other way). Basically similar to how we are often amazed by assumptions some people have about the way that computers/software works. The paradox of this though is that to deliver truly good software for a particular industry one must really know that industry as well as somebody who has years of experience in that area. At which point you start to feel that you could do their job too. Another alternative is to have a very tight feedback loop with someone who has tons of domain knowledge and is the target user. Basically sit next to them as you develop the software and have them review every single change you make.
- vicbrooker 14y ago
  I think in order to create good software that fits the industry's current best practices then you do need a really in-depth domain-specific knowledge. If that's the goal then learn everything possible about that industry. To create good solutions that are also innovative, I think that programmers only really need enough knowledge in that domain to be dangerous - to know more or less what happens but with enough gaps to not really know what's doable and what isn't. But, this knowledge is only useful if you can combine it with bits and pieces from several completely separate domains. If innovation is the goal then I think there is a point where domain knowledge actually hurts you. Of course there still needs to be at least one specialist to keep everything on track though - and I completely agree with you about having someone with deep knowledge on board. I think it should be someone who has been in that industry and will stay in that industry. I can't see the point in programmers trying to become experts in areas that they're not interested in working in for the long haul. It's easier to learn (and actually apply) a little about a lot than a lot about a little.
east2west 14y ago
HN is understandably filled with programmers who value programming above all else and believe domain knowledge can be picked up quickly. It would be true for someone hired as a programmer who implement a published algorithm or port existing software to newer systems or different platform; in other words, improving existing systems. I believe it requires equal depth if not not more for domain specific knowledge to innovate in non-computational fields. For my field of bioinformatics/computational biology, many open questions cannot be answered because of computational hurdles. Reading textbooks won't lead anyone to them, nor would reading survey papers. It requires expert knowledge in both biology and computer science to even know what is feasible and interesting. We want to find the driver mutations for cancer, discover mechanism of gene regulation, uncover subtle interaction of heredity and environment. Computation is intimately involved because observation is indirect, noisy, and massive. Only with knowledge in both fields can answer questions that domain experts care about but cannot do because of computational/quantitative obstacles. I find researchers (I am in academia) in both fields cross-talk too much. Computational/quantitative experts try to shoe-horn biological questions into their small corner of research area, while biologists don't care about computational efficiency, statistical robustness/efficiency, and the mathematical implication of their simple methods; no one cares about software quality. This is unfortunate. It is also unfortunate hiring committees do not look favorably towards multi-disciplinary researchers because one is not a trained geneticist/cancer researcher. I definitively do not encourage computer scientists go into support roles in academia; they are poorly paid and get little respect.
joshwa 14y ago
Reminds me of the old "what the customer actually wanted" comic: http://imgur.com/NXHbZ http://imgur.com/NXHbZ
trvd1707 14y ago
I think domain knowledge is important, but sometimes it can be a trap that hinders inovation. As Pierre Bayard put very nicely in "How to talk about books you haven't read", when we don't have domain knowledge we open ourselves to questions that we wouldn't make otherwise.

203 points by ajhai 14y ago

I am currently working on a web software project in the medical/health space. Of course its really good to know how the people that use the software work and what they expect and i am eager to learn those things, but i will not dive into a sector i am not really interested and absorb any and all information i can find, since i probably wont need it for the next project and it personally does not interest me a lot. For technical domain knowledge i agree, but it also happens that this is often of personal interest to me anyway.

kevingadd 14y ago

The lack of this kind of domain knowledge - either from being unwilling to go find it, or not being given the time and the resources to do so - is a big problem for tools development in games. People doing the scheduling and planning tend to assume that if you put enough smart programmers in a room and give them detailed specifications, they'll be able to solve all the problems faced by some artists and writers in another room, even though they don't know anything about art or writing. Sometimes this results in the kind of obvious mistakes that you might otherwise assume only happen when you're writing software for the government. In a sense it's kind of sad: In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work, or ask them questions on a regular basis to understand their issues - you're separated from them by a sales process and probably some automated support tickets and a PR guy who won't let you have open discussion in public. People building tools for game development can have their customers sitting a few desks away!

calpaterson 14y ago
> In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work One of the nicest things about the Agile fashion at the moment is that my customer (or a reasonable representative of him) is a couple of desks away and I really can go and ask him about what he wants. It's incredible how much time is saved by avoiding doing things he doesn't care about and how much happier he is with the end result.
jeltz 14y ago
> Sometimes this results in the kind of obvious mistakes that you might otherwise assume only happen when you're writing software for the government. My personal guess is that the reason there too is lack of domain knowledge. That many government projects are ran by handing out specs to the people tasked with implementing them.
Jare 14y ago
I always asked my tools programmers (and do myself) to spend some time regularly sitting by an artist or designers while she works, and observe. HUGE difference.

kenkam 14y ago

This particularly resonates with me. I have been working in a bank as a graduate developer for the last year and a half, getting by without much care about how instruments are priced, traded, recorded, audited, etc. To me, I was moving a button around and colouring things using WPF. To me, the "Yield" column was just another column with numbers that were supposed to be 2 decimal places and turns red if negative. All the while I was dreaming I wished I worked at Google on 'technology'. Which, in hindsight, was a little misguided. There is so much value in learning and understanding the domain in which we are working in because it makes us so much more effective. The realisation came when I became frustrated in meetings where we would discuss ideas on financial models and I could not add any value to the discussion. I then imagined working at a company like Google, let's say, doing stuff I wouldn't normally be interested in. Would I bother learning the domain knowledge? I couldn't say yes with 100% conviction. I love programming but I also want to be effective in what I do. To me now, understanding the problems we are solving is probably just as important as writing maintainable code.

gcv 14y ago

Ah yes, the tragedy of studying computing to the exclusion of everything else. Graduates with computer science degrees have an in-depth understanding of... text editors. And maybe compilers, databases, and operating systems — all items of zero value until they are applied to solving some real problems in some other field. Problems which the computer science graduate knows nothing about. Then these graduates go into the working world, and they typically can't help solve problems in economics, physics, applied mathematics, or medicine except by coding up some system to someone else's specs. The more dedicated ones make an effort to understand what they're doing. They exceedingly rare, gifted, and motivated ones might learn enough about an industry to make real contributions. I hope that our field eventually evolves to the point that programming skill becomes an accessory. People will then learn to program just like they learn to write, and use the craft to help them solve their actual problems. The profession of programmers who only know how to program should disappear like the profession of scribes who only know how to write.

GFischer 14y ago
I hope we'll have specialized careers someday (Medical CS, Economics CS, etc...). In many careers, it's the other way round - most physics I know taught themselves how to program, for example, but they're lacking some tools that would probably help them a lot (and physics are very close to CS). I've seen accountants do a lot of programming in Excel macros and Access too. Doctors and Economists might have less interest in learning, but academy-minded ones will really appreciate the modeling they can do with the use of computers (I think I saw a medical paper where they reinvented some basic CS concept :) ). Some of the domain training comes from practical experience - I started out working for the credit report sector, and then moved on to the insurance sector, so I've picked along some domain knowledge in those areas. Other colleagues have worked in logistics, and now know about their problems, etc...
- eru 14y ago
  > (I think I saw a medical paper where they reinvented some basic CS concept :) Perhaps more shocking: https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/ https://fliptomato.wordpress.com/2007/03/19/medical-research...

alemhnan 14y ago

My take is that one of the main improvements in programming languages will be to reduce the gap between domain experts and programming experts. Now we should face three cases: 1) the programmer needs to learn the domain; 2) the domain expert needs to know how to develop; 3) something in between. Most of developing is done in 1). There are some brilliant cases of 2). I once talked with a biologist doing some multi sequence alignment that told me that some routine in C was quite slow and was far better rewrite that routine from scratch in assembly. I was quite speechless. In order to create better programming languages we could: 1) express domains in a better way. Actually, just be able to express domains would be really cool (and not so easy). 2) provide higher levels of abstraction.

jacquesm 14y ago
Every programming language from the dawn of time up to the present has made that claim at some point. This time it would be different. I think it is just a people problem. Programmers tend to seclude themselves from their customers. Whole IT departments tend to seclude themselves from their users. When you are looking for underwater welders for the off-shore industry, what is better, teaching a welder to dive or teaching a diver to weld? (that's a serious question with a real answer). It is much easier for a programmer to acquire the required domain knowledge than it is (usually) for a domain specialist to learn how to program. But you can only do so if you're prepared to 'dive in'.
- unwind 14y ago
  My guess would be the first, teaching a welder to dive. I base that on the fact that the core value, the welds, will be higher-quality and that's more important than the diving execution. The diving might be off, but it shouldn't be that hard to learn well enough to survive and be effective. Of course, I'm neither a diver nor a welder. Did I win?
  - jacquesm 14y ago
    > Did I win? Yep. But a drowned welder is also not much ;)

Nursie 14y ago

Actually I think domain knowledge is overrated. A good hacker can pick up the domain knowledge needed to contribute to a project very quickly. Now, if you're designing a product to solve a problem in a particular domain you need the knowledge ahead of time, sure. But if you're a shit-hot coder and a fast learner, brought in to make things work, who cares?

rcfox 14y ago
Because, as the article clearly mentions, there's a difference between making something work and making something work right.
- Nursie 14y ago
  I disagree that it's important unless you're the person designing the solution. If you're not (and in many/most teams you won't be) it's far less important.
  - rcfox 14y ago
    Sure, if the problem is solved, then you probably don't need to know anything. But implementing other people's solutions is only satisfying for so long.
    - Nursie 14y ago
      I don't disagree with that, but there are plenty of people that make a career out of it. In the case of ex-colleagues who find a corner at a big multinational and get told what to do, someone else translates domain knowledge for them, and they come up with tech solutions to the presented requirements. And then there are people like me who get called in because we're good at picking up large, existing, broken codebases quickly and somebody's in the sort of trouble that's best summed up as "Oh $&%£ we've got three months to deliver, the product's less than half-done and it's full of weird intermittent bugs". I'm finding it satisfying enough for now :)
      - 14y ago

qxf2 14y ago

The line I found most insightful: >>"Software is much more intertwined with the institution commissioning it and the people using it ..." Given that, I'm surprised at the emphasis on 'reading' as the primary source of gaining domain knowledge. There are just 2-3 mentions of speaking/collaborating with the domain experts that are driving you to build the software itself. If you are lucky enough to work with acknowledged domain experts, spend time talking to them. Pick their brain. Two techniques I use: a) forward interesting articles to them with a question or an analogous concept in my domain b) resist the urge to show them a 'better' way. E.g.: In the HealthCare domain, I learnt people don't really want quicker or lesser clicks. They would prefer a set, stable workflow. If you cater to senior-citizens, keyboard shortcuts are less important than instructions in large font. I still suck at engaging domain experts. So, I'd love to learn more techniques to hold an interesting conversation with domain experts. Ideas?

jacques_chester 14y ago
I can read a book faster than I can speak or hear words. In addition, books can usually be found, on almost any topic, which are intended to bring a novice up to a basic level of conversancy. Reading one (or two) such books will make discussions with the experts much more productive. Plus: it's fun to learn about new things.
- qxf2 14y ago
  Oh, no argument on the fact that reading is extremely useful. I read too. In expressing the balance between reading and collaborating, I felt that your article emphasized too much on reading - but I fully recognize that you do collaborate. >>"I can read a book faster than I can speak or hear words." I disagree that this is an argument in favor of reading. Reading is easier but I think its not always as high in quality as collaborating. >> "Reading one (or two) such books will make discussions with the experts much more productive." Totally agree. I try my best to read as much with the end goal being picking the brains of my colleagues.
  - jacques_chester 14y ago

adrianbg 14y ago

Programmers should learn more about other domains and domain experts should learn how to program. Programming is becoming nearly as basic a skill as writing.

mathattack 14y ago

This is a very true post, but I don't think the solution is at the university, at least for undergrads. Schools barely have enough time in 4 years to create entry level programmers. Lopping off half the coursework in exchange for domain knowledge leaves grads unprepared. My observation is if you can learn to program well, you can learn any field of knowledge. A programmer can learn consumer products easier than a brand manager can learn Python. Similarly a CS major can learn bond math easier than a Finance major can learn database design. (Yes, these are generalizations) What's this mean? That bridging the Tech/Client divide is on the shoulder of the technologist. In addition to learning the latest frameworks, we have to learn our customers business. In addition to tech credentials, we should add theirs too.

jacques_chester 14y ago
The degree gives you a tour of the foundations of the solution domain. Mastery of the solution domain and knowledge of various problem domains cannot be taught in a 3 or 4 year degree.
- mathattack 14y ago
  Agree completely.

marpalmin 14y ago

Perhaps we should all work in compilers ;)

dschiptsov 14y ago

Another metaphor for necessity of Domain Knowledge is translation. Give an average translator a technological text full of idioms from this technology domain and you will laugh your ass off. I remember reading reports about some SAP installation process (written by idiots) full of stupid special terms such as "System Landscape" and "User Privileges" translated by an English teacher.) Why we have this as a top news? Isn't it obvious that you must know the slang before you're trying to speak, let alone translate? Well, most of coders use a slightly different strategy - jump right into IDE and pile it up, using all that containers and design patterns they have heard of (without understanding, why bother?) until it satisfies the manager (compiles) who know absolutely nothing (this is why everything must be statically typed). Then test it like a black-box - correct input produces correct output (never try incorrect or unexpected) and then ship and forget.)

euroclydon 14y ago

I've held programming jobs in: Water Treatment Process Control, Workflow Management, Clinical Trials Management, Electrical Power Monitoring and Simulation, and Medical Imaging. In each case my natural curiosity led me to learn as much as I could about the field. I always enjoyed talking to the domain experts at each company and asking them as many questions as they could handle. There are usually one or two people who relish answering these questions and I especially like the ones who talk fast, mock astonishment at my ignorance, and proceed to try to fix it. In water treatment, I learned about oxic/anoxic bacteria and how they're grown and recycled in the process to treat waste, how to write chemical metering logic in a PLC because I was the only one left on-site to do it, how motors are controlled with variable frequency drives, how proportional-integral-derivatives are used to prevent oscillation in feedback based controls. In clinical trials management, I learned how double-blind studies are made, the process of randomization and dispensing drugs, medical coding and the life cycle of the trial. In power systems simulation, I learned about three-phase power transmission, resistive and capacitive loads, active and reactive power, the challenges of hooking up a lot of solar panels our existing grids, how data centers are build and cooled and powered, microgrids, power meters, intelligent breakers and more. I feel privileged that my skills at software development have exposed me to a variety of fields and that I know how several important processes in our word work. And I got to take home a nice salary while gaining that exposure. I do lament that, unless I pick a field and stick with it, I'll never get to work on the difficult programming problems that require a decade or more of experience and knowledge.

danboarder 14y ago

This article is talking about what I learned as classic Systems Analysis and Design. It is an interdisciplinary intentional practice of seeking to fully understand the business processes involved. See http://en.m.wikipedia.org/wiki/Systems_analysis_and_design http://en.m.wikipedia.org/wiki/Systems_analysis_and_design as a starting point.

treerock 14y ago
So have you ever tried what the articles suggest? Ever spent a week working on the 'shop floor'? In my experience of public sector software projects, the programmers seem to be kept as far away from the end users as possible.
- danboarder 14y ago
  As a independent contractor / systems analyst, yes I have. However I agree, most solutions providers don't bother.

markbao 14y ago

Turning the tables around: perhaps the software (or even physical products) that we use where we think, "this is stupid! why would anyone build it this way?" were similarly built by people with no domain expertise in the product. Introduce a software engineer with an interest in the subject matter into that environment, and provided it isn't too big (like government), it might be self-correcting.

smoyer 14y ago

I believe this is actually the major problem faced by outsourcing projects. When I was in the cable industry, I saw several projects fail when the "Systems Engineers" (who were in charge of writing the specifications) were unable to convey enough information to the developers. There's a real limit to the amount of "truth" you can put into a specification, and at some point you have to trust the developer to "do the right thing". If they don't have domain knowledge in the industry they're writing for, they don't stand a chance ... and so your project fails even when everyone works hard and has good intentions.

martinced 14y ago

Sadly in some domain it's not possible. I've had cases where to understand a particularly complex business logic case it turned out nobody had any idea how that case should be resolved. Lawyers / jurists, employees, managers, etc. We tried to go up the chain up to someone, anyone, who could explain how that case should be resolved but nobody could tell. Why? Legalese. And just like most legalese documents: there's more than one interpretation possible because the very concept of legalese (based on thousands years old tradition) is inherently flawed and open to interpretation. So, somewhere, some politicians, probably out of bikeshedding fashion, decides to vote an amendment exceedingly badly written and exceedingly complex. Then it becomes law. And law has to be followed. So that case has to be taken into account by the software but isn't (yet). Everything is all and well until that specific case appear. The day it appears all bets are of. It's not working and nobody knows why. Nobody knows what is not working and why it's not working, nor how it should work. The law is bogus and the business case cannot be resolved without contradicting with the law(s). Trying to determine this ended up with the programming team receiving a PDF with tens and tens of pages of legalese text that nobody up the chain (including lawyers) could explain to us. Probably that the politicians who voted these changes couldn't even themselves explain it. We simply couldn't make the changes. How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse. And there came the technopriests: fixing the production DB...

raverbashing 14y ago
> How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse. This is usually how it ends up. I've been close to similar situations (but while creating something, not changing something that already exists), but usually, if nobody understands how this is to be done, the fact that you can't code it is second in importance.

jedbrown 14y ago

In science and engineering, it's common to hear of "Computational X" where X is "Plasma Physics", "Geodynamics", "Ocean Science", "Electrical Engineering", "Aerodynamics", or any of a myriad of other disciplines. In these fields, the primary task is formulating Science/Engineering questions that can be effectively answered using computation, then writing the software to do so. There is also the more generic field "Computational Science and Engineering" (CS&E) [1], which is primarily the development of methods and software that is applied in many specific disciplines. I work primarily on the library development side of CS&E, working with many applications to understand their needs, often finding ways to satisfy those needs with better software reuse and better ultimate flexibility, then implementing any components that belong in libraries. Additionally, we do basic research by recognizing unexploited common themes and designing new algorithms that will benefit existing and potential users. There is currently a great deal of demand for Computational X, for CS&E through the postdoc level, and for CS&E in industry and national labs. It is recognized that a versatile practitioner of CS&E can effectively assist Computational X for many values of X, since there really is a lot in common. Once you've put in 10000 hours of CS&E, plus 1000 hours in a handful of disciplines, you should be able to get orientedy quickly in new applications. At the faculty level, support for CS&E is highly variable. At some universities, X departments have the view that all their faculty should "X first, computation second", while the mathematics department wants to see theorems and isn't overly concerned with following through to see the method applied to obtain a new science or engineering result. When hiring committees say "we need CS&E, but some other department should have the CS&E faculty", it drives a lot of talent away from teaching roles (e.g., into industry), perpetuating the relative lack of early education in CS&E fundamentals. On the other hand, the trend is for universities to have more direct interest in CS&E (sometimes recognizing it with a dedicated department or inistitute). It helps that computation stimulates many interdisciplinary collaborations, leading to more diverse funding possibilities. [1] We have a big meeting next week: http://www.siam.org/meetings/cse13/ http://www.siam.org/meetings/cse13/

pmahoney 14y ago
What sort of backgrounds do people working in this field have? More specifically, I have a bachelor's degree in materials science and currently work doing run-of-the-mill web development (were the typical background of my peers is a bachelor's in CS). What steps might I take to move into computational engineering?

hawleyal 14y ago

OP might be right if they choose to write software without collaboration with clients or subject matter experts. IMHO, with collaboration, it's more a matter of providing a hyper-rational process by which the purpose and priorities of the target audience(s) are focused and honed, and then producing the software as feedback, automation, and a usable compliment to their needs.

keithpeter 14y ago

Does anyone here find the work of Lave and Wenger ('situated learning') of any use in their attempts to understand a problem area for a customer? http://www.infed.org/biblio/communities_of_practice.htm http://www.infed.org/biblio/communities_of_practice.htm Basically, some organisations make their processes easily visible to most employees, and some don't. The ones that do can train people more quickly and improve their processes more quickly. Organisations that make 'peripheral participation' easy might be better customers for developers than those that don't. You might not have to make many sails or use a height gauge too often to 'see' what the issues are, just interview knowledgeable people carefully and cross reference the results with observation. As a concrete example, the original author invested a considerable amount of (quite expensive) time in learning about sail making and lathe operating. In some of the 'business applications' I have to use daily, it is pretty obvious that the customer was a manager higher up and that the interface we have to use to populate the database that generates the wonderful reports has not really had much time spent on it!

DirtyMonkey 14y ago

I agree, but this is something I've found resistance to in the past. For example, last year I was building an automated shipping system for a client and I wanted to spend a few days in the shipping department - this was flat out refused as the management didn't see it how it would be useful, since the process was fully documented. Turns out, much had to be rewritten because the shipping staff were (predictably) doing 'undocumented' things to speed up their workflow, the new shipping system messed this up and made them slower.

mark_l_watson 14y ago

Jacques is right-on. For a lot of the work in my life, most of the effort was coming up to speed on a domain rather than just technology. Much of my work has been in 'artificial intelligence' (for some definition; never liked that term because it is too general) so domain knowledge is central if you are trying to get human level or better performance in some narrow domain. Having to spin up on new domains also keeps work and life interesting.

bwooceli 14y ago

So I don't work in a scientific field, and I'm not a "programmer" or "software engineer". I am a business analyst. I am a business analyst who CAN code. My code isn't always pretty, and I recycle a crap ton of boilerplate and hack the hell out of django apps. And I read the heck out of HN and have spent 5 years slowly refining my design chops. But most importantly, I know top-to-bottom the ins and outs of my business. So when our call center shopped around for a software solution to provide a call center full of agents realtime access to their stats, we were given a 3 month time frame and quoted $100k+ for a solution. So I wrote and deployed a web based results dashboard of realtime stats over two weekends. And we were just quoted a price on a quality management tool for $225k. Guess what's next for me? I've been an agent on the phones. I've sold in a retail environment. I've been the corporate trainer dealing with the LMS. Domain knowledge plus tech chops has proven a powerful combination for me.

jiggy2011 14y ago

I agree about the importance of gathering as much domain knowledge as possible. One thing that I have always had issues with when gathering requirements around something for an unfamiliar industry is that people who are in that industry will often neglect to mention things that are both important and non-obvious/counter-intuitive to someone unfamiliar with the industry. This is probably because they are so familiar with the quirks of the way that the particular industry works that such things seem so obvious as to not need mentioning (and they probably don't see how it could possibly be any other way). Basically similar to how we are often amazed by assumptions some people have about the way that computers/software works. The paradox of this though is that to deliver truly good software for a particular industry one must really know that industry as well as somebody who has years of experience in that area. At which point you start to feel that you could do their job too. Another alternative is to have a very tight feedback loop with someone who has tons of domain knowledge and is the target user. Basically sit next to them as you develop the software and have them review every single change you make.

vicbrooker 14y ago
I think in order to create good software that fits the industry's current best practices then you do need a really in-depth domain-specific knowledge. If that's the goal then learn everything possible about that industry. To create good solutions that are also innovative, I think that programmers only really need enough knowledge in that domain to be dangerous - to know more or less what happens but with enough gaps to not really know what's doable and what isn't. But, this knowledge is only useful if you can combine it with bits and pieces from several completely separate domains. If innovation is the goal then I think there is a point where domain knowledge actually hurts you. Of course there still needs to be at least one specialist to keep everything on track though - and I completely agree with you about having someone with deep knowledge on board. I think it should be someone who has been in that industry and will stay in that industry. I can't see the point in programmers trying to become experts in areas that they're not interested in working in for the long haul. It's easier to learn (and actually apply) a little about a lot than a lot about a little.

east2west 14y ago

HN is understandably filled with programmers who value programming above all else and believe domain knowledge can be picked up quickly. It would be true for someone hired as a programmer who implement a published algorithm or port existing software to newer systems or different platform; in other words, improving existing systems. I believe it requires equal depth if not not more for domain specific knowledge to innovate in non-computational fields. For my field of bioinformatics/computational biology, many open questions cannot be answered because of computational hurdles. Reading textbooks won't lead anyone to them, nor would reading survey papers. It requires expert knowledge in both biology and computer science to even know what is feasible and interesting. We want to find the driver mutations for cancer, discover mechanism of gene regulation, uncover subtle interaction of heredity and environment. Computation is intimately involved because observation is indirect, noisy, and massive. Only with knowledge in both fields can answer questions that domain experts care about but cannot do because of computational/quantitative obstacles. I find researchers (I am in academia) in both fields cross-talk too much. Computational/quantitative experts try to shoe-horn biological questions into their small corner of research area, while biologists don't care about computational efficiency, statistical robustness/efficiency, and the mathematical implication of their simple methods; no one cares about software quality. This is unfortunate. It is also unfortunate hiring committees do not look favorably towards multi-disciplinary researchers because one is not a trained geneticist/cancer researcher. I definitively do not encourage computer scientists go into support roles in academia; they are poorly paid and get little respect.

joshwa 14y ago

Reminds me of the old "what the customer actually wanted" comic: http://imgur.com/NXHbZ http://imgur.com/NXHbZ

trvd1707 14y ago

I think domain knowledge is important, but sometimes it can be a trap that hinders inovation. As Pierre Bayard put very nicely in "How to talk about books you haven't read", when we don't have domain knowledge we open ourselves to questions that we wouldn't make otherwise.