14 ms·

You Can't Buy Integration

275 points by mpweiher 5y ago

vog 5y ago
I have always suspected this to be the case, but it is great to see it confirmed and well-explained in detail by an almost-authority in our field. I wonder if in the near future we'll see a similar article regarding security - "You Can't Buy Security".
- purerandomness 5y ago
  Wow, didn't expect to read something as condescending as calling Martin Fowler an "almost-authority" in this thread...
  - gnabgib 5y ago
    While it's Martin's blog, most posts these days are by other people.. this particular one is by "Brandon Byars"
jefflombardjr 5y ago
Really great read. Long but worth it. This is a good critique of low-code/no-code in general. The root problem I see of "Unfortunately, when we frame the problem space that way, we have allowed our tools to think for us." is that in reality there are just not enough qualified software engineers out there.
- gnabgib 5y ago
  The article isn't finished yet (see the footer)... so it'll be a longer read
- keyle 5y ago
  I got bounced off a front-end job once, for not being good enough at React. Mind you I've been building SPA's since they weren't "a thing" and today I mostly use Vue. But for that role, they didn't like my lack of .env variables in the front-end (such as for things that end up in the HTML), after a 3 hours coding test, and a couple of minor React-y tid bits they couldn't even clarify. Basically, not "idiomatic". Meanwhile, none of them could tell me how React actually works, beyond throwing jargon vomit. They couldn't write a web application without React. The recruiter, which we had to go through, basically had no empathy and saw me as a failed resume. I felt pretty helpless, even though I've said many times to both parties that I wasn't a "React developer". That is the sad state of affairs today; and I'm finding it's not just in the front-end, where you could argue that you need sanity on this pile of rubbles. It's also in the backend, especially on the auto-magic "DevOps!"
  - w-j-w 5y ago
    There's an even worse shortage of qualified interviewers out there.
  - gitfan86 5y ago
    I have been in a similar boat. I was spending a bunch of time avoiding extra API calls and extra renders in React and the team I was working with didn't seem to understand that there were potential performance issues around that. They said "just use memo" On the other hand I think it is a natural progression. 30 years ago experienced engineers would look at young people who didn't know how to fix a PCB or read bytecode and think it was a shame. If everyone had to get a computer engineering degree and work at several different roles before they could even build a UI, that would be bad for productivity.
    - debaserab2 5y ago
      It does make you wonder what is going to get missed though, doesn't it? Each generation gets to work on a level of abstraction higher than the one before it. I think about that often when I think about all the things I don't understand about hardware and EE.
      - gitfan86 5y ago
        I did computer engineering in school 20 years ago, and have always enjoyed understanding technology. So I know what is happening from the gate in the cpu all the way up to the pixel in the monitor. The downside is that I'm spread pretty thin. I can't pass leetcode exams in the 30 minutes of allotted time, ops people can't understand why I don't use containers, and my knowledge is probably useless in making a modern CPU.
        keyle 5y ago
        This resonates with me. I'm actually thinking that at some point, no one will be building custom software. Everything will be available off the shelf, as some product, running on some cloud within a few clicks. So "custom software" developers are a dying breed. For example, when Jira is down, a Jira "specialist" will tinker and bring it back and look like some guru walking down the mountain of knowledge. Arguably, this person probably wouldn't know anything about SIMD. Yet the SIMD guy is looking for a job, no one cares about that guy. But the "salesforce developers" and "Jira specialist" are permanently employed at very high wages.
- hinkley 5y ago
  I’ve hear the way to succeed with SAP is to reorganize your business to match either the default world view or some other cookie cutter variant. Essentially using it no code style. Otherwise you’re exerting yourself doing “normal” things. That’s not sustainable.
  - r00fus 5y ago
    This outlook is actually endemic in the ERP implementation space. The part you're not going to like hearing is that, due to laws/regulations, that "default world view" may be the best way to approach that problem for that given module/area (e.g. how to structure benefits or compensation). "Creativity" in the ERP space is costly. Also reducing your variation to something that's configurable as opposed to custom codebase means you don't have to do extensive testing whenever a patch or upgrade to the ERP software comes out.
  - NikolaNovak 5y ago
    All ERPs are fundamentally like that. At the end of the day you are purchasing a COTS product specifically to gain efficiency of industry best practices already coded for you. If you take a COTS product and then try to rewrite it to fit your "unique" business processes, we'll, that's why half of them fail. Implementation of erp is incorrectly and disastrously viewed as an IT project. It is first and foremost a business transformation project. A company should be self-aware to say "HR|Accounting|whatever is NOT our core business or competitive advantage; it's not what we are great at. It's not a thing we should be unique in. It's a cost centre and we need to standardize and minimize that cost with help of people and software that were successful with many other companies". (Source: I've been implementing Peoplesoft, a competing erp,for 20 years for dozens of companies, as a technical resource eventually wise enough to be aware it's fundamentally not a technical endeavour :-)
    - kayodelycaon 5y ago
      This is why my company has custom applications that talk to the ERP for certain departments. Inventory control in the ERP isn’t going to work with reality. The data modeling is good enough but the interface is not. Rather than train everyone we hire to use horrifically bad and expensive barcode readers, we purchased cheap smart phones, put them in cases, and locked them down to run an app that uses the camera to scan barcodes. Drop a phone? It’s encased in rubber. Run it over with a forklift? Toss them another one from the box and then figure out why this only happens on Tuesday afternoons. We already need a dev team for other reasons. Adding another decent programmer to create and support internal apps isn’t that expensive in the long run. Plenty of good devs like me who are happy to work remote from the Midwest at less than FAANG rates. :)
- hef19898 5y ago
  I have it seen time and again. Organizations and leadership being faced with a problem and jumping directly to tool / software selection and implementation before really having understood the problem itself, the processes and requirements. After selection, directly to implementation. Because, you know, we a re agile and working in sprints. So we can sort out architecture and interfaces as we go. End result, tools force you into the use not understood processes that have nothing to do with your business requirements whatsoever, an architecture that is cobbled together and users that don't understand neither. Even better if the tools are selected based on previous experience alone.
csours 5y ago
You Can't Buy Understanding of Your Problems You Can't Buy Someone Else Caring about Your Problems It seems like you can hire experts to understand and consultants to care about your problems, but someone has to bring the intent and to close the loop.
- malermeister 5y ago
  You can hire experts to pretend they care and do whatever's the most convenient for them and since you don't understand, you'll never know the difference.
- RealityVoid 5y ago
  Hah, I posit that there is a _very_ low possibility of buying someone to care about your problems.
  - mypalmike 5y ago
    This is why visiting a doctor can be a rather frustrating experience.
    - kansface 5y ago
      Exactly right! The doctor has her own problems to care about!
  - charcircuit 5y ago
    Everyone has a price.
    - RealityVoid 5y ago
      I posit that paying money only buys the appearance of caring, not the actual caring. That is the reason celebrities get (got? Not up to date) doctors who prescribed addictive drugs, because they just gave the client what they wanted, not what they needed. You really want someone who speaks truth to power, not just pleasing nonsense. And the power must be able to discern the two.
debacle 5y ago
Fowler doesn't ship. When you read his articles, especially when they make you feel like you know something others (especially those awful management types) don't, remember that he hasn't made anything. He is a software life coach.
- robbintt 5y ago
  It's not true. I am literally reading "Refactoring" right now. It's very good.
- Jtsummers 5y ago
  Fowler isn't the author so if you're reading this article you aren't reading one of his (Fowler's) articles. Check the byline: Brandon Byars. I have no knowledge of this person so can't comment on his particular relevance, but seriously, took 1 second to see who the author is.
- hobofan 5y ago
  Not sure why you are writing this. Fowler isn't the author of the article. martinfowler.com nowadays is just the Thoughtworks blog. I can't remember the last time reading a article by Fowler on there.
  - debacle 5y ago
    Even better, we're listening to someone who has accomplished less than Martin Fowler.
    - dang 5y ago
      Please don't do personal attacks on HN. The damage to the ecosystem outweighs any benefit such harshness provides. You can make your substantive points without that. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
988747 5y ago
It is a weird article. Fowler obviously knows a lot about integration, but since his business is about writing software he tries to discredit "low code tools", so that more people use his services. I think that those tools are great for exactly the reasons indirectly stated in the article - they take the mind off the implementation and let you focus on bigger picture. I was involved in tons of integration project and they were always 90+% about communication: agreeing on interfaces. The actual implementation is almost always trivial, with tools like TIBCO or MuleSoft you can sometimes do it in 15-30 minutes (for simple flows, the more complex ones can take up to 3 days).
- random314 5y ago
  This article seems to be by Brandon, not Martin
cardosof 5y ago
The post isn't bad or anything, its just that coming from Fowler (or a partner) who sells software services and books on software for a living, it is as biased as it gets. I would love to read more on this particular topic but from the point of view of someone who needed to get the job done and tried no/low code tools and building his own thing and learned the pros and cons of each approach. Everything else looks like salespeople talking.
- bbyars 5y ago
  I find the bias claim an odd critique. I am of course biased -- in the appropriate context (e.g., architecture), bias is just another word for "judgment" -- but I don't think a claim of commercial bias is a strong argument. It is true that my company (Thoughtworks) is primarily a custom software delivery firm, but the point you might not have awareness of is that many (and most or all of the large) software services firms choose to build capabilities in low code platforms (in the past year alone, I've worked with Accenture and BCG Digital Venture Mulesoft experts and EY around Pega). In fact, many of the larger services firms get EXTRA commercial lift through partnerships with those low code companies ("finders fees"). We could easily do the same, make good money in the process, and take on a broader book of business in doing so. I'd have an easier time buying a related critique that neither I nor the colleagues I spoke to in shaping the article have exposure to other contexts where low code integration tools are the right choice. If you have some, I'd love to hear them.
  - Mandatum 5y ago
    Why do I have the opinion that ThoughtWorks is better than Accenture (or EY, Deloitte, etc)? Do they pay more for talent? About half the projects I see come out of non-boutique consultancies are shit. Actually, no. More than half lately. Probably 80%. I work in a role that exposes me to a wide range of partner-led engagements across large enterprise, but I admit my role is limited by region. ThoughtWorks on the other hand is massive, and seems to pick and choose engagements a lot more. They also seem to stack their teams with relevant talent. If they're assigning juniors or mid-level devs, they almost always seem to get some oversight by someone with a lot of relevant experience and input before going live.
ako 5y ago
Regarding visual diffing: "The delta between source code commits can only be represented textually; graphical palettes are not designed to represent change over time." I would think visual systems might be in a better spot to visualize change over time. Some of Tufte's diagrams have interesting ideas of how this could be represented statically, but turning static code diffs into illustrated animations might even be better than code diffing...
gorgoiler 5y ago
Business logic happens line by line between interfaces. Coding may be all ifs and for loops, but when it comes to implementing a new feature, having chosen carefully where to draw the line between, say, your model and your view can be the difference between a 10 line patch and a complete rewrite. If you don’t have a culture of code review then maybe the rewrite will fly, bugs and all, through to production. If however you have the ability to reason logically about the salient changes diff by prescient diff — where by definition each prescient diff represents nothing but the salient changes — then you at least stand a fighting chance of version N+1 having fewer bugs than version N while also moving your business forwards. Anything else is a quagmire that will never converge on stability and it all starts with there being a 1:1 mapping between what you think and what you write. Java is particularly unforgiving for code where ones thoughts are unclear. As a complete aside: buy your dev team a fountain pen and a pad of paper for Christmas. Writing meaningfully is everything.
- thewarrior 5y ago
  The line “coding is just if statements” really annoys me. I used to think this earlier in my career. It’s technically true but it means that you haven’t really grasped what makes programming hard. The real challenge of programming is how you organize a million if statements in a way that the system can be understood and evolved predictably and cheaply over time. And any tool that takes away the control of how you organize your code and allows for the creation of complexity without proper version control, debugging, testing and code review is doomed to failure. No code is just code at a higher level of abstraction. No code works when you don’t need a lot of code to solve the problem to begin with. Which is kind of a mind bender.
  - randomdata 5y ago
    Coding is just if statements, more or less. Programming and coding are different words with different meanings, though. What is the reason to conflate them?
    - codebje 5y ago
throwaway984393 5y ago
"Integration software products ... are not products that directly solve a business problem. ... They are programming languages, bundled with a toolchain and a runtime to support the compilation process." By this definition, Excel is a form of integration tool. A spreadsheet doesn't "directly solve a business problem". It's basically just a piece of paper with lines on it, along with some "programming formulas". A hammer would be an integration tool if it were digital. The technology industry is wholeheartedly committed to selling you the idea that you need to write software. The industry knows that will guarantee new customers (and profits) for decades to come, for all the things you'll need to buy into to "solve" the software engineering problems you'll undoubtedly encounter. But do you need to write software to solve it? Most other industries do not make things themselves when they need to solve a problem. They don't buy anvils when they need nails, they don't buy sewing machines when they need to clothe their workers, and they don't build industrial machines when they need to produce factory goods. They don't build cars when they need to deliver pizzas, nor build ovens to bake them, nor knives to cut them. There are always exceptions, but for rare, exceptional cases. I think the meat of this article is predicated on the idea that it's expected that you need to be somehow developing software to solve business problems. But really, none of the technologies in this article should be needed to solve most business problems. If there isn't an existing product for sale that's general enough to solve your problem, that's the problem we should be solving; not writing software for every single business case under the sun.
- Jtsummers 5y ago
  > "Integration software products ... are not products that directly solve a business problem. ... They are programming languages, bundled with a toolchain and a runtime to support the compilation process." > By this definition, Excel is a form of integration tool. A spreadsheet doesn't "directly solve a business problem". It's basically just a piece of paper with lines on it, along with some "programming formulas". A hammer would be an integration tool if it were digital. You're misreading that quote. The direction is actually important in conveying meaning, it's like trying to misread "Every ancient poet known as Homer was a man" as "all men are ancient poets known as Homer".
- mr_toad 5y ago
sam_lowry_ 5y ago
In Microsoft BizTalk one could make transformation rules not by drag-and-dropping but by coding in some weird XSLT dialect. It was a joy to use compared to the usual three-pane screen.
- lloydatkinson 5y ago
  What actually is BizTalk? I'm a .NET dev pretty involved in the Microsoft stack and never once have I seen it in use or worked anywhere that uses it.
  - gnabgib 5y ago
    Did you try wikipedia[0] or microsoft[1]? It's been around 20 years, but it's been fading away the last 5 [0]: https://en.wikipedia.org/wiki/Microsoft_BizTalk_Server https://en.wikipedia.org/wiki/Microsoft_BizTalk_Server [1]: https://partner.microsoft.com/en-us/solutions/microsoft-biztalk-server https://partner.microsoft.com/en-us/solutions/microsoft-bizt...
  - pjmlp 5y ago
    For how long? BizTalk heyday was during the Visual Studio.NET early days when SOAP was going to take the up the world of distributed computing as main replacement for CORBA/DCOM.
dsanchez97 5y ago
I agree with most the tradeoffs that were brought up with "low code" integrations tools, but I feel like none of the tradeoffs of using a general purpose program language were discussed. My viewpoint is that most organizations reach for these solutions because they provide highly available runtimes that people can quickly access. If I am at an organization and not directly on a software development team and want to do a simple data transformation between two systems, how would I go about getting an environment to run my process? Even if I know how to write it using a general purpose language there are probably too many steps to go through to get a server environment to run my code on. With a low code solution, I can get my simple process running and automated much easier. A little while ago there was an article about the python environment run at big banks that allow banking analysts to easily use python to solve their day to day problems. The banking python system was built to allow these people to quickly write python without much organizational overhead and therefore saw wide spread adoption among non traditional software developers. I have been thinking about this space because I am trying to build a platform on top of serverless (FaaS) platforms that would make it easy for companies to empower workers to use general purpose programming languages to solve their business problems.
- hoten 5y ago
  The referenced article on Python in banking: https://news.ycombinator.com/item?id=29104047&p=2 https://news.ycombinator.com/item?id=29104047&p=2
- lmm 5y ago
  It's the configuration complexity clock all over again. Code errors cause outages, then we impose a bunch of process and restrictions on code deployment, then we make our code configurable so that business users can easily change the behaviour, then configuration errors cause outages.
- Stranger43 5y ago
  How do you get time/instances on an no code platform? This problem was solved in 80ies on most timeshare platforms(read unix) where you would simply let users access an general purpose environment on a shared server and run processes in the background or kick up batch job using some kind of scheduler(crontab). The old unix "shell account" should really be the benchmark for FaaS to beat and so far i dont think anyone have gotten close for small scale project.
barnyfried 5y ago
Ok so here's the deal, although this is sort of well written-ish. There is absolutely no surprise waiting for us in COTS, phony 4GLs that purport to "solve a problem". I mean really? The fact that people still huck these tools means the software business might not be dead. Now if you work for a company that buys these things, I hope your not in tech. Because these fool things which are "container engines for consultants", will cost you a ton of money. Maybe you prefer it that way?, its your money. The technical argument which sort of tried to make some point relative to event driven architecture was pretty obviously weak. Whether or not I personally agree with that architecture, I think that there could be arguments made there. But we have MUCH bigger problems in tech. Some of them we cant talk about openly. Like labor sourcing and scrum. So we may have people problems folks. Let tool salesmen do their jobs and tend your own field and your own flock.
nwhatt 5y ago
It's not really clear if this advice is for within the enterprise (a company that maintains so many systems they spend millions to integrate them) or at the edge (a company that needs to integrate with multiple third parties). I can see the argument for the intra-enterprise work being valuable, but at the edge I'm less convinced.
- chasil 5y ago
  I arrived in my position in the late 90's, greeted by HP-UX, VAX VMS, and Unisys OS2200. All of this was tied by our homegrown queue software. Some years later, I was tasked with migrating HP-UX to Linux, in stages, and I was lucky enough to have found all the queueing source code, and I rebuilt it, imperfectly, on Linux. We hired back our past workers, including the original author, who made perfect my original port. As the years have passed, I continued to improve this code, compiling with directives from "hardening-check" after bug hunts. It has tied me to my role, but I suppose that I am happy here.
- bbyars 5y ago
  Good feedback. Most of the article is about integration inside an enterprise, although in the next installment I do talk about integration between organizations (hint: there are scenarios where I think low-code tools make sense there).
fitzoh 5y ago
Two things missing from the Step Functions portion of the article: * Step Functions module in the AWS CDK: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-stepfunctions-readme.html https://docs.aws.amazon.com/cdk/api/latest/docs/aws-stepfunc... * Step Functions Data Science SDK (python): https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/ https://aws-step-functions-data-science-sdk.readthedocs.io/e... There is first class support for defining Step Functions using real programming languages (although it does compile down to the states language dls before deployment)
pqyzwbq 5y ago
Yes, Best programmers can win against any integration tools. But in real world, it rare to find enough "good programmers", so they can not abstract good components and good layers. So, at this case, the business integration tools will win. Those tools mainly sell best practices and well-tested components, so, although you are limited to what those tools provided, but will solve those tricky small things without your worry. In the meantime, the business integration tools can learn more concepts from general programming languages, like: version control, unit test, external variables, integrate sub-components into another combined component, shared macros, etc.
quintes 5y ago
The article was unfortunately all over the place for me, I got little out of it.
trabant00 5y ago
There's a lot of things you can't buy in IT. There's a lot of things you don't need to either because they are free. For all this you must find people who are passionate and have interests aligned to your own. This is how I aim to hold the interviews I conduct for infrastructure engineers. The most important question I ask is "Why do you want to work in this position?". 10 years ago I used to care mostly about candidate know-how, now I care mostly about drive.
- DocTomoe 5y ago
  That particular question is a major red flag to me as an interviewee, maybe because most people who ask it are not interested in an honest answer (the most honest being: "you offer money for work, I offer work for money"), but have been told by some HR voodoo master that this almost-ritual incantation reveals deep psychological insights into the interviewee's mind. At best, it only tells you if they prepared by memorizing ideal answers and can lie convincingly. An infrastructure engineer's top motivation does not have to experience a deep appreciation of the faux philosophy in your founder's ghostwritten business tales book for him/her to be an effective employee. Things like soft skills and technical know-how are a lot more important - things that cannot be easily faked in an interview situation. The best interviews are those that do not follow a script. Remember: In a job interview, the potential hire is not the only one on trial.
  - trabant00 5y ago
    That particular question springs from my own motivation at fulfilling that role. I do not know why you think I copied it from some faux book. And "you offer money for work, I offer work for money" is not a valid response in some cases, some positions require high internal motivation. This is all from my experience of hiring people to work under me, I don't care about the company or HR or anything like that. I care if I will have to push them constantly to deliver or if they desire this for themselves.
- sokoloff 5y ago
  I’ll happily write working computer programs for free and do it in my spare time and with my kids. What companies are paying me for is to focus on and solve the company’s specific problems.
jl6 5y ago
What are peoples’ thoughts on industry data models? These feel like integration by a different name. They often come with integration tooling as well as the core logical/physical model. That tooling is often “config not code”. And the database(s) which instantiate that data model act as the integration nexus for an organization, like a passive ESB. They become the architectural center of the organization, and the primary concern is conformance to the model (with the aim of achieving some future reusability benefit) rather than creating clean interfaces over capabilities. Seems like all the same concerns from the article apply.
davidgl 5y ago
> It’s easy to over-index on reuse as a primary goal of APIs (I believe taming complexity is a more important goal) So true !
AtNightWeCode 5y ago
Pretty much the same as my opinion on the topic. The nature of integrations is that you must expect anything. I worked with some of these flow engines and sooner or later you end up writing small integrations that are used by the integration tool. In many cases it would have been easier to just have everything in one integration code base. At least for debugging.
rawgabbit 5y ago
I have worked with Mulesoft before. It is a classic case of over-promise and underdeliver. It promises drag and drop functionality to integrate systems using its Mulesoft Composer. While that may work for many situations, often I wound using Anypoint Studio (eclipse shell), struggle with its dataweave data manipulation language, and googling the java error messages it spat out. It is also expensive as all Salesforce products are. With Mulesoft's Cloudhub, we pay for "virtual cores". We schedule Mule package to the tenth of a virtual core and stack their schedules to not exceed our assigned limits. Each virtual core is essentially a virtual machine/java process/network that is independently restartable. My biggest complaint is dataweave. It mixes objects/arrays/literals without aiding the developer in any meaningful way. https://docs.mulesoft.com/dataweave/2.4/dataweave-cookbook-extract-data https://docs.mulesoft.com/dataweave/2.4/dataweave-cookbook-e...

275 points by mpweiher 5y ago

vog 5y ago

I have always suspected this to be the case, but it is great to see it confirmed and well-explained in detail by an almost-authority in our field. I wonder if in the near future we'll see a similar article regarding security - "You Can't Buy Security".

purerandomness 5y ago
Wow, didn't expect to read something as condescending as calling Martin Fowler an "almost-authority" in this thread...
- gnabgib 5y ago
  While it's Martin's blog, most posts these days are by other people.. this particular one is by "Brandon Byars"

jefflombardjr 5y ago

Really great read. Long but worth it. This is a good critique of low-code/no-code in general. The root problem I see of "Unfortunately, when we frame the problem space that way, we have allowed our tools to think for us." is that in reality there are just not enough qualified software engineers out there.

gnabgib 5y ago
The article isn't finished yet (see the footer)... so it'll be a longer read
keyle 5y ago
I got bounced off a front-end job once, for not being good enough at React. Mind you I've been building SPA's since they weren't "a thing" and today I mostly use Vue. But for that role, they didn't like my lack of .env variables in the front-end (such as for things that end up in the HTML), after a 3 hours coding test, and a couple of minor React-y tid bits they couldn't even clarify. Basically, not "idiomatic". Meanwhile, none of them could tell me how React actually works, beyond throwing jargon vomit. They couldn't write a web application without React. The recruiter, which we had to go through, basically had no empathy and saw me as a failed resume. I felt pretty helpless, even though I've said many times to both parties that I wasn't a "React developer". That is the sad state of affairs today; and I'm finding it's not just in the front-end, where you could argue that you need sanity on this pile of rubbles. It's also in the backend, especially on the auto-magic "DevOps!"
- w-j-w 5y ago
  There's an even worse shortage of qualified interviewers out there.
- gitfan86 5y ago
  I have been in a similar boat. I was spending a bunch of time avoiding extra API calls and extra renders in React and the team I was working with didn't seem to understand that there were potential performance issues around that. They said "just use memo" On the other hand I think it is a natural progression. 30 years ago experienced engineers would look at young people who didn't know how to fix a PCB or read bytecode and think it was a shame. If everyone had to get a computer engineering degree and work at several different roles before they could even build a UI, that would be bad for productivity.
  - debaserab2 5y ago
    It does make you wonder what is going to get missed though, doesn't it? Each generation gets to work on a level of abstraction higher than the one before it. I think about that often when I think about all the things I don't understand about hardware and EE.
    - gitfan86 5y ago
      I did computer engineering in school 20 years ago, and have always enjoyed understanding technology. So I know what is happening from the gate in the cpu all the way up to the pixel in the monitor. The downside is that I'm spread pretty thin. I can't pass leetcode exams in the 30 minutes of allotted time, ops people can't understand why I don't use containers, and my knowledge is probably useless in making a modern CPU.
      - keyle 5y ago
        This resonates with me. I'm actually thinking that at some point, no one will be building custom software. Everything will be available off the shelf, as some product, running on some cloud within a few clicks. So "custom software" developers are a dying breed. For example, when Jira is down, a Jira "specialist" will tinker and bring it back and look like some guru walking down the mountain of knowledge. Arguably, this person probably wouldn't know anything about SIMD. Yet the SIMD guy is looking for a job, no one cares about that guy. But the "salesforce developers" and "Jira specialist" are permanently employed at very high wages.
hinkley 5y ago
I’ve hear the way to succeed with SAP is to reorganize your business to match either the default world view or some other cookie cutter variant. Essentially using it no code style. Otherwise you’re exerting yourself doing “normal” things. That’s not sustainable.
- r00fus 5y ago
  This outlook is actually endemic in the ERP implementation space. The part you're not going to like hearing is that, due to laws/regulations, that "default world view" may be the best way to approach that problem for that given module/area (e.g. how to structure benefits or compensation). "Creativity" in the ERP space is costly. Also reducing your variation to something that's configurable as opposed to custom codebase means you don't have to do extensive testing whenever a patch or upgrade to the ERP software comes out.
- NikolaNovak 5y ago
  All ERPs are fundamentally like that. At the end of the day you are purchasing a COTS product specifically to gain efficiency of industry best practices already coded for you. If you take a COTS product and then try to rewrite it to fit your "unique" business processes, we'll, that's why half of them fail. Implementation of erp is incorrectly and disastrously viewed as an IT project. It is first and foremost a business transformation project. A company should be self-aware to say "HR|Accounting|whatever is NOT our core business or competitive advantage; it's not what we are great at. It's not a thing we should be unique in. It's a cost centre and we need to standardize and minimize that cost with help of people and software that were successful with many other companies". (Source: I've been implementing Peoplesoft, a competing erp,for 20 years for dozens of companies, as a technical resource eventually wise enough to be aware it's fundamentally not a technical endeavour :-)
  - kayodelycaon 5y ago
    This is why my company has custom applications that talk to the ERP for certain departments. Inventory control in the ERP isn’t going to work with reality. The data modeling is good enough but the interface is not. Rather than train everyone we hire to use horrifically bad and expensive barcode readers, we purchased cheap smart phones, put them in cases, and locked them down to run an app that uses the camera to scan barcodes. Drop a phone? It’s encased in rubber. Run it over with a forklift? Toss them another one from the box and then figure out why this only happens on Tuesday afternoons. We already need a dev team for other reasons. Adding another decent programmer to create and support internal apps isn’t that expensive in the long run. Plenty of good devs like me who are happy to work remote from the Midwest at less than FAANG rates. :)
hef19898 5y ago
I have it seen time and again. Organizations and leadership being faced with a problem and jumping directly to tool / software selection and implementation before really having understood the problem itself, the processes and requirements. After selection, directly to implementation. Because, you know, we a re agile and working in sprints. So we can sort out architecture and interfaces as we go. End result, tools force you into the use not understood processes that have nothing to do with your business requirements whatsoever, an architecture that is cobbled together and users that don't understand neither. Even better if the tools are selected based on previous experience alone.

csours 5y ago

You Can't Buy Understanding of Your Problems You Can't Buy Someone Else Caring about Your Problems It seems like you can hire experts to understand and consultants to care about your problems, but someone has to bring the intent and to close the loop.

malermeister 5y ago
You can hire experts to pretend they care and do whatever's the most convenient for them and since you don't understand, you'll never know the difference.
RealityVoid 5y ago
Hah, I posit that there is a _very_ low possibility of buying someone to care about your problems.
- mypalmike 5y ago
  This is why visiting a doctor can be a rather frustrating experience.
  - kansface 5y ago
    Exactly right! The doctor has her own problems to care about!
- charcircuit 5y ago
  Everyone has a price.
  - RealityVoid 5y ago
    I posit that paying money only buys the appearance of caring, not the actual caring. That is the reason celebrities get (got? Not up to date) doctors who prescribed addictive drugs, because they just gave the client what they wanted, not what they needed. You really want someone who speaks truth to power, not just pleasing nonsense. And the power must be able to discern the two.

debacle 5y ago

Fowler doesn't ship. When you read his articles, especially when they make you feel like you know something others (especially those awful management types) don't, remember that he hasn't made anything. He is a software life coach.

robbintt 5y ago
It's not true. I am literally reading "Refactoring" right now. It's very good.
Jtsummers 5y ago
Fowler isn't the author so if you're reading this article you aren't reading one of his (Fowler's) articles. Check the byline: Brandon Byars. I have no knowledge of this person so can't comment on his particular relevance, but seriously, took 1 second to see who the author is.
hobofan 5y ago
Not sure why you are writing this. Fowler isn't the author of the article. martinfowler.com nowadays is just the Thoughtworks blog. I can't remember the last time reading a article by Fowler on there.
- debacle 5y ago
  Even better, we're listening to someone who has accomplished less than Martin Fowler.
  - dang 5y ago
    Please don't do personal attacks on HN. The damage to the ecosystem outweighs any benefit such harshness provides. You can make your substantive points without that. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

988747 5y ago

It is a weird article. Fowler obviously knows a lot about integration, but since his business is about writing software he tries to discredit "low code tools", so that more people use his services. I think that those tools are great for exactly the reasons indirectly stated in the article - they take the mind off the implementation and let you focus on bigger picture. I was involved in tons of integration project and they were always 90+% about communication: agreeing on interfaces. The actual implementation is almost always trivial, with tools like TIBCO or MuleSoft you can sometimes do it in 15-30 minutes (for simple flows, the more complex ones can take up to 3 days).

random314 5y ago
This article seems to be by Brandon, not Martin

cardosof 5y ago

The post isn't bad or anything, its just that coming from Fowler (or a partner) who sells software services and books on software for a living, it is as biased as it gets. I would love to read more on this particular topic but from the point of view of someone who needed to get the job done and tried no/low code tools and building his own thing and learned the pros and cons of each approach. Everything else looks like salespeople talking.

bbyars 5y ago
I find the bias claim an odd critique. I am of course biased -- in the appropriate context (e.g., architecture), bias is just another word for "judgment" -- but I don't think a claim of commercial bias is a strong argument. It is true that my company (Thoughtworks) is primarily a custom software delivery firm, but the point you might not have awareness of is that many (and most or all of the large) software services firms choose to build capabilities in low code platforms (in the past year alone, I've worked with Accenture and BCG Digital Venture Mulesoft experts and EY around Pega). In fact, many of the larger services firms get EXTRA commercial lift through partnerships with those low code companies ("finders fees"). We could easily do the same, make good money in the process, and take on a broader book of business in doing so. I'd have an easier time buying a related critique that neither I nor the colleagues I spoke to in shaping the article have exposure to other contexts where low code integration tools are the right choice. If you have some, I'd love to hear them.
- Mandatum 5y ago
  Why do I have the opinion that ThoughtWorks is better than Accenture (or EY, Deloitte, etc)? Do they pay more for talent? About half the projects I see come out of non-boutique consultancies are shit. Actually, no. More than half lately. Probably 80%. I work in a role that exposes me to a wide range of partner-led engagements across large enterprise, but I admit my role is limited by region. ThoughtWorks on the other hand is massive, and seems to pick and choose engagements a lot more. They also seem to stack their teams with relevant talent. If they're assigning juniors or mid-level devs, they almost always seem to get some oversight by someone with a lot of relevant experience and input before going live.

ako 5y ago

Regarding visual diffing: "The delta between source code commits can only be represented textually; graphical palettes are not designed to represent change over time." I would think visual systems might be in a better spot to visualize change over time. Some of Tufte's diagrams have interesting ideas of how this could be represented statically, but turning static code diffs into illustrated animations might even be better than code diffing...

gorgoiler 5y ago

Business logic happens line by line between interfaces. Coding may be all ifs and for loops, but when it comes to implementing a new feature, having chosen carefully where to draw the line between, say, your model and your view can be the difference between a 10 line patch and a complete rewrite. If you don’t have a culture of code review then maybe the rewrite will fly, bugs and all, through to production. If however you have the ability to reason logically about the salient changes diff by prescient diff — where by definition each prescient diff represents nothing but the salient changes — then you at least stand a fighting chance of version N+1 having fewer bugs than version N while also moving your business forwards. Anything else is a quagmire that will never converge on stability and it all starts with there being a 1:1 mapping between what you think and what you write. Java is particularly unforgiving for code where ones thoughts are unclear. As a complete aside: buy your dev team a fountain pen and a pad of paper for Christmas. Writing meaningfully is everything.

thewarrior 5y ago
The line “coding is just if statements” really annoys me. I used to think this earlier in my career. It’s technically true but it means that you haven’t really grasped what makes programming hard. The real challenge of programming is how you organize a million if statements in a way that the system can be understood and evolved predictably and cheaply over time. And any tool that takes away the control of how you organize your code and allows for the creation of complexity without proper version control, debugging, testing and code review is doomed to failure. No code is just code at a higher level of abstraction. No code works when you don’t need a lot of code to solve the problem to begin with. Which is kind of a mind bender.
- randomdata 5y ago
  Coding is just if statements, more or less. Programming and coding are different words with different meanings, though. What is the reason to conflate them?
  - codebje 5y ago

throwaway984393 5y ago

"Integration software products ... are not products that directly solve a business problem. ... They are programming languages, bundled with a toolchain and a runtime to support the compilation process." By this definition, Excel is a form of integration tool. A spreadsheet doesn't "directly solve a business problem". It's basically just a piece of paper with lines on it, along with some "programming formulas". A hammer would be an integration tool if it were digital. The technology industry is wholeheartedly committed to selling you the idea that you need to write software. The industry knows that will guarantee new customers (and profits) for decades to come, for all the things you'll need to buy into to "solve" the software engineering problems you'll undoubtedly encounter. But do you need to write software to solve it? Most other industries do not make things themselves when they need to solve a problem. They don't buy anvils when they need nails, they don't buy sewing machines when they need to clothe their workers, and they don't build industrial machines when they need to produce factory goods. They don't build cars when they need to deliver pizzas, nor build ovens to bake them, nor knives to cut them. There are always exceptions, but for rare, exceptional cases. I think the meat of this article is predicated on the idea that it's expected that you need to be somehow developing software to solve business problems. But really, none of the technologies in this article should be needed to solve most business problems. If there isn't an existing product for sale that's general enough to solve your problem, that's the problem we should be solving; not writing software for every single business case under the sun.

Jtsummers 5y ago
> "Integration software products ... are not products that directly solve a business problem. ... They are programming languages, bundled with a toolchain and a runtime to support the compilation process." > By this definition, Excel is a form of integration tool. A spreadsheet doesn't "directly solve a business problem". It's basically just a piece of paper with lines on it, along with some "programming formulas". A hammer would be an integration tool if it were digital. You're misreading that quote. The direction is actually important in conveying meaning, it's like trying to misread "Every ancient poet known as Homer was a man" as "all men are ancient poets known as Homer".
mr_toad 5y ago

sam_lowry_ 5y ago

In Microsoft BizTalk one could make transformation rules not by drag-and-dropping but by coding in some weird XSLT dialect. It was a joy to use compared to the usual three-pane screen.

lloydatkinson 5y ago
What actually is BizTalk? I'm a .NET dev pretty involved in the Microsoft stack and never once have I seen it in use or worked anywhere that uses it.
- gnabgib 5y ago
  Did you try wikipedia[0] or microsoft[1]? It's been around 20 years, but it's been fading away the last 5 [0]: https://en.wikipedia.org/wiki/Microsoft_BizTalk_Server https://en.wikipedia.org/wiki/Microsoft_BizTalk_Server [1]: https://partner.microsoft.com/en-us/solutions/microsoft-biztalk-server https://partner.microsoft.com/en-us/solutions/microsoft-bizt...
- pjmlp 5y ago
  For how long? BizTalk heyday was during the Visual Studio.NET early days when SOAP was going to take the up the world of distributed computing as main replacement for CORBA/DCOM.

dsanchez97 5y ago

I agree with most the tradeoffs that were brought up with "low code" integrations tools, but I feel like none of the tradeoffs of using a general purpose program language were discussed. My viewpoint is that most organizations reach for these solutions because they provide highly available runtimes that people can quickly access. If I am at an organization and not directly on a software development team and want to do a simple data transformation between two systems, how would I go about getting an environment to run my process? Even if I know how to write it using a general purpose language there are probably too many steps to go through to get a server environment to run my code on. With a low code solution, I can get my simple process running and automated much easier. A little while ago there was an article about the python environment run at big banks that allow banking analysts to easily use python to solve their day to day problems. The banking python system was built to allow these people to quickly write python without much organizational overhead and therefore saw wide spread adoption among non traditional software developers. I have been thinking about this space because I am trying to build a platform on top of serverless (FaaS) platforms that would make it easy for companies to empower workers to use general purpose programming languages to solve their business problems.

hoten 5y ago
The referenced article on Python in banking: https://news.ycombinator.com/item?id=29104047&p=2 https://news.ycombinator.com/item?id=29104047&p=2
lmm 5y ago
It's the configuration complexity clock all over again. Code errors cause outages, then we impose a bunch of process and restrictions on code deployment, then we make our code configurable so that business users can easily change the behaviour, then configuration errors cause outages.
Stranger43 5y ago
How do you get time/instances on an no code platform? This problem was solved in 80ies on most timeshare platforms(read unix) where you would simply let users access an general purpose environment on a shared server and run processes in the background or kick up batch job using some kind of scheduler(crontab). The old unix "shell account" should really be the benchmark for FaaS to beat and so far i dont think anyone have gotten close for small scale project.

barnyfried 5y ago

Ok so here's the deal, although this is sort of well written-ish. There is absolutely no surprise waiting for us in COTS, phony 4GLs that purport to "solve a problem". I mean really? The fact that people still huck these tools means the software business might not be dead. Now if you work for a company that buys these things, I hope your not in tech. Because these fool things which are "container engines for consultants", will cost you a ton of money. Maybe you prefer it that way?, its your money. The technical argument which sort of tried to make some point relative to event driven architecture was pretty obviously weak. Whether or not I personally agree with that architecture, I think that there could be arguments made there. But we have MUCH bigger problems in tech. Some of them we cant talk about openly. Like labor sourcing and scrum. So we may have people problems folks. Let tool salesmen do their jobs and tend your own field and your own flock.

nwhatt 5y ago

It's not really clear if this advice is for within the enterprise (a company that maintains so many systems they spend millions to integrate them) or at the edge (a company that needs to integrate with multiple third parties). I can see the argument for the intra-enterprise work being valuable, but at the edge I'm less convinced.

chasil 5y ago
I arrived in my position in the late 90's, greeted by HP-UX, VAX VMS, and Unisys OS2200. All of this was tied by our homegrown queue software. Some years later, I was tasked with migrating HP-UX to Linux, in stages, and I was lucky enough to have found all the queueing source code, and I rebuilt it, imperfectly, on Linux. We hired back our past workers, including the original author, who made perfect my original port. As the years have passed, I continued to improve this code, compiling with directives from "hardening-check" after bug hunts. It has tied me to my role, but I suppose that I am happy here.
bbyars 5y ago
Good feedback. Most of the article is about integration inside an enterprise, although in the next installment I do talk about integration between organizations (hint: there are scenarios where I think low-code tools make sense there).

fitzoh 5y ago

Two things missing from the Step Functions portion of the article: * Step Functions module in the AWS CDK: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-stepfunctions-readme.html https://docs.aws.amazon.com/cdk/api/latest/docs/aws-stepfunc... * Step Functions Data Science SDK (python): https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/ https://aws-step-functions-data-science-sdk.readthedocs.io/e... There is first class support for defining Step Functions using real programming languages (although it does compile down to the states language dls before deployment)

pqyzwbq 5y ago

Yes, Best programmers can win against any integration tools. But in real world, it rare to find enough "good programmers", so they can not abstract good components and good layers. So, at this case, the business integration tools will win. Those tools mainly sell best practices and well-tested components, so, although you are limited to what those tools provided, but will solve those tricky small things without your worry. In the meantime, the business integration tools can learn more concepts from general programming languages, like: version control, unit test, external variables, integrate sub-components into another combined component, shared macros, etc.

quintes 5y ago

The article was unfortunately all over the place for me, I got little out of it.

trabant00 5y ago

There's a lot of things you can't buy in IT. There's a lot of things you don't need to either because they are free. For all this you must find people who are passionate and have interests aligned to your own. This is how I aim to hold the interviews I conduct for infrastructure engineers. The most important question I ask is "Why do you want to work in this position?". 10 years ago I used to care mostly about candidate know-how, now I care mostly about drive.

DocTomoe 5y ago
That particular question is a major red flag to me as an interviewee, maybe because most people who ask it are not interested in an honest answer (the most honest being: "you offer money for work, I offer work for money"), but have been told by some HR voodoo master that this almost-ritual incantation reveals deep psychological insights into the interviewee's mind. At best, it only tells you if they prepared by memorizing ideal answers and can lie convincingly. An infrastructure engineer's top motivation does not have to experience a deep appreciation of the faux philosophy in your founder's ghostwritten business tales book for him/her to be an effective employee. Things like soft skills and technical know-how are a lot more important - things that cannot be easily faked in an interview situation. The best interviews are those that do not follow a script. Remember: In a job interview, the potential hire is not the only one on trial.
- trabant00 5y ago
  That particular question springs from my own motivation at fulfilling that role. I do not know why you think I copied it from some faux book. And "you offer money for work, I offer work for money" is not a valid response in some cases, some positions require high internal motivation. This is all from my experience of hiring people to work under me, I don't care about the company or HR or anything like that. I care if I will have to push them constantly to deliver or if they desire this for themselves.
sokoloff 5y ago
I’ll happily write working computer programs for free and do it in my spare time and with my kids. What companies are paying me for is to focus on and solve the company’s specific problems.

jl6 5y ago

What are peoples’ thoughts on industry data models? These feel like integration by a different name. They often come with integration tooling as well as the core logical/physical model. That tooling is often “config not code”. And the database(s) which instantiate that data model act as the integration nexus for an organization, like a passive ESB. They become the architectural center of the organization, and the primary concern is conformance to the model (with the aim of achieving some future reusability benefit) rather than creating clean interfaces over capabilities. Seems like all the same concerns from the article apply.

davidgl 5y ago

> It’s easy to over-index on reuse as a primary goal of APIs (I believe taming complexity is a more important goal) So true !

AtNightWeCode 5y ago

Pretty much the same as my opinion on the topic. The nature of integrations is that you must expect anything. I worked with some of these flow engines and sooner or later you end up writing small integrations that are used by the integration tool. In many cases it would have been easier to just have everything in one integration code base. At least for debugging.

rawgabbit 5y ago

I have worked with Mulesoft before. It is a classic case of over-promise and underdeliver. It promises drag and drop functionality to integrate systems using its Mulesoft Composer. While that may work for many situations, often I wound using Anypoint Studio (eclipse shell), struggle with its dataweave data manipulation language, and googling the java error messages it spat out. It is also expensive as all Salesforce products are. With Mulesoft's Cloudhub, we pay for "virtual cores". We schedule Mule package to the tenth of a virtual core and stack their schedules to not exceed our assigned limits. Each virtual core is essentially a virtual machine/java process/network that is independently restartable. My biggest complaint is dataweave. It mixes objects/arrays/literals without aiding the developer in any meaningful way. https://docs.mulesoft.com/dataweave/2.4/dataweave-cookbook-extract-data https://docs.mulesoft.com/dataweave/2.4/dataweave-cookbook-e...