5 ms·
I don't resonate with this at all, and think this attitude is toxic. To take the engineering analogy: this is like reducing civil engineering to "90% of buildin
by larve 4y ago
I don't resonate with this at all, and think this attitude is toxic. To take the engineering analogy: this is like reducing civil engineering to "90% of building a bridge is making shapes out of cement of varying quality". How you use cement of varying quality to build a bridge up to spec, that's the engineering part, and it's hard. Cement will always be of varying quality, because it's not unobtainium that you can mold into any bridge you want without effort.
Noawadays, I am in the business of writing CRUD e-commerce applications, using PHP, wordpress and jquery of all things. In the past 20 years I've worked on anything from bare-metal embedded to building search engines and doing "decent scale" data engineering. I'm basically dividing software engineering in two broad categories.
One is what I call "copying bytes". You get some business relevant input, and as a software engineer, your job is copy those input bytes to their business relevant destinations. For example, you receive an order from the customer (line items + shipping address), and you have to copy those line items and shipping address to the fulfillment department, recording a few log entries along the way. You might calculate some tax or apply a coupon along the way, slightly modifying those bytes, forward a few tracking metrics here and there, but roughly, you are just wrapping these bytes making up the order in fancy ways.
APIs for "copying bytes" are one of:
* "if you call readBytes, I will return someBytes",
* "if you call writeBytes, I will store some bytes somewhere"
* "please tell me when someone calls writes some bytes"
* "hey, someone just wrote some bytes"
The other is what I call "inventing bytes", or "transforming bytes in really complicated ways". This is the realm of funky algorithms, data science, mathematics, complicated stuff. This is taking an input 3D model and outputting some CAM instructions. This is taking a search string and computing features to quickly locate relevant information in indexes. The coding part of this style of software engineering is the tiniest part of the actual undertaking. Often, the hard thinking is done by people at universities or research labs. One could wonder if they're software engineers at all, and not "3D model transformation engineers", or "search index data structure engineers", as often their code needs to be rewritten by someone who knows how to do it pragmatically (say, port from the ugliest hacky matlab to java). The APIs here are weird, and frankly I don't really ever deal with them, so can't say too much as to their quality.
Chances are, you fall into the "copying bytes" category. Which bytes to copy, when, where, what log entries to add along the way, how to scale the copying, how to ensure the copied bytes are transactionally aligned with other copied bytes, that stuff is complicated. But it's turtles all the way down, one copyBytes API upon another, until you hit the hardware, and boy are the hardware APIs to copy bytes utterly nonsense.
You might think it's inane. You learned how to balance trees, and how to schedule processes, and how to do some differential curve geometry, how to write compilers and how design database engines, how to implement distributed consensus, and why don't you get to write all those beautiful algorithms as part of your day job? Thing is, clever people have already done so, and often they made a business out of it so that you can pay them for their clever thinking, hoping that they were indeed more clever than you, and save you time.
If you are not in the business of thinking clever ideas or making pretty things like video games, but in the business of making money, you will probably just write code to copy bytes, using code from other people that also copies bytes, until those bytes finally reach the rare unicorn that makes better bytes out of them. (here's a secret: if you do adtech or e-commerce or banking or embedded software, that actually doesn't happen. The business value of your software is to copy bytes to the right destination at the right time).
People want to transfer data for all kind of different reasons, in different ways, with different constraints. That means that even the best written API documentation will have to be generic enough to cover what was intended, and with some chance, what people are using it for. Most probably, it won't fit your use case to a T, and you call it "poorly documented" when really it means that people have different problems than yours. I actually think API documentation is insanely better than it ever was. All the APIs I had to deal with lately come with copypasta for 15 languages, a snappy searchable index, a plethora of tutorials and articles and videos.
But here's why I find my job copying bytes with subpar APIs singularly fulfilling:
Copying bytes is not easy, especially in today's distributed world. Doing it well and reliably is a tough problem. Figuring out how to do it reliably, securely and at scale requires a lot of thought. Figuring out which data to transfer to whom needs business understanding, knowledge of people and systems thinking. It requires planning a project, managing teams, dealing with changing requirements. It requires planning for maintenance, keeping a codebase healthy, mentoring other engineers, aligning with stakeholders, dealing with legacy code.
The APIs I ultimately use are the least of my problems. Some are nice, some are terrible, some are really problematic, but they don't have much to do with the "engineering" part of my job. My engineering is drawing pictures with boxes and arrows. The boxes are other people's systems, the arrows are other people's APIs. What arrows I draw and where they point to and what the little bubbles along them contain, and which boxes I chose to use, that's what I do as an engineer.
As a concrete example of a hard problem, fully implemented by glueing together shoddy APIs: fulfilling an order on an e-commerce store.
* seamlessly show payment process and checkout status to the customer using a slick UX in their browser (APIs: some shoddy REST, some janky web socket or whatever, some hacked react and some dubious CSS. Ultimately just shoveling the status code from the payment processor to the user's browser, potentially adding some curly braces and escaping a string or two.)
* log the transaction and provide metrics and analytics to business and ops (some inane pixel data, some putrid tracking JS, some shoddy SQL, maybe some janky JSON going to a SQS queue)
* process the payment information, order information and display it in the customers account section / backend CRM / fedex (I'm not even going to list how subpar the APIs are here)
* transform the shipping address into a proper shipping label with barcode (have fun with zebra and ZPL)
* dispatches the line items to the required fulfillment centers, provides picking information to the warehouse workers, with tracking, metrics, inventory handling, ...
Designing something like this so that it works correctly, efficiently, with visibility, reliability, scalability, and fulfills business needs, and sparks joy for every human involved. That's engineering.