12 ms·
For ten years, I worked on codebases written in C++ without using any third party libraries (no, not even STL -- we needed faster, thread safe versions of strin
by hliyan 3y ago
For ten years, I worked on codebases written in C++ without using any third party libraries (no, not even STL -- we needed faster, thread safe versions of strings and maps which were not available at the time; Boost? What's that?), its own custom messaging middleware built directly on sockets, its on distributed process management system built with unix syscalls (obviously). There was not a single line of HTML, Java etc. anywhere. It was a comfortable, high-paid life and I solved cutting edge problems (HFT, distributed, concurrent computing). In 2013, I forced myself to quit, start with a small company that paid me half the salary so that I could learn the web tech stack from the ground up. It was all to avoid this fate. I'd like to think I've succeeded.
- Aloha 3y agoWas it something that ran in an embedded environment?
- hliyan 3y agoHigh frequency algorithmic trading. Often ran on colocated hardware within the exchange / trading system data centers. Started with stuff like Sun SPARC, Solaris and Oracle and we slowly made our way to Suse, Intel and... surprise, our own database optimised for fast writes.
- eamonnsullivan 3y agoBloomberg? Comdb? If not, sounds very similar, anyway. Been there, done that!
- bboygravity 3y agoNaive question: is this like creating fancy tech that is helping rich people to front-run the poor and middle-class? Because that's what a lot of people think online. Maybe it's more nuanced. I'm curious how someone from that carreer sees it?
- semi-extrinsic 3y agoNo, the poor and middle-class are not day trading stocks in any sort of meaningful volume. It is to help the super-rich who own hedge funds to frontrun the rich upper-middle-class people with significant disposable income.
- hliyan 3y agoUnfortunately, yes. It never sat well with me. Colocated systems saw latencies no mom-and-pop investor could ever dream to achieve. That is one of the reasons why the second company I joined was a non-profit that built systems to help track human rights violations, enforce labor rights, monitor democratic elections etc. Third job was in the food industry and the last one, in housing / property management. So I'd like to think I've made sufficient amends.
- ReactiveJelly 3y agoI wonder why mom and pop are investing in individual stocks at all. Buying the index has been a passable "get rich slow" scheme for me.
- lordnacho 3y agoThe whole business model is misunderstood by your average internet commentator. When you have a market, you need someone to be there to provide liquidity. Imagine if you're a farmer and you show up to the market with your wheat, but all the bakers have gone home that day. Or the baker shows up and there's no farmer. The market maker stands around all day offering to buy and sell so that you don't have to wait for the guy you're really trading with. Of course this middle man wants to get paid for it, but your cost as an average Joe is next to nothing. This is trading in time. Now imagine you want to cook a meal and your ideal meat is beef, but actually you're ok with pork, so long as the pork is cheap enough to make it worth it. How much cheaper should it be? Well your fellow who knows all the pork and beef people will be able to gauge where the balancing spread is, given the amount of interest. In fact he will from time to time do the trade when the spread is out of line. This is trading in space. So why all the fancy tech? After all market makers used to stand around in a pit in a colored jacket, and they didn't have degrees. My first boss in the market was one of these guys. Well, things have gotten very tech heavy because as soon as prices are out of line, there is money to be made. Or rather, lost. As a market maker, you are constantly out there with your prices, offering to buy or sell at a very small spread. If some news happens that affects prices in a big way, you can be sure that you will buy when it's going down and sell when it's going up. In order to both have tight prices and avoid this "adverse selection", you really want to be able to react as quickly as you can when your system decides that something's up.
- zabzonk 3y ago> so that I could learn the web tech stack from the ground up. It was all to avoid this fate same fate > I'd like to think I've succeeded. possibly not
- bowsamic 3y agoWhat about it would you like to avoid? I don't understand the fears that people have in this thread. To me this sounds like the Platonic ideal of a programming job. EDIT: I'll reply to my replier here because /u/Dang rate limited me (again), though I must commend him for letting me have one more comment than usual before he pushes the button ;) > I would say the ending -- the part where you are unemployed and unable to support yourself in your old age because you invested decades of your brainpower into a highly specialised, non-transferrable skill. I would contend that such a person, unless very reckless with finances or extremely underpaid, should have enough money saved up to make what is mentioned there very unlikely. That said, I know some high-income earners struggle to not blow their entire bank account each month
- hliyan 3y agoI would say the ending -- the part where you are unemployed and unable to support yourself in your old age because you invested decades of your brainpower into a highly specialised, non-transferrable skill.
- michaelteter 3y agoI should think that years of specialized C++ work in hft environments would have come with impressively high pay… in which case one could retire early and self study anything for fun, not hope to find a job for financial reasons.
- hliyan 3y agoIdeally, yes! Unfortunately, there is such a thing as being young and not knowing your own worth / not knowing how to negotiate / not having a manager who looks out for your career growth.
- michaelteter 3y agoUnderstood. In that case, if web development is your interest, then you have many possible paths. The primary choices are full stack, backend, or frontend. Obviously full stack is most flexible because you can theoretically do it all. But keeping up with so many technologies simultaneously is really tiring. Take a look at Phoenix framework. It is a fantastic way to get started, and it will grow with you into just about any future project need. Plus Elixir is an excellent language built on top of the incredibly powerful Erlang ecosystem.
- stefan_ 3y agoAnd now you are jumping from one "stack" to another, none the wiser, your knowledge hopelessly outdated just a year in. The principles you learned in HFT? Those apply all the same.
- armitron 3y agoSo you left a job that paid 2x and exposed you to hard problems that require reliable solutions for a job inside the horror clusterfuck that's the modern web where nothing really "works" and there's no real engineering to speak of besides monkeys throwing crap in a shared pile and shovelling it out the door ASAP. Good for you I guess?
- intelVISA 3y agoMost people I know, inc. myself, would call this a "bruh moment". I can't fathom willingly entering the web mines if you've already sunk in the hours to master high perf native code, OSdev, dist systems etc. People usually enter web to avoid having to learn such things. To discard that knowledge and instead compete with an endless stream of fresh grads, boot camp grinders and LLMs for less money? A madness if I've seen one.
- sys_prog 3y agoFun fact: I'm actually a bootcamp grad in web dev doing the opposite. I agree.
- shadowgovt 3y agoSpeaking as someone who does both: Web tech is more immediately rewarding. With C++ (and especially with modern web dev agile processes), you can spend a week building enough framework and infrastructure crap to stand up the barest inkling of a demo of the solution you plan to implement, only to have the whole thing pivot out from under you and become wasted work. More complicated languages and frameworks that manage memory and do UI and everything for you let you get to a prototype much faster, so you can see whether it'll work or whether you want to toss out the idea much sooner.
- seanthemon 3y agoTell me you know nothing about web dev without telling me you know nothing about web dev
- ncann 3y ago
- DrBazza 3y ago> so that I could learn the web tech stack from the ground up. It was all to avoid this fate. I'd like to think I've succeeded. I'm not sure if that's intentional humour or not.
- sph 3y agoOut of the frying pan, into the cow manure.
- hliyan 3y agoThere is a massive advantage in being able to see the web tech stack for what it is and wielding it without over-engineering or fanaticism, while your competitors are spending 2-3x on monstrosities that essentially does the same thing, but slower.
- ticviking 3y agoThe day a professor made me implement a simple http server was the day I achieved code enlightenment. At that level of abstraction it’s all just strings
- tambourine_man 3y agoThat was the web, yeah. Absurdly simple. Now all new protocols are binary and require reading several academic papers instead of a single RFC.
- bob1029 3y ago> At that level of abstraction it’s all just strings Exactly... I worry about the masses learning of this and gaining literacy over the landed gentry. PHP/CGI basically had it solved. I emulate PHP in .NET6 today using verbatim and interpolated string operations.
- a_subsystem 3y agoRelevant: https://twitter.com/serbantanasa/status/1582019078533046272 https://twitter.com/serbantanasa/status/1582019078533046272
- gnfargbl 3y agoI know it's not all about the money, but over the course of your entire career, do you think that move was financially positive or negative? I'm very roughly estimating that you could probably have worked in the HFT job for around two-thirds the time you're going to have spent working on generic web tech, and still been better off.
- hliyan 3y agoAbsolutely 100% positive. I was largely an individual contributor all that time, and only did some unstructured, untrained, unguided people & project management. Perhaps because it was such a niche, engineer-dominated environment. Several years after leaving that company, I quickly learned to effectively manage people and projects, architect systems using more mainstream/open source tech and eventually reached exec level (but that's a different story).
- markus_zhang 3y agoThat's really some fields I'd love to get into -- less need to talk to people and more need to talk to the machine. What a dream! Alas I only scratched the surface of C++.
- gnfargbl 3y agoAh, that's the missing punchline: you had ambitions to be something else other than an engineer, and you (quite correctly) recognised that you were never going to grow your wider skills at such a place. For someone who had wanted to stay an IC, I don't think the move would have been quite as wise. Being in a niche, engineer-dominated environment, and being paid extremely well for it, is nirvana for many lifelong ICs. It certainly feels very different from the experience of Arthur from the linked article.
- LudwigNagasena 3y agoI don’t think anyone working with low-level C++ for HFT can face similar fate. Most of skills gained at such job seem very transferable. The article seems to be more about jobs where 95% of work consists in implementing CRUD in accordance with very specific business logic of the company. The only reason to change such job I can imagine is to sit out your non-compete package. But I am not in this industry so maybe I’m missing something.
- bluGill 3y agoYeah, I find it amazing someone would transfer to standard web CRUD development from that. Sure you can have enough of all the custom stuff, but there are plenty of companies that will hire you for your current salary for non-web work. (you can probably guess how little I like web work)
- shrimp_emoji 3y agoSame! The poster's pivot reads like horror to me. Like they lost their mind and joined a cult where they worship Memnon with sermons of JavaScript. > It never sat well with me. Colocated systems saw latencies no mom-and-pop investor could ever dream to achieve. That is one of the reasons why the second company I joined was a non-profit that built systems to help track human rights violations, enforce labor rights, monitor democratic elections etc. Third job was in the food industry and the last one, in housing / property management. So I'd like to think I've made sufficient amends. Ah, nevermind. Turns out they were worshipping Memnon with C++ and renounced that cult with web tech! :p
- lordnacho 3y agoIt's not that you won't have the skills, it's somehow demonstrating them. I also worked at an HFT C++ shop that used no STL. Every data structure has a weird name and API that you get used to. The whole architecture has its own idioms, and hasn't needed much upgrading in terms of language features. Imagine you learn the grammar of English. You learn how to pluralise nouns, when to put an s at the end of a verb, how to to use commas, and so on. You then go to work at a place where they have a different word for coffee, drink, and order. Along with every other word. You end up learning where everything is and how to talk about it, and you're productive. How do you apply for another job? Chances are you will be asked how to order a coffee and drink it. Very real issue btw, I still have friends there.
- rr808 3y agoI did the same, I thought C++ was heading for obsolescence. Now a master of Java, Javascript and can write great full stack apps. Meanwhile HFT C++ jobs are paying double, I'm thinking of going back, even if the systems are boring.
- esMazer 3y agobeen working on java jobs and easy code bases since I graduated 10 years ago, nothing had been really challenging and I'm not complaining. Would be interested in a job that pays more even do you think working on C++ jobs is more challenging or just boring?
- rr808 3y agoDepends on the job of course. Probably more technical problems and tuning. Often code is low level which is interesting if its your thing but not if you like business problems.
- bla3 3y agoSounds like your old place had a good idea what's good about c++. It's not the STL and certainly not boost.
- intelVISA 3y agoLegit, is that place hiring? Sounds like an ideal stack free from the tragedy of the commons.
- sumtechguy 3y agoThose stacks can be full of 'not invented here'. They are unique and have their own quirks. Quirks that you more than likely can not google for any sort of guidance as to what is going on. I wrote one of those stacks a few years ago. No std libs at all. Why? must fit in 32k and most of that space is needed to hold data. That means throwing a lot of those helper bits out or spend weeks figuring out how to bend the compiler to your will and not include everything. When it is easier just to roll your on CS version of whatever. But it sucks because you probably have odd edge cases.
- intelVISA 3y agoYeh it's a rough but rewarding life; I spend most of my hours on problems that aren't on Google, or any LLM, so I'm kinda used to it now. (Stockholm) Would love to know of more spaces which require this level of programming - usually I feel kinda underworked at most shops. Though, true enlightenment is combining the urge to NIH with some, light, sensible borrowing from those smarter than us :) very much a key skill unto itself.
- HPsquared 3y agoGame engines have a lot of low-level C++... Often not the best working conditions though from what I hear.
- sumtechguy 3y agoThe space I was in was IoT type items. Usually hub controllers. The thing is that is getting less and less as you can get a fairly robust thing now with a couple hundred meg of flash and ram for a very reasonable price. So those 'must fit in 32k' items are few and far between these days. With that sort of space you may as well use the proper libs that are usually nicely documented and all the quirks have been seen by hundreds of other poor souls such as yourself.
- pantulis 3y agoI have a pretty similar story. I spent 7 years working as a subcontractor for a big telco with CORBA, pthreads and the ACE reactor library. First week, I halted a whole contact center service by adding a sleep call to the codebase (it was on UAT environment). Fast forward several years and I could effortlessly debug cores with thousands of threads, reason about semaphores, reentrancy and even tell the difference between the POA and the BOA Orbix adaptors. I began to become the mythical guy who knew what everything did not because I was technically good, it was because I understood the business and how it mapped to the different architecture elements. Then I felt like I could spend my whole career there and decided there was a whole world out there in the web, began to learn PHP in 2004 until the mythical "blog in 5 minutes" Rails demo appeared. I switched jobs in 2007 and I have never looked back.
- hliyan 3y agoInteresting. I've used semaphores with pthreads (only once -- after that I realised plain old mutexes will do). Like your sleep call, I once brought an entire service to a grinding halt by adding a simple `fsync` call (to fix a recovery failure because once a sequenced message we thought we had written to disk before a crash had actually got lost in the buffer). Understanding things at that level makes any new tech fairly easy to grok.
- adolph 3y ago> I understood the business and how it mapped to the different architecture elements If you have access to key systems, you can definitely reverse-Conway to understand how an organization really works. https://en.wikipedia.org/wiki/Conway's_law https://en.wikipedia.org/wiki/Conway's_law Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. — Melvin E. Conway
- pantulis 3y agoThat's so correct, this was Conway's Law in action!
- otabdeveloper4 3y ago> learn the web tech stack from the ground up Ouch. Should have stayed in legacy C++.
- bfrog 3y agoC++ is such a damn chore. This is partly why, everyone always assumes they can do it better, they don't trust any other code, so you end up with monoliths of NIMBY code and esoteric build setups. Code reviews are incredibly painful for the same sort of C++ reasons, everything can effect everything, the whole idea of encapsulation tends to fall apart when memory, time, ordering behavioral effects get relied on after some amount of time usage. Every change is a footgun to be found in production under just the right circumstances. Hard to be productive when you can't trust the language, the tools, or other developers to be correct.
- coliveira 3y agoI hope it worked for you, but I don't see any big difference. Web technology will become dated faster than the old C++ codebase. If you learn some web tech stack today I expect that it will be obsolete in 5 to 10 years. What are you gonna do next? Move to another small company for half the salary to learn everything again and compete with 20 year olds?
- commandlinefan 3y ago> in C++ without using any third party libraries (no, not even STL As God intended it!
- deleted 3y ago[deleted]
- deadletters 3y agoThat first place sounds awesome