7 ms·

I'm going to come clean, I am a shitty software engineer. I'd like to have things like CRDTs under my radar to pull from when architecting technical solutions.

I'm going to come clean, I am a shitty software engineer. I'd like to have things like CRDTs under my radar to pull from when architecting technical solutions. But I don't. I have been coddled by the bullshit of "just-good-enough" web development for nearly a decade, and I feel like I will be haunted by it forever at this point. I WANT to employ mathematically proven solutions, but not only has it been 10+ years since studying computer science (and with it, the relevant mathematics) but I never even graduated. So here I am, another under-educated over-paid GED holding Rails engineer putting together shit solutions I know will eventually leak. So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start? I feel like going back to school full-time is out of the question, because I have a life now. Maybe online classes? I'm curious to hear thoughts from people who maybe came from a similar pit of despair, or otherwise understand what I am talking about. It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again. Is this just the profession I have locked myself into?

rememberlenny 6y ago
Check out http://www.roomservice.dev http://www.roomservice.dev Turns your front end app multiplayer with a library. They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.
- mentat 6y ago
  > The major downside of having an “always syncable” data structure is that it gets really big, really quick Perhaps it would be prudent to read the actual article before tossing off this sort of comment since it's indeed the whole point?
  - btilly 6y ago
    Normally I would downvote a comment that amounts to "read the article", except that this one is entirely on point. One of the classic problems was size and performance. However both problems have now disappeared. Operations are `O(log(n))` and the size overhead is a factor of 1.5-2.0. That's quite workable.
- jlu 6y ago
  Mind elaborating more on how’s is it better than CRDT?
tobyhede 6y ago
I am in this photo and I don't like it.
adventureadmin 6y ago
I feel ya, I did not graduate either, and didn't go for CS. After having Linux as a hobby, I made my way into a web dev job. The "good enough" work is killing me. I feel like I am pushing and pulling my team to the light, even simple things like DRY. I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating. I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.
- solipsism 6y ago
  You guys are overestimating the degree to which concepts from CS curricula are relevant to the average engineer's work. "Pushing and pulling your team to the light" is normal, can't be solved by getting a CS degree, and is more about your team dynamics than anything.
mrdoops 6y ago
Picking up Elixir is a decent way to get into backend distributed systems from a Ruby/Rails background.
- lawik 6y ago
  I think this is surprising levels of true. Elixir really works as an on-ramp from web to a lot of other stuff.
tootie 6y ago
When this guy discusses OT vs CRDT he's discussing algorithms for some very specific use cases. Namely, concurrent document editing a la Google Docs. Realistically, there's maybe 50 people in the world who are versed in these kinds of algorithms at a really deep level. 99% of us are users. If I was pressed into it, I maybe have the chops to implement a reference implementation for something like this, but I've never come with a country mile of an organization that has the resources to invest in something like that. You can be a great programmer with a very successful career as someone who only knows how to read an API and use it. And unless you are tasked with creating a concurrent document editor, you will probably never need to understand the API for a CRDT.
- zurfer 6y ago
  generally agree, but I bet there is more like 500-5000 people that understand this kind of stuff. Most modern web tools support at least some basic functionality of synchronously working on the same document.
  - tootie 6y ago
    Concurrency is really broad problem space with a million applications, but not real-time editing of an indeterminate data object. Concurrent modification of a row in a database is a perfect example of deeply complex problem that 99% of us don't really need to understand beyond how to use the API. If you know how to commit a transaction, you don't need to read the thousands of dissertations on how to process them optimally.
peterwwillis 6y ago
No, you have not locked yourself into anything, you can always grow. Comfort can breed complacency, so my initial advice is to quit your job and try to get a new one that's more demanding. It forces you to kick your own butt into learning new stuff. I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work. I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.
Ericson2314 6y ago
CRDTs are just applied partial order theory. Partial orders are just mazes where you can drag your finger in a certain vague direcion, taking arbitrary turns in that vague direction, and still find the exit. Get your intuition down really good and 10+ years is not required.
jiggawatts 6y ago
For every million bricklayers, there may be one or two material scientists devising new clay formulations to make stronger, cheaper bricks. The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future. The material scientist may hit a dead end, or their company may axe their entire department to "cut costs". One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded. I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.
- cageface 6y ago
  This is exactly why I've focused on boring, mainstream tech skills. I'd rather be doing audio dsp or machine learning or fancy AR apps but I figure vanilla js/web chops are going to be in demand for the foreseeable future and pay well enough to support a decent lifestyle.
  - fnord123 6y ago
    Good thing this is just a dress rehearsal for when you can spend time working on things that interest you!
  - jiggawatts 6y ago
    Look at it this way: Other people tell us what they think we should do with our lives. What we should find interesting, or fulfilling, or is good for our career. Meanwhile, I feel like I'm an adult and can make my own decisions. I don't care if what I like is not "cool" or "hip" or some "up and coming technology". I don't have to be in research or the cutting edge of something to make interesting personal discoveries. In my career I've accidentally stumbled upon multidimensional OLAP cubes (in the form of SQL Server Analysis Services) and discovered that they're quite fun to work with, actually. The reputation of business analytics as something "dry and boring" is undeserved. It's a wonderful mix of human problems like "manufacturing calendars" and quirky sales bonus calculations on top of fancy optimisation tricks to make calculations over terabytes of data nearly instant. I've also had the incredibly gratifying experience of making a bog-standard web app with only a few mildly tricky technical bits for a department of education. It helped about a million kids read more than they would have otherwise. That's something. In fact, I'm certain that my one library app made kids read more books they wouldn't have otherwise than a dozen English teachers. My parents were educators, and I outperformed them a hundred fold without ever stepping foot in a classroom as a teacher.
yowlingcat 6y ago
You are not a shitty software engineer who's been "coddled." What you likely are is dissatisfied with where your career is going (and who amongst us is not in some way) and nervous about not having a name brand pedigree so you are turning it inward and trying to understand it in ways that imply the cause and effect are localized to yourself. That's understandable but I wonder if it will get you anywhere. Stop beating yourself up and start exploring what you really want. Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all. Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way. Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better. See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move. Good luck. I believe in you.
- choicenotchance 6y ago
  To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better. This. Especially that last line. I feel most of the time I'm just wanting to get more and more, both financially and challenges wise. But to what end? Your words here really calmed me down. I got me thinking. I felt like a dog running after a car.
datagram 6y ago
> So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start? HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms. Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly). For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.
josephg 6y ago
Author here. I love this comment. > So, having said all of that, is there hope for adding things like this to my toolbelt? Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know. We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.
- jezclaremurugan 6y ago
  Loved your blog - but I think the parent's question was how to gain the skills of working/designing OT/CRDTs etc on their own. (I am clarifying the question - since I'm interested in the answer as well)
  - vanderZwan 6y ago
    I honestly think a good skill to have as a programmer is to know when something is a complex, niche problem with inherently "fragile" implementations (as in, it is really easy to make mistakes) that is therefore best solved by a few domain experts working on an open source library, and just use that library. Based on everything I've seen about them, OT/CRDTs sound like they are that.
    - josephg 6y ago
      (Author here) I agree with that conclusion. I think they're really fun to implement and explore. I've been working in the area for a decade. And I've implemented over a dozen variants of OT systems by now. And despite all that I still consistently make mistakes that are only found with a fuzzer. Even thorough unit testing is never quite enough. I recommend implementing them for the fun of it. I love it. But don't put anything you write into production until you can leave a randomized test suite running overnight without errors. Example: Random op generator: https://github.com/ottypes/text-unicode/blob/master/test/genOp.js https://github.com/ottypes/text-unicode/blob/master/test/gen... Which powers this fuzzer: https://github.com/ottypes/fuzzer/blob/master/lib/index.js https://github.com/ottypes/fuzzer/blob/master/lib/index.js
perfectritone 6y ago
I've worked for multiple Rails shops with high quality engineers. The trick is to interview them too before accepting any offer. The best teams I've been on have been small. Past a team/project size, maintaining high quality code becomes much more difficult.
erikerikson 6y ago
I might suggest starting with one thing at a time, pulling in more and more. No one learns it all at once. I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0]. [0] https://www.confluent.io/stream-processing/ https://www.confluent.io/stream-processing/
rswail 6y ago
I never graduated, but that was back in the 80s. The important part is not getting "official" education, but learning how to self-learn. You don't need to necessarily understand the underlying maths, if you can understand the general theory. And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore. If you need to understand certain things at a deeper level, then explore the classes and other guided tuition. I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft. You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.
deleted 6y ago
[deleted]
namelosw 6y ago
This is my daily struggle routine. I cannot get rid of the idea that proceeding my career only accumulate how much I earn, instead of making myself a better programmer.
cs02rm0 6y ago
Sounds like you've got an awareness of CRDTs now. That's a win and a differentiator from many other engineers. I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas. I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough. That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!
bdg 6y ago
I wrote an article for folks like you (because I also felt like this was my life at one point). In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crdts https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory. I included all the resources I used to learn the concept on the page too. It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources https://statagroup.com/articles/editing-shared-resources ).
thallukrish 6y ago
I can give a philosophical answer rather than a technical one. Our lives are filled with learning various things. And we all realize there is no end to it. But truly what makes you feel better about what you do is not this endless learning. It is when you can drop what you learnt and think freely about a problem that you can solve. Until you reach there, you are in a boat tossed by the waves trying to grab whatever you can to keep yourself afloat. Please disregard this answer if it is not of any help.
vlfig 6y ago
I love the candor in your post but you're being too hard on yourself. Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.
daviddaviddavid 6y ago
The article mentions Martin Kleppmann. Go buy a copy of his "Designing Data-Intensive Applications". It may well have been titled "Practical Distributed Systems for the Everyday Developer". It is an absolutely fantastic book with a perfect ratio of theory/practice. Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)
humantorso 6y ago
Hey are you me? Cause this sounds like me.
silverlake 6y ago
> And where do I even start? There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.

rememberlenny 6y ago

Check out http://www.roomservice.dev http://www.roomservice.dev Turns your front end app multiplayer with a library. They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.

mentat 6y ago
> The major downside of having an “always syncable” data structure is that it gets really big, really quick Perhaps it would be prudent to read the actual article before tossing off this sort of comment since it's indeed the whole point?
- btilly 6y ago
  Normally I would downvote a comment that amounts to "read the article", except that this one is entirely on point. One of the classic problems was size and performance. However both problems have now disappeared. Operations are `O(log(n))` and the size overhead is a factor of 1.5-2.0. That's quite workable.
jlu 6y ago
Mind elaborating more on how’s is it better than CRDT?

tobyhede 6y ago

I am in this photo and I don't like it.

adventureadmin 6y ago

I feel ya, I did not graduate either, and didn't go for CS. After having Linux as a hobby, I made my way into a web dev job. The "good enough" work is killing me. I feel like I am pushing and pulling my team to the light, even simple things like DRY. I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating. I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.

solipsism 6y ago
You guys are overestimating the degree to which concepts from CS curricula are relevant to the average engineer's work. "Pushing and pulling your team to the light" is normal, can't be solved by getting a CS degree, and is more about your team dynamics than anything.

mrdoops 6y ago

Picking up Elixir is a decent way to get into backend distributed systems from a Ruby/Rails background.

lawik 6y ago
I think this is surprising levels of true. Elixir really works as an on-ramp from web to a lot of other stuff.

tootie 6y ago

When this guy discusses OT vs CRDT he's discussing algorithms for some very specific use cases. Namely, concurrent document editing a la Google Docs. Realistically, there's maybe 50 people in the world who are versed in these kinds of algorithms at a really deep level. 99% of us are users. If I was pressed into it, I maybe have the chops to implement a reference implementation for something like this, but I've never come with a country mile of an organization that has the resources to invest in something like that. You can be a great programmer with a very successful career as someone who only knows how to read an API and use it. And unless you are tasked with creating a concurrent document editor, you will probably never need to understand the API for a CRDT.

zurfer 6y ago
generally agree, but I bet there is more like 500-5000 people that understand this kind of stuff. Most modern web tools support at least some basic functionality of synchronously working on the same document.
- tootie 6y ago
  Concurrency is really broad problem space with a million applications, but not real-time editing of an indeterminate data object. Concurrent modification of a row in a database is a perfect example of deeply complex problem that 99% of us don't really need to understand beyond how to use the API. If you know how to commit a transaction, you don't need to read the thousands of dissertations on how to process them optimally.

peterwwillis 6y ago

No, you have not locked yourself into anything, you can always grow. Comfort can breed complacency, so my initial advice is to quit your job and try to get a new one that's more demanding. It forces you to kick your own butt into learning new stuff. I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work. I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.

Ericson2314 6y ago

CRDTs are just applied partial order theory. Partial orders are just mazes where you can drag your finger in a certain vague direcion, taking arbitrary turns in that vague direction, and still find the exit. Get your intuition down really good and 10+ years is not required.

jiggawatts 6y ago

For every million bricklayers, there may be one or two material scientists devising new clay formulations to make stronger, cheaper bricks. The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future. The material scientist may hit a dead end, or their company may axe their entire department to "cut costs". One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded. I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.

cageface 6y ago
This is exactly why I've focused on boring, mainstream tech skills. I'd rather be doing audio dsp or machine learning or fancy AR apps but I figure vanilla js/web chops are going to be in demand for the foreseeable future and pay well enough to support a decent lifestyle.
- fnord123 6y ago
  Good thing this is just a dress rehearsal for when you can spend time working on things that interest you!
- jiggawatts 6y ago
  Look at it this way: Other people tell us what they think we should do with our lives. What we should find interesting, or fulfilling, or is good for our career. Meanwhile, I feel like I'm an adult and can make my own decisions. I don't care if what I like is not "cool" or "hip" or some "up and coming technology". I don't have to be in research or the cutting edge of something to make interesting personal discoveries. In my career I've accidentally stumbled upon multidimensional OLAP cubes (in the form of SQL Server Analysis Services) and discovered that they're quite fun to work with, actually. The reputation of business analytics as something "dry and boring" is undeserved. It's a wonderful mix of human problems like "manufacturing calendars" and quirky sales bonus calculations on top of fancy optimisation tricks to make calculations over terabytes of data nearly instant. I've also had the incredibly gratifying experience of making a bog-standard web app with only a few mildly tricky technical bits for a department of education. It helped about a million kids read more than they would have otherwise. That's something. In fact, I'm certain that my one library app made kids read more books they wouldn't have otherwise than a dozen English teachers. My parents were educators, and I outperformed them a hundred fold without ever stepping foot in a classroom as a teacher.

yowlingcat 6y ago

You are not a shitty software engineer who's been "coddled." What you likely are is dissatisfied with where your career is going (and who amongst us is not in some way) and nervous about not having a name brand pedigree so you are turning it inward and trying to understand it in ways that imply the cause and effect are localized to yourself. That's understandable but I wonder if it will get you anywhere. Stop beating yourself up and start exploring what you really want. Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all. Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way. Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better. See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move. Good luck. I believe in you.

choicenotchance 6y ago
To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better. This. Especially that last line. I feel most of the time I'm just wanting to get more and more, both financially and challenges wise. But to what end? Your words here really calmed me down. I got me thinking. I felt like a dog running after a car.

datagram 6y ago

> So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start? HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms. Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly). For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.

josephg 6y ago

Author here. I love this comment. > So, having said all of that, is there hope for adding things like this to my toolbelt? Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know. We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.

jezclaremurugan 6y ago
Loved your blog - but I think the parent's question was how to gain the skills of working/designing OT/CRDTs etc on their own. (I am clarifying the question - since I'm interested in the answer as well)
- vanderZwan 6y ago
  I honestly think a good skill to have as a programmer is to know when something is a complex, niche problem with inherently "fragile" implementations (as in, it is really easy to make mistakes) that is therefore best solved by a few domain experts working on an open source library, and just use that library. Based on everything I've seen about them, OT/CRDTs sound like they are that.
  - josephg 6y ago
    (Author here) I agree with that conclusion. I think they're really fun to implement and explore. I've been working in the area for a decade. And I've implemented over a dozen variants of OT systems by now. And despite all that I still consistently make mistakes that are only found with a fuzzer. Even thorough unit testing is never quite enough. I recommend implementing them for the fun of it. I love it. But don't put anything you write into production until you can leave a randomized test suite running overnight without errors. Example: Random op generator: https://github.com/ottypes/text-unicode/blob/master/test/genOp.js https://github.com/ottypes/text-unicode/blob/master/test/gen... Which powers this fuzzer: https://github.com/ottypes/fuzzer/blob/master/lib/index.js https://github.com/ottypes/fuzzer/blob/master/lib/index.js

perfectritone 6y ago

I've worked for multiple Rails shops with high quality engineers. The trick is to interview them too before accepting any offer. The best teams I've been on have been small. Past a team/project size, maintaining high quality code becomes much more difficult.

erikerikson 6y ago

I might suggest starting with one thing at a time, pulling in more and more. No one learns it all at once. I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0]. [0] https://www.confluent.io/stream-processing/ https://www.confluent.io/stream-processing/

rswail 6y ago

I never graduated, but that was back in the 80s. The important part is not getting "official" education, but learning how to self-learn. You don't need to necessarily understand the underlying maths, if you can understand the general theory. And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore. If you need to understand certain things at a deeper level, then explore the classes and other guided tuition. I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft. You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.

deleted 6y ago

[deleted]

namelosw 6y ago

This is my daily struggle routine. I cannot get rid of the idea that proceeding my career only accumulate how much I earn, instead of making myself a better programmer.

cs02rm0 6y ago

Sounds like you've got an awareness of CRDTs now. That's a win and a differentiator from many other engineers. I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas. I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough. That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!

bdg 6y ago

I wrote an article for folks like you (because I also felt like this was my life at one point). In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crdts https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory. I included all the resources I used to learn the concept on the page too. It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources https://statagroup.com/articles/editing-shared-resources ).

thallukrish 6y ago

I can give a philosophical answer rather than a technical one. Our lives are filled with learning various things. And we all realize there is no end to it. But truly what makes you feel better about what you do is not this endless learning. It is when you can drop what you learnt and think freely about a problem that you can solve. Until you reach there, you are in a boat tossed by the waves trying to grab whatever you can to keep yourself afloat. Please disregard this answer if it is not of any help.

vlfig 6y ago

I love the candor in your post but you're being too hard on yourself. Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.

daviddaviddavid 6y ago

The article mentions Martin Kleppmann. Go buy a copy of his "Designing Data-Intensive Applications". It may well have been titled "Practical Distributed Systems for the Everyday Developer". It is an absolutely fantastic book with a perfect ratio of theory/practice. Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)

humantorso 6y ago

Hey are you me? Cause this sounds like me.

silverlake 6y ago

> And where do I even start? There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.