8 ms·
I think that the job of a CTO is to minimize the surface area of "built in-house" so that your team can focus on things for which your customers actually pay yo
by zkirill 4y ago
I think that the job of a CTO is to minimize the surface area of "built in-house" so that your team can focus on things for which your customers actually pay you.
Sure, PostgreSQL can be used for sessions, but Redis has solved this problem on virtually every single platform a long time ago. Your customer probably doesn't even know what a session is, but they will definitely learn about them when your in-house implementation inevitably encounters an edge case.
Of course, PostreSQL has wonderful search capabilities but it was never intended to be used as a search engine. Solr was created in 2004 and is still being improved and used in production every day. Do you know what will happen to your full-text search when your customer types in a mix of English and Chinese characters?
Yes, PostgreSQL addition of SKIP LOCKED was neat but AFAIK the author of that feature himself recommended using a traditional job queue unless you had a very good reason against it. RabbitMQ was designed as a message queue and when you read the documentation you realize that they had encountered virtually every single problem and figured out a way to deal with it so that you don't have to. The documentation pretty much tells you what problems you are going to have later so that you can plan for them today.
Choose boring technology (tm), follow industry best practices, and enable your team to get their work done using the right tools for the job, so they can deliver the product to the customer, and leave the office on time.
- jmull 4y agoRedis doesn’t “solve” sessions.
- danielvaughn 4y agoI’m not sure I follow. Postgres has been around forever, it’s basically what I would call boring technology. Sure they’re still innovating, but the core tech is mature. An example of a “not boring” database would be like Cockroach or Planetscale.
- Delemono 4y agoChoose PostgreSQL as long as possible. This is probably good enough for 99% of things out there running stuff. For everything else you have architects which will do the right thing for you.
- robertlagrant 4y agoThis is build vs buy vs run. Yes, what you're saying might mean slightly less is built, but there's a lot more to run. Having ops people who know Postgres inside out might go better than having ops people who have to try and know four totally different technologies a bit.
- MasterYoda 4y agoI think "use postgres for everything" is a good solid foundation to start with (if you dont have a really good reason not to and know the exact use case beforehand). Can redis, rabbitmq or solr etc do a better job for a specific user case, of course becasue they are dedicated tools for the job. But the question is, is the builtin tool in postgres "good enough"? And for many thing they are. I see postgres like a tractor. It's the most important machine and you can attach alot of different equipment's to get alot of jobs done. Some times you notice a task need a specific machine that the tractor is not suited for, then you invest in that machine. To invest in all specialized machines from the beginning just because they can do one single job better than the tractor is not an efficient way of farming, there is a price to have all those machines too, that does not necessarily result in a better outcome.
- vb-8448 4y agoThis is the usual debate on "hammers and nails". You are absolutely right, but what if(just an example) you need transaction behavior between your queue system and your database? Good luck with 2pc or using saga and similar. Or, as other people stated, how much will cost to maintain postgres + redis + rabbitmq vs only postgres? In my opinion, the golden rule is: *use the most general purpose tool you know unless you hit a hard limit of the tool, in that case start moving to a specialized tool*.
- master_crab 4y agoCouldn’t say the first sentence any better. If your company doesn’t make money directly from the “thing you built” than it’s losing money, lot’s of money.
- manigandham 4y agoPart of being a CTO is knowing when that specialization, both in tools and in-house development, make sense for your context.
- WhiteOwlEd 4y agoBuilding on this, for many companies, the leader of the IT org has the main responsibility of focusing on end (or outside) customer needs at the highest quality with a low "Total Cost of Ownership".
- serverholic 4y agoI very much disagree with the idea that a CTOs job is to minimize code built in-house. Every app has different needs and there's always some code that would be a lot simpler if it were written in-house and specifically built for the needs of the application. Abstraction has a cost and if you take everything off the shelf you'll end up with a much higher overall level of abstraction in your codebase. Plus, your engineers aren't going to understand third-party code as well as in-house code. I've worked at companies that wrote almost everything in-house and I've worked at companies that had a phobia of in-house code. Both had problems and I think the real solution is somewhere in the middle. Edit: I also get a "nobody was fired for buying IBM" vibe from this.
- didibus 4y agoNot only that, but in-house offerings can be a competitive advantage. Sometimes it's over-engineering, and sometimes it's the secret to what makes your business successful over competitiors. In house has the ability to build only what is really needed and nothing more, and can adapt to your specific needs, it can also identify unique to your domain challenges and tailor solutions specific to that. A good CTO has good intuition into when and what makes sense to invest in an in-house solution and what is best using a self-managed open source solution, and what is best using a paid managed offering, and all manners of hybrids.
- BigJono 4y agoHonestly the biggest competitive advantage isn't in the thing you're building in-house, it's just the fact that you can debug it 10x faster than your competition. I never understood this until I saw some dysfunctional enterprise companies. I've seen entire teams spend 3 months doing fuck all because they can't get a local dev environment or build pipeline to work. If you only have Postgres, and you're weighing up whether to add a second dependency instead of installing a Postgres plugin or writing 50 lines of code, of course it's going to seem like an obvious choice ("I can get this working right now instead of spending a day or two on it! Wow!"). If you make a habit of making those decisions again and again, then your project is going to spiral out of control before you even realise it.
- otabdeveloper4 4y ago> I think that the job of a CTO is to minimize the surface area of "built in-house" so that your team can focus on things for which your customers actually pay you. The "built in-house" things are the only things that actually make your company competitive and provide shareholder value. There is no value in downloading and installing commonly-available stuff from the internet.
- pushedx 4y agoWhich is exactly why if you aren’t a database vendor, you shouldn’t have your engineers spend their time maintaining a proprietary database technology.
- otabdeveloper4 4y agoIf you maintaining a proprietary database technology solves a problem that provides some shareholder value, then go ahead and do it. Maintaining off the shelf software that you got from Github provides no shareholder value whatsoever, however. It is a pure cost center. As a CTO you do not want to be the guy that runs a pure cost center department.
- WJW 4y agoThere is negative value in rebuilding things that could have been downloaded for free from the internet. Postgres, MySql, Sqlite, RabbitMQ, Redis, Kafka and all the other common tools each have had thousands upon thousands of hours sunk into bugfixes, correctness guarantees and performance work. Rebuilding a poor version of that does not provide shareholder value at all.
- Winsaucerer 4y agoMany others have said this elsewhere in this thread, but the cost of using all those things is not free. Every added service is an extra burden to maintain, all the way from writing code through to running and maintained in production. Sometimes, the cost (both immediately and long term) of introducing a new component into your stack will exceed the cost of building and maintaining an inferior in-house solution, and sometimes it won't. Either way, the cost of these freely downloadable services is not just their cost to download.
- cube2222 4y agoI get where you're coming from, but I think that folks often underestimate the maintenance burden of additional components in their system. Sure, if your use case requires it, use Redis for sessions or RabbitMQ for queues. But you can also use a library with postgres, or even write the 40 lines of code yourself. Each component has its own debugging requirements and tooling. Each component adds a bunch of complexity. Sometimes it's worth it, sometimes it's not. It's not as clear-cut as you're making it be. There are pros and cons to both alternatives.
- xwolfi 4y agoThose 40 lines of codes give me nightmare. What if the dev disappear and we know only the C binary, source disappeared ? Little tiny custom stop gaps are the worst in a bit system.
- serverholic 4y agoI actually had this debate at work a few years ago. We needed a queue system and I wrote a postgres prototype in about 100 lines of code that would have worked fine for our use-case and would have likely scaled with us for years of growth. My boss said he just wanted to install the best thing and be done with it forever. So we ended up spending 10x the time and wrote like 10x the code to integrate a third-party solution and we only ended up using the most basic features. Not to mention additional infrastructure.
- jbverschoor 4y agoIn other words: boss has no idea, boss doesn’t trust your technical nor business skills Conclusion: abandon this company.. you’re probably just a cost center and nagging headache in his eyes. If you’re not valued or respected, there’s only way way up, and that’s out
- grenoire 4y agoI think there exists a midway, where you rely on the heavily-tested open source solution that implements what you want, and a sprinkle more. RabbitMQ for example is that layer that your 40 lines of code would do, but pays off dividends in good nights' sleep.
- spiffytech 4y ago> Choose boring technology (tm) I'd call Postgres pretty solidly "boring technology", including for session storage and job queues. People were storing sessions in SQL databases when I got my start in 2005! It won't address every scale and every use case, but then, that's never your project's requirement anyway. I frequently see the term "boring technology" treated as a euphemism for "what I'm accustomed to".
- abraxas 4y agoAbsolutely on point. Postgres can and will scale to workloads that average developers can't comprehend. I've used Postgres professionally since 2003 doing everything from run of the mill web work to high volume log aggregation systems with massive datasets. Postgres can ingest and index hundreds of thousands of records per second on pretty modest hardware by today's standards. One just needs to learn the tool. Postgres is far more "boring" and battle tested than Redis. All the hipster tech that came out of Web 2.0 companies (including Cassandra and Redis) has hallmarks of being built by junior developers who refused to learn and then build upon state of the art and went on to reinvent many wheels quite poorly.
- dalyons 4y agoYou had some reasonable points but then you segued into name calling and baseless negativity. Both redis and Cassandra are well built, well understood common industry tools that have been use at scale for many years now. With their own trade offs yes , but quit it with the “built by junior hipster developers” BS, it’s not a good look
- abraxas 4y agoI stand by my point. It was all reactionary stuff. I went to the conferences and saw the talks by those bright eyed young developers. Their disdain for RDBMS and SQL was as fervent as it was misguided. They hailed a new era of Big Data and NoSQL. Hadoop was gonna be the way to store it all or you weren't webscale. Forward a decade and most of it is in the dustbin of unmanageable tech while good ole RDBMS outlived them all. Mongo is about the only one I still hear about and see pop up in job ads. Might share the fate of Hadoop too when people learn that Postgres can index JSON too and even be sharded if you need to (you likely don't).
- Justsignedup 4y agoCounter argument: 1 postgres instance is far easier to manage than a slew of servers. I can spin up a heroku environment with a postgres very quickly, and with little effort. If you're starting out with a team of say 4, and that team will stay small, don't waste your time integrating complex tech. Solr is DEFINITELY better than postgres fulltext search. However pg_search and a clever index has let me solve 95% of all our search needs in almost no time with zero maintenance. However once you want to get into search complexities, Solr quickly outperforms postgres, but you have a lot of setup and env management to do now. So it is all about team size. My instinct is at around 10 engineers you should start thinking about which complexity is worth pulling out of postgres and into its own service based on needs.
- mountainriver 4y agoSolr is better in snots use cases but does things like sparse indexing very poorly and Postgres is actually worlds better
- kristianp 4y ago"Snots use cases"? Is that an unusual autocorrection for COTS?
- maayank 4y agoI think this may underplay the additional operational cost and risk in deploying multiple classes of services. You added RabbitMQ for that one queue use case? You suddenly need to handle some health check edge case in prod since your programmers doesn’t have experience with it. Just added redis? You now have an extra set of server and client sdks to regularly patch up. Etc. Sure, there’s a point where it’s logical to add a new class of services, but it’s not remotely close to zero (which is how I read the comment).
- zkirill 4y agoThe enemy's gate is down. If a CTO brings the surface area of "built in-house" down to zero while accomplishing all of their objectives, ad infinitum, they win the game. Obviously, it would be impossible to maintain such an advantage in real life for any extended period of time. However, orienting your team towards that goal gives the CTO a way to quantify risks and costs associated with accomplishing their objectives. Health checks and SDKs are standardized commodities that can be implemented and maintained at a predefined market rate that's always approaching zero. Finding and fixing a bug in your proprietary code has a potentially infinite cost.
- slashdev 4y agoFinding and fixing a bug in your own code that interfaces with a Kafka cluster that's tripping a disturbuted systems edge case is infinite squared. Solving it requires a huge volume of knowledge about complicated systems and topics as well as debugging across system and server boundaries. It's all trade-offs. Complexity is complexity, whether the code is yours or not.
- serverholic 4y agoThinking about software costs in this way is so oversimplified that it ends up being wrong in practice. I'm reminded of the McNamara Fallacy. Also, SDKs are not standardized. Third-party software still gets updated which means code maintenance for your team. And, even worse, it's software that is generally opaque because, by definition, it wasn't written in-house. I worked at a place where they had a phobia of in-house code and the end result was huge amounts of time wasted on updating libraries and debugging issues caused because we didn't really understand the code we were using. The real answer is deeply understanding your product and finding the right mixture of in-house and third-party software that maximizes simplicity while also allowing for flexibility and growth in ways that matter for your specific product.
- hkon 4y agoWhat is a session?
- tormeh 4y agoNow your engineers need to understand all these technologies instead of just postgres. As usual the correct answer to when to repurpose existing tech in your stack and when to add new tech depends on your specific needs, team size, etc.
- sonthonax 4y agoThe majority of engineers understand basic SELECT and INSERT semantics. Correctly building queuing and caching systems on top of database concurrency primitives is an order of magnitude harder than just using RabbitMQ and Redis.
- nhumrich 4y agoThe problem isn't using one of these things, the problem happens when you use each of these things. Now you have a distributed system where data is in different systems and you have data inconsistencies or have to deal with distributed transactions. The advantage to using postgres is you can continue to use a single transaction for data integrity.