5 ms·

Your whole first paragraph literally describes our situation exactly, same stack and all. Classic premature optimisation. It’s made it so clear to me that so m

Your whole first paragraph literally describes our situation exactly, same stack and all. Classic premature optimisation. It’s made it so clear to me that so much of what you read on HN about the latest and greatest scaling tricks are only relative to a tiny tiny fraction of business.

jjice 5y ago
FWIW, setting up a Redis server and setting up some basic caching middleware is pretty straightforward in my experience. Did this at my job a month ago in an afternoon. I'd say the biggest overhead is adding Redis if you don't already have it, and that addition's difficulty will vary based on how you host Redis. We use Elasticache on AWS, so just a few clicks and set it in the same VPC. I guess the real question comes down to how you feel about an extra moving part. Redis is probably the part of our system that has had the least hiccups (very much set and forget and no resource issues so far), but I can understand in the case where you'd rather not add more than a DB. I'd say it's just as easy to setup as Postgres. Elastic search I hear is a pain, though I have no personal experience.
- blowski 5y ago
  The pain is not initially setting it up, it’s in the ongoing maintenance. Redis is one of the less painful services to support, especially if using a managed version. But I don’t like the trend of defaulting to using Redis without really justifying it.
  - jjice 5y ago
    You're right for sure if the service is small - Redis would be overkill. I guess I'm coming at it from the perspective I'm most used to where we use it for data caching and session data since we have multiple servers, so handling it any other way would be more work.
  - lmm 5y ago
    Redis is a better default-without-thinking than Postgres IMO. Transactions and referential integrity make some big tradeoffs and it's easy to accidentally rely on them without thinking through the consequences. Global serializability is theoretically even worse, but the direct analogy with application-layer threading makes it easier to understand and avoid the pitfalls.
    - naasking 5y ago
      Funny, I think the exact opposite is the case. It's rare that an application needs more throughput than Postgres on modern hardware can provide. Unless you know you actually need that throughput, you're better off defaulting to Postgres and enjoy that referential integrity.
      - lmm 5y ago
        > It's rare that an application needs more throughput than Postgres on modern hardware can provide. Agreed, but it's extremely common to need more resilience and/or better multi-DC support than Postgres can provide, and if you've implicitly built your datamodel on Postgres' guarantees then you might find it very hard to migrate to something more distributable. Frankly true master-master HA should have been table stakes for any serious datastore for at least the past decade, as soon as it became clear that cloud was a thing or even before that. (Redis also falls down on that front in terms of what it offers itself, but it's much easier to migrate to something more distributable, IME).
- barrkel 5y ago
  Setting up caching usually isn't the problem, it's invalidation and eviction that bites you.
- smoe 5y ago
  From my view, the issue in the case is not how difficult it is to setup Redis for caching (it is indeed just a couple of clicks/commands), but the new issues one has to deal with when resorting to caching things too prematurely, instead of making the app fast enough with minimal effort.
jrumbut 5y ago
Getting good at solving problems with relational databases is a highly underrated skill. They are really underutilized by many projects. Not to say any other software is bad, just that keeping the stack simple can help small teams move quickly and not get bogged down fighting fires. Also, the paths to scale up the major RDBMSes are well documented and understood. With a newer service and many interacting systems in your back end you end up having to be a pioneer (which takes time away from implementing new features).
- jaxrtech 5y ago
  Absolutely. I've also seen my fair share of horrendous home-grown "ETL" programs that waste more time shuffling bytes to and from database with poor ORM queries in loops, that could be done with a couple half decent SQL queries. Probably the most useful things for me was learning relational algebra in college, and having been thrown on the deep end on a team that was very SQL heavy (not withstanding attempting to debug Oracle PL/SQL syntax errors while pulling your hair out about a missing closing parenthesis -- of which isn't the problem). The usual challenge seems to be fetching data from external services or performing complex business that may conditionally load things -- things that can be awkward in procedural SQL. At the end of the day, you're building a messy ad-hoc dependency graph that is being manually executed very inefficiently. Would be better to just have your code just describe the dependency graph and treat each value transparently as a promise/future and then have a separate engine execute it. Anyhow, something something monads with lipstick, I digress...
  - KptMarchewa 5y ago
    ETL using ORM? Really?
    - PeterisP 5y ago
      Actual ETL systems would not, however, if you have a homegrown pile of scripts fulfilling the ETL role, I would not be surprised if those scripts would use ORM.
- 5y ago

jjice 5y ago

FWIW, setting up a Redis server and setting up some basic caching middleware is pretty straightforward in my experience. Did this at my job a month ago in an afternoon. I'd say the biggest overhead is adding Redis if you don't already have it, and that addition's difficulty will vary based on how you host Redis. We use Elasticache on AWS, so just a few clicks and set it in the same VPC. I guess the real question comes down to how you feel about an extra moving part. Redis is probably the part of our system that has had the least hiccups (very much set and forget and no resource issues so far), but I can understand in the case where you'd rather not add more than a DB. I'd say it's just as easy to setup as Postgres. Elastic search I hear is a pain, though I have no personal experience.

blowski 5y ago
The pain is not initially setting it up, it’s in the ongoing maintenance. Redis is one of the less painful services to support, especially if using a managed version. But I don’t like the trend of defaulting to using Redis without really justifying it.
- jjice 5y ago
  You're right for sure if the service is small - Redis would be overkill. I guess I'm coming at it from the perspective I'm most used to where we use it for data caching and session data since we have multiple servers, so handling it any other way would be more work.
- lmm 5y ago
  Redis is a better default-without-thinking than Postgres IMO. Transactions and referential integrity make some big tradeoffs and it's easy to accidentally rely on them without thinking through the consequences. Global serializability is theoretically even worse, but the direct analogy with application-layer threading makes it easier to understand and avoid the pitfalls.
  - naasking 5y ago
    Funny, I think the exact opposite is the case. It's rare that an application needs more throughput than Postgres on modern hardware can provide. Unless you know you actually need that throughput, you're better off defaulting to Postgres and enjoy that referential integrity.
    - lmm 5y ago
      > It's rare that an application needs more throughput than Postgres on modern hardware can provide. Agreed, but it's extremely common to need more resilience and/or better multi-DC support than Postgres can provide, and if you've implicitly built your datamodel on Postgres' guarantees then you might find it very hard to migrate to something more distributable. Frankly true master-master HA should have been table stakes for any serious datastore for at least the past decade, as soon as it became clear that cloud was a thing or even before that. (Redis also falls down on that front in terms of what it offers itself, but it's much easier to migrate to something more distributable, IME).
barrkel 5y ago
Setting up caching usually isn't the problem, it's invalidation and eviction that bites you.
smoe 5y ago
From my view, the issue in the case is not how difficult it is to setup Redis for caching (it is indeed just a couple of clicks/commands), but the new issues one has to deal with when resorting to caching things too prematurely, instead of making the app fast enough with minimal effort.

jrumbut 5y ago

Getting good at solving problems with relational databases is a highly underrated skill. They are really underutilized by many projects. Not to say any other software is bad, just that keeping the stack simple can help small teams move quickly and not get bogged down fighting fires. Also, the paths to scale up the major RDBMSes are well documented and understood. With a newer service and many interacting systems in your back end you end up having to be a pioneer (which takes time away from implementing new features).

jaxrtech 5y ago
Absolutely. I've also seen my fair share of horrendous home-grown "ETL" programs that waste more time shuffling bytes to and from database with poor ORM queries in loops, that could be done with a couple half decent SQL queries. Probably the most useful things for me was learning relational algebra in college, and having been thrown on the deep end on a team that was very SQL heavy (not withstanding attempting to debug Oracle PL/SQL syntax errors while pulling your hair out about a missing closing parenthesis -- of which isn't the problem). The usual challenge seems to be fetching data from external services or performing complex business that may conditionally load things -- things that can be awkward in procedural SQL. At the end of the day, you're building a messy ad-hoc dependency graph that is being manually executed very inefficiently. Would be better to just have your code just describe the dependency graph and treat each value transparently as a promise/future and then have a separate engine execute it. Anyhow, something something monads with lipstick, I digress...
- KptMarchewa 5y ago
  ETL using ORM? Really?
  - PeterisP 5y ago
    Actual ETL systems would not, however, if you have a homegrown pile of scripts fulfilling the ETL role, I would not be surprised if those scripts would use ORM.
5y ago