7 ms·

Controversial opinion: we don't need more databases. We need better application design. Why do you need a redis/memcache? Because you want to look up a shit-to

Controversial opinion: we don't need more databases. We need better application design. Why do you need a redis/memcache? Because you want to look up a shit-ton of random data quickly. Why does it have to be random? Do you really need to look up any and all data? Is there not another more standard (and not dependent on a single db cluster) data storage and retrieval method you could use? If you have a bunch of nodes with high memory just to store and retrieve data, and you have a bunch of applications with a tiny amount of memory.... Why not just split the difference? Deploy your apps to nodes with high amounts of memory, add parallel processing so they scale efficiently, store the data in memory closer to the applications, process in queues to prevent swamping the system and more reliable scaling. Or use an SSD array and skip storing it in memory, let the kernel VM take care of it. If you're trying to "share" this memory between a bunch of different applications, consider if a microservice architecture would be better, or a traditional RDBMS with more efficient database design. (And fwiw, organic networks (as in biological) do not share one big pot of global state, they keep state local and pass messages through a distributed network)

dvt 4y ago
Weird hill to die on. There’s like a million valid reasons for using an in-memory data store. Off the top of my head: chat apps, real-time video games, caching, (some) data pipelines, etc.
- deleted 4y ago
  [deleted]
- paulgb 4y ago
  As I understand it, high-performance real-time video games do tend to use the architecture that throwaway proposed, i.e. state is in the game server itself rather than a separate caching layer.
  - AtNightWeCode 4y ago
    Caching is for poor designs. Most "modern" designs only rely on cache for data distribution speed. Cache is an optimization.
    - hinkley 4y ago
      > Caching is for poor designs. Caching is the death of design. People who like to make decisions and then studiously avoid ever looking at the consequences of their actions don't seem to notice that once caching is introduced, the data model in the system either stops improving or begins to unravel. It becomes very difficult to benchmark the system looking for design issues because the real cost of a cold cache is very difficult to measure. And the usual argument that we can turn off the cache for performance analysis doesn't hold water for more than a few months. Pulling the same data twice, three times, ten times in a given transaction are all about the same cost, and it gets me out of trying to figure out how to plumb new data through the sequence diagrams. I can just snatch values out of global state wherever, whenever, and as often as I need them. Once people give up and start doing that, you can't turn off the caches anymore. When you started it cut the cost of the lookup in half, or a third. Now it's four, or five, or sixteen, and that call is now over-represented in the analysis, hiding other, more tractable problems. You've settled for your broken design, but also cut the legs out from under any of the tools that can dig you out of the hole you've dug. Caches are just global state under a different name, but slathered in RPC issues.
    - romange 4y ago
      Some valid business use-cases allow eventual consistency. Think of social apps, for example. FB and TW have excellent engineering teams and they built huge systems that rely on caching layers. Similarly, with gaming companies.
      - dvt 4y ago
        Once you reach a certain scale, caching becomes basically a necessity. I mean, heck, even DNS is cached. The idea that caching is a failure of design is just wrong.
- Cwizard 4y ago
  Could share some more on the data pipelines you mention? What kind of data store do you use for this purpose? We are reading a lot of the same data from disk to do transformation (joins) with new data, I’ve been looking for something that might speed this process up but would like to keep the setup small and simple.
  - dvt 4y ago
    I've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things. [1] https://arrow.apache.org/ https://arrow.apache.org/
deleted 4y ago
[deleted]
gxt 4y ago
Or you need the best of both worlds. A tool that can store data, serve data, convert code to data, interpret data, display data. Some kind of tool that has at its core a kind of phpmyadmin but that can be distributed on an app store, but for end-users. Something that will handle authentication, authorization, audit, provide confidentiality, scalability, remove the need for third party tools for end users and developers, just one big ecosystem that let's you host entire enterprise software , extend it, customize it for your end users and your power user. Just one big ass tool that will replace all other information management services.
zackmorris 4y ago
One of the main uses in PHP/Laravel (probably borrowed from Ruby on Rails) is caching views for Russian doll caching and caching SQL query results: https://laracasts.com/series/russian-doll-caching-in-laravel https://laracasts.com/series/russian-doll-caching-in-laravel (subscription only) Unfortunately, it looks like they retired their database caching lesson, which was a mistake on their part since it was so good: https://laracasts.com/discuss/channels/laravel/cache-tutorials https://laracasts.com/discuss/channels/laravel/cache-tutoria... (links to defunct https://laracasts.com/lessons/caching-essentials https://laracasts.com/lessons/caching-essentials) Laracasts are the best tutorials I've ever seen, regardless of language, outside of how php.net/<keyword> search and commenting was structured 20 years ago. They would be the best if they got outside funding to provide all lessons for free. Anyway, one HTTP request can easily generate 100+ SQL queries under an ORM. Which sounds bad, but is trivially fixable globally without side effects via global "select" query listeners and memoization. I've applied it and seen 7 second responses drop to 100 ms or less simply by associating query strings and responses with Redis via the Cache::remember() function. I also feel that there's a deeper problem in how web development began as hacking but devolved into application-heavy bike shedding. We have a generation of programmers taught to apply decorators by hand repeatedly, rather than take an aspect-oriented (glorified monkey patching) approach that fixes recurring problems at the framework or language level. I feel that this is mainly due to the loss of macros without runtimes providing alternative ways of overriding methods or even doing basic reflection. So code today often isn't future-proof (requires indefinite custodianship of conventions) and has side-effect-inducing or conceptually-incorrect changes made in the name of secondary concerns like performance. The antidote to this is to never save state in classes, but instead pipe data through side-effect-free class methods, then profile the code and apply momoization to the 20-% of methods that cost 80+% of execution time. Also microservices are definitely NOT the way to go for rapid application development or determinism. Or I should say, it's unwise to adopt microservices until there are industry-standard approaches for reverting back to monoliths.
oofbey 4y ago
Yes! We just need to get our users to visit the website in order of their user_id, at their assigned time, and then we wouldn't have these random access patterns. /s Unfortunately, reality.

dvt 4y ago

Weird hill to die on. There’s like a million valid reasons for using an in-memory data store. Off the top of my head: chat apps, real-time video games, caching, (some) data pipelines, etc.

deleted 4y ago
[deleted]
paulgb 4y ago
As I understand it, high-performance real-time video games do tend to use the architecture that throwaway proposed, i.e. state is in the game server itself rather than a separate caching layer.
- AtNightWeCode 4y ago
  Caching is for poor designs. Most "modern" designs only rely on cache for data distribution speed. Cache is an optimization.
  - hinkley 4y ago
    > Caching is for poor designs. Caching is the death of design. People who like to make decisions and then studiously avoid ever looking at the consequences of their actions don't seem to notice that once caching is introduced, the data model in the system either stops improving or begins to unravel. It becomes very difficult to benchmark the system looking for design issues because the real cost of a cold cache is very difficult to measure. And the usual argument that we can turn off the cache for performance analysis doesn't hold water for more than a few months. Pulling the same data twice, three times, ten times in a given transaction are all about the same cost, and it gets me out of trying to figure out how to plumb new data through the sequence diagrams. I can just snatch values out of global state wherever, whenever, and as often as I need them. Once people give up and start doing that, you can't turn off the caches anymore. When you started it cut the cost of the lookup in half, or a third. Now it's four, or five, or sixteen, and that call is now over-represented in the analysis, hiding other, more tractable problems. You've settled for your broken design, but also cut the legs out from under any of the tools that can dig you out of the hole you've dug. Caches are just global state under a different name, but slathered in RPC issues.
  - romange 4y ago
    Some valid business use-cases allow eventual consistency. Think of social apps, for example. FB and TW have excellent engineering teams and they built huge systems that rely on caching layers. Similarly, with gaming companies.
    - dvt 4y ago
      Once you reach a certain scale, caching becomes basically a necessity. I mean, heck, even DNS is cached. The idea that caching is a failure of design is just wrong.
Cwizard 4y ago
Could share some more on the data pipelines you mention? What kind of data store do you use for this purpose? We are reading a lot of the same data from disk to do transformation (joins) with new data, I’ve been looking for something that might speed this process up but would like to keep the setup small and simple.
- dvt 4y ago
  I've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things. [1] https://arrow.apache.org/ https://arrow.apache.org/

deleted 4y ago

[deleted]

gxt 4y ago

Or you need the best of both worlds. A tool that can store data, serve data, convert code to data, interpret data, display data. Some kind of tool that has at its core a kind of phpmyadmin but that can be distributed on an app store, but for end-users. Something that will handle authentication, authorization, audit, provide confidentiality, scalability, remove the need for third party tools for end users and developers, just one big ecosystem that let's you host entire enterprise software , extend it, customize it for your end users and your power user. Just one big ass tool that will replace all other information management services.

zackmorris 4y ago

One of the main uses in PHP/Laravel (probably borrowed from Ruby on Rails) is caching views for Russian doll caching and caching SQL query results: https://laracasts.com/series/russian-doll-caching-in-laravel https://laracasts.com/series/russian-doll-caching-in-laravel (subscription only) Unfortunately, it looks like they retired their database caching lesson, which was a mistake on their part since it was so good: https://laracasts.com/discuss/channels/laravel/cache-tutorials https://laracasts.com/discuss/channels/laravel/cache-tutoria... (links to defunct https://laracasts.com/lessons/caching-essentials https://laracasts.com/lessons/caching-essentials) Laracasts are the best tutorials I've ever seen, regardless of language, outside of how php.net/<keyword> search and commenting was structured 20 years ago. They would be the best if they got outside funding to provide all lessons for free. Anyway, one HTTP request can easily generate 100+ SQL queries under an ORM. Which sounds bad, but is trivially fixable globally without side effects via global "select" query listeners and memoization. I've applied it and seen 7 second responses drop to 100 ms or less simply by associating query strings and responses with Redis via the Cache::remember() function. I also feel that there's a deeper problem in how web development began as hacking but devolved into application-heavy bike shedding. We have a generation of programmers taught to apply decorators by hand repeatedly, rather than take an aspect-oriented (glorified monkey patching) approach that fixes recurring problems at the framework or language level. I feel that this is mainly due to the loss of macros without runtimes providing alternative ways of overriding methods or even doing basic reflection. So code today often isn't future-proof (requires indefinite custodianship of conventions) and has side-effect-inducing or conceptually-incorrect changes made in the name of secondary concerns like performance. The antidote to this is to never save state in classes, but instead pipe data through side-effect-free class methods, then profile the code and apply momoization to the 20-% of methods that cost 80+% of execution time. Also microservices are definitely NOT the way to go for rapid application development or determinism. Or I should say, it's unwise to adopt microservices until there are industry-standard approaches for reverting back to monoliths.

oofbey 4y ago

Yes! We just need to get our users to visit the website in order of their user_id, at their assigned time, and then we wouldn't have these random access patterns. /s Unfortunately, reality.