3 ms·

Does this fix how much of an insane memory hog materialize is? Some queries are just impossible if you can’t use disk. This is why I was forced to stick with Fl

Does this fix how much of an insane memory hog materialize is? Some queries are just impossible if you can’t use disk. This is why I was forced to stick with Flink. Even though materialize makes things appear stupid simple and easy with SQL, I found that you can only do the most simple streaming views with it. You can’t even do unique counts with this for very long without breaking—and there’s no probabilistic alternatives. Bad for big data. Great for small and simple data sets. But who is using Kafka with small data? Also they do not integrate at all with custom data types in Postgres IME. E.g. an enumeration in your table will mean materialize can’t read the table as a source. Lame.

benesch 4y ago
As mentioned in the blog post, clusters allow horizontal scalability and daisy chaining, so you can allocate more memory for your views even if you run up against the limits of how much memory you can fit on a single machine. We've got plans in the works to support out-of-core execution, too. > Also they do not integrate at all with custom data types in Postgres IME. E.g. an enumeration in your table will mean materialize can’t read the table as a source. Lame. We're aware of this and are working on a fix. There are two tracking issues, if you'd like to follow along: * #6818 (https://github.com/MaterializeInc/materialize/issues/6818 https://github.com/MaterializeInc/materialize/issues/6818) is specifically about supporting PostgreSQL enum types * #15073 (https://github.com/MaterializeInc/materialize/issues/15073 https://github.com/MaterializeInc/materialize/issues/15073) is about handling PostgreSQL types that are unknown to Materialize in a more general purpose way *