3 ms·

Materialize, today, looks to be limited to running on a single node. https://materialize.io/docs/overview/architecture https://materialize.io/docs/overview/arch

Materialize, today, looks to be limited to running on a single node. https://materialize.io/docs/overview/architecture https://materialize.io/docs/overview/architecture Netflix's workload would likely exhaust the resources of even a vertically-scaled single node.

derefr 7y ago
Mind you, given the “Timely Dataflow” abstraction Materialize operates on top of, if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows (and, further back, retrieve the source rows) required to “render” the particular result-rows you ask for. (Sort of like how Excel, in memory-constrained conditions, only computes the intermediate cell-values required to render the cells currently in view, and thus, if an intermediate cell requires an `XmlHttpRequest` call to resolve, that call won’t fire if the cell’s value doesn’t need resolving yet.) Because of that, you don’t really need to scale Materialize in a sharding sense. You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query launched by, say, Citus—where Citus was thinking it was talking to a bunch of Citus shard nodes each holding a table-shard named X, but was actually talking to Materialize nodes each holding a matview named X. As long as the query sent from the map-reduce job to each node is constrained in its WHERE clause to only the part of the data it expects to get from that node—rather than relying on the node to know what data it has—then the Materialize nodes would each just do the work required to supply that data (including only pulling in the parts of the configured sources required to compute that result.) That’s just my intuition from how Materialize presents itself as PG-wire-protocol compatible, though; I haven’t tried this myself, and there might be some footguns in the path of anyone really trying to implement it. And, of course, this is all irrelevant the moment you write a query that needs a pure reduce (e.g. the computation of a current finite-state-machine state over an event-stream source) rather than a map-reduce. Druid/Clickhouse/etc. can probably “scale” those, in at least the Hadoop “move the job around between each serial stage, so each stage has data-locality for the data of that stage” sense; while Materialize would give you no benefit at all in such a job over just querying a plain PG view defined on top of a Foreign Data Wrapper source.
- quodlibetor 7y ago
  > if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows This is absolutely correct! > You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query This should work, but we have been thinking about it/testing it differently internally. In general you should be able to create materialized views on different "shards" that have different `where` conditions, allowing you to control memory that way. This technique does require data that is actually partitionable in this way, same as it must be partitionable in mapreduce. > this is all irrelevant the moment you write a query that needs a pure reduce Of course, with materialize's sinks you can spin up a bunch of `materialized`s and connect them for a final reduce after data has gone through e.g. kafka or shared files. Being able to write joins and aggregates across heterogenous sources makes this kind of workload actually pretty pleasant.
polskibus 7y ago
I can swear I saw they will be offering multi-node commercially soon. I can't find it now, apart from their cloud offering at https://materialize.io/download/ https://materialize.io/download/

derefr 7y ago

Mind you, given the “Timely Dataflow” abstraction Materialize operates on top of, if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows (and, further back, retrieve the source rows) required to “render” the particular result-rows you ask for. (Sort of like how Excel, in memory-constrained conditions, only computes the intermediate cell-values required to render the cells currently in view, and thus, if an intermediate cell requires an `XmlHttpRequest` call to resolve, that call won’t fire if the cell’s value doesn’t need resolving yet.) Because of that, you don’t really need to scale Materialize in a sharding sense. You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query launched by, say, Citus—where Citus was thinking it was talking to a bunch of Citus shard nodes each holding a table-shard named X, but was actually talking to Materialize nodes each holding a matview named X. As long as the query sent from the map-reduce job to each node is constrained in its WHERE clause to only the part of the data it expects to get from that node—rather than relying on the node to know what data it has—then the Materialize nodes would each just do the work required to supply that data (including only pulling in the parts of the configured sources required to compute that result.) That’s just my intuition from how Materialize presents itself as PG-wire-protocol compatible, though; I haven’t tried this myself, and there might be some footguns in the path of anyone really trying to implement it. And, of course, this is all irrelevant the moment you write a query that needs a pure reduce (e.g. the computation of a current finite-state-machine state over an event-stream source) rather than a map-reduce. Druid/Clickhouse/etc. can probably “scale” those, in at least the Hadoop “move the job around between each serial stage, so each stage has data-locality for the data of that stage” sense; while Materialize would give you no benefit at all in such a job over just querying a plain PG view defined on top of a Foreign Data Wrapper source.

quodlibetor 7y ago
> if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows This is absolutely correct! > You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query This should work, but we have been thinking about it/testing it differently internally. In general you should be able to create materialized views on different "shards" that have different `where` conditions, allowing you to control memory that way. This technique does require data that is actually partitionable in this way, same as it must be partitionable in mapreduce. > this is all irrelevant the moment you write a query that needs a pure reduce Of course, with materialize's sinks you can spin up a bunch of `materialized`s and connect them for a final reduce after data has gone through e.g. kafka or shared files. Being able to write joins and aggregates across heterogenous sources makes this kind of workload actually pretty pleasant.

polskibus 7y ago

I can swear I saw they will be offering multi-node commercially soon. I can't find it now, apart from their cloud offering at https://materialize.io/download/ https://materialize.io/download/