3 ms·
Join performance is a bit of a hairy topic. If you're pulling data from an external source that supports query folding[1] such as a database, it'll try to push
by cosmie 6y ago
Join performance is a bit of a hairy topic.
If you're pulling data from an external source that supports query folding[1] such as a database, it'll try to push those joins to the original source.
If you're pulling in data from a source that doesn't (or the join occurs after query folding is no longer possible), there are a host of performance considerations/optimizations.
By default PowerQuery knows nothing about your source data and makes no assumptions. If you're joining data using the GUI, the generated code will likely use the NestedJoin[2] function. A single "step" in the processing will have its ram capped at 256MB[3], so depending on the size of the data you're joining and absent of any data processing steps that would give it guarantees about your data, you may or may not be paging massive amounts of data to disk as it loops through the entire dataset you're joining for each row that is being joined.
That said, PowerQuery has 6-7 join algorithms available, and a variety of techniques that can be used to optimize the processing. For example, if you add a primary key to the table (via Table.AddKey or Table.Distinct or Table.Group), it can short-circuit the data processing the moment it finds a match. This[4] article series is really helpful for optimizing joins in particular and the link-outs and tidbits scattered within the series are fantastic for getting a better understanding of PowerQuery's inner workings and performance considerations.
[1] https://docs.microsoft.com/en-us/power-query/power-query-folding https://docs.microsoft.com/en-us/power-query/power-query-fol...
[2] https://docs.microsoft.com/en-us/powerquery-m/table-nestedjoin https://docs.microsoft.com/en-us/powerquery-m/table-nestedjo...
[3] Mentioned halfway down this article: https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-container-size/ https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-co...
[4] https://blog.crossjoin.co.uk/2020/05/31/optimising-the-performance-of-power-query-merges-in-power-bi-part-1/ https://blog.crossjoin.co.uk/2020/05/31/optimising-the-perfo...