5 ms·
Interesting extensions. The nice part of the syntax is that: 1. it is clear when you are going external. 2. procedural code can be written in C# -- big
by joshuaellinger 11y ago
Interesting extensions.
The nice part of the syntax is that:
1. it is clear when you are going external.
2. procedural code can be written in C# -- big win there.
But it misses the two big issues in ETL that I always run into:
1. You have a bunch files that are 95% the same but have 2-5 columns that are different.
2. Someone sticks the wrong type of data a field.
For those issues, you'd be better off mapping a file (or a group of files) to a virtual table so you can specify the schema up front.
- M_Rys_Msft 11y agoActually U-SQL gives you the ability to map files to a virtual table (aka view :)). CREATE VIEW V AS EXTRACT <insert schema here> FROM "/a/b/{*}.csv" USING Extractors.Csv(); If the data is more complex, you can write your own Extractors that for example handle the divergence in columns. Although we are planning on some features here in the future.