3 ms·
I'll add to this that because AWS is simply piecing different technologies together under the hood, there are a lot of data type issues. Another example is tha
by MSM 6y ago
I'll add to this that because AWS is simply piecing different technologies together under the hood, there are a lot of data type issues.
Another example is that some date/time columns got brought in and crawled as a string. That's a bummer because obviously you want to do native operations of these, datediff, datepart, etc. without having to cast all over the place. We manually set them to timestamp and they work perfect (awesome!), even in Athena, so we thought the problem was solved.
However, once we did anything with those columns in Glue ETL, those fields got set as nulls.
The problem can be fixed of course, but these types of issues happen fairly often and they quietly fail (no errors, just values set to null).