4 ms·
I don't have any write ups but off the top of my head I remember issues with data type casting from ddb -> athena via glue. If a ddb number type was an int in o
by aketchum 6y ago
I don't have any write ups but off the top of my head I remember issues with data type casting from ddb -> athena via glue. If a ddb number type was an int in one item and a float in a different Item, glue transformed it to a struct ( something like struct(long:null,double:50.50) and struct(long:20,double:null)). The suggested fix of a cast function didn't work.
- mobjack 6y agoThat brings back bad memories from working with Glue. If the source data isn't 100% clean and compatible with the destination, it is such a pain to get it working. I eventually got it set up, but for the amount of effort involved, I could have just wrote my own custom ETL solution in less time. The scheduling jobs and triggers is nice once set up. I do hope AWS makes improvements because Glue has potential, but it doesn't feel like it is ready for prime time yet.
- MSM 6y agoI'll add to this that because AWS is simply piecing different technologies together under the hood, there are a lot of data type issues. Another example is that some date/time columns got brought in and crawled as a string. That's a bummer because obviously you want to do native operations of these, datediff, datepart, etc. without having to cast all over the place. We manually set them to timestamp and they work perfect (awesome!), even in Athena, so we thought the problem was solved. However, once we did anything with those columns in Glue ETL, those fields got set as nulls. The problem can be fixed of course, but these types of issues happen fairly often and they quietly fail (no errors, just values set to null).