AWS Glue transform a struct into dynamicframe


I am a little new to AWSGlue. I am working on transform a raw cloudwatch json out into csv with AWSGlue. The transformation script is pretty straight forward, however documentation and example doesn’t seem to be comprehensive.
The data structure is something like this:

The tricky part is to transform it from single dynamic frame(lable,string, datapoint array) into dynamic frames (Timestamp,string,Sum,Double,Unit,String).
I am not sure which method to use in dynamic dataframe.


I don’t think AWSGlue provide any mapping method for it. After some struggling, I found the transformation was relatively easy in the pyspark. Here is the pseudo code:

  • Retrieve datasource from database
  • Convert it into DF and transform it in spark
  • Convert back to DynamicFrame and continue the rest of ETL process

Thanks to this reference

Leave a Reply