Adding a column in AWS glue dynamic dataframe


I am very new to AWS Glue. I am working on a small project and the ask is to read a file from S3 bucket, transpose it and load it in a mysql table. The source data in S3 bucket looks as below

The target table structure is
Product_id, Parameter, value

I am expecting target table to have following values

p1, cost, 5

P1, data, 1000

I am able to load the target table with ID and Value. But I am not able to populate the parameter column. This column is not present in the input data and I want to populate a string depending on which column value I am populating.

Here is the code I used for cost.

Can somebody help me to add this new column to my data frame so that it can be made available in the table?



For a smaller datsframe you can do the following

  1. convert the dynamic frame to spark dataframe
  2. add a column
  3. convert back to dynamic frame

step 1

step 2

step 3

The issue is when you have a large dataset the operation toDF() is very expensive!

Leave a Reply