Question:
I’m trying to set up AWS Glue to read from a RDS Postgres using CloudFormation. In order to do that I need to create a crawler using the JdbcTarget option. (Or do I not?)
|
1 2 3 4 5 6 7 8 9 |
Records: Type: 'AWS::Glue::Crawler' Properties: DatabaseName: transact Targets: JdbcTargets: - Path: "jdbc:postgresql://host:5432/database" Role: !Ref ETLAgent |
But creating the stack on CloudFormation wil fail with:
CREATE_FAILED | AWS::Glue::Crawler | Records | Connection name cannot be equal to null or empty. (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException;
Even though the docs say:
ConnectionName
The name of the connection to use for the JDBC target.
Required: No
What is the correct AWS Glue setup using CloudFormation that will allow me to read from RDS?
Answer:
You’re really missing the ConnectionName property, which should carry the name of connection resource which you’re missing. The Path property you’re setting is used to select the schemas/tables to crawl (dbname/%/% to include all). Consult CloudFormation docs on Crawler JDBCTarget for details.
Your template should look something like
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
MyDbConnection: Type: "AWS::Glue::Connection" Properties: CatalogId: !Ref 'AWS::AccountId' ConnectionInput: Description: "JDBC Connection to my RDS DB" PhysicalConnectionRequirements: AvailabilityZone: "eu-central-1a" SecurityGroupIdList: - my-sec-group-id SubnetId: my-subnet-id ConnectionType: "JDBC" ConnectionProperties: "JDBC_CONNECTION_URL": "jdbc:postgresql://host:5432/database" "USERNAME": "my-db-username" "PASSWORD": "my-password" Records: Type: 'AWS::Glue::Crawler' Properties: DatabaseName: transact Targets: JdbcTargets: - ConnectionName: !Ref MyDbConnection Path: "database/%/%" Role: !Ref ETLAgent |