Question:
I am using bigquery connector for aws glue in my glue job. It was working fine few days ago but now suddenly it is giving me below error:
|
1 2 3 |
LAUNCH ERROR | Glue ETL Marketplace - failed to download connector.Please refer logs for details. |
Below is the full error that i am getting on cloudwatch
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
2021-11-08T11:33:02.045+05:00 Traceback (most recent call last): File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main 2021-11-08T11:33:02.070+05:00 "__main__", mod_spec) File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 361, in 2021-11-08T11:33:02.070+05:00 main() File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 351, in main 2021-11-08T11:33:02.070+05:00 res += download_jars_per_connection(conn, region, endpoint, proxy) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 304, in download_jars_per_connection 2021-11-08T11:33:02.070+05:00 download_and_unpack_docker_layer(ecr_url, layer["digest"], dir_prefix, http_header) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 168, in download_and_unpack_docker_layer 2021-11-08T11:33:02.070+05:00 layer = send_get_request(layer_url, header) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 80, in send_get_request 2021-11-08T11:33:02.070+05:00 2021-11-08T11:33:02.070+05:00 response.raise_for_status() File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 765, in raise_for_status 2021-11-08T11:33:02.071+05:00 raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request 2021-11-08T11:33:02.119+05:00 Glue ETL Marketplace - failed to download connector, activation script exited with code |
Answer:
When Glue job tried using connector, it has to download the connector in the form of container. The containers for connectors are available in amazon public ECR repo. To pull the container from the AWS public repo, we have to add “AmazonEC2ContainerRegistryFullAccess” policy to your IAM role. We can limit the access to read only as well.