How to end or fail AWS Glue job with error?


Consider an AWS Glue job snippet:

How do I exit the job with an error status? If I just skip execution, it simply ends as successful; if I throw an exception, it fails with an exception. Can I just invoke something to stop the job with a fail/error status but without throwing an exception?


At first glance I can:


  1. SparkContext is from the inner framework and ending the job can lead to unpredictable (unstable) results.
  2. org.apache.spark.SparkContext#cancelJob receives Int while AWS Glue has a String JOB_ID like this: j_aaa11111a1a11a111a1aaa11a11111aaa11a111a1111111a111a1a1aa111111a. So it cannot be passed to cancelJob directly.


This is written as pyspark, as it’s what I know

The spark job and glue job are different things, which is why you can’t interchange their IDs.

Leave a Reply