I have a dataset registered in Glue / Athena, call it
my_db.table. I’m able to query it via Athena and everything generally seems to be in order.
I’m trying to use this table in a Glue job, but am getting the following fairly opaque error message:
py4j.protocol.Py4JJavaError: An error occurred while calling o54.getCatalogSource.
: java.lang.Error: No classification or connection in my_db.table
This would appear to indicate that Glue can’t see the catalog entry for my table, or can’t use the information in that entry, but I don’t have any further visibility than that.
Has anyone experience with this error and what might be causing it?
The error message actually describes the problem well – there was no classification for the table being queried.
Tables created via Glue are registered with a Classification –
json. See Creating Tables Using Athena for AWS Glue Jobs.
The table I created ‘manually’ via Athena did not have a classifcation. See the below screenshot from the Glue ‘tables’ page.
The solution is easy: at the end of the
CREATE TABLE script user must append a
classification property like so
CREATE EXTERNAL TABLE IF NOT EXISTS my_db.my_table (
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = ',',
'field.delim' = ',',
'collection.delim' = 'undefined',
'mapkey.delim' = 'undefined'
) LOCATION 's3://my_bucket/'
Now the table has a classification within the Glue interface and is accessible via a Glue job.