Amazon s3a returns 400 Bad Request with Spark


For checkout purpose I try to set up an Amazon S3 bucket as checkpoint file.

but it stops with this exception

I don’t understand why I got this error and I can’t find any example.


This message correspond to something like “bad endpoint” or bad signature version support.

like seen here frankfurt is the only one that not support signature version 2. And it’s the one I picked.

Of course after all my reserch can’t say what is signature version, it’s not obvious in the documentation. But the V2 seems to work with s3a.

The endpoint seen in the S3 interface is not the real endpoint it’s just the web endpoint.

you have to use one of theses endpoint like that
sc.hadoopConfiguration.set("fs.s3a.endpoint", "")

But it’s work by default with US endpoint

Leave a Reply