I have a directory of M4V files (each around 1 GB) on my machine that I want to upload to my S3 bucket. I decided to try the AWS CLI so I can execute a command and let my computer do the rest, but it doesn’t seem to work.

The command I’m issuing is:

But running this command returns output like the following:

upload failed: ./54cffd1ad106d.m4v to s3://yourfightsite-vod/videos/output/m4v/54cffd1ad106d.m4v
HTTPSConnectionPool(host=’yourfightsite-vod.s3.amazonaws.com’, port=443): Max retries exceeded with url: /videos/output/m4v/54cffd1ad106d.m4v?partNumber=4&uploadId=oG.0CBqIpsRcxO.ZqLIgOOBi8g9JFOKD8wQrmrNFa6Cx9LvGY9_PXiqaaVm6X3fIzXbCor8QSMEeqCfovtivHNFVyea8UNoxrVTpTEvM3ibGBxF30HGPkrxWuA83k6gj (Caused by : Errno 32 Broken pipe)

What does this mean? What is a “broken pipe” and how can I rectify this so my uploads are successful?

What is a “broken pipe” and how can I rectify this so my uploads are
successful?

“Broken Pipe” means you’ve lost your connection. It could be a problem on Amazon’s side, could be a problem on your side… who knows…. the point is that you were communicating, and now you are not.

Best resolution is to use multi-part uploads. In their own documentation, Amazon recommend that you use multi-part uploads for large files over 100MB. It looks like the CLI tool might be using this already.

The second half of the resolution is for your code to catch and handle errors such as this gracefully (i.e. retry a couple of times and then ring alarm bells).