I am using API gateway to call lambda function that imports a mpeg file (10 mb) from s3 and saves in /tmp folder of lambda and the lambda uploads it to external API (youtube etc)
Recently the API gateway call to lambda is failing intermittently with error

Here is how i am downloading the file

urllib.urlretrieve (s3_mpeg_url, '/tmp/{}'.format(mpeg_filename))

If i create a new version of that same lambda function and assign to alias API gateway pointing to , it starts to work and again at some point it keeps getting the same error

When i test that lambda function from lambda console it always works

Any idea ?

Your lambda function has approximately 500MB of disk space available on /tmp. However, for performance reasons, AWS Lambda might retain and reuse instances of your function on subsequent calls. There are two possible paths you can take here:

1. If your function is totally stateless (i.e. you don’t need the mpeg file after uploading it to the external API), just delete it from the /tmp folder after the upload.
2. If you need to keep the data around, consider storing it on another media, like S3 or DynamoDB after processing.