Local filesystem as a remote storage in Django

Question 1

I use Amazon S3 as a part of my webservice. The workflow is the following:

User uploads lots of files to web server. Web server first stores them locally and then uploads to S3 asynchronously
User sends http-request to initiate job (which is some processing of these uploaded files)
Web service asks worker to do the job
Worker does the job and uploads result to S3
User requests the download link from web-server, somedbrecord.result_file.url is returned
User downloads result using this link

To work with files I use QueuedStorage backend. I initiate my FileFields like this:

    user_uploaded_file = models.FileField(..., storage=queued_s3storage, ...)
    result_file = models.FileField(..., storage=queued_s3storage, ...)

1

2

3

user_uploaded_file = models.FileField(..., storage=queued_s3storage, ...)

result_file = models.FileField(..., storage=queued_s3storage, ...)

Where queued_s3storage is an object of class derived from ...backends.QueuedStorage and remote field is set to '...backends.s3boto.S3BotoStorage'.

Now I’m planning to deploy the whole system on one machine to run everything locally, I want to replace this '...backends.s3boto.S3BotoStorage' with something based on my local filesystem.

The first workaround was to use FakeS3 which can “emulate” S3 locally. Works, but this is not ideal, just extra unnecessary overhead.

I have Nginx server running and serving static files from particular directories. How do I create my “remote storage” class that actually stores files locally, but provides download links which lead to files served by Nginx? (something like http://myip:80/filedir/file1). Is there a standard library class for that in django?

Question 2

The default storage backend for media files is local storage.

Your settings.py defines these two environment variables:

MEDIA_ROOT (link to docs) — this is the absolute path to the local file storage folder
MEDIA_URL (link to docs) — this is the webserver HTTP path (e.g. '/media/' or '//%s/media' % HOSTNAME

These are used by the default storage backend to save media files. From Django’s default/global settings.py:

# Default file storage mechanism that holds media.
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'

1

2

3

# Default file storage mechanism that holds media.

DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'

This configured default storage is used in FileFields for which no storage kwarg is provided. It can also be accessed like so: rom django.core.files.storage import default_storage.

So if you want to vary the storage for local development and production use, you can do something like this:

# file_storages.py
from django.conf import settings
from django.core.files.storage import default_storage
from whatever.backends.s3boto import S3BotoStorage

app_storage = None
if settings.DEBUG == True:
    app_storage = default_storage
else:
    app_storage = S3BotoStorage()

1

2

3

4

5

6

7

8

9

10

11

# file_storages.py

from django.conf import settings

from django.core.files.storage import default_storage

from whatever.backends.s3boto import S3BotoStorage

app_storage = None

if settings.DEBUG == True:

app_storage = default_storage

else:

app_storage = S3BotoStorage()

And in your models:

# models.py
from file_storages import app_storage

# ...
    result_file = models.FileField(..., storage=app_storage, ...)

1

2

3

4

5

6

# models.py

from file_storages import app_storage

# ...

result_file = models.FileField(..., storage=app_storage, ...)

Lastly, you want nginx to serve the files directly from your MEDIA_URL. Just make sure that the nginx URL matches the path in MEDIA_URL.

Local filesystem as a remote storage in Django

Question:

Answer:

Leave a Reply Cancel reply