Image Upload Storage Strategies

Image upload storage strategies

I've answered a similar question before but I can't find it, maybe the OP deleted his question...

Anyway, Adams solution seems to be the best so far, yet it isn't bulletproof since images/c/cf/ (or any other dir/subdir pair) could still contain up to 16^30 unique hashes and at least 3 times more files if we count image extensions, a lot more than any regular file system can handle.

AFAIK, SourceForge.net also uses this system for project repositories, for instance the "fatfree" project would be placed at projects/f/fa/fatfree/, however I believe they limit project names to 8 chars.

I would store the image hash in the database along with a DATE / DATETIME / TIMESTAMP field indicating when the image was uploaded / processed and then place the image in a structure like this:

images/
  2010/                                      - Year
    04/                                      - Month
      19/                                    - Day
        231c2ee287d639adda1cdb44c189ae93.png - Image Hash

Or:

images/
  2010/                                    - Year
    0419/                                  - Month & Day (12 * 31 = 372)
      231c2ee287d639adda1cdb44c189ae93.png - Image Hash

Besides being more descriptive, this structure is enough to host hundreds of thousands (depending on your file system limits) of images per day for several thousand years, this is the way Wordpress and others do it, and I think they got it right on this one.

Duplicated images could be easily queried on the database and you'd just have to create symlinks.

Of course, if this is not enough for you, you can always add more subdirs (hours, minutes, ...).

Personally I wouldn't use user IDs unless you don't have that info available in your database, because:

Disclosure of usernames in the URL
Usernames are volatile (you may be able to rename folders, but still...)
A user can hypothetically upload a large number of images
Serves no purpose (?)

Regarding the CDN I don't see any reason this scheme (or any other) wouldn't work...

What is the best place for storing uploaded images, SQL database or disk file system?

I generally store files on the file-system, since that's what its there for, though there are exceptions. For files, the file-system is the most flexible and performant solution (usually).

There are a few problems with storing files on a database - files are generally much larger than your average row - result-sets containing many large files will consume a lot of memory. Also, if you use a storage engine that employs table-locks for writes (ISAM for example), your files table might be locked often depending on the size / rate of files you are storing there.

Regarding security - I usually store the files in a directory that is outside of the document root (not accessible through an http request) and serve them through a script that checks for the proper authorization first.

Image Upload Strategy with Clusters And Amazon S3

This is all about how you display the images. Let's say an image has been uploaded and you stored the record about it in some shared storage (like DB), you saved the image id and the node specific url where the image was temporary placed. I hope you can access each individual node in your cluster.

When you display the image by its id, you go to the DB and pick that node specific url, so it will be visible across the cluster. When the image has been uploaded to S3, you just swap that DB url for the new one on S3 and delete the image from that specific node.

Recommended architecture for handling user image uploads

Storing these in a database and pulling them for each one and displaying via PHP doesn't seem like a good way to go, but throwing all images in to a single directory doesn't either.

You can take a hybrid approach.

Store the images in a heirarchy of folders (according to whatever scheme you determine to be appropriate for your application). Store the full path to each image in your database.

In a background job, have thumbnails of the images produced (say with ImageMagick) whose filenames slightly differ from the images themselves (eg: add 'thumb-' on the front) but which are stored alongside the real images. You can have a field in the database for each image which means "My thumbnail is ready, so please include me in galleries".

When you get a request for a gallery, slice and dice the group of images using database fields, then produce a piece of HTML which refers to the appropriate thumbnail and image paths.

Edit:
What Aaron F says is important when you need to handle a very large number of requests. Partitioning of the image / sql data is a good road to scalability. You'll need to look at access patterns in your application to determine where the partition points lie. Something you can do even sooner is to cache the generated HTML for the galleries in order to reduce SQL load.

How to store images in your filesystem

Just split your userid from behind. e.g.

UserID = 6435624 
Path = /images/24/56/6435624

As for the backup you could use MySQL Replication and backup the slave
database to avoid problems (e.g. locks) while backuping.

What do I do so an image in Storage knows to which doc it belongs to in Cloud Firestore? React

There are two primary strategies for associating an object in Cloud Storage to some other record in a database.

Use object metadata to store a locator to the database record. Object metadata is essentially a set of key/value pairs of strings attached to the object. You could store the path to the related Firestore document in metadata, and use that to find the document easily.
Name the object in storage the same as it's named in Firestore. Firestore documents often have random IDs. You can use the same random ID in the name of the object path. All you have to do is parse the path of that object to find the document ID, and use that to build the path to the related document. The way you create the path to the document in Firestore and the path to the object in Storage don't have to be exactly the same, but it should be clear how to convert one to the other.

Laravel 5.7 getting images from storage

If you generate the symbolic link, then your path will be :

/storage/uploads/files/image-1.png

Image Upload Storage Strategies