: Strategy for managing lots of pictures for a website I'm starting a new website that will (hopefully) have a lot of user generated pictures. I'm trying to figure out the best way to store
I'm starting a new website that will (hopefully) have a lot of user generated pictures. I'm trying to figure out the best way to store and serve these pictures.
The CMS I'm using (umbraco) has a media library that puts a folder on the server for each image. Inside of there you can have different sizes of that same image. That folder has an ID on it and the database has additional information for that image along with the ID of the folder.
This works great for small sites, but what if the pictures get up to 10,000, 100,000 or 1,000,000? It seems like the lookup on the directory would take a long time to find the correct folder. I'm on windows 2008 if that makes a difference.
I'm not so worried about load. I can load balance my server pretty easily and replicate the images across the servers. The nature of the site won't have a lot of users on it either, but it could have a lot of pics.
Thanks.
-Nate
EDIT
After some thought I think I'm going to create a directory for each user under a root image folder then have user's pictures under that. I would be pretty stoked if I had even 5,000 users, so that shouldn't be too bad of a linear lookup. If it does get slow I will break it down into folders like /media/a/adam/image123.png.
If it ever gets really big I will expand the above method to build a bigger tree. That would take a LOT of content though.
More posts by @Odierno851
2 Comments
Sorted by latest first Latest Oldest Best
Generate a hash value for each picture, based on the contents of the picture (like SHA-1 or SHA-2) and separate the directory structure based on the beginning value of the hash (i.e. 64 directories covering a range of the hash values:
/images/00-03
/images/04-07
/images/08-0B
... (etc)
OR another breakdown
/images/0000
/images/0001
/images/0002
...
/images/000A
... (etc)
Some file would be named 0003ABC2EFA23.png. It would be found in directory: /images/0003
The number of the directory would represent the first digits of the hash value. You can configure it to use a wider OR smaller range of hash values. This allows you to break up the files into separate directories and quickly find the file you want based on this hash.
NOTE: make sure you consider collision resolution of the hash (because it can and probably will happen). Something like 0003ABC2EFA23-01.png for the first collision, 0003ABC2EFA23-02.png for the second.
You mentioned that information was stored in the database... why not search the database then go to the folder directly?
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.