: Requesting advice for accepting large file uploads I asked this question over at StackExchange but was advised to try here... I'm looking at building a web app that includes a file upload element.

@Speyer207

There is an open source php5 project similar to yousendit that you might be able to get some ideas from: zend.to/
This solution works nicely if you are not going over 2GB file uploads, and works okay for larger uploads as long as you are aware of the limitations of http uploads using browsers:


Not all browsers support file uploads greater than 2GB. In fact.. Safari, Chrome and Opera are the only browsers that I know of that do.
The uploading computer needs as much free space on their disks as the file size they are uploading. i.e., if you are uploading a 5GB file, you will need 5Gig of free space during the upload process. The space is freed after completion or if they cancel.
The upload process on large files can take a long time on old/slow computers.


Plus sides to http uploads:


No client to install/configure
No java apps to deal with
Simple https to secure uploads/downloads


If you decide to not use http based uploads, then you will need to implement some kind of client/server based tool: java, browser based plugins or a downloadable app. At this point your hurdles will be the normal client/server issues such as authentication, resuming uploads, etc.

From a hosting point of view (off the top of my head):


Virus/Malware scanning
Retention policies
File integrity (i.e., md5 sums on client and server)
Firewall/NAT/Proxy issues
Bandwidth throttling
Maybe look into data-dedup stores to reduce storage costs and bandwidth fees. S3 is cheap on storage, but the costs can run up on transfer fees.
Preventing hot linking to stored assets (i.e., keeping people from passing out the url's to files)
Caching files
Location aware hosting like a CDN (Amazon has a CDN like s3 plan)


node.js might be a place to look into for a client/server socket app.

If you do not have the budget for enterprise storage, maybe a ZFS based file server with some SSD's for write caching and a boat load of ram: nexentastor.org, openindiana.org, etc.

Really just brainstorming here. I have to deal with very large file transfers daily and it is always a pain (cross-platform 10GB+ file transfers). I use the zend.to for quick and easy email based transfers, SFTP for transfers that need to linger a while, and we pay $$$ for an enterprise file exchange tool (aspera faspex) for our large file deliveries to external clients.

Hope this helps some!

Vote Up Vote Down

@Shelley277

If you are willing to look at other languages my advice will be to go with Django (Python webframework), here is exactly what you need: docs.djangoproject.com/en/dev/topics/http/file-uploads/
Documentation is extensive, all round cases are handled by framework, so you just need to learn it. It's not hard but takes some time. This is not a simplest or fastest route but if you are going to do some webdev in the future consider this option as it will pay out definitely.

If python + django is too much to learn try to go with some php webframework. A good framework always has some helper functions for you handle this situation in professional manner. I'm no expert in php webframeworks so I can advice no one.

Vote Up Vote Down