: SMTP Email volume My company's application has a bulk email system, all custom built, used by our clients to send emails based on opt-ins. While doing performance monitoring during some particularly

@Kevin317

As much as I love PHP, that is where your bottleneck is in your system. PHP just doesn't have the efficiency of other languages when it comes to text processing. stackoverflow.com/questions/603163/is-perl-a-good-option-for-heavy-text-processing
Perl is going to be a better choice for processing and sending emails. A few years back I had written a simple Perl program that would send personalized emails out to our opt-ins. I was able to send about 80,000 emails in 6 hours just using Perl to create the email and submit them to the local Sendmail program. This was on a pretty standard Virtual Private Server with 2GB of RAM.

Are you submitting the jobs to the local Sendmail process or is PHPMailer using SMTP? The local Sendmail program will be faster since PHP is not having to open up any network sockets to send the email.

So in summary you should:


Use Perl instead of PHP (I hate Perl too, but it's a better text processing tool)
Submit the job to the local Sendmail program (that can be configured to forward the jobs to external SMTP servers if it needs to)
Use an external SMTP server if the local one gets overloaded.

Vote Up Vote Down

@Reiling115

As long as you're using a non-blocking SMTP server (in case it hits a bad DNS/SMTP server) then it is a matter of how many different domains are the emails destined for and what is the bandwidth (roughly speaking because the exceptions to the rule appear at the extremes). I would suspect the latter (bandwidth) due to you seeing similar results with an external server.

Any packaging/encryption on modern hardware takes a fraction of the time needed to send an email to a slow server. If you host your own DNS servers then make sure these return fast results as the receiving SMTP server will probably want to double check your records (slowing things down further). Having two mail servers (with enough bandwidth) is normally way faster (due to the blocking on a slow receiving server) than spending the same amount of money on a better spec'ed single server.

Nasty hacks for quick wins


Sort the email addresses by domain name. If you create a new database table hanging of the table where you store the email address, it can contain the foreign key, the user part and the domain part. A run to do the split on existing data and some changes in the the CRUD to update the records will achieve this. By sorting on domain name you'll allow the sending server to re-use its connection to the remote mail server (be careful your not SPAM blocked).
Where the code is hard to change and you want to alternate the mail servers you can just alter the function so it accepts a reference (prefix it with an & in PHP) then you can use a $_GLOBAL variable that is changed every x seconds by a scheduler that calls a hidden PHP page.
Use a local DNS cache and query the necessary MX records before the run (or at least kick off a script to run in parallel). Most caches will hold records for 24 hours (normally written as 3600 seconds). It can reduce the initial connection latency by around 100ms. If you have multiple SMTP servers and the receiver has multiple MX records you can spoof the results. So your first sending server sees:


MX Record 1 Priority 10
MX Record 2 Priority 20
MX Record 3 Priority 30
MX Record 4 Priority 40


and your second sending server sees


MX Record 1 Priority 40
MX Record 2 Priority 10
MX Record 3 Priority 20
MX Record 4 Priority 30


etc. This way you can increase parallel behaviour, but it erodes any gains from point 1.


If you can run a packet trace to identify the bottlenecks, it will help find the necessary quick wins.

Vote Up Vote Down