All posts Paper color Change page color Announcements

“Multi Send Handler” IEM Addon Released!

Introduction

Our newest product is the Multi Send Handler(MSH) [Pre-launch name “Multi Process Manager”(MPM)] addon for Interspire Email Marketer(IEM). This addon tries to leverage IEM from it’s single process, single threaded design using some super cool process control extensions(PCNTL) available for PHP in Unix derivative operating systems.

Technical

Ever put efforts trying to find a faster transport agent to use with IEM, only to find them wasted later on? Well, it’s no wonder. The fact is, reputated mail transfer agents, such as Postfix have a mail sending capability much higher than IEM could ever fully utilize. IEM is the lazy ass here, not the transfer agent. The main bottleneck of IEM is it’s sequential one-by-one mail processing system. Not to mention multi-core, multi-processor servers; wasting time, waiting for I/O is a huge draw­back, even in single core, single processor systems. To counter this problem, MSH addon takes control from IEM right before it starts processing a job. Then it spawns some additional processes, and distributes the whole mail sending job to all of them. Doing things in parallel greatly improves mail dispatch rate, as much as an impressive 0.85 million mail/hour in our lab test!

Benchmarking the Addon

Software

  • PHP 5.3.2-1ubuntu4.9 with Suhosin-Patch (cli)
  • Ubuntu 10.04.3 LTS with Linux kernel 2.6.32–316-ec2
  • Interspire Email Marketer 6.1.0
  • Multi Send Handler Addon 1.0
  • A dummy SMTP server written in python was used as the MTA. It just logs the recipient name and mail reception time to a text file.

Hardware:

An Amazon EC2 High-CPU Medium Instance was used as the hardware -

  • 1.7 GB of memory
  • 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each)
  • 350 GB of instance storage
  • 32-bit platform
  • I/O performance: Moderate
  • API name: c1.medium

Test Conditions:

There are three performance defining configuration variables available in MSH -

  • MSH_NUMBER_OF_WORKER_PROCESS which determines the number of extra processes forked. The total number of PHP processes in action will be one plus this number.
  • MSH_WORK_CHUNK_SIZE determines the max number of works handed off to a single process in every turn.
  • MSH_PAUSE_CHECKING_INTERVAL determines interval(in seconds) between checking for a job pause event.

In the test, we set MSH_WORK_CHUNK_SIZE = 30 and MSH_PAUSE_CHECKING_INTERVAL = 1 constant through­out. But MSH_NUMBER_OF_WORKER_PROCESS was varied from 2 to 32, effectively changing active PHP process count from 3 to 33.

As the test data, we made a procedurally generated list of 10,000 contacts. But as we can see, in every case, the number of mails sent were 10,000 + 2, because of the two extra notification mails sent to the campaign owner. The duration was measured by calculating the difference between the arrival time of the last email and the first email. Each of the test emails were 6 KB in size.

Test Results

Test no.MSH EnabledProcess CountDuration (second)No. of Mails SentRate (mail/second)Rate (mail/hour)
1No140610000 + 224.688687.7
2Yes2 + 1224.910000 + 244.5160103.2
3Yes4 + 1118.210000 + 284.6304629.4
4Yes8 + 161.810000 + 2161.8582640.8
5Yes16 + 145.010000 + 2222.3800160.0
6Yes24 + 142.010000 + 2238.1857314.3
7Yes32 + 145.310000 + 2220.8794860.9

Test results in graph form

Conclusion

As we can see from the graph, doubling the MSH_NUMBER_OF_WORKER_PROCESS variable almost doubles the sending rate for lower process counts. For higher process counts, inter-process communication and scheduling over­heads start to dominate; slowing down the increment rate. In our test, 24 seems to be the optimum value with maximum mail sending rate.

If you are interested about this addon, please take a look at the product page for more information.

Software Solutions