IMAP benchmark

Introduction

This benchmark is a quick and dirty comparison of 3 different mailbox formats (mbox, mbx, maildir) on two different filesystems (ext3, reiserfs). Popular wisdom is that:

• mbox sucks
• both mbx and maildir are a significant improvement on mbox
• maildir should benefit from a fs with directory indexes (such as reiser rather than ext3)

The mailbox

The mailbox was created by taking my Jan 2002 read-mail folder (20Mb, 1000 messages) and appending 2000 random messages of around 4k:

bash-2.05a$ ls -sh testfolder2 
 29M testfolder2
bash-2.05a$ grep -c '^From ' testfolder2 
2969

The benchmark script

Was the same as the one used in http://www.courier-mta.org/mbox-vs-maildir/.

Hardware

1 x PIII 500MHz
256Mb RAM
MegaRaid h/ware RAID (model tbc) on using 3 x 9gb SCSI (details tbc)

The Configs

MBOX refers to WU-IMAPd from the RedHat 7.3 RPM (2001.315rh) using mbox mailboxes
MBX refers to the same IMAP server using mbx mailboxes
MAILDIR refers to Courier imap 1.5.3 using maildir mailboxes
ext3 refers to files on an ext3 partition
ReiserFS refers to files on a ReiserFS partition using the R5 hashing algorithm
ReiserFS2 refers to the same as ReiserFS but mounted with the "noatime,nodiratime" options.

The Results

real	0m9.836s
user	0m1.000s
sys	0m0.790s
      

real	0m19.833s
user	0m1.030s
sys	0m0.850s
      

real	0m5.487s
user	0m0.150s
sys	0m0.220s
      

real	0m9.509s
user	0m0.150s
sys	0m0.260s
      

real	0m4.033s
user	0m0.090s
sys	0m3.930s
      

real	0m0.518s
user	0m0.110s
sys	0m0.400s
      

real	0m0.531s
user	0m0.090s
sys	0m0.440s
      

SELECT.2	ext3	ReiserFS	ReiserFS2
MBOX	real 0m0.624s user 0m0.340s sys 0m0.150s	real 0m0.650s user 0m0.390s sys 0m0.120s
MBX	real 0m0.336s user 0m0.090s sys 0m0.050s	real 0m0.253s user 0m0.120s sys 0m0.010s
MAILDIR	real 0m0.126s user 0m0.120s sys 0m0.000s	real 0m0.133s user 0m0.120s sys 0m0.010s	real 0m0.137s user 0m0.110s sys 0m0.030s

DELETE.1	ext3	ReiserFS	ReiserFS2
MBOX	real 0m8.281s user 0m0.780s sys 0m0.690s	real 0m8.483s user 0m0.820s sys 0m0.650s
MBX	real 0m7.396s user 0m0.090s sys 0m0.490s	real 0m8.215s user 0m0.090s sys 0m0.540s
MAILDIR	real 0m0.264s user 0m0.250s sys 0m0.020s	real 0m0.280s user 0m0.230s sys 0m0.050s	real 0m0.286s user 0m0.230s sys 0m0.050s

FETCH.1	ext3	ReiserFS	ReiserFS2
MBOX	real 0m5.056s user 0m4.370s sys 0m0.520s	real 0m5.031s user 0m4.310s sys 0m0.590s
MBX	real 0m2.676s user 0m2.030s sys 0m0.480s	real 0m2.649s user 0m2.050s sys 0m0.490s
MAILDIR	real 0m2.865s user 0m2.310s sys 0m0.290s	real 0m3.021s user 0m2.370s sys 0m0.300s	real 0m2.627s user 0m2.270s sys 0m0.330s

FETCH.2	ext3	ReiserFS	ReiserFS2
MBOX	real 0m5.018s user 0m4.350s sys 0m0.540s	real 0m4.975s user 0m4.280s sys 0m0.620s
MBX	real 0m2.626s user 0m1.950s sys 0m0.540s	real 0m2.621s user 0m2.070s sys 0m0.460s
MAILDIR	real 0m2.766s user 0m2.330s sys 0m0.290s	real 0m2.673s user 0m2.280s sys 0m0.400s	real 0m2.589s user 0m2.360s sys 0m0.230s

SEARCH.1	ext3	ReiserFS	ReiserFS2
MBOX	real 0m6.237s user 0m5.490s sys 0m0.650s	real 0m6.241s user 0m5.350s sys 0m0.790s
MBX	real 0m2.953s user 0m2.200s sys 0m0.620s	real 0m2.964s user 0m2.260s sys 0m0.610s
MAILDIR	real 0m6.461s user 0m5.650s sys 0m0.630s	real 0m8.661s user 0m5.780s sys 0m0.560s	real 0m6.279s user 0m5.710s sys 0m0.550s

SEARCH.2	ext3	ReiserFS	ReiserFS2
MBOX	real 0m6.273s user 0m5.390s sys 0m0.750s	real 0m6.253s user 0m5.350s sys 0m0.790s
MBX	real 0m2.958s user 0m2.120s sys 0m0.700s	real 0m2.978s user 0m2.200s sys 0m0.670s
MAILDIR	real 0m6.480s user 0m5.800s sys 0m0.480s	real 0m7.974s user 0m5.760s sys 0m0.580s	real 0m6.313s user 0m5.830s sys 0m0.440s

Conclusion

Maildir + ReiserFS is a win on pretty much every front, and a significant improvement in several crucial operations (opening a new mailbox and deleting from a mailbox). The only loss is in search speed, and this can be improved by tweaking some FS options (turning off atime updates).

Unsurprisingly, MBX beats MBOX on every front, but the only place in which it has a significant advantage over Maildir is in searching.

Given other good reasons for not using MBX (poor support from procmail etc., lack of documentation, tied to WU imapd) the verdict seems to be: use maildir on reiserfs. The only downside is the significant effort involved in migrating our system, given that neither imapd supports both formats.