OwlCyberSecurity - MANAGER

Edit File: formats.txt

/* ======================================================================== * Copyright 1988-2006 University of Washington * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * * ======================================================================== */ Mailbox Format Characteristics Mark Crispin 11 December 2006 When a mailbox storage technology uses local files and directories directly, the file(s) and directories are layed out in a mailbox format. I. Flat-File Formats In these formats, a mailbox and all the messages inside are a single file on the filesystem. The mailbox name is the name of the file in the filesystem, relative to the user's "mail home directory." A flat-file format mailbox is always a file, never a directory. This means that it is impossible to have a flat-file format mailbox that has inferior mailbox names under it (so-called "dual-usage" mailboxes). For some inexplicable reason, some people want this. The mail home directory is usually the same as the user login home directory if that concept is meaningful; otherwise, it is some other default directory (e.g. "C:\My Documents" on Windows 98). This can be redefined by modifying the c-client source code or in an application via the SET_HOMEDIR mail_parameters() call. For example, a mailbox named "project" is likely to be found in the file "project" in the user's home directory. Similarly, a mailbox named "test/trial1" (assuming a UNIX system) is likely to be found in the file "trial1" in the subdirectory "test" in the user's home directory. Note that the name "INBOX" has special semantics and rules, as described in the file naming.txt. The following flat-file formats are supported by c-client as of the time of this writing: . unix This is the traditional UNIX mailbox format, in use for nearly 30 years. It uses a line starting with "From " to indicate start of message, and stores the message status inside the RFC822 message header. unix is not particularly efficient; the entire mailbox file must be read when the mailbox is open, and when reading message texts it is necessary to convert the newline convention to Internet standard CR LF form. unix preserves UIDs, and allows the creation of keywords. Only one process may have a unix-format mailbox open read/write at a time. . mmdf This is the format used by the MMDF mailer. It uses a line consisting of 4 <CTRL/A> (0x01) characters to indicate start and end of message. Optionally, there may also be a unix format "From " line. It otherwise has the same characteristics as unix format. . mbx This is the current preferred mailbox format. It can be handled quite efficiently by c-client, without the problems that exist with unix and mmdf formats. Messages are stored in Internet standard CR LF format. mbx permits shared access, including shared expunge. It preserves UIDs, and allows the creation of keywords. . mtx This is supported for compatibility with the past. This is the old Tenex/TOPS-20 mail.txt format. It can be handled quite efficiently by c-client, and has most of the characteristics of mbx format. mtx is deficient in that it does not support shared expunge; it has no means to store UIDs; and it has no way to define keywords except through an external configuration file. . tenex This is supported for compatibility with the past. This is the old Columbia MM format. This is similar to mtx format, only it uses UNIX-style bare-LF newlines instead of CR LF newlines, thus incurring a performance penalty for newline conversion. . phile This is not strictly a format. Any file which is not in a recognized format is in phile format, which treats the entire contents of the file as a single message. II. File/Message Formats In these formats, a mailbox is a directory, and each the messages inside are separate files inside the directory. The file names of these files are generally the text form of a number, which also matches the UID of the message. In the case of mx, the mailbox name is the name of the directory in the filesystem, relative to the user's "mail home directory." In the case of news and mh, the mailbox name is in a separate namespace as described in the file naming.txt. A file/message format mailbox is always a directory. This means that it is possible to have a file/message format mailbox that has inferior mailbox names under it (so-called "dual-usage" mailboxes). For some inexplicable reason, some people want this. Note that the name "INBOX" has special semantics and rules, as described in the file naming.txt. The following file/message formats are supported by c-client as of the time of this writing: . mx This is an experimental format, and may be removed in a future release. An mx format mailbox has a .mxindex file which holds the message status and unique identifiers. Messages are stored in Internet standard CF LF form, so the file size of the message file equals the size of the message. mx is somewhat inefficient; the entire directory must be read and each file stat()'d. We found it intolerable for a moderate sized mailbox (2000 messages) and have more or less abandoned it. . mh This is supported for compatibility with the past. This is the format used by the old mh program. mh is very inefficient; the entire directory must be read and each file stat()'d, and in order to determine the size of a message, the entire file must be read and newline conversion performed. mh is deficient in that it does not support any permanent flags or keywords; and has no means to store UIDs (because the mh "compress" command renames all the files, that's why). . news This is an export of the local filesystem's news spool, e.g. /var/spool/news. Access to mailboxes in news format is read only; however, message "deleted" status is preserved in a .newsrc file in the user's home directory. There is no other status or keywords. news is very inefficient; the entire directory must be read and each file stat()'d, and in order to determine the size of a message, the entire file must be read and newline conversion performed. news is deficient in that it does not support permanent flags other than deleted; does not support keywords; and has no expunge. Soapbox on File/Message Formats If it sounds from the above descriptions that we're not putting too much effort into file/message formats, you are correct. There's a general reason why file/message formats are a bad idea. Just about every filesystem in existance serializes file creation and deletions because these manipulate the free space map. This turns out to be an enormous problem when you start creating/deleting more than a few messages per second; you spend all your time thrashing in the filesystem. It is also extremely slow to do a text search through a file/message format mailbox. All of those open()s and close()s really add up to major filesystem thrashing. What about Cyrus and Maildir? Both formats are vulnerable to the filesystem thrashing outlined above. The Cyrus format used by CMU's Cyrus server (and Esys' server) has a special associated flat file in each directory that contains extensive data (including pre-parsed ENVELOPEs and BODYSTRUCTUREs) about the messages. Put another way, it's a (considerably) more featureful form of mx. It also uses certain operating system facilities (e.g. file/memory mapping) which are not available on older systems, at a cost of much more limited portability than c-client. These considerably ameliorate the fundamental problems with file/message formats; in fact, Cyrus is halfway to being a database. Rather than support Cyrus format in c-client, you should run Cyrus or Esys if you want that format. The Maildir format used by qmail has all of the performance disadvantages of mh noted above, with the additional problem that the files are renamed in order to change their status so you end up having to rescan the directory frequently to locate the current names (particularly in a shared mailbox scenario). It doesn't scale, and it represents a support nightmare; it is therefore not supported in the official distribution. Maildir support code for c-client is available from third parties; but, if you use it, it is entirely at your own risk (read: don't complain about how poorly it performs or bugs). So what does this all mean? A database (such as used by Exchange) is really a much better approach if you want to move away from flat files. mx and especially Cyrus take a tenative step in that direction; mx failed mostly because it didn't go anywhere near far enough. Cyrus goes much further, and scores remarkable benefits from doing so. However, a well-designed pure database without the overhead of separate files would do even better.