OwlCyberSecurity - MANAGER

Edit File: mixfmt.txt

/* ======================================================================== * Copyright 1988-2006 University of Washington * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * * ======================================================================== */ Last update: 18 December 2006 INTRODUCTION This file is the descendant of a design document used to specify the mix format. An attempt is being made to keep this document more or less current with the way the mix format actually works. 1. Mix mailbox naming Mailbox names correspond to directory names; thus mix format mailboxes are "dual-use" (lack both \NoInferiors and \NoSelect). This will satisfy some long-standing requests. 2. Mailbox files A mix format mailbox is a directory with regular files with filenames of: .mixmeta mailbox metadata file .mixindex message index file (message static data) .mixstatus message status file (message dynamic data) .mix######## (where ######### is a <hex8>) secondary message data files. .mix primary message data file (used in experimental versions, supported for compatibility only) 2.1 Metadata, index, and status files The mailbox metadata, index, and status files contain a sequence of CRLF-terminated lines. These files have an update sequence, which is a strictly-ascending sequence value. Any time the file is changed, the update sequence is increased; this allows easy detection of whether the file has been changed by another process. For now, this update sequence is a modseq (see below). 2.1.1 Metadata file The mailbox metadata file is called ".mixmeta". It contains a series of CRLF-terminated lines. The first character of the line is a key that identifies the payload of the line, and the remainder of the line is the payload. Key Payload --- ------- S <hex8> ;; update sequence V <hex8> ;; UIDVALIDITY L <hex8> ;; UIDLAST N <hex8> ;; current new message file K [atom 0*(SP atom)] ;; keyword list All other keys are reserved for future assignment and must be ignored (and may be discarded) by software which does not recognize them. The mailbox metadata file is rewritten as part of new mail delivery (so APPENDUID/COPYUID can work) and when new keywords are added. 2.1.2 Message static index file The mailbox message static index file is called ".mixindex". It contains a series of CRLF-terminated lines. The first character of the line is a key that identifies the payload of the line, and the remainder of the line is the payload. Key Payload --- ------- S <hex8> ;; update sequence : <uid>:<date>:<size>:<file>:<pos>:<isiz>:<hsiz> ;; per-message record The per-message records contain the following data: <uid> = <hex8> ;; message UID <date> = <yyyymmddhhmmss+zzzz> ;; internal date <size> = <hex8> ;; rfc822.size <file> = <hex8> ;; message data file (0 = .mix file) <pos> = <hex8> ;; message position in file <isiz> = <hex8> ;; message internal data size <hsiz> = <hex8> ;; header size (offset to body) All other keys, and subsequent fields in per-message records, are reserved for future assignment and must be ignored (and may be discarded) by software which does not recognize them. The mailbox metadata file is appended by new mail delivery and rewritten by expunge "burping", and otherwise is not altered. 2.1.3 Message dynamic status file The mailbox message dynamic status file is called ".mixstatus". It contains a series of CRLF-terminated lines. The first character of the line is a key that identifies the payload of the line, and the remainder of the line is the payload. Key Payload --- ------- S <hex8> ;; update sequence : <uid>:<uf>:<sf>:<mod>: ;; per-message record The per-message records contain the following data: <uid> = <hex8> ;; message UID <keys> = <hex8> ;; keyword flags <flag> = <hex4> ;; system flags <mod> = <hex8> ;; date/time last modified (modseq) All other keys, and subsequent fields in per-message records, are reserved for future assignment and must be ignored (and may be discarded) by software which does not recognize them. The mailbox dynamic idex file is rewritten by flag changes (or any future change that alters dynamic data) and is re-read when a session sees that the mtime has changed (atime and ctime are not used). The modseq is an unsigned 32-bit date/time, along with a guarantee that this value can not go backwards. It currently corresponds to the time from time(); however, since it is unsigned, it won't run out until the year 2106. In the future, this may be used as a basic for implementing the IMAP CONDSTORE extension. 2.2 Message data files A mix message file is a regular file with filename starting with ".mix" followed by a <hex8> suffix which indicates the file number. It contains a series of CRLF-terminated lines. By special dispensation, the filename ".mix" is used for file number 0, which was used in experimental versions of mix as a "primary" file (this concept no longer exists). A file number is set to the current modseq when it is created. If a copy or append causes the file to exceed the compiled-in file size limit, a new file is started and the metadata is updated accordingly. Preceeding each message is per-message record with the following format: Key Payload --- ------- ;; per-message record : :<code>:<uid>:<date>:<size>: The per-message records contain the following data: <code> = "msg" ;; fixed code <uid> = <hex8> ;; message UID <date> = <yyyymmddhhmmss+zzzz> ;; internal date <size> = <hex8> ;; rfc822.size The message data begins on the next line Subsequent fields are reserved for future assignment and must be ignored. 3. New mail delivery To deliver a new message, it is necessary to share lock the destination metadata file, then get an exclusive lock on the destination index and status files. Once this is done, the new message data is appended to the new message file. The metadata (UIDLAST value), index, and status files are all updated to add the new message. Then all the destination mailbox files are closed. 4. Mailbox pinging The index and status files are share locked. Initially, sequences are remembered as zero, so at open time they are always "altered". The sequence from the index file is checked; if it is altered the index file is read and processed as follows: . If expunge is permitted, then any messages that are not in the index are reported as having been expunged via mm_expunged(). . new messages are announced via mm_exists()/mm_recent(). Next, the sequence from the status file is checked. If it is altered, the status file is read and the status updated for any message which is new or has an altered modseq in the status file. Altered modseq messages are announced via mm_flags(). Then the index and status files are closed. 4. Flag alteration The status file is exclusive locked. The sequence from the status file is checked. If it is altered, the status file is read and the status updated for any message which is new or has an altered modseq in the status file. Altered modseq messages are announced via mm_flags(). The alterations are then applied for all requested messages, updating the modseq for each requestedmessage which changes flags as a result of the alteration (alterations which do not result in a change do not alter the modseq). Then the status file is rewritten with a new sequence, but only if flags of at least one message was changed. Then the status file is closed. 5. Checkpoint and expunge Checkpoint is identical to expunge, however it skips the step of expunging deleted messages. The index and status files are locked exclusive. If expunging, all deleted messages are expunged from the index and announced via mm_expunged(). The message data is notremoved at this time. If a checkpoint was requested, or if any messages were expunged, or if it remembered that a "burp" was needed, then: . the metadata file is locked exclusive. If this fails, remember that a burp is needed. Otherwise perform a burp: . calculate the file byte ranges occupied by expunged messages . for each file needing "burping", open and slide down subsequent file data on top of the expunged messages . update the index and status files Then the index and status files are closed. 5.1 More details on expunging and "burping" Shared expunge presents a problem due to the requirements of the IMAP protocol. You can't "burp" away a message until you are certain that no sharers have a pointer to any longer. Consequently, for the nonce "burping" out expunged data be defered to an exclusive expunge as in mbx format. If shared burping is ever implemented, then care will be needed not to burp data that a session still relies upon. It's easy enough to burp the index files; just create new index files, deleting the old, and require that you look for a new one appearing at mailbox ping time (when it's safe). The data files are a problem, since we intentionally don't want to keep them open and do want to avoid quota problems by overwriting in place. Also, when you burp you have to change the pointers in the index file. Bottom line: shared burping is too hairy right now, so the first version will do exclusive-only burping and not worry about it. If shared burping is really needed, then that routine will need to be rewritten. Shared burping has been a problem for every other IMAP server. Most get it wrong, and cause terrible confusion to clients (including client crashes). 6. Message data file file roll out strategy The current new message file is finalized, and a new one started, when an append or copy is done that would cause the file to grow to larger than a preconfigured size (MIXDATAROLL). A multi-message copy or append is written into its entirety to a single new message file. In the case of multi-copy, the new message file is switched when the sum of the sizes of all messages to be copied would cause the current new message file to exceed MIXDATAROLL. In the case of multi-append, only the first message is considered; this is due to technical limitations. 7. Error detection Mix detects bad data in the metadata, index, and status files; and declares the stream dead. It does not unilaterally reassign UIDVALIDITY the way that the flat file formats do. When mix reads a header from the message file, it also reads the per-message record and verifies that there is a per-message record there. This is a simple test for message file corruption. It doesn't declare the stream dead; it simply issues an error message and returns a zero-length string for the message header. This makes it possible for the user to fix the mailbox simply by deleting and expunging any messages that are in this state. 8. Reconstruct tool [None of this is implemented yet.] The layout of these files is designed to make the reconstruct tool be as simple as possible. Much of the need for the reconstruct tool is eliminated since the mix format has a much more limited scope of writing than the flat file formats; thus there is "less collateral damage." If the metadata file is lost or corrupted, then all keywords are lost; if the mailbox has any keywords used in the .mixstatus file, it'll be necessary to create some placeholder names. Otherwise, a new UIDVALIDITY can be assigned, and a good UIDLAST value calculated by the reconstruct tool. Since this file is very small, it's not likely to be damaged. If the index file is lost or corrupted, it is possible to reconstruct it with no loss by reading all the data files. However, this could cause expunged but not yet burped messages to reappear. If the status file is lost or corrupted, then flags are lost and will revert to a default state of no flags set. Just deleting the corrupted file is good enough. The reconstruct tool can use the per-message record in the message file to locate messages if the recorded sizes and/or messages are corrupt. If that happens, it will need to rebuild the index file (with associated changes to the metadata file to change the UIDVALIDITY). That should probably be a manual operation and not be part of the default operation or auto-reconstruct. 9. Locking strategy The mix format does not use the traditional c-client /tmp file locking. The metadata file is open and locked whenever the mailbox is open. Normally this is a shared lock, but it will be upgraded to exclusive if the mailbox is expunged. As a guard (since there is no true lock-upgrade/downgrade on UNIX), the index exclusive lock must be acquired first before upgrading to exclusive. The index file is shared locked when reading the index, and exclusive locked (and read) when appending new messages to the index or when expunging (note that expunging also requires an exclusive lock on metadata). Normally, the index file is not open or locked. The status file is shared locked when reading status, and exclusive locked (and read) when updating status. Normally, the status file is not open or locked. It isn't necessary to lock any of the data files as long as we only have exclusive burping. 10. Memory usage The mix format returns a file stringstruct, which is the modern c-client behavior. This prevents imapd from growing to enormous sizes due to a godzillagram (how it affects other programs depends upon what they do with the returned stringstruct). 11. Future extensions Cached ENVELOPE, BODYSTRUCTURE. Cyrus does, and this will eliminate most of the reason to access the data files. Possibly cached overviews, ala NNTP, instead? Support for ANNOTATION. 12. RENAME issues Mix currently makes no attempt to address the IMAP RENAME problem. This occurs when a mailbox is deleted, and another mailbox is renamed with that name in place, no attempt is made to reassign UIDVALIDITY for this mailbox and all the inferior mailboxes. This potentially can cause problems for a disconnected-use client that has cached status for the old mailbox which had that name. The RENAME problem is a well known flaw in the IMAP protocol. Few servers correctly handle it (among other things, not only do all the UIDVALIDITY values have to be changed but this has to be done atomically!). It was a mistake to add RENAME into IMAP, but it's much too late to remove it now.