Design Notes On Fetchmail

Introduction

This document is supposed to complement Eric S. Raymond's (ESR's) design notes. The new maintainers don't agree with some of the decisions ESR made previously, and the differences and new directions will be laid out in this document. It is therefore a sort of a TODO document, until the necessary code revisions have been made.

Security

Fetchmail was handed over in a pretty poor shape, security-wise. It will happily talk to the network with root privileges, use sscanf() to read remotely received data into fixed-length stack-based buffers without length limitation and so on. A full audit is required and security concepts will have to be applied. Random bits are:

• code talking to the network does not require root privileges and needs to run without root permissions
• all input must be validated, all strings must be length checked, all integers range checked
• all types will need to be reviewed whether they are signed or unsigned

SMTP forwarding

Fetchmail's multidrop and rewrite options will process addresses received from remote sites. Special care must be taken so these features cannot be abused to relay mail to foreign sites.

ESR's attempt to make fetchmail use SMTP exclusively failed, fetchmail got LMTP and --mda options – the latter has a lot of flaws unfortunately, is inconsistent with the SMTP forwarder and needs to be reviewed and probably bugfixed. --mda doesn't properly work with multiple recipients, it cannot properly communicate errors and is best avoided for now.

Server-side vs. client-side state.

Why we need client-side tracking

ESR asserted that server-side state were essential and those persons repsonsible for removing the LAST command from POP3 deserved to suffer. ESR is right in stating that the POP3 UID tracks which messages have been read by this client – and that is exactly what we need to do.

If fetchmail is supposed to retrieve all mail from a mailbox reliably, without being disturbed by someone occasionally using another client on another host, or a webmailer, or similar, then client-side tracking of the state is indispensable. This is also needed to match behavior to ETRN and ODMR or to support read-only mailboxes in --keep mode.

Present and future

Fetchmail supports client-side state in POP3 if the UIDL option is used (which is strongly recommended). Similar effort needs to be made to track IMAP state by means of UIDVALIDITY and UID.

This will also mean that the UID handling code be revised an perhaps use one file per account or per folder.

Concurrent queries/concurrent fetchmail instances

ESR refused to make fetchmail query multiple hosts or accounts concurrently, on the grounds that finer-grained locks would be hard to implement portably.

The idea of using one file per folder or account to track UIDs on the client-side will make solving this locking problem easy – the lock can be placed on the UID file instead.

Multidrop issues

Fetchmail tries to guess recipients from headers that are not routing relevant, for instance, To:, Cc:, or Resent-headers (which are rare anyways). It is important that fetchmail insists on the real envelope operation for multidrop. This is detailed in my article "Requisites for working multidrop mailboxes".

As Terry Lambert pointed out in the FreeBSD-arch mailing list on 2001-02-17 under the subject "UUCP must stay; fetchmail sucks", fetchmail performs DNS MX lookups to determine domains for which multidrop is valid, on the assumption that the receiving SMTP host upstream were the same as the IMAP or POP3 server.