Email Protocols have evolved since their early inception and are now the underpinnings of much modern communication software. The careful considerations for security, the ability to send large files, and wide-scale access weren’t always available through Internet email.
Generations of email protocol revisions and advances have led us to the modern systems we have today. Whether you’re building a new email client from the ground up or simply checking your Gmail account—chances are SMTP, POP3, and IMAP protocols are going to play a major role in the process.
Highlights
- ARPANET was the stage on which the early messaging protocols were developed.
- SMTP was the first major advance to simple File Transfer Protocol and was designed specifically for internet messaging.
- POP3 and IMAP are later protocols developed to provide more robust security, access, and interaction
- Popular consumer applications like Gmail, Outlook, and Thunderbird all facilitate the usage of these protocols in abstracted user interfaces.
Quick History
In 1958 the U.S. Defense Department (DoD) founded the Advanced Research Projects Agency (ARPA) (DOD Directive 5105.15). ARPA was billed as an agency to “to help maintain U.S. technological superiority and guard against unforeseen technological advances by potential adversaries.” At the time this likely wholly referred to Russia’s Sputnik program.
ARPANET is Created
In 1969, ARPA contracted BBN Technologies to build the first routers which allowed scientists to connect “geographically separated” computers. This was called ARPANET and shortly thereafter transformed into the earliest expressions of the Internet we know today.
First Network Created
The initial ARPANET network was a four-node design consisting of Internet Message Processors (IMP) located at UCLA, Standford, University of California at Santa Barbara, and the University of Utah. The first successful network message transmission between separate machines occurred on October 29, 1969.
TCP/IP is Born
In 1973 DARPA, having been re-branded a year prior, introduced the TCP/IP protocol. This project was done in collaboration with researchers from Stanford University. The TCP/IP protocol remains the foundation of much of the modern Internet’s communications—including a large portion of modern Internet Mail Exchanges.
Localized Message Services
Before there was email, there were internetwork communications. Initially, users would log on to a shared resource and save files in a local directory with names like ToAlice or ToBob indicating the desired recipient. The follow-up to this rudimentary system came in 1965 when MIT researchers developed the MAIL command as outlined in a circulated staff note circa 1965:
A new command should be written to allow a user to send a private message to another user which may be delivered at the receiver’s convenience This will be useful for the system to notify a user that some or all of his files have been backed-up. It will also be useful for users to send authors any criticism.
This protocol established the basics of the inbox: a user can receive directed messages such that newer ones are differentiated from older ones. While still running around on local networks, this system laid the conceptual groundwork for message queuing that would later serve as backbone services to email and internet messaging protocols.
Internet Email is Born
In a 1971 Request for Comments (RFC196 ), Richard W. Watson outlined a proposed Mail Box Protocol specifically for the send/receipt of textual messages. This RFC outlined such aspects of the Mail Box Protocol as follows:
- Sequential user files to which new messages would be appended
- Transmission format of 7-Bit ASCII in 8-bit bytes
- Endlines, newlines, and site-specific encoding opportunities
- Message width and document length
- Transmissions protocols
- Connection lifespan
- Mail Box numbering systems
The backbone of early email messaging was done via File Transfer Protocol outlined in the 1972 RFC354. This outlined the new usage of several existing protocols to form a more useful data transfer protocol. The existing protocols were HOST-HOST, Initial Connection Protocol, and TELNET. This RFC outlined FTP as the following considerations:
- Establishing Data Connections
- Data Representation & Storage
- File Structure & Transfer Modes
- Error Recovery & Restart
- Files Transfer Functions
- FTP Commands
- Access Control Commands
- Data Transfer Commands
- FTP Service Commands
This document laid the groundwork for describing a protocol that would then be used as the backbone of early internet messaging services.
In a later RFC469, further specifications were made that include the use of several new additions to the File Transfer Protocol that was sending/receiving messages:
- @ – to denote user/host relationships
- TO – to address the recipient
- FROM – to denote the sender
- AUTHOR – to denote possible multiple message authors
- TITLE – the early term for “subject”
- ACKNOWLEDGEMENT – protocol for notifying the sender of a transmission success
- RECORDED – optional field to store a copy on network servers
- TYPE – sender direction for how the message should be stored (long, urgent, short)
- TEXT – the field for the main body of the message
Simple Mail Transfer Protocol
Unsecured Port: 25
Secured Port: 465
In 1981, Jonathan B. Postel officially laid out the Simple Mail Protocol in RFC 821, the contents of which are not brief. This proposal stood to revolutionize the current FTP system in a way that would be much more specific to mail-sending based needs. Among many outlined procedures, RFC821 outline the following:
- Forwarding
- Verification
- Opening & Closing messages
- Message Relaying
- Domains
- SMTP commands
The goal of SMTP was to create a system to transfer textual messages reliably and efficiently. SMTP was seen as an evolution of protocol because it was to exist agnostic of the lower network layers and their respective transmission protocols. In other words, SMTP was an application layer service.
SMTP still uses the 7-bit ASCII model outlined in RFC196 which requires binary data from sender/receiver sides to be encoded upon transmission and then decoded upon receipt accordingly.
The SMTP Messaging Model
SMTP brought a lot of new concepts to the table. At the core, SMTP was designed to facilitate the communication between two hosts connected via the same transport service. A basic outline of this process is below:
- Establish Transmission channel
- SMTP Sender sends a MAIL command
- SMTP Receiver responds with an OK command
- SMTP Sender sends RCPT command, identifying the intended recipient
- SMTP Receiver responds with OK if the message can be received
- SMTP sends mail data, using a specific termination sequence
- SMTP Receiver responds with OK on successful receipt of message data
SMTP uses a Store-and-Forward protocol which is characterized by its PUSH protocol where the messaging service is initiated by the sending server. A user enters a command that is transmitted (Pushed) via the user’s SMTP server to the receiver’s SMTP server. The recipient server can then store the file onto the recipient users’ local file system as shown in the following illustration:
Initially, SMTP was designed to be a message transfer service intended to route completed messages between hosts. There was little accommodation for incorrect message formatting or error checking. As SMTP became more popular and its usage more widespread, it was realized more as a message submission service. Published in December of 1998, RFC2476 notes this and outlines a protocol for more robust Message Submission use.
Post Office Protocol – Version 3
Unsecured Port: 110
Secured Port: 995
POP3 was outlined in May of 1996 via RFC1939 and was intended to outline a mail service where SMTP may not be practical given limited local resources. As such, POP3 allows a user to retrieve mail from a remote mail server on which messages addressed to them are being “held” while still relying on SMTP protocol between mail servers.
POP3 works via TCP on port 110 for non-secured messaging and consists of the following phases:
- Authorization
- Transaction
- Update
During these phases, POP3 mainly consists of the following four commands issued by the user:
- list – provides the user with a list of messages and sizes
- retr – requests contents of a particular message
- dele – marks a message on the mail server for deletion
- quit – ends the session, the server executes dele commands.
After each of these commands, the mail server responds with either an +OK or -ERR message to inform the user regarding the success of their previous command. There are several other POP3 commands available that indicate things like multi-line responses, unique identification numbers for messages, and MD5 encryption directives for more secure transactions. These are considered “optional” POP3 commands.
Authorization requires that a user provides credentials such as a username and password to access data on the remote mail server. This phase has two primary commands: user and pass each followed by the appropriate data.
It’s worth noting that POP3 when operating via TCP on port 110, is an unsecured means of transferring credentials. If the user and pass data are accepted, denoted by an +OK response from the server, the session then proceeds to the next phase.
Note: Modern POP3 is available as a secured service operating on port 995
Transaction Phase
During the transaction phase, a user can delete messages, mark messages as read/unread, and also read mail-related statistics such as the size of a particular message.
During the transaction Phase of POP3, the user can operate in one of two “modes:” download-and-delete or download-and-keep. This instructs the mail server how to handle messages after being retrieved by the user. This requires the list, retr, and dele commands.
Operating in delete-and-keep mode, the user
Update Phase
The third phase takes place after the user issues a QUIT command. During this phase, the mail server deletes any messages that were marked for deletion by the user. Per the RFC, the QUIT command, signaling the UPDATE Phase, results in the following actions:
- Mail server removes all messages marked for deletion
- Mail server replies with the status of this operation (+OK or -ERR)
- Mail server releases any access-locks on messages not marked for deletion.
- Mail server closes TCP connection with the user.
Mail servers do not issue dele instructions when a POP3 session is interrupted for reasons other than the QUIT command being issued by a user. Also, a QUIT command issued during the authorization state will not result in the UPDATE state being completed. For example, consider the following scenario:
- User enters Authorization Phase, successfully authenticating w/server
- User enters Transaction phase, successfully marking messages for deletion
- User’s network connection drops, the user is disconnected from the server.
- User attempts to re-connect and, during the Authentication phase, drops connection again.
POP3 protocol dictates that during step 3 (the first time the user is disconnected) if the QUIT command is not issued the messages aren’t deleted. If, while trying to reconnect during step 4, the user is disconnected again, POP3 again prevents the QUIT command from being issued.
This is a fail-safe to ensure the user has finished marking/unmarking/remarking messages for deletion before the final Update process is initiated.
Internet Message Access Protocol (IMAP)
Unsecured Port: 143
Secured Port: 993
IMAP is similar to the POP3 protocol in that it is a retrieval protocol. IMAP was designed to help manage a single mailserver using multiple clients, as illustrated below:
Whereas POP3 protocol generally results in users downloading copies to their local machine, IMAP leaves copies of messages on the remote server until explicitly deleted.
IMAP has undergone several iterations, beginning with IMAP2 and now technically IMAP4rev4 per RFC3501. IMAP is different than POP3 in several ways, including the following:
- IMAP allows for a continuous connection
- IMAP doesn’t delete server messages unless explicitly instructed
- IMAP allows multiple connected clients to the same mailbox
- IMAP uses MIME-type access to data, such that users can download a message w/o attachments or other data.
- IMAP allows server-side searches
Review
Internet Messaging Protocols are built-in to many popular applications, though there is often little client-facing awareness of such. Gmail doesn’t list the instructions it issues between mail servers, Outlook doesn’t give an error message stating “UPDATE phase not completed.”
However, most email clients still contain hints of their underpinning protocols in the settings. When connecting email clients like Outlook or Mozilla Thunderbird, one is often given the option of whether to use POP3 or IMAP to check messages. This faint hint of the underlying technology is as close as many consumers will ever get to underlying protocols—and that’s ok!
