SMTP / SMTPS


If you have ever hosted a SMTP server you will know how problematic it can be. The last server "problem" I fixed was due to one setting in the IIS configuration allowing anyone to send email from the server. There were 3GB of spam queued! Hosting a website is something you might want to do yourself, but email can be a problem unless you have some experience.

After using many commercial solutions for different clients, I now recommend Gmail because the service is so reliable. Google has become a dominant force in providing email services. They also provide spam filtering and best of all tie gmail to google apps. This means that you can register a domain name for $9.48 using my favorite registrar, and host the domain on googles servers (free for the standard edition). When you enable email for this domain you now have a gmail account that uses your domain (Me@gmail.com becomes Me@mydomain.com).

This is music to most peoples ears. You can login and get the gmail web interface or send email from a desktop email client (Mail Transfer Agent) like Outlook.The only sour note is that google requires an encrypted connection to send email from their servers. They use the standard TLS (SSH) encryption on port 465 instead of port 25.

I have detailed the implementation of a very high quality open source library, cryptlib, and if you are writing in the windows environment, a certificateless Schannel implementation, but first you need to understand what happens once the secure connection is established.



SMTP is the Simple Mail Transfer Protocol, an Internet standard for e-mail transmission. SMTP was first defined in RFC 821 and last updated by RFC 5321 which includes the extended SMTP (ESMTP) additions, and is the protocol in widespread use today.

While mail servers use SMTP to send and receive mail messages, a user-level client mail application or MTA (mail transfer agent) typically only uses SMTP for sending messages to a mail server for relaying. For receiving messages, client applications usually use either the Post Office Protocol (POP) or the Internet Message Access Protocol (IMAP) to access their mail box accounts on a mail server. (IMAP is evil and should be avoided at all costs IMHO)

(Example code for implementing a SMTP dialog in C++ or BASIC is available download)

SMTP is essentially a long winded conversation that you would typically have with a small child. It begins with HELO sent by the <C>lient and is responded to with a 3 digit code by the <S>erver:

  C: HELO yourdomain.com
S: 250 yourdomain.com

C: AUTH LOGIN
S: 334
C: <Username>
S: 334
C: <Password>
S: 235 Accepted

C: MAIL FROM:<you@yourdomain.com>
S: 250 OK
C: RCPT TO:<john@aol.com>
S: 250 OK
C: RCPT TO:<peter@yahoo.com>
S: 250 OK
C: RCPT TO:<fred@gmail.com>
S: 250 OK

C: DATA
S: 354 go ahead
C: <email body>
C: .
S: 250 OK

C: QUIT
S: 221 closing connection
That's 10 separate send/reply events with all the latency of the server/internet compounded. (This can cut in half with ESMTP.)


The email body should contain something like:
  • Date
  • Subject
  • Cc
  • Bcc
  • From
  • DateTime

An example session with google using TLS on port 465:

    C:                <establish a connection>
    S: 44Bytes:  220 mx.google.com ESMTP 6sm661677yxg.48

    C:  7 Bytes:   EHLO
    S: 148Bytes: 250-mx.google.com at your service, [46.311.221.11]
                        250-SIZE 35651584
                        250-8BITMIME
                        250-AUTH LOGIN PLAIN
                        250-ENHANCEDSTATUSCODES
                        250 PIPELINING
   
    C: 13 Bytes:  AUTH LOGIN
    S: 18 Bytes:  334 VXNlcm5hbWU6
   
    C: 46 Bytes:  bm9yZXBseUBmbfedV3LmNvbQ==     (MIME encoded username)
    S: 18 Bytes:  334 UGFsdfedvcmQ6
   
    C: 14 Bytes:  aXJvabcfbffGw=     (MIME encoded password)
    S: 20 Bytes:  235 2.7.0 Accepted
   
    C: 46 Bytes:  MAIL FROM: <noreply@mydomain.com>
    S: 30 Bytes:  250 2.1.0 OK 6sm661677yxg.48
   
    C: 29 Bytes:  RCPT TO: <recipient@theirdomain.com>
    S: 30 Bytes:  250 2.1.5 OK 6sm661677yxg.48
   
    C: 07 Bytes:  DATA
    S: 31 Bytes:  354  Go ahead 6sm661677yxg.48
   
    C: 176Bytes: <the email body goes here>
    S: 41 Bytes:  250 2.0.0 OK 1258345359 6sm661677yxg.48
   
    C: 07 Bytes: QUIT
    S: 46 Bytes: 221 2.0.0 closing connection 6sm661677yxg.48   



If you use port 587, then the session begins as plain text and must transition to an encrypted TLS session. In this case the initial response will be something like:

    C:                <establish a connection>
    S: 44Bytes:  220 mx.google.com ESMTP 6sm661677yxg.48

    C:  7 Bytes:   EHLO
    S: 125Bytes: 250-mx.google.com at your service, [46.311.221.11]
                        250-SIZE 35651584
                        250-8BITMIME
                        250-STARTTLS
                        250 ENHANCEDSTATUSCODES
   
    C: 13 Bytes:  AUTH LOGIN
    S: 67 Bytes:  530 5.7.0 Must issue a STARTTLS command first. 6sm661677yxg.48

(The session has failed here because you did not transition to a TLS session.)



ESMTP

An email client may ask a mail server for the SMTP extensions that the server supports, by using the EHLO greeting instead of HELO. The server will respond with a list. This is extended SMTP (ESMTP). :
  • HELO on port 587 Google returns:
    C:                <establish a connection>
    S: 44Bytes:  220 mx.google.com ESMTP 6sm661677yxg.48

    C:  7 Bytes:   HELO
    S: 125Bytes: 250-mx.google.com at your service


  • EHLO on port 587 Google returns:
    C:                <establish a connection>
    S: 44Bytes:  220 mx.google.com ESMTP 6sm661677yxg.48

    C:  7 Bytes:   EHLO
    S: 125Bytes: 250-mx.google.com at your service, [46.311.221.11]
                        250-SIZE 35651584
                        250-8BITMIME
                        250-STARTTLS
                        250 ENHANCEDSTATUSCODES

  • EHLO on port 465 Google returns:
    C:                <establish a connection>
    S: 44Bytes:  220 mx.google.com ESMTP 6sm661677yxg.48

    C:  7 Bytes:   EHLO
    S: 125Bytes: 250-mx.google.com at your service, [46.311.221.11]
                        250-SIZE 35651584
                        250-8BITMIME
                        250-AUTH LOGIN PLAIN
                        250-ENHANCEDSTATUSCODES
                        250-PIPELINING
                        250 STARTTLS



The most useful ESMTP feature here is PIPELINING (see below).

The other google extensions mean:
SIZE  - maximum message size that will be accepted by the server
8BITMIME - 8 bit data transmission
STARTTLS - The session must transition to a secure TLS connection
PIPELINING - Command pipelining
ENHANCEDSTATUSCODES - more informative explanations of error conditions,

Google returns URLs to the MIME spec along with an error code to help.
These codes appear in all 2xx, 4xx, and 5xx response lines (other than initial greeting and any
response to HELO or EHLO). Note that 3xx responses are NOT included in this list.



PIPELINING

(For examples of both sessions see here.)

Pipelining will cause your SMTP server to send commands in a stream without waiting for a response.
Pipelining indicates a server can accept multiple commands in a single TCP send operation.
Using a single TCP send operation for multiple commands can improve SMTP performance significantly.

(Note: the maximum total length of a text line including the <CRLF> is 1000 characters)
For example code see here

An example pipelined session with google:

C:  EHLO
S: 250-mx.google.com at your service, [23.431.142.46]
S: 250-SIZE 35651584
S: 250-8BITMIME
S: 250-AUTH LOGIN PLAIN
S: 250-ENHANCEDSTATUSCODES
S: 250 PIPELINING

C:  AUTH LOGIN
C:  bm9yZXBseUBmbfedV3LmNvbQ==     (MIME encrypted username)
C:  aXJvabcfbffGw=                                 (MIME encrypted password)
S: 334 VXNlcm5hbWU6
S: 334 UGFzc3dvcmQ6

C:  MAIL FROM: <noreply@mydomain.com>
C:  RCPT TO: <you@yourdomain.com>
S: 235 2.7.0 Accepted
S: 250 2.1.0 OK 6sm1037382ywd.37
S: 250 2.1.5 OK 6sm1037382ywd.37

C:  DATA
S: 354  Go ahead 6sm1037382ywd.37
S: 250 2.0.0 OK 1258854770 6sm1037382ywd.37

C:  QUIT
S: 221 2.0.0 closing connection 6sm1037382ywd.37


This dialog was the result of opening a socket and connecting using a plain text protocol from this code.







SMTP END marker

You may have notice above, that the response to EHLO was more than one line. Since the internet can break responses into several pieces and delay one with respect to another, we cannot always assume that when we retrieve the response it is complete.


SMTP marks the last line of a single response, by using a space between the 3 digit code and the text that follows it. In other words, if the fourth character of a complete line of the response is a space (ASCII 32) , and that line is terminated with the required <CRLF> the response is complete and you may return it.

This is slightly complicated when using ESMTP in that each response to the pipelined requests is sent as a complete response, meaning,  three completed responses may be sent one after the other if you submitted 3 requests like AUTH,  MAIL FROM and RCPT TO.

You need to recover each of these responses as complete separate responses before proceeding with the next request (in this case DATA).

To do this, the code I wrote takes the number of expected responses as an argument and ensures that all responses have been received before returning (or it times out)

I suppose it would be possible to write a generic case that parses all the pipelined lines to find SMTP reserved words, but this would also need to track the user name and password lines somehow. Error handling could be a headache so I just elected to specify it explicitly.




MIME Encoding
All user names and passwords must be MIME (UTF-8) encoded. I included a function to do this in the source code as it can be a little tricky to write without getting too deep. This is obviously a crucial piece of code to make this all work as google, like most mail hosts these days, require authentication. The authentication step is the only place where text must be encoded before being sent. If you think about it, this a probably more a hangover from the port 25 plain text SMTP days, because everything sent via an encrypted connection is encrypted.

I tried specifying AUTH LOGIN PLAIN, but could not get google to accept this SMTP extension. Considering they support most other SMTP extensions, this makes little sense. If you just cant deal with writing a MIME encoder and will not be using multipart files, then you can just encode your user name and password and feed the encoded string to the gmail server. A free online base64 encoder and decoder is available here

The username and password may be pipelined with AUTH LOGIN so that authentication can be accomplished in a single request. What I found odd, was that the code:
"235 2.7.0 Accepted", was sent only after the next request (MAIL FROM and RCPT TO) This may just be a quirk of google, but if you know where to expect it, its not a problem.






Multipart MIME

This is where email gets cool. Bank of the West sent my client a very nice multipart email recently, complete with a MIME encoded image of their trademark bear. They had encoded a 90k image + HTML..

He was very impressed that the image showed up right away without being blocked as most email clients like outlook do these days. Blocking HTML links that would otherwise be used to display an image in your email is important to prevent the sender from doing things you might not approve of. This means that all those beautiful graphics you spent so much time on are replaced with the good old red x.



This tends to irritate banks, especially when they are trying to separate you from your hard earned money by promoting their endless banking "products" (like variable rate credit cards, late fees, and overdraft "protection")

Notice in the version above that the HTML is still working even though the images were blocked. We have a nice color background and font control and clickable links to connect you to their website. This is all very useful in a business setting, especially when you are branding your product and trying to make it easy for users to click through and buy your product.

My client wanted to replicate this so I created an example that does all three of these parts:
  • default text
  • HTML
  • embedded file - logo (image)
Obviously you can just use the default text and the HTML if you want to be size friendly. As you can see above, even Banks find it hard to pump millions of 90k of MIME encoded images to people, and have elected to use the HTML links instead for most of their traffic.

This code simply creates an email body with with all the parts separated with boundary's that are given a unique Content-ID. For simplicity I use a a single ID in all my emails (for which I am sure I will receive a stiff sentence at the email crimes tribunal) but the multi part spec says a Content-ID is supposed to be globally and permanently unique.

To send HTML with a text alternative (without an attachment) I created a simple example to insert the multipart boundaries. It would be a simple matter to replace the string constant with a unique string like a GUID for example.

The Multi-part mime example sends both an HTML and TEXT part of an email message in a single email.  When the recipients email client receives a multipart message, it displays the HTML version if it can render HTML, otherwise it displays the plain text alternative.

Sending both an HTML version and a TEXT alternative allows you to reach the maximum audience; those that have an advanced email client get a richer message format.

If you are fascinated by the details of multi part MIME, read the Wiki. To start playing with it immediatley, use the code here.




SMTPS - SSL/TLS
- sending email becomes irksome!
There are many solutions to providing a secure connection without having to roll up[ your sleeves and get gritty handshake details under your fingernails. I will review of most of them including the built in Windows solution Schannel.



First a little history. Transport Layer Security (TLS), a network protocol and successor to Secure Sockets Layer (SSL) comes in a few versions 1.0, 1.1 and the current 1.2. As with any security designed by comittee, it has issues. v1.0 relied upon MD5 which is an insecure cipher. v1.2 can use Advanced Encryption Standard (AES) ciphers, but this is negotiated upon connection with the server. Obviously if the server can only support v1.0, your "Secure" connection is anything but, and even private keys are not secure. Leaving that aside for the moment, the central issue is how do we just get something working so we can send emails!

A "secure" session can be created explicitly at the handshake, or upgraded after the connection is made using STARTTLS (which google will prompt for if you connect on port 25 without TLS. Some languages (Python, php etc ) support this command and thus allow to implicitly create a "secure" connection. Apart from that, you will need a library to manage all this for you.

Many people use STUNNEL to provide this capability. It needs to be installed and runs as a service, but apart from that, reports are good. If asking your customers to install another piece of software on their machines might be a bit of an inconvenience, then there are some other options.

Commercial ActiveX (COM) libraries from Chilkat also require an installer. This product is mostly delivered as an Activex (COM) "dll" (requiring an installer on the users machine to 'register' the dll - another bad .net idea). The installer loads "ChilkatMime.dll" 1.33Mb, "ChilkatCert.dll" 1.26MB, "ChilkatUtil.dll" 720k. The developers were not at all interested in cooperating on a true C .dll library that could be called from any language including C/C++/BASIC/Python etc etc. Apart from the cheesy name/artwork, their products are reasonably priced, but the one I tried, connected on port 25 despite being told to use port 465.

For top of the line tools, Socket Tools from catalyst (http://www.catalyst.com) cannot be beat.

http://www.catalyst.com/products/sockettools/secure/library/index.html

SMTP is now available as component of the main socket tools product for 1/3 the price. These tools are first class! yes, you get what you pay for. The developers are responsive and open to suggestions. They offer ALL flavors of .dll including a stand alone .dll that can be shipped with you product that is only 230k! The tools even handle the SMTP dialog and error checking. For commercial solutions they win hands down.


For Open Source, there are two options, OpenSSL and CryptLib. Since STUNNEL is written around OpenSSL, I have to conceed that it does work, even if it does require two or three .dll's ("libeay32.dll" 1.35MB + "libssl32.dll" 310k + "zlib1.dll" 75k) that add up to 1.75MB.

Apart from that, OpenSSL has some perfomance issues (http://josefsson.org/gnutls4win/), 
"Initializing libgcrypt takes a long time on some systems, there has been reports that it can take around 10 seconds."
and does some non standard things:
(http://marc.info/?l=openssl-dev&m=108982648816140&w=2) here and there.
"OpenSSL (as you're using it) is using an undocumented homebrew data format that isn't compatible with anything else in existence." http://article.gmane.org/gmane.comp.encryption.cryptlib/2364
It is not a trivial task to get this library to work as mentioned here or here.
Despite many efforts, I was unable to find anything working with this library. If you do, please send me an example in any language that statically links the OpenSSL Dlls at runtime.

By contrast, I got CryptLib to work almost immediately. cryptLib is an open source solution that facilitates an SSL/TLS tunnel with a C style .dll in only 1MB (full compilation). Since the source is available, it is possible to compile a version of the dll with just the components you need that should come in somewhat less than that.

http://www.cs.auckland.ac.nz/~pgut001/cryptlib/download.html

The author was very responsive even though I got the library to work immediately. There is even a 330 page manual!

This library is not an MTA (mail transfer agent) so you must write the SMTP conversation detailed above, but it is FREE!

Cryptlib is an extremely robust cryptography toolkit developed by Peter Gutmann. It differs from other toolkits such as OpenSSL, in that it is more of a cryptography kernel than a C library.  If you are writing code than may need access to everything at some point down the road, this is the library for you. It has a very simple high level interface which I have written the examples that follow to use.

CryptLib does not manage any of the SMTP dialog, it just creates an encrypted connection  Once the connection has been made, a "tunnel" exists through which a SMTP session may proceed as normal.

For a discussion of the implementation using cryptlib, read more here.


Windows Schannel is shipped with all OS's going back to Windows 95. You access the functionality via the Security Support Provider Interface, SSPI. For a discussion of the implementation read here.