libfcgi2

SMTP / SMTPS‎ > ‎

TLS with cryptlib

Sending email via an encrypted (SSL) connection
Since google introduced gmail and required the use of an encrypted session, the need to incorporate TLS (which is an SSL connection or tunnel) in applications has arisen. Since SSL is a complex protocol that requires handshaking, negotiation and encryption, this is usually more than most developers can or want to deal with just to send an email via smtp.gmail.com.

Google apps and gmail provide free reliable email hosting, which is difficult to find even in well rated commercial solutions. Sending email from an application then becomes a little more tricky than just opening a TCP connection on port 25 and proceeding with the SMTP dialog. Now the connection must be an SSL tunnel through which the SMTP dialog is conducted.

A commercial solution like the excellent socket tools manage the connection and the SMTP dialog for you, but cryptlib is an open source library that can provide the SSL tunnel if you are willing to write the code for the SMTP dialog. This Article will show you how to do both.

This article is designed for languages that do not SMTPS natively like some flavors of BASIC and C, and even for some that do and run into trouble.


SMTP ports
1. Port 25 (SMTP) is plain text. The conversation between server and client is held in plain text. This is where the majority of internet MTAs are and have been since SMTP was introduced. The big problem with plain text being that user credentials and passwords are easy to intercept by sniffing the transmission media.

2. Port 465 (SMTPS) is TLS encrypted. The conversation between server and client is held through an encrypted connection.

3. Port 587 (STARTTLS) is a transitional hybrid. Think of it as a special interim measure between 1 and 2. Although 'interim' on the internet likely means the next decade or so. STARTTLS is an enhanced SMTP command; an optional extension to the SMTP protocol. STARTTLS allows the MTA and client to switch (one way) from plain text to TLS encrypted transmission. STARTTLS is often provisioned on Port 25 for compatibility with either plain text or encrypted transmission. When provisioned on port 587 it is implicitly a requirement that transmission must become encrypted prior to authentication.


SMTPS Client - Source code

The code referred to in this article is available for download:

Example BASIC SMTP client and BASIC headers

Example C++ SMTP client and C headers (VS2008 Project also available for download)

An alternative to writing a base 64 MIME encoded is a free online base64 encoder and decoder, available here



Cryptlib

cryptlib is a very complete encryption library developed by Peter Gutmann over a number of years. cryptlib allows even inexperienced programmers to easily add SLL (and encryption) to their software. We are very lucky to have such a high quality library available as open source. cryptlib uses industry-standard X.509, S/MIME, and ssh/SSL/TLS data formats, the resulting encrypted or signed data can be easily transported to other systems and processed there, and cryptlib itself runs on any commonly-user operating system. Multi-threaded cryptlib Operation cryptlib is re-entrant and completely thread-safe.

The source can be downloaded here, and v3.33 the compiled 32bit windows dll from here.


cryptlib is available under the GPL-compatible Sleepycat dual license, which means you can use it under the GPL terms or under standard commercial terms, depending on your preference. More information on use under the commercial terms is given in the cryptlib brochure which explains the features of cryptlib and details the licensing terms (it's free for non-commercial and some types of commercial use, shareware, academic research, and so on).

The examples that follow use this library to create and encrypted tunnel with a SMTPS server (gmail in this case) through which an email can be sent. This functionality can be crucial to an application even if it is not an MTA (Mail Transfer Agent). The ability to perform this task without requiring the install of any third part libraries is invaluable when providing solutions for users. No one likes delivering software with the disclaimer "requires the install of xxx compnant to work"

Google provided a very reliable free email service. Google apps allow you to register a domain mydomain.com, and point the DNS servers to google for the email portion. Having used several hosted email solutions over the years, some of which were very expensive, I have always been dogged with unreliablity and the perenial queuing on Sunday nights as all the spammers blast the world with their unwanted solicitations for your viewing pleasure on Monday morning.

I did find a free alternative that does not require an encrypted TLS connection
http://www.gmx.com/
but have not had an opportunity to test them out over a period of time. My guess is that a company like google has the resources to deliver this service more reliably at present.


Creating an SSL Session

To use the library, we must first intitialize it:

    RetVal = CryptInit()

Since TLS is really just SSL, the first step is to create an SSL session with:

    RetVal = CryptCreateSession( VARPTR(hSess), CRYPT_UNUSED, CRYPT_SESSION_SSL )

Notice that the first argument is a pointer to the LONG INTEGER session handle. All CryptLib functions that modify a vlaue, take a pointer to that value (in this case a LONG POINTER), so you must create this hSess pointer in your language.

After the call to CryptCreateSession(), hSess will contain a session handle in a LONG value that is fed to subsequent function calls like:

    RetVal = CryptSetAttributeString( hSess, CRYPT_SESSINFO_SERVER_NAME, STRPTR(sSrvr), LEN(sSrvr) )

This function sets a string attribute.  sSrvr is a string containing the server name (or IP):  "smtp.gmail.com"

      RetVal = CryptSetAttribute( hSess, CRYPT_SESSINFO_SERVER_PORT, nPort )

This function sets a LONG attribute, in this case the port 465.

Finally we must activate the session with:

      RetVal = CryptSetAttribute( hSess, %CRYPT_SESSINFO_ACTIVE, 1 )

This causes CryptLib to perform all the connection handshaking, certificate exchange and protocol negotiation that fills volumes in university libraries.

The by product is that google gets an empty request which trigges the error response:
220 mx.google.com ESMTP 6sm665677yxg.48

Here Google is prompting us to send a ESMTP request EHLO, which marks the beginning of the SMTP dialog detailed here.


Push/Pop Data Functions

This is where the rubber meets the road in this library. In the manual, Peter describes the benefits of using a full I/O completion ports implementation in Windows IIS servers, but that is a lot of work just to send an email. All we need to do, is handle response timing, error codes and internet connection issues, and we will have a reliable TLS email client.

The request is sent with

    RetVal = CryptPushData( hCrypt, STRPTR(sSend), LEN(sSend), VARPTR(BytesSent) )

This returns the number of bytes actually sent, and as far as I can tell should always be equal to the number of bytes in the string for smaller data sets

Next the request must be flushed (sent to the server) with:

    RetVal = CryptFlushData(hCrypt)


And finally the response is returned with a call to:

      RetVal = CryptPopData( hCrypt, STRPTR(sBuff), LEN(sBuff), VARPTR(BytesReply) )

Its that simple in concept. The problem is that CryptPopData() just returns what has been received at the moment you call it.
This function does not block.

This means that we need to loop and keep checking for a response. It also means that we need to examine the response to make sure that it is complete and that it contains what we expect to find (lines of SMTP response terminated with <CRLF>)

As described here, most responses are single lines, terminated with a <CRLF> (0d0a),
  • 250 2.1.0 OK 16sm183640gxk.15

but one in particular is not:
  • 250-mx.google.com at your service, [46.311.221.11]
  • 250-SIZE 35651584
  • 250-8BITMIME
  • 250-AUTH LOGIN PLAIN
  • 250-ENHANCEDSTATUSCODES
  • 250 PIPELINING
This is a single response. Notice that the last line does not have a hyphen - between the code 250 and PIPELINING.This signals the last line of the response.

One way to process this, is to begin with the last byte and work backwards through the string until we arrive at the begging of the line. Then we can check the fourth byte to see if it is a space. If it is not we need to wait for some more lines.

This also raises the possibility that the line may not be complete (ie terminated with <CRLF>, so this must be checked before we look at the fourth character. Since the minimum line length is 6 characters, we could check that too.

  • BYTE     0   1   2     3     4    5  
  • ASCII    C1 C2 C3   ??   CR LF

Pipelining adds a small layer of complexity as many individual responses can be returned at once. For example, PIPELINING AUTH LOGIN followed by the username and password results in the following two responses.
334 VXNlcm5hdWU6
334 UGFzc3dvfdmQ6

Each one is a complete response. These is not a multi-lined response like the response to EHLO. Each one of these must pass the tests above in order for the function to return success. Any SMTP error codes or incomplete returns will cause the PIPELINED request to fail.


Request/Response handling

    FUNCTION TLSPushPop( hCrypt AS LONG, sErr AS STRING, sReply AS STRING, sSend AS STRING ) AS LONG

This function is designed to return the complete server replies or an error (which can also be a timeout). The function uses string concatenation for simplicity. In translating this to your language, a Vector or StringBuilder style of class would probably be more suitable.

Since it is possible that another server may do something very unexpected, the first thing we do after ensuring a buffer for the replies, is to check for any data that might have been returned since the last call. This should not happen, but if it is not explicitly handled a situation could arise where some extra unexpected data gets pre-pended onto the response to the current response causing unexpected results.

Next the function pushes all the data to CryptLib and flushes out to the server. Then we immediately enter a loop that begins counting milliseconds of delay. Should we get to SMTP_RESPONSE_TIMEOUT milliseconds without a complete return the function will return a timeout error code.

When CryptLib processes return data, CryptPopData() returns the data in the buffer and we check it as described above. If your language does not support pointers, just use MID$() to pull out characters one by one.

Should you need to send a large number of bytes, then you may need to create a loop around CryptPushData() if it chokes, and break the file up into chunks.

BASIC pseudo code:

    RetVal = CryptPopData( hCrypt, STRPTR(sBuff), LEN(sBuff), VARPTR(BytesReply) ) 
    IF BytesReply > 0 THEN RETURN ERROR

    RetVal = CryptPushData( hCrypt, STRPTR(sSend), LEN(sSend), VARPTR(BytesSent) )  

    RetVal = CryptFlushData(hCrypt) 

      DO 

        SLEEP 20 ' Wait for a response 
        Totms = Totms + 20 '
        IF Totms > %SMTP_RESPONSE_TIMEOUT THEN RETURN ERROR
 
        RetVal = CryptPopData( hCrypt, STRPTR(sBuff), LEN(sBuff), VARPTR(BytesReply) ) 
        IF BytesReply > 0 THEN
              sReply = sReply + LEFT$(sBuff, BytesReply) 

              Last   = LEN(sReply)
              pByte = STRPTR(sReply) 
              IF Last > 5 AND @pByte[Last-1] = 10 AND @pByte[Last-2] = 13 THEN
              FOR k = 3 TO Last 
                  IF @pByte[Last-k] = 10 AND @pByte[Last-k+4] = 32 THEN EXIT DO
                  IF k = last                    AND @pByte[Last-k+3] = 32 THEN EXIT DO
             NEXT
          END IF 

        END IF 

      LOOP


The loop must be executed for each response for all the responses to be returned.

This page and the code derived from it is the result of about 60hrs reading and development testing. cryptlib is the result of years of work and can do a lot more than create an SSL tunnel. If you are interested in further reading, a good place to start is here:




Cryptlib and SSL (from the manual)


  cryptlib works with two classes of objects, container objects and action objects. A
  container object is an object that contains one or more items such as data, keys or
  certificates. An action object is an object which is used to perform an action such as
  encrypting or signing data. The container types used in cryptlib are envelopes (for
  data), sessions (for communications sessions), keysets (for keys), and certificates (for
  attributes such as key usage restrictions and signature information). Container
  objects can have items such as data or public/private keys placed in them and
  retrieved from them. In addition to containing data or keys, container objects can
  also contain other objects that affect the behaviour of the container object. For
  example pushing an encryption object into an envelope container object will result in
  all data which is pushed into the envelope being encrypted or decrypted using the
  encryption object. 

  Encryption contexts are the action objects used by cryptlib. Action objects are used
  to act on data, for example to encrypt or decrypt a piece of data or to digitally sign or
  check the signature on a piece of data.

  The usual mechanism for processing data is to use an envelope or session container
  object. The process of pushing data into an envelope and popping the processed data
  back out is known as enveloping the data. The reverse process is known as deenveloping
  the data. Session objects work in a similar manner, but are used to
  encapsulate a secure session with a remote client or server rather than a local data
  transformation. The first section of this manual covers the basics of enveloping data,
  which introduces the enveloping mechanism and covers various aspects of the
  enveloping process such as processing data streams of unknown length and handling
  errors. Once you have the code to perform basic enveloping in place, you can add
  extra functionality such as password-based data encryption to the processing. After
  the basic concepts behind enveloping have been explained, more advanced techniques
  such as public-key based enveloping and digital signature enveloping for S/MIME
  and PGP are covered.
      
  Session objects are very similar to envelope objects except that they represent a
  communications session with a remote client or server. The next section covers the
  use of session objects for protocols such as SSL, TLS, and SSH to secure
  communications or work with protocols such as CMS, SCEP, RTCS, OCSP, and TSP
  that handle functions such as certificate status information and timestamping.
  The use of public keys for enveloping requires the use of key management functions,
  and the next section covers key generation and storing and retrieving keys from
  keyset objects and crypto devices. The public portions of public/private key pairs are
  typically managed using X.509 certificates and certificate revocation lists. The next
  sections cover the management of certificates including certificate issue, certificate
  status checking, and certificate revocation list (CRL) creation and checking, as well
  as the full CA management process. This covers the full key life cycle from creation
  through certification to revocation and/or destruction.


  High-level Interface
 
  This interface requires no knowledge of encryption and digital signature techniques,
  and is easiest for use with languages like Visual Basic and Java that don’t interface to
  C data structures very well. The container object interface provides services to create
  and destroy envelopes and secure sessions, to add security attributes such as
  encryption information and signature keys to a container object, and to move data
  into and out of a container. Because of its simplicity and ease of use, it’s strongly
  recommended that you use this interface if at all possible.
      

  Initialisation
  Before you can use any of the cryptlib functions, you need to call the cryptInit
  function to initialise cryptlib. You also need to call its companion function cryptEnd
 

  The Visual Basic interface to cryptlib is otherwise mostly identical to the standard C/C++ one.

  Return Codes
  Every cryptlib function returns a status code to tell you whether it succeeded or
  failed. If a function executes successfully, it returns %CRYPT_OK. If it fails, it
  returns one of the status values detailed in “Error Handling” on page 275.       

      
  Secure Sessions - SSH, SSL, and TLS sessions,
      
  As with envelopes, cryptlib takes care of all of the session details for you so that all
  you need to do is provide basic communications information such as the name of the
  server or host to connect to and any other information required for the session such as
  a password or certificate. cryptlib takes care of establishing the session
   
  Secure sessions are very similar to envelopes, with the main difference being that
  while an envelope is a pure data object into which you can push data and pop the
  processed form of the same data, a session is a communications object into which you
  push data and then pop data that constitutes a response from a remove server or
  client. This means that a session object can be viewed as a bottomless envelope
  through which you can push or pop as much data as the other side can accept or
  provide.
 
  For example to connect to a server using
  SSH and obtain a directory of files using the ls command you would do the
  following:
  - create the session;
  - add the server name, user name, and password;
  - activate the session;
  - push data
  - pop the return
  - destroy the session
         
 
  Client vs. Server Sessions
  cryptlib distinguishes between two types of session objects, client sessions and server
  sessions. Client sessions establish a connection to a remote server while server
  sessions wait for incoming communications from a remote client.
       
  Establishing a Session
  it’s generally better to explicitly activate the session
  You can activate a session by setting its %CRYPT_SESSINFO_ACTIVE attribute to
  true (any nonzero value).
 
  SSL/TLS Sessions
  SSL and TLS are actually variations of the same protocol, the protocol known by the
  generic term SSL. cryptlib will automatically negotiate the highest protocol
  version supported by the other side.
 
  SSL/TLS is a secure data transfer protocol that provides confidentiality, integrity protection,
  protection against replay attacks, and a variety of other services. The SSL
  server is authenticated via a certificate, and the client isn’t authenticated (in rare
  circumstances client certificates may be used, but these are usually avoided due to the
  high degree of difficulty involved in working with them). Alternatively, the client
  and server may be mutually authenticated via a secret-key mechanism such as a user
  name and password, which avoids the need for certificates altogether. cryptlib
  supports SSL version 3, TLS version 1.0 (a.k.a SSL version 3.1), and TLS version 1.1
  (a.k.a SSL version 3.2).
 
  SSL/TLS Client Sessions
  Establishing a session with an SSL/TLS server requires adding the server name or IP
  address and an optional port number if it isn’t using the standard SSL/TLS port.
 
  Once you’ve added this information, you can activate the session and establish the
  connection:
 
  The Visual Basic form of the code is:
 
  Dim cryptSession As Long
 
  ' Create the session
  cryptCreateSession cryptSession, cryptUser, %CRYPT_SESSION_SSL 
 
  ' Add the server name and activate the session
  cryptSetAttributeString cryptSession, %CRYPT_SESSINFO_SERVER_NAME, serverName, Len( serverName )
 
  cryptSetAttribute cryptSession, %CRYPT_SESSINFO_ACTIVE, 1
 
       
  Activating a session results in cryptlib performing a lot of work in the background.
  For example when activating the SSL/TLS session shown above cryptlib will connect
  to the remote host, read the server’s certificate, generate a secret data value to
  exchange with the server using the key contained in the certificate, create the
  appropriate encryption contexts and load keys based on the secret data value into
  them, negotiate general session parameters, and complete negotiating the encrypted
  link with the server.
 
  Obtaining Session Status Information
  When a session is established a lot of state information is exchanged between the
  client and server and status information is generated by both sides. After the session
  has been activated you can query the session object for information such as the
  session status,
 
 
  Exchanging Data
  Once a general-purpose secure communications session has been established, you can
  exchange data with the remote client or server over the encrypted, authenticated link
  that it provides. This works exactly like pushing and popping data to and from an
  envelope, except that the session is effectively a bottomless envelope that can accept
  or return (depending on the remote system) an endless stream of data.

    
  When you close a session, cryptlib will immediately shut down the session as is,
  without flushing data in internal session buffers.


IIS

Using SSPI Schannel in a Windows Server setting raises a problem. Accessing user certificates requires running as a user with permissions to open the certificate store, and accessing the user private key requires either that the user has logged on locally with the password, or that the server is enabled for delegation. Machine certificates are readable by everyone, but accessing the private key requires admin rights (or local user) unless you adjust the ACL.

Since cryptlib is a stand alone library it does not need to access the certificate store. This makes it ideal for use in a windows server environment where a CGI application may need to create a secure connection with another server. CGI applications run under IUSR, a guest account with limited privileges, that is restricted from access to the certificate store. This is covered fully here.

When cryptlib is intialized, it performs various sanity-check tests including the ability to read files Specifically, it checks to see if it can read/write to the path CSIDL_APPDATA which on Windows Server 2003 is:

  • C:\Documents and Settings\Administrator.MY-SERVER\Application Data
when logged in as Administrator or
  • C:\WINDOWS\system32\config\systemprofile\Application Data

when running as IUSR.

To get around this, the author removed this check. The re-configured DLL can be downloaded.