Sending email via an encrypted (SSL) connection Since google introduced gmail and required the use of an encrypted session, the need to incorporate TLS (which is an SSL connection or tunnel) in applications has arisen. Since SSL is a complex protocol that requires handshaking, negotiation and encryption, this is usually more than most developers can or want to deal with just to send an email via smtp.gmail.com. Google apps and gmail provide free reliable email hosting, which is difficult to find even in well rated commercial solutions. Sending email from an application then becomes a little more tricky than just opening a TCP connection on port 25 and proceeding with the SMTP dialog. Now the connection must be an SSL tunnel through which the SMTP dialog is conducted. A commercial solution like the excellent socket tools manage the connection and the SMTP dialog for you, but cryptlib is an open source library that can provide the SSL tunnel if you are willing to write the code for the SMTP dialog. This Article will show you how to do both. This article is designed for languages that do not SMTPS natively like some flavors of BASIC and C, and even for some that do and run into trouble. SMTP ports 1. Port 25 (SMTP) is plain text. The conversation between server and client is held in plain text. This is where the majority of internet MTAs are and have been since SMTP was introduced. The big problem with plain text being that user credentials and passwords are easy to intercept by sniffing the transmission media. 2. Port 465 (SMTPS) is TLS encrypted. The conversation between server and client is held through an encrypted connection. 3. Port 587 (STARTTLS) is a transitional hybrid. Think of it as a special interim measure between 1 and 2. Although 'interim' on the internet likely means the next decade or so. STARTTLS is an enhanced SMTP command; an optional extension to the SMTP protocol. STARTTLS allows the MTA and client to switch (one way) from plain text to TLS encrypted transmission. STARTTLS is often provisioned on Port 25 for compatibility with either plain text or encrypted transmission. When provisioned on port 587 it is implicitly a requirement that transmission must become encrypted prior to authentication. SMTPS Client - Source code The code referred to in this article is available for download: Example BASIC SMTP client and BASIC headers Example C++ SMTP client and C headers (VS2008 Project also available for download) An alternative to writing a base 64 MIME encoded is a free online base64 encoder and decoder, available here Cryptlib cryptlib is a very complete encryption library developed by Peter Gutmann over a number of years. cryptlib allows even inexperienced programmers to easily add SLL (and encryption) to their software. We are very lucky to have such a high quality library available as open source. cryptlib uses industry-standard X.509, S/MIME, and ssh/SSL/TLS data formats, the resulting encrypted or signed data can be easily transported to other systems and processed there, and cryptlib itself runs on any commonly-user operating system. Multi-threaded cryptlib Operation cryptlib is re-entrant and completely thread-safe. The source can be downloaded here, and v3.33 the compiled 32bit windows dll from here. cryptlib is available under the GPL-compatible Sleepycat dual license, which means you can use it under the GPL terms or under standard commercial terms, depending on your preference. More information on use under the commercial terms is given in the cryptlib brochure which explains the features of cryptlib and details the licensing terms (it's free for non-commercial and some types of commercial use, shareware, academic research, and so on). The examples that follow use this library to create and encrypted tunnel with a SMTPS server (gmail in this case) through which an email can be sent. This functionality can be crucial to an application even if it is not an MTA (Mail Transfer Agent). The ability to perform this task without requiring the install of any third part libraries is invaluable when providing solutions for users. No one likes delivering software with the disclaimer "requires the install of xxx compnant to work" Google provided a very reliable free email service. Google apps allow you to register a domain mydomain.com, and point the DNS servers to google for the email portion. Having used several hosted email solutions over the years, some of which were very expensive, I have always been dogged with unreliablity and the perenial queuing on Sunday nights as all the spammers blast the world with their unwanted solicitations for your viewing pleasure on Monday morning. I did find a free alternative that does not require an encrypted TLS connection http://www.gmx.com/ but have not had an opportunity to test them out over a period of time. My guess is that a company like google has the resources to deliver this service more reliably at present. Creating an SSL Session To use the library, we must first intitialize it: RetVal = CryptInit() Since TLS is really just SSL, the first step is to create an SSL session with: RetVal = CryptCreateSession( VARPTR(hSess), CRYPT_UNUSED, CRYPT_SESSION_SSL ) Notice that the first argument is a pointer to the LONG INTEGER session handle. All CryptLib functions that modify a vlaue, take a pointer to that value (in this case a LONG POINTER), so you must create this hSess pointer in your language. After the call to CryptCreateSession(), hSess will contain a session handle in a LONG value that is fed to subsequent function calls like: RetVal = CryptSetAttributeString( hSess, CRYPT_SESSINFO_SERVER_NAME, STRPTR(sSrvr), LEN(sSrvr) ) This function sets a string attribute. sSrvr is a string containing the server name (or IP): "smtp.gmail.com" RetVal = CryptSetAttribute( hSess, CRYPT_SESSINFO_SERVER_PORT, nPort ) This function sets a LONG attribute, in this case the port 465. Finally we must activate the session with: RetVal = CryptSetAttribute( hSess, %CRYPT_SESSINFO_ACTIVE, 1 ) This causes CryptLib to perform all the connection handshaking, certificate exchange and protocol negotiation that fills volumes in university libraries. The by product is that google gets an empty request which trigges the error response: 220 mx.google.com ESMTP 6sm665677yxg.48 Here Google is prompting us to send a ESMTP request EHLO, which marks the beginning of the SMTP dialog detailed here. Push/Pop Data Functions This is where the rubber meets the road in this library. In the manual, Peter describes the benefits of using a full I/O completion ports implementation in Windows IIS servers, but that is a lot of work just to send an email. All we need to do, is handle response timing, error codes and internet connection issues, and we will have a reliable TLS email client. The request is sent with RetVal = CryptPushData( hCrypt, STRPTR(sSend), LEN(sSend), VARPTR(BytesSent) ) This returns the number of bytes actually sent, and as far as I can tell should always be equal to the number of bytes in the string for smaller data sets Next the request must be flushed (sent to the server) with: RetVal = CryptFlushData(hCrypt) And finally the response is returned with a call to: RetVal = CryptPopData( hCrypt, STRPTR(sBuff), LEN(sBuff), VARPTR(BytesReply) ) Its that simple in concept. The problem is that CryptPopData() just returns what has been received at the moment you call it. This function does not block. This means that we need to loop and keep checking for a response. It also means that we need to examine the response to make sure that it is complete and that it contains what we expect to find (lines of SMTP response terminated with <CRLF>) As described here, most responses are single lines, terminated with a <CRLF> (0d0a),
but one in particular is not:
One way to process this, is to begin with the last byte and work backwards through the string until we arrive at the begging of the line. Then we can check the fourth byte to see if it is a space. If it is not we need to wait for some more lines. This also raises the possibility that the line may not be complete (ie terminated with <CRLF>, so this must be checked before we look at the fourth character. Since the minimum line length is 6 characters, we could check that too.
Pipelining adds a small layer of complexity as many individual responses can be returned at once. For example, PIPELINING AUTH LOGIN followed by the username and password results in the following two responses. 334 VXNlcm5hdWU6 334 UGFzc3dvfdmQ6 Each one is a complete response. These is not a multi-lined response like the response to EHLO. Each one of these must pass the tests above in order for the function to return success. Any SMTP error codes or incomplete returns will cause the PIPELINED request to fail. Request/Response handling FUNCTION TLSPushPop( hCrypt AS LONG, sErr AS STRING, sReply AS STRING, sSend AS STRING ) AS LONG This function is designed to return the complete server replies or an error (which can also be a timeout). The function uses string concatenation for simplicity. In translating this to your language, a Vector or StringBuilder style of class would probably be more suitable. Since it is possible that another server may do something very unexpected, the first thing we do after ensuring a buffer for the replies, is to check for any data that might have been returned since the last call. This should not happen, but if it is not explicitly handled a situation could arise where some extra unexpected data gets pre-pended onto the response to the current response causing unexpected results. Next the function pushes all the data to CryptLib and flushes out to the server. Then we immediately enter a loop that begins counting milliseconds of delay. Should we get to SMTP_RESPONSE_TIMEOUT milliseconds without a complete return the function will return a timeout error code. When CryptLib processes return data, CryptPopData() returns the data in the buffer and we check it as described above. If your language does not support pointers, just use MID$() to pull out characters one by one. Should you need to send a large number of bytes, then you may need to create a loop around CryptPushData() if it chokes, and break the file up into chunks. BASIC pseudo code: RetVal = CryptPopData( hCrypt, STRPTR(sBuff), LEN(sBuff), VARPTR(BytesReply) ) IF BytesReply > 0 THEN RETURN ERROR RetVal = CryptPushData( hCrypt, STRPTR(sSend), LEN(sSend), VARPTR(BytesSent) ) RetVal = CryptFlushData(hCrypt) DO SLEEP 20 ' Wait for a response Totms = Totms + 20 ' IF Totms > %SMTP_RESPONSE_TIMEOUT THEN RETURN ERROR RetVal = CryptPopData( hCrypt, STRPTR(sBuff), LEN(sBuff), VARPTR(BytesReply) ) IF BytesReply > 0 THEN Last = LEN(sReply) pByte = STRPTR(sReply) IF Last > 5 AND @pByte[Last-1] = 10 AND @pByte[Last-2] = 13 THEN FOR k = 3 TO Last IF @pByte[Last-k] = 10 AND @pByte[Last-k+4] = 32 THEN EXIT DO IF k = last AND @pByte[Last-k+3] = 32 THEN EXIT DO NEXT END IF END IF LOOP The loop must be executed for each response for all the responses to be returned. This page and the code derived from it is the result of about 60hrs reading and development testing. cryptlib is the result of years of work and can do a lot more than create an SSL tunnel. If you are interested in further reading, a good place to start is here: Cryptlib and SSL (from the manual) cryptlib works with two classes of objects, container objects and action objects. A container object is an object that contains one or more items such as data, keys or certificates. An action object is an object which is used to perform an action such as encrypting or signing data. The container types used in cryptlib are envelopes (for data), sessions (for communications sessions), keysets (for keys), and certificates (for attributes such as key usage restrictions and signature information). Container objects can have items such as data or public/private keys placed in them and retrieved from them. In addition to containing data or keys, container objects can also contain other objects that affect the behaviour of the container object. For example pushing an encryption object into an envelope container object will result in all data which is pushed into the envelope being encrypted or decrypted using the encryption object. Encryption contexts are the action objects used by cryptlib. Action objects are used to act on data, for example to encrypt or decrypt a piece of data or to digitally sign or check the signature on a piece of data. The usual mechanism for processing data is to use an envelope or session container object. The process of pushing data into an envelope and popping the processed data back out is known as enveloping the data. The reverse process is known as deenveloping the data. Session objects work in a similar manner, but are used to encapsulate a secure session with a remote client or server rather than a local data transformation. The first section of this manual covers the basics of enveloping data, which introduces the enveloping mechanism and covers various aspects of the enveloping process such as processing data streams of unknown length and handling errors. Once you have the code to perform basic enveloping in place, you can add extra functionality such as password-based data encryption to the processing. After the basic concepts behind enveloping have been explained, more advanced techniques such as public-key based enveloping and digital signature enveloping for S/MIME and PGP are covered. Session objects are very similar to envelope objects except that they represent a communications session with a remote client or server. The next section covers the use of session objects for protocols such as SSL, TLS, and SSH to secure communications or work with protocols such as CMS, SCEP, RTCS, OCSP, and TSP that handle functions such as certificate status information and timestamping. The use of public keys for enveloping requires the use of key management functions, and the next section covers key generation and storing and retrieving keys from keyset objects and crypto devices. The public portions of public/private key pairs are typically managed using X.509 certificates and certificate revocation lists. The next sections cover the management of certificates including certificate issue, certificate status checking, and certificate revocation list (CRL) creation and checking, as well as the full CA management process. This covers the full key life cycle from creation through certification to revocation and/or destruction. High-level Interface This interface requires no knowledge of encryption and digital signature techniques, and is easiest for use with languages like Visual Basic and Java that don’t interface to C data structures very well. The container object interface provides services to create and destroy envelopes and secure sessions, to add security attributes such as encryption information and signature keys to a container object, and to move data into and out of a container. Because of its simplicity and ease of use, it’s strongly recommended that you use this interface if at all possible. Initialisation Before you can use any of the cryptlib functions, you need to call the cryptInit function to initialise cryptlib. You also need to call its companion function cryptEnd The Visual Basic interface to cryptlib is otherwise mostly identical to the standard C/C++ one. Return Codes Every cryptlib function returns a status code to tell you whether it succeeded or failed. If a function executes successfully, it returns %CRYPT_OK. If it fails, it returns one of the status values detailed in “Error Handling” on page 275. Secure Sessions - SSH, SSL, and TLS sessions, As with envelopes, cryptlib takes care of all of the session details for you so that all you need to do is provide basic communications information such as the name of the server or host to connect to and any other information required for the session such as a password or certificate. cryptlib takes care of establishing the session Secure sessions are very similar to envelopes, with the main difference being that while an envelope is a pure data object into which you can push data and pop the processed form of the same data, a session is a communications object into which you push data and then pop data that constitutes a response from a remove server or client. This means that a session object can be viewed as a bottomless envelope through which you can push or pop as much data as the other side can accept or provide. For example to connect to a server using SSH and obtain a directory of files using the ls command you would do the following: - create the session; - add the server name, user name, and password; - activate the session; - push data - pop the return - destroy the session Client vs. Server Sessions cryptlib distinguishes between two types of session objects, client sessions and server sessions. Client sessions establish a connection to a remote server while server sessions wait for incoming communications from a remote client. Establishing a Session it’s generally better to explicitly activate the session You can activate a session by setting its %CRYPT_SESSINFO_ACTIVE attribute to true (any nonzero value). SSL/TLS Sessions SSL and TLS are actually variations of the same protocol, the protocol known by the generic term SSL. cryptlib will automatically negotiate the highest protocol version supported by the other side. SSL/TLS is a secure data transfer protocol that provides confidentiality, integrity protection, protection against replay attacks, and a variety of other services. The SSL server is authenticated via a certificate, and the client isn’t authenticated (in rare circumstances client certificates may be used, but these are usually avoided due to the high degree of difficulty involved in working with them). Alternatively, the client and server may be mutually authenticated via a secret-key mechanism such as a user name and password, which avoids the need for certificates altogether. cryptlib supports SSL version 3, TLS version 1.0 (a.k.a SSL version 3.1), and TLS version 1.1 (a.k.a SSL version 3.2). SSL/TLS Client Sessions Establishing a session with an SSL/TLS server requires adding the server name or IP address and an optional port number if it isn’t using the standard SSL/TLS port. Once you’ve added this information, you can activate the session and establish the connection: The Visual Basic form of the code is: Dim cryptSession As Long ' Create the session cryptCreateSession cryptSession, cryptUser, %CRYPT_SESSION_SSL ' Add the server name and activate the session cryptSetAttributeString cryptSession, %CRYPT_SESSINFO_SERVER_NAME, serverName, Len( serverName ) cryptSetAttribute cryptSession, %CRYPT_SESSINFO_ACTIVE, 1 Activating a session results in cryptlib performing a lot of work in the background. For example when activating the SSL/TLS session shown above cryptlib will connect to the remote host, read the server’s certificate, generate a secret data value to exchange with the server using the key contained in the certificate, create the appropriate encryption contexts and load keys based on the secret data value into them, negotiate general session parameters, and complete negotiating the encrypted link with the server. Obtaining Session Status Information When a session is established a lot of state information is exchanged between the client and server and status information is generated by both sides. After the session has been activated you can query the session object for information such as the session status, Exchanging Data Once a general-purpose secure communications session has been established, you can exchange data with the remote client or server over the encrypted, authenticated link that it provides. This works exactly like pushing and popping data to and from an envelope, except that the session is effectively a bottomless envelope that can accept or return (depending on the remote system) an endless stream of data. When you close a session, cryptlib will immediately shut down the session as is, without flushing data in internal session buffers. IIS Using SSPI Schannel in a Windows Server setting raises a problem. Accessing user certificates requires running as a user with permissions to open the certificate store, and accessing the user private key requires either that the user has logged on locally with the password, or that the server is enabled for delegation. Machine certificates are readable by everyone, but accessing the private key requires admin rights (or local user) unless you adjust the ACL. Since cryptlib is a stand alone library it does not need to access the certificate store. This makes it ideal for use in a windows server environment where a CGI application may need to create a secure connection with another server. CGI applications run under IUSR, a guest account with limited privileges, that is restricted from access to the certificate store. This is covered fully here. When cryptlib is intialized, it performs various sanity-check tests including the
ability to read files Specifically, it checks to see if it can read/write to the path CSIDL_APPDATA which on Windows Server 2003 is:
when logged in as Administrator or
when running as IUSR. To get around this, the author removed this check. The re-configured DLL can be downloaded. |


