TLS with Schannel


  • 32 bit windows DLL SSPI Schannel examples are available for download along with an example SMTPS application.

Secure network communication has become increasingly important. One of the most widely used protocols is Secure Sockets Layer (SSL) which is gradually being replaced by its successor, Transport Layer Security (TLS). These protocols ensure:

  • confidentiality (the data is kept secret from the third party that is not involved in the data exchange)
  • message integrity (the message received is the message sent)
  • endpoint authentication (one of the communication endpoints or both are verifiably known)
For Windows developers, Microsoft has provided a version of Generic Security Service Application Program Interface (GSS-API) , called Security Support Provider Interface (SSPI). SSPI allows developers to implement SSL/TLS without needing to know the details of SSL. The C libraries and header files are freely available from the the platform SDK (any one) and are included with my Visual Studio project available for download.

This article provides everything you need to implement a secure connection. You will not need to buy and install any certificates, or ship a 3rd party dll. I have written procedural code in C and BASIC so that the flow can easily be followed, although an OOP implementation would probably be more suitable for production code.


The TLS (Transport Layer Security) protocol is a further development of the SSL (Secure Socket Layer) protocol developed by Netscape Communications Corp. TLS utilizes TCP for a reliable connection. Microsoft operating systems include Schannel, the Microsoft implementation of SSL/TLS. SChannel is an integral part of the operating system and depending on the version of the Cryptographic libraries used, may be FIPS-1 or even FIPS-2 certified. SSPI is a set of generic functions that can be used to access a specific security provider like Schannel, to obtain an authenticated connection. These calls do not require extensive knowledge of the security protocol's details. SChannel calls follow GSS API standards, unlike OpenSLL for example.

Implementing this API is not trivial however as you can see from the example. Although these 2000 lines of code hide the gory details, creating a secure connection is not simple to implement.

SSPI allows an application to call one of several security providers to obtain an authenticated connection. In this case,  we will call SSPI routines directly as a socket-based application. The routines use request/response messages to carry data over the secure connection.

When a dynamic-link library (DLL) loads into the address space of an .exe file, the trust level of the .exe file determines the final access level of the DLL. Also, the DLL runs in the address space of the .exe which is not desirable for a secure connection. To get around this,  SECUR32.DLL creates a secure virtual address space (outside of the .exe address space) when InitSecurityInterface() is called. 

Accessing the data and functions in this space is done via a SecurityFunctionTable structure. This is a dispatch table that contains pointers to the functions defined in SSPI.h. It contains the following functions:

  • dwVersion                  
  • EnumerateSecurityPackagesA 
  • QueryCredentialsAttributesA
  • AcquireCredentialsHandleA  
  • FreeCredentialHandle  
  • InitializeSecurityContextA 
  • AcceptSecurityContext      
  • CompleteAuthToken          
  • DeleteSecurityContext      
  • ApplyControlToken          
  • QueryContextAttributesA    
  • ImpersonateSecurityContext 
  • RevertSecurityContext      
  • MakeSignature              
  • VerifySignature            
  • FreeContextBuffer          
  • QuerySecurityPackageInfoA  
  • ExportSecurityContext      
  • ImportSecurityContextA     
  • AddCredentialsA            
  • QuerySecurityContextToken  
  • EncryptMessage             
  • DecryptMessage             
  • SetContextAttributesA      
  • SetCredentialsAttributesA  

Each member of this structure is a DWORD (unsigned 32 bit Integer), and is the entry point for the function in the secure address space. To call the function, you must use the function definition provided in SSPI.h, then use a technique like:

    g_pSSPI->DeleteSecurityContext( &hContext );

This de-references the SSPI pointer and invokes the structure member DeleteSecurityContext() using the function prototype that follows. To do this in another language like BASIC:

    CALL DWORD @gpSSPI.DeleteSecurityContext USING DeleteSecurityContext( phContext )


  • Initialize Security Interface
  • Initialize Windows Sockets
  • Create an SSPI credential
  • Connect to the server
  • Perform handshaking
  • Authenticate the server's credentials
  • Get the server's certificate.
  • Verify the server's certificate
After these steps, a secure connection is in place and we may begin our request/response session.

Keep in mind that we are using Winsock to make a socket connection. When writing any code using Windows Sockets, be sure to read The Windows Sockets Lame List, a great list of what to avoid when programming with Windows Sockets ranging from the "Nauseatingly lame" to "Stooping to unspeakable depths of lameness".

First the client sends a ClientHello message to the server which includes a starting cipher suite choice and a random value. The server responds with a ServerHello message which includes the chosen cipher suite, a session ID and another random value. The cipher suite determines which algorithm should be used for the key exchange and the encryption. 

Depending on the chosen cipher suite, the server sends a certificate and parameters for the key exchange followed by a ServerHelloDone message. Next the client sends parameters for the key exchange. Next a ChangeCipherSpec message and a Client Finished message (which is now encrypted with the key just exchanged) is sent. The server responds with the same. After this is completed and verified on both ends the handshake is finished. Now both the client and the server have a verified key with which all communications are encrypted.


This is a quick overview. The process of encryption takes plain text and turns into an unreadable cipher using a key:

AES Rijndael Overview

A digital signature is used to verify a message (or document) was authored by a certain source, and no one else. It also serves to verify that the message was not modified in any way. This is known as Authentication

Encryption and digital signatures can be used seperately or together for example:

  • A message may be encrypted but not digitally signed meaning; only people with the key can read it, but the reader cannot be certain who wrote it.
  • A message may be digitally signed but not encryted meaning; anyone can know who wrote it ad everyone can read it
  • A message may be encrypted first then digitally signed meaning; only someone with the key can read it, but anyone can tell who wrote it
  • A message may be digitally signed THEN encrypted meaning; only someone with the key can read it and know who wrote it.


A Client certificate is NOT required for a secure connection. Remember, it IS possible to have a one sided certificate exchange during a secure connection handshake!

When security transport-layer protocols first emerged, their primary purpose was to guarantee that a client was connecting to an authentic server and help protect the privacy of data while in transit. However, SSL 3.0 and TLS 1.0 also include support for the transmission of a client's certificate during the protocol's handshake. This optional feature enables the mutual authentication of the client and server. The decision of whether to use a client certificate should be made in the context of the application. Client certificates are unnecessary if the primary requirement is authenticating the server. 

In the example code, finding a certificate in the system store will work if you already have a certificate and it was successfully imported into the "MY" certificate store. Since you probably don't, you will see none when you use Internet Explorer to view your certificate store:

In this case the code creates a NULL credential which is stored in the paCred member of the SCHANNEL_CRED structure. This is sent to the Server which will accept it and return its certificate allowing us to create a connection. We will validate the Server Certificate, but obviously the Server cannot validate the Client certificate as there isn't one.

As specified in rfc3760, a request using a NULL will prompt the server to return all credentials stored under the current user account. The client recovers this certificate and then validates it by calling QueryContextAttributes().

If you want to use a Client certificate, ultimately it must be purchased. An easy place to get a certificate with a recognized authority is GoDaddy. You can get an SSL cert valid for 1 year for $30. There are other places that might be cheaper if you hunt around a bit. It's not too difficult to do. Most of the sites are very user friendly and make it easy to buy a cert.

Remember, since you are doing mutual authentication, you'll need a certificate with a valid authority, and not just a self-signed one which you can create yourself because the Gmail server would have no way to verify it.

INSTALLING A CERTIFICATE (if you decide to go this route)

Once you have bought a certificate it needs to be installed:
  1. Install the certificate into the machine store. Go to Start | Run and enter in MMC to get a blank MMC console. Go to File | Add/Remove Snap-in. Click the Add button to select the snap-in. Choose Certificates from the list and click Add. Choose the Computer account option, then Next, Finish, Close, and finally OK.
  2. Importing the certificate into the Machine Store. Expand out the tree and choose the Personal folder. Right click on Personal and choose All Tasks, Import. Follow the wizard to complete the process. The certificate should now be in the machine store.
  3. Adjusting the private key permissions. On Windows 2008, this is added to the Certificate Manager GUI. In Windows 2003, you'll need to download the Windows Server 2003 Resource Kit Tools. Use the following command's format to set permissions:
C:\Program Files\Windows Resource Kits\Tools\winhttpcertcfg.exe -g -c LOCAL_MACHINE\My -s "mycert" -a "USER_ACCOUNT"

Where 'LOCAL_MACHINE\My' is the Personal Certificate store in the local machine, 'mycert' is the name of your certificate, and USER_ACCOUNT is the account which will be given read permissions for the private key. You can try the IUSR account, but you might need to use the 'NETWORK SERVICE' depending on the exact context your CGI app is running under. You will need to restart IIS for changes to take effect. The documentation for WinHttpCertCfg covers what it's doing.

You should now have permissions to access the private key. You will still use CertOpenSystemStore with the MY flag.

The server certificate chain will be examined by a call to CertGetCertificateChain()This function builds a certificate chain context starting from an end certificate and going back, if possible, to a trusted root certificate. 

When an application requests a certificate chain, the structure returned is in the form of a CERT_CHAIN_CONTEXT. This context contains an array of CERT_SIMPLE_CHAIN structures where each simple chain goes from an end certificate to a self-signed certificate. The chain context connects simple chains through trust lists. Each simple chain contains the chain of certificates, summary trust information about the chain, and trust information about each certificate element in the chain.

The is passed to CertVerifyCertificateChainPolicy() which checks a certificate chain to verify its validity, including its compliance with any specified validity policy criteria. This returns true/false. There are many possible error codes, but even if the server certificate cannot be verified, the connection may optionally still be used. Browsers will pop up a dialog informing the user that certificate could not be verified for example.

Server Certificate Errors:


Schannel error 36872 or 36870 on a server usually indicates a problem with the server certificate. This occurs when the verification of the private and public key pair of the server certificate fails, meaning the certificate is not usable for encryption.

The five main reasons for this are:

  • Incorrect ACL’s on the MachineKeys folder on the system disk
Usually caused by modifying the ACL’s on the system disk post-install, for example by applying a restrictive security policy to the server that prevents the System account from reading the private or public key.
This can be caused by the certificate having been revoked *or* by the system account not being able to download any of the CRL’s stamped on the certificates in the chain. Proxy servers requiring authentication are an example of this because thee System account can’t authenticate using NTLM  but it can authenticate using Kerberos if the proxy supports it. To check this, export all the certificates in the chain and examine them.
  • Unable to build a certificate chain to a trusted root Certificzate Authority
This usually means the certificate of the Root CA at the top of the certificate chain is not in the list of Trusted Root Certification Authorities so the chain is invalid.
  • Bad Certificate format
This can be caused by incorrectly formatted or missing information in the Subject or the Subject Alternate Name (SAN) of the certificate. A certificate must have a SAN DNS name that matches the DNS name of the domain controller!
  • Expired Certificate
Time to buy a new Certificate.

For a full list of windows error codes see here.

At this point a secure connection is established and we may begin the task we connected for. This takes the form of a request and a response, just like an HTTP session for example. You should expect the server to immediatly send a response once the connection is esatablished even though a request was not issued. This is usually the welcome message. If you do not read it prior to issuing your first request, it will appear at the beginning of the response to your first request as all date is buffered.

Data is read on the secure connection by calling Windows Sockets recv(). This is a blocking function, so you need to check the data to know when the response is complete. In the case of a SMPTS session, we are looking for a CRLF preceeded by a space. Once this is parsed, the response has been read. If you call recv() again, it will block until the server sends more data.

The send() and recv() operations are sending/reading encrypted data. In the case of sending the data is encrypted prior to sending and in the case of reading it is decrypted after being read by calling the SChannel functions EncryptMessage() and DecryptMessage(0 in SECUR32.DLL.

SMTP session is now possible. Now keep in mind the "Mind bogglingly lame" assuming that stream sockets maintain message frame boundaries is wrong. Stream sockets (TCP) are called stream sockets, because they provide data streams (duh). As such, the largest message size an application can ever depend on is one-byte in length. No more, no less.

The Server may respond so fast that the welcome message is recovered by the handshake and reported as extra bytes. This can be decrypted by calling DecryptMessage(). If not, it will be found as the first part of the response to the first EHLO request. The example code simply detects the first occurrence of a CRLF, so the Read Decrypt function will need to be called to recover the welcome message before the actual response can be recovered.

Requests and responses are Sent or Received as Encrypted Envelopes. Each envelope contains a complete request or response. Each time you call recv() it blocks until data is available. The buffer returned contains a complete envelope. This envelope must be decrypted and checked to see it more data is expected.


SMTP ports:
  1. Port 25 (SMTP) is plain text. The conversation between server and client is held in plain text. This is where the majority of internet MTAs are and have been since SMTP was introduced. The big problem with plain text being that user credentials and passwords are easy to intercept by sniffing the transmission media.
  2. Port 465 (SMTPS) is TLS encrypted. The conversation between server and client is held through an encrypted connection.
  3. Port 587 (STARTTLS) is a transitional hybrid. Think of it as a special interim measure between 1 and 2. Although 'interim' on the internet likely means the next decade or so. STARTTLS is an enhanced SMTP command; an optional extension to the SMTP protocol. STARTTLS allows the MTA and client to switch (one way) from plain text to TLS encrypted transmission. STARTTLS is often provisioned on Port 25 for compatibility with either plain text or encrypted transmission. When provisioned on port 587 it is implicitly a requirement that transmission must become encrypted prior to authentication.

A Plain Text session can be upgraded to a fully encrypted session with some servers, particularly google on port 25 or port 587 after the connection is innitiated. So STARTTLS is really an extension to plain text communication protocols. It offers a way to upgrade a plain text connection to an encrypted (TLS or SSL) connection instead of using a separate port for encrypted communication as described in RFC 5246 which states:
One advantage of TLS is that it is application protocol independent. Higher-level protocols can layer on top of the TLS protocol transparently. The TLS standard, however, does not specify how protocols add security with TLS; the decisions on how to initiate TLS handshaking and how to interpret the authentication certificates exchanged are left to the judgment of the designers and implementors of protocols that run on top of TLS.[1]
Several library implementations of TLS like RFC 3207 SMTP extension use this style for connecting client and server with a plain text session that can start a secure session.

The example code can support this mode. If you fail to implement a secure connection, you will get an error message that looks something like:
"530 5.7.0 Must issue a STARTTLS command first"

An example SMTP session with google:

    C:                <establish a connection>
    S: 44Bytes:  220 ESMTP 6sm661677yxg.48

    C:  7 Bytes:   EHLO
    S: 125Bytes: at your service, [46.311.221.11]
                        250-SIZE 35651584
                        250 ENHANCEDSTATUSCODES

    C:  initiate TLS session and begin the SMTP dialog


Using SSPI Schannel in a Windows Server setting raises a problem. Accessing user certificates requires permissions of a user, and accessing the users private key requires either that the user has logged on locally with the password, or that the server is enabled for delegation. Machine certificates are readable by everyone, but accessing the private key requires admin rights (or local user) unless you adjust the ACL.

If you want a CGi/FastCGI process to send an email (to confirm a purchase and supply download information for example) the CGI application will need this access if you intend to use a Client Certificate for the secure session. Since IIS runs a CGI application under IUSR (which has very limited priviledges to maintain security) this poses a problem. 

The handshake process begins by checking for a Client certificate in the "MY" certificate store.Without permission, this function returns zero and GetLastError() also returns zero. A solution I did not try is to buy a Client Certificate and then grant permission to the IUSR account to read the private key of the certificate as covered above.

Also in IIS 6, you may receive an error message when you try to start a CGI program. This is by design! You will need to set CreateProcessAsUser to false, and specify the application pool's identity to local system.

Since the only reason we want to open the "MY" certificate store is to look for a potential certificate that can be used for client authentication as part of the initial handshake, it makes more sense to just use a NULL certificate if we don't care about the Server validating our certificate, and for sending email via gmail, you probably don't, because all we really care about is making sure our client is talking to the server it thinks it is.

This makes life a lot easier, and we just need to adjust the code to skip the call to CertOpenSystemStore(0, "MY"). With that done the handshake process will complete but with a wrinkle. Some servers (like gmail) return a little welcome message (63 bytes of encrypted data to be precise) that decoded looks like:

  • 220 ESMTP 4sm2476575ywg.43

This raises a problem because it is returned in the buffer as  SECBUFFER_EXTRA. This data must be processed as it contains a check sum. If it is incomplete, the next recv() will complete it and the decryption will succeed. If you do not process it, the next read is likely to return SEC_E_MESSAGE_ALTERED or perhaps SEC_E_INCOMPLETE_MESSAGE. The example code was not updated to do this, but you have the road map here.

Subpages (1): SSPI Log