WinHTTP

HTTP is a request/response protocol. You request some resource like the HTML of a webpage and the response comes back with the HTML attached. Like SMTP, the basic protocol is fairly simple, but high performance HTTP handling is not at all trivial and cannot be effectively implemented with a simple use of TCP sockets.

This article deals with writing HTTP clients for windows. There is a lot to say! If you just want to get up and running as quickly as possible, use the code examples in the wrapper for each of the WinHTTP function calls. To determine which function calls are needed, take a look at the wrapper function WinHTTPSession().

Microsoft went to a good deal of trouble to write a library that implements HTTP effectively. The Library provides a direct C style API interface and a COM API interface (using the IWinHttpRequest and IWinHttpRequestEvents ). The WinHTTP API is designed to provide client side HTTP services to server applications.

Under the hood, Win HTTP utilizes TCP which is a way to transmit data that is reliable. By this I mean:  if you send a message over a network using TCP, it will arrive, and it won't be garbled or corrupted. TCP manages all the lost packets, re-send requests, order of data arrival and hashing transparently. Thanks goodness for that!

WinInet
The predecessor to WinHTTP was WinInet.  WinINet was designed as an HTTP API client platform that allowed the use of interactive message dialogs such as entering user credentials. By contrast, WinHTTP's API set is geared towards a non-interactive environment allowing for use in service-based applications where no user interaction is required or needed.

WinHTTP is more secure and robust than WinINet and has fewer dependencies on platform-related API's. However, single-user applications that require FTP or Gopher protocol functionality, cookie persistence, caching, automatic credential dialog handling, Internet Explorer compatibility, or downlevel platform support should still consider using WinInet, even though it is officially deprecated.

As mentioned in the MSDN Article Windows with C++: Asynchronous WinHTTP;
"
A common misconception is that you need to use the Microsoft® .NET Framework if you want your application to access the Web.The truth is that developers using managed code still must deal with many of the same issues"


WinHTTP OBJECTS
The code is an object oriented solution to the problem of modeling three separate operations:
  1. A Session Object   - One session per application.
  2. A Connection Object - One connection per HTTP SERVER
  3. A Request Object - One per Request

        1                          2                        3



Choosing an Interface
Probably the biggest benefit of the direct C style API, is the ability to do multiple Asynchronous HTTP calls. The COM interface does not support AutoProxy, and before a response can begin to be processed, the entire response must be received and buffered. This is obviously a serious limitation. The direct C style API also does not have the added layer of COM, with all its quirks and complexity, getting in the way.


WinHTTP 5.1
The C API header file WinHTTP.h  (and WinHTTP.lib) is available for download from Microsoft but also in the examples I have provided (current as of Dec 2009). The reason I mention this, is because if you want to get these headers (and 1.25GB of other material) you will need to download, burn and install the Windows SDK  ISO (new name for the Platform SDK) . Don't worry about the name "SDK for Windows Server 2008", it's also for Server 2003, XP and Vista.

Next run the setup.exe on the DVD, select full installation and twiddle your thumbs for 40 minutes or so. After extracting the items you need, you can enjoy watching the progress bar as you delete the package from your hard drive. If that is not the refreshing break you were looking for right at this minute, then just download the HTTP examples. There in there. If you are indeed going to utilize this package in Visual Studio 2008 (any version) , then be sure to read the workarounds documented here and here.


The following features have been added in version 5.1 of WinHTTP:
 - IPv6 support.
 - AutoProxy capabilities.
 - HTTP/1.0 protocol, including support for keep-alive (persistent) connections and session cookies.
 - HTTP/1.1 chunked transfer support for HTTP responses.
 - Keep-alive pooling of anonymous connections across sessions.
 - Secure Sockets Layer (SSL) functionality, including CLIENT certificates. Supported SSL protocols: SSL 2.0/3.0, Transport Layer Security 1.0.
 - Support for SERVER and proxy authentication, including integrated support for Microsoft Passport 1.4 and the Negotiate/Kerberos package.
 - Automatic handling of redirects unless suppressed.
 - Scriptable interface in addition to the API.
 - Trace utility to help troubleshoot problems.  http://msdn2.microsoft.com/en-us/library/aa384119.aspx
  
The WinHTTP team has posted some helpful blogs here.


Request Verbs
There are many defined in the HTTP spec, but since most servers block all but a few like GET, POST and HEAD.

GET
Literally a request to GET the resource, like a web page from a server

POST
Sends DATA TO a web SERVER USING POST method AND returns the response DATA IN sReturn 
 ERROR TEXT contained IN gsWinHTTPLastError  

  A POST request is used TO SEND DATA TO the SERVER TO be processed IN some way, like by a CGI script.
  A POST request is different FROM a GET request IN the following ways:

      * There's a block of data sent with the request, in the message body. There are usually extra headers to
        describe this message body, like Content-TYPE: AND Content-Length:.
      * The request URI is NOT a resource TO retrieve; it's usually a program to handle the data you're sending.
      * The HTTP response is normally program OUTPUT, NOT a STATIC file.

  The most common use OF POST, by far, is TO submit HTML form DATA TO CGI scripts. In this case, the Content-Type:
  header is usually application/x-www-form-urlencoded, AND the Content-Length: header gives the length OF the
  URL-encoded form data. The CGI script receives the message body through STDIN, AND decodes it. 


    sSend = sSend + "HTTP/1.1" + $CRLF
    sSend = sSend + "Content-Type: text/html" + $CRLF  ' application/x-www-form-urlencoded
    sSend = sSend + "Content-Length: " + STR$(LEN(sData)) + $CRLF + $CRLF
    sSend = sSend + sData

 A valid Content-Length is required ON ALL HTTP/1.0 POST requests - unless its just BINARY DATA!

HEAD

PUT
This is a verb that is not supported on web servers without some setup for obvious reasons. This capability is considered a vulnerability in todays world and hence IIS6 does not have PUT support out of the box; you need to enable the WebDAV extension.

The steps to set up WebDAV are:

  1. Enable the WebDAV extension on the Web Service Extensions detail pane in IIS.
  2. Create a filesystem directory for a WebDAV share. Set the NTFS permissions appropriately.
  3. Create a virtual directory in IIS that points to the filesystem directory.
  4. Set the IIS directory security appropriately. Microsoft recommends not using integrated Windows authentication for internet-accessible shares.
  5. Launch a client application that makes a PUT request

This works quite easily, but is not a solution to any real world problems. See below for a robust secure solution.

  If the Request-URI refers to an already existing resource, the enclosed entity should be considered as a modified version of the one residing on the origin server.


ASYNCHRONOUS MODE

WinHTTP can handle multiple requests and multiple servers at the same time. It simply creates a new object (another instance of the class) for each item. There is a nice object oriented solution provided on MSDN, but sometimes an application may be written in a language that does not support OOP, or you may just want a simple procedural solution.

Managing multiple requests is done with a callback in much the same way the windows API manages dialogs (forms). The main function initiates the request and then instead of calling a blocking function to wait for the request, marches onward issuing more requests or interacting with the user. The job of recovering the response is left to the callback function. The Callback is specified with WinHttpSetStatusCallback() passing a pointer to the callback function as an argument.

A different callback can be specified for any Object, meaning any Object in all three classes
  1. A Session Object   - One session per application.
  2. A Connection Object - One connection per HTTP SERVER
  3. A Request Object - One per Request
can have it's own callback. I can't think of a good reason for this right now, but this could be very useful if needed.


The callback function must have 5 unsigned 32bit integers (DWORD) as arguments, one of which is user assignable! (see below) This becomes very useful as a way to pass control and data between the callback function and the main application code. This is most easily accomplished by creating a structure and passing a pointer to this structure to the callback via this context variable.

Assigning the context variable can be done in two ways:
1. by calling WinHttpSetOption() with WINHTTP_OPTION_CONTEXT_VALUE
2. WinHttpSendRequest() can be used to associate a context value with a Request handle

With this accomplished all the variables in the structure are available in the main application code. Application specific handles like events, threads or a critical section to control flow and signal events are good candidates for inclusion in the structure.

To use WinHTP functions in full asynchronous mode, you must do things in the right order:

   1. Open the session with WinHttpOpen() - Use INTERNET_FLAG_ASYNC 
        Check for errors

   2. Set a callback using WinHttpSetStatusCallback()
       Check for errors       
                     
   3. Open the connection using WinHttpConnect() - Specify the URL 
      Check for errors     
                  
   4. Open a request with WinHttpOpenRequest() - Specify path and VERB
      Check for errors
      WINHTTP_CALLBACK_STATUS_HANDLE_CREATED notification indicates that a request handle was created.
                       
   5. Open a request with WinHttpSendRequest() - Specify the data length
      Check for errors 
      This function optionally can send data immediately after the request headers.
      An application can use the same HTTP request handle in multiple calls to WinHttpSendRequest <-------- ***
      to re-send the same request, but the application must read all data returned from the previous
      call before calling this function again.
      Upon receiving the WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE statuscallback, the application
      can start to receive a response from the server with WinHttpReceiveResponse. Before then,
      no other asynchronous functions can be called, otherwise, ERROR_WINHTTP_INCORRECT_HANDLE_STATE is returned.
                                                                                                       
   6. Send the Data with WinHttpWriteData()
      Check for errors
      Warning  When using WinHTTP asynchronously, always set the lpdwNumberOfBytesWritten parameter to NULL
      and retrieve the bytes written in the callback function; otherwise, a memory fault can occur.
      When the application is sending data, it can call WinHttpReceiveResponse to end the data transfer. 
      If WinHttpCloseHandle is called, then the data transfer is aborted.
                              
                              
   7. Receive the Response with WinHttpReceiveResponse()
      If this function returns TRUE, the application should expect
      WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE completion callback indicating success
                            
                                     
   8. Find Bytes available with WinHttpQueryDataAvailable()
      When WinHTTP is used in asynchronous mode, always set the lpdwNumberOfBytesAvailable parameter to NULL
      and retrieve the bytes available in the callback function
      If no data is available and the end of the file has not been reached,
      the function returns TRUE, and when data becomes available, calls the callback function with
      WINHTTP_STATUS_CALLBACK_DATA_AVAILABLE and indicates the number of bytes immediately available
      to read by calling WinHttpReadData.
      The amount of data that remains is not recalculated until all available data indicated by the call to
      WinHttpQueryDataAvailable is read.
                                                 
   8. Read the Available Data with WinHttpReadData()
      When WinHTTP is used in asynchronous mode, always set the lpdwNumberOfBytesRead parameter to NULL
      and retrieve the bytes read in the callback function.
      Warning: The Buffer and BuffLen values are modified asynchronously by WinHTTP!
      If this function returns TRUE, use theWINHTTP_CALLBACK_STATUS_READ_COMPLETE completion to determine
      whether this function was successful and the value of the parameters.
      When the read buffer is very small, WinHttpReadData might complete synchronously.  If you call
      WinHttpReadData after receiving WINHTTP_CALLBACK_STATUS_READ_COMPLETE the result is a stack overflow.
      In general, it is best to use a read buffer that is comparable in size, or larger than the internal
      read buffer used by WinHTTP of 8 KB
      Use the return value of WinHttpReadData rather than that of WinHttpQueryDataAvailable to
      determine whether the end of the response has been reached, because an improperly terminated
      response can cause WinHttpQueryDataAvailable to continue to indicate that more data is
      available even when the response has been completely read.
                                             
   9. Close the Request and connection handles with WinHttpCloseHandle and wait for INTERNET_STATUS_HANDLE_CLOSING
      and the facultative INTERNET_STATUS_REQUEST_COMPLETE notification (sent only if an error occurs –
      like a sudden closed connection - you must test thees cases).
      At this state, you can either begin a new connection process or close the session handle. But before
      closing it, you should un-register the callback function.


PIPELINING
It is possible to make more requests on a connection before receiving the full response to the previous request. This is pipelining. However it doesn’t change the need to receive all the data from the previous requests first. So if an application is asking for another request to a server while the current one is in progress, it can either wait till the connection is available or start a new connection. This decision is mostly controlled by a configurable maximum number of simultaneous connections to a server.





WINHTTP_STATUS_CALLBACK CALLBACK FUNCTION
This callback function must be thread-safe and should be able to handle re-entrance for the same request while it is processing. Because callbacks are made during the processing of the request, the application should spend as little time as possible in the callback function to avoid degrading data throughput on the network.

If you set the callback on the session handle before creating the request handle, the request handle inherits the callback function pointer from its parent session. At the end of asynchronous processing, the application must set the callback function to NULL. This prevents the client application from receiving additional notifications; if it is not set to NULL, the client application could become unstable.

The Status CALLBACK takes five DWORD arguments:
  1. hSession - Session HANDLE.
  2. dwContext - application-defined context value associated with hSession
  3. dwInternetStatus - Status code indicates why CALLBACK FUNCTION is called.
  4. lpvStatusInformation - Pointer to a buffer that specifies information about this call
  5. dwStatusInformationLength - Size in bytes of the lpvStatusInformation buffer.    
Many WinHTTP functions perform several operations on the network. Each operation can take time to complete and can fail.
After initiating the WinHttpSetStatusCallback FUNCTION, the CALLBACK FUNCTION can be accessed from within WinHTTP
for monitoring time-intensive network operations.
at the end of asynchronous processing, the application must set the CALLBACK FUNCTION to NULL.
This prevents the client application from receiving additional notifications; if it is not set to NULL,
the client application could become unstable.
       
When WinHTTP is used in asynchronous mode, always set lpdwNumberOfBytesAvailable to NULL in WinHttpQueryDataAvailable(), and retrieve data in the CALLBACK FUNCTION, not doing so can cause a memory fault. Also set lpdwNumberOfBytesRead to NULL in WinHttpReadData(), and retrieve data in the CALLBACK FUNCTION, not doing so can cause a memory fault.
 
The status callback function receives updates on the status of asynchronous operations through notification flags. Notifications that indicate a particular operation is complete are called completion notifications, or just completions. The following table lists the six completion flags and the corresponding function that is complete when this flag is received.
  1. WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE                   -  WinHttpQueryDataAvailable
  2. WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE            -  WinHttpReceiveResponse
  3. WINHTTP_CALLBACK_STATUS_READ_COMPLETE                   -  WinHttpReadData
  4. WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE   -  WinHttpSendRequest
  5. WINHTTP_CALLBACK_STATUS_WRITE_COMPLETE                 -  WinHttpWriteData
  6. WINHTTP_CALLBACK_STATUS_REQUEST_ERROR                  -  Any of the above functions when an error occurs.
The callback is ideal for an application using an Object Oriented design, deriving WinHTTP from your applications class. All the reading and writing can be done by calling member functions of your application main class as demonstrated here. In this case the callback is essential for calling class functions that process the response, after the request has been issued.

A callback has less relevance when using WinHTTP procedurally unless you use the application variable to pass a structure containing pointers to all the functions you will need to process the response. This is probably not going to be very practical, so the callback becomes more usefull for indicating progess of the request and response.

Any unprotected data that is being shared with the callback should not be released until the WINHTTP_STATUS_­CALLBACK_HANDLE_CLOSING notification arrives, as the callback may recieve notifications after your code issues a WinHttpCloseHandle(). Request cancellation with asynchronous WinHTTP can get you into deep water if you do not keep this in mind.





LARGE FILE XFER
And finally the problem that began this journey.
The problem of uploading a file to a server securely can be solved by using the PUT verb with a request. Apart from the security issues of allowing users to upload files to your server, there is the size issue. The content length integer variable is a DWORD in all the C API functions. This variable has a maximum size of 4294,967295 Bytes = content length (just over 4GB)

In Windows Vista, WinHttp is got an update in this category. Applications can specify a Content-Length > 4GB. To do that, they have to manually append a Content-Length header with the desired value. WinHttp will monitor for this header and if it is present and its value is bigger than 2^32 it will use it as an indicator of the total request payload size and will ignore the dwTotalLength parameter. WinHttp will internally store the header value in a ULONGLONG, which is a 64 bit unsigned variable (QUAD).

Well that's all very well, but uploading a file that large is fraught with dangers, like connection drops and timeouts, so it becomes clear that a design is needed that can resume where it left off if need be. That means that there needs to be some code on the server and that means a CGI application is needed.

Probably the best approach to sending a large file, is to break it into smaller chunks and send them one by one. Ideally you would hash each chunk and confirm the hash on the other end. Fortunately this is done for us by TCP/IP.

In sending multiple chunks, it's clear that this will take many request/response cycles, so on the client side we do not want to be initializing the library and connecting to the server each time. Ideally we just create a sequence of requests with the chunk payload attached. On the server side we do not want to be initializing a CGI application for each request. FastCGI is designed for this.

If Chink size is too large, the time taken to xfer it may exceed the response timeout. WinHTTP will then generate an error and the xfer will fail. Chunks need to small enough to ensure enough time to deliver them and get a response.

So now we need to write a client application and a FastCGI application using the POST method, that can handle binary data like a .jpg image for example.

HTTPS
'To initiate a secure session using SSL in WinHTTP, you must call WinHttpOpenRequest using the WINHTTP_FLAG_SECURE flag. This topic is covered here, and I have not written an example for it yet.

WRITING DATA
If you are POSTing data then you will need to specify the data length in WinHttpSendRequest() and then write that much data with WinHttpWriteData(). Up until Windows 7, it used to be possible to specify n Bytes and write 2n Bytes. The extra bytes would sent after the response. Windows 7 traps this error. So the if you need to send data from two separate strings for example (Header and Body as in the File Upload example) then you must specify the total number of bytes in your call to WinHttpSendRequest(), followed by two separate calls to WinHttpWriteData() that together write to number of bytes specified.


RECOVERING HEADER INFORMATION
WinHttpQueryHeaders() provides a set of information flags that, for the most part, map to an individual header like the status code returned by the server using the WINHTTP_­QUERY_STATUS_CODE flag. This provides a way to implement response error checking. WinHTTP can even return an integer instead of a string for flags like this by specifying WINHTTP_QUERY_CONTENT_LENGTH in the call. (see the wrapper code for the implementation) Once we know the request was successful, we can call WinHttpReadData() to begin reading the response.
These are all the possible flags and server response codes.
The wrapper has functions with these enumerated, so that a text description may be returned to the user instead of a code.


COOKIES

HTTP session data is passed between the client and server in the cookie header of the request or the response. The server sends cookies to the client in the Set-cookie header of the response and the WinHTTP API resends the server cookie to the server in the cookie header of the request.

While cookie handling can be dome manually a very nice Automatic Cookie Handling functionality exists in WinHTTP which automatically handles cookies, meaning the client application performs no custom cookie handling.

WinHTTP obtains the cookie from the servers Set-Cookie header and stores it in a cache on a per-session basis. This cookie is resent on subsequent requests in the same WinHTTP session where the target matches the source of the cookie. The WinHTTP API regenerates the request cookie header for each leg in the request.

In the manual mode, your WinHTTP client application has to manually specify all cookies, using WinHttpAddRequestHeaders (The client application should clear all cookie headers before resending the request.)


WRAPPER
This article describes the operation of the WinHTTP functions encapsulated by the wrapper. The wrapped is designed to implement all of the WinHTTP functionality while trapping errors, making HTTP functionailty much easier to implement in a larger application. The wrapper could be used in either an object oriented or procedural design for synchronous or asynchronous operation.

There are many ways to implement the WinHTTP functionality, but in my experience, most developers want an easy to use libray that will do what is needed or present an explanation of why it failed. Trapping errors is tedious painstaking work that most skip over in the push to get something working, so this wrapper offers *a* solution that attempts to encompass that need.

WinHTTPSession() provides a flexible implementation of an HTTP request/response. I have created examples to demonstrate GET/POST/PUT using this function. They reduce the process to a single function call, the desired goal. The tradeoff is that the library is intitialized for each request/response, which is not going to work for some situations, but the Upload example demonstrates how to make multiple calls once a sessions handle and a connect handle have been established.

The next question is how expensive is the library initialization? After some testing, it turns out that it only takes xx clocks to accomplish this. Thats equivelent to converting xx numbers to strings for example.









Comments