A High-Performance Windows Web Server Interface with SIGTERM Handling
 1. Introduction The original FastCGI spec (whitepaper) does not define named pipes; in fact, support for named pipes is not a requirement of the FastCGI specification. named pipes are used by mod fastcgi and the FastCGI application library as a replacement for Unix sockets. mod_fastcgi uses named pipes on Windows (Unix sockets on Unix) by default. Most FastCGI servers support a connection to a FastCGI application via sockets, so sockets are fairly well described in the Spec. Since the pipe protocol is not defined, there is some room for interpretation. So what is a Windows named pipe anyway? Well... it's a pipe with a name. Named pipes allow two unrelated processes to communicate with each other. They are also known as FIFOs (first-in, first-out) and can be used to establish a two-way (full-duplex) flow of data. Named pipes exist as entries in the file system, but, in fact, when you use them, all operations occur in memory, meaning named pipes exist in memory, not on disk. So although you access them with commands used to open and read from files, they are about as fast as shared memory. The real question for FCGI is how to get hold of the name of the named pipe (as designated by the
Web server) because you need the name of the pipe to connect to it. Ideally it would be sent as an environment variable (_FCGI_X_PIPE_) to avoid enumerating all the pipe names,
but this is not always the case. In most Web server implementations, STD_INPUT_HANDLE contains a
handle to the pipe, but as you will see later, that only gets you part way to a SIGTERM solution. FastCGI calls the "packets" or "chunks"
that flow in either direction records. It's just a name. A record consists of an
8 byte header describing the content of the body of the record followed by the
body itself. Simple enough. Check out an example of the record flow. The idea is that each record belongs to a single user
request (created by the user clicking in the browser, for example). If two users
are clicking at the same time, then in theory, the two requests will be given
different RequestIDs, and your application will handle each one as a unique
request without having to fire up another instance of your application. To do this, the Web server needs to multiplex the two requests. I say in theory because to my knowledge, there are no Windows Web
servers that implement FastCGI multiplexing. Two flavors of variables are sent from the Web server to the application: 1) environment variables 2) request variables Guess which set comes with the HTTP request? So the order of things is: FIRST, the Web server sends the environment variables and then, after some hocus pocus, the request variables. The meat of this process is the hocus pocus which I have termed...  This is unknown because pipes are not described in the spec. The next source of information is the obtuse C code for libfcgi.dll. After five of us took a look at this code for three months, trying to unravel this legacy "spec", it still was not clear. To compound the difficulty, it was written to compile for UNIX, WINDOWS, MAC and anything else you can shake a stick at and has a few "hacks". Having written over 50 emails to at least 20 different
developers (who write in an assortment of
languages for different OSs) all over the world regarding the FastCGI implementation, one thing is very clear: this libfcgi.dll is the de facto
source of all information pertaining to the implementation of FastCGI. Tearing this code apart required testing and debugging to figure out what is actually going on in certain situations. The original idea was just to adapt libfcgi.dll specifically for Windows IIS and implement SIGTERM. It rapidly became clear that doing so was just a pipe dream (pun intended). We needed to write a completely new implementation from the ground up. To begin, we worked from the other end... the IIS Web server. It turns out, Windows IIS does something a little creative with the implementation of pipes, so we decided to begin with the Abyss Web server. After a lot of discussion with Aprellium's lead developer, we saw that they went to some trouble to interpret the spec in a way that not only works with libfcgi.dll but allows a FastCGI application to set initial variables. Setting the initial variables introduces the first concept in the FCGI pipe dance: the Web server is not the Pipe server. The idea is that the Web server becomes the pipe client. Now if you think about this for a moment, you see that it's a little odd that the process calling your application is deemed to be the client when, after all, it's calling the FastCGI application and populating the environment variables! But there is a reason. The concept behind a pipe server created (with blocking enabled) is that it waits for the client (Web server) to connect. Since the whole concept of FastCGI is that your application survives the request and waits for another request, it is going to have to sit there waiting for that second request to come in (or for a termination signal). This is why your FastCGI application must be the pipe server. The problem then becomes how to have the Web server become the pipe client. To do this, the server could just come up with a pipe name
and send it to your application via the environment variable _FCGI_X_PIPE_.
Your application would create the pipe server using that name, and the Web
server would then connect to it and begin sending records. But both IIS and Abyss do it a little differently. The solution implemented by Abyss is for the Web
server to create a pipe initially (making it the pipe server), connect to it,
and send the first "record" -- FCGI_GET_VALUES. It also sends the PIPE NAME
contained in the environment variable _FCGI_X_PIPE_. For example: "\\.\pipe\abws-fcgi-00000002-01c98a402cfa3a4a00000874" This means that retrieving the pipe handle from STD_INPUT_HANDLE and using it to read from the pipe can be done ONLY for the first record. We can then return the record FCGI_GET_VALUES_RESULT with the values we choose. This is how the first exchange on the Abyss Web server accomplishes its initialization. The Abyss Web server sends the record immediately after launching your application: Header Version = 1 Header ContentType = FCGI_GET_VALUES Header RequestId = 0 Header ContentLength = 48 Header PaddingLength = 0 The Body is essentially a request for you to populate three variables: FCGI_MAX_CONNS FCGI_MAX_REQS FCGI_MPXS_CONNS You must then call ReadFile() to read the record bytes. Since the pipe has been created with blocking enabled, both ConnectNamedPipe() and ReadFile will block, meaning that ConnectNamedPipe() waits until the client (your app) connects, and ReadFile() waits until the Write operation (by the Web server) has been completed. As a side note, PeekNamedPipe() does not block. At one point in my ReadRecord() procedure, I use this to peek ahead 8 bytes and read the next header (if present). If it is the NUL termination record, I just swallow it right away with another ReadFile() before returning. After receiving the initial record, you should return a FCGI_GET_VALUES_RESULT record with the values defined as: FCGI_MAX_CONNS = 1 FCGI_MAX_REQS = 1 FCGI_MPXS_CONNS = 0 (I suppose you could just skip this by closing the handle, and the Web server may use default values, but I did not test this.) The return should be a total of 72 bytes, actually two records because you need to signal the end of a "transmission" with a NUL Record. I guess you could call it the termination record. (If the data to be returned is large, it will span many records, hence the need to signal the end of the data with a termination record.) The return should look like this: BYTE 0x Dec Ascii Description --------------------- Header (Record 1) 001 = 01 001 1 Version 002 = 0A 010 10 ContentType 003 = 00 000 0 RequestIdB1 004 = 00 000 1 RequestIdB0 005 = 00 000 0 ContentLengthB1 006 = 33 051 51 ContentLengthB0 007 = 05 005 5 PaddingLength 008 = 00 000 0 Reserved --------------------- Body 009 = 0E 014 010 = 01 001 011 = 46 070 F 012 = 43 067 C 013 = 47 071 G 014 = 49 073 I 015 = 5F 095 _ 016 = 4D 077 M 017 = 41 065 A 018 = 58 088 X 019 = 5F 095 _ 020 = 43 067 C 021 = 4F 079 O 022 = 4E 078 N 023 = 4E 078 N 024 = 53 083 S 025 = 31 049 1 (Notice that the value follows the name without an equals sign.) 026 = 0D 013 027 = 01 001 028 = 46 070 F 029 = 43 067 C 030 = 47 071 G 031 = 49 073 I 032 = 5F 095 _ 033 = 4D 077 M 034 = 41 065 A 035 = 58 088 X 036 = 5F 095 _ 037 = 52 082 R 038 = 45 069 E 039 = 51 081 Q 040 = 53 083 S 041 = 31 049 1 042 = 0F 015 043 = 01 001 044 = 46 070 F 045 = 43 067 C 046 = 47 071 G 047 = 49 073 I 048 = 5F 095 _ 049 = 4D 077 M 050 = 50 080 P 051 = 58 088 X 052 = 53 083 S 053 = 5F 095 _ 054 = 43 067 C 055 = 4F 079 O 056 = 4E 078 N 057 = 4E 078 N 058 = 53 083 S 059 = 30 048 0 060 = 00 000 Padding 061 = 00 000 Padding 062 = 00 000 Padding 063 = 00 000 Padding 064 = 00 000 Padding --------------------- --------------------- Header (record 2) - termination record 001 = 01 001 Version 002 = 0A 010 ContentType 003 = 00 000 RequestIdB1 004 = 00 001 RequestIdB0 005 = 00 000 ContentLengthB1 006 = 00 000 ContentLengthB0 - no body 007 = 00 000 PaddingLength 008 = 00 000 Reserved --------------------- I should add that apparently, although the padding is specified in the spec, Web servers generally do not care if you do not pad your STDOUT and generally do not pad STDIN data! Now we have completed the first step. At this point, the Abyss server is expecting to connect to a pipe with the name it provided in _FCGI_X_PIPE_, so first you must use CloseHandle() to close the initial pipe created by the Abyss Web server. The next step is for your FastCGI application to become the pipe server. This is covered in MSDN beginning with CreateNamedPipe(). The flags are important here. (PIPE_WAIT defines blocking mode.) Initially, and because multiplexing is not supported, we used a synchronous pipe. Also if synchronous I/O is used, the problem of synchronizing reading so that it occurs after the Web server has finished writing is handled automatically. Now that we have a pipe server created, we need to let the Abyss server connect. We must wait to allow the server to connect; otherwise we will begin reading from the pipe before a connection has been made (and anything has been written). Luckily there is a windows mechanism designed to block until a pipe client connects: ConnectNamedPipe(). This call returns when a client connects (as defined by the PIPE_WAIT flag). Next the Abyss Web server populates the request variables and sends them as a FCGI_PARAMS record (or records). At this point the pipe dance is complete, and we move to the meat of the FCGI protocol.After the sent records have been processed and a return sent, (i.e., every request is complete), the pipe must be recycled with DisconnectNamedPipe() and then CloseHandle(). Then a new pipe server is created, and ConnectNamedPipe() once again blocks until the Web server sends the first record of the next request.  As you can see, the pipe dance is full of pitfalls and
assumes that you understand the concepts above. Since most of us couldn't care
less about the mechanics of pipe connections and just want to process the
records, the Microsoft FastCGI development team decided to make our lives easier
by shortcutting the entire pipe dance on the Web server side. They elected to just send a
handle to the named pipe to your application in the variable STD_INPUT_HANDLE. This relieves you of the complications of
implementing asynchronous I/O with I/O completion ports, for example. The handle is in fact a pipe server handle to which IIS connects as a client, but who cares. You simply use the handle to read data using ReadFile() and write a response to the pipe using WriteFile(). What could be simpler. The FCGI_GET_VALUES request (issued by the Abyss server) is derived from the .INI file used by IIS to configure the FastCGI ISAPI module. So far this is much simpler, and we can progress directly to reading the request records. 4. FCGI_STDIN
If any data was sent using the POST method, you will find it contained in these records. This is simply one or more records containing the data (followed by a terminating record, of course). About all you need to do with these records is trim off the headers and concatenate them. A good implementation should probably use something like the C++ string builder class or a stream because string concatenation is very slow. In managing memory, the class should double its size every time you exceed capacity, as Joel Spolsky points out in his article on strings. 5. Decoding There are several decoding/encoding procedures that need to
be dealt with, and they can be a little time consuming to write, streamline, and test. Earlier I said that FastCGI is supposed to be language and platform independent. The concept is, but the implementation in a few areas is clearly biased toward C and Unix. In the case of the header layout, this is very obvious. The header is composed of 8 bytes:  They're defined as: typedef struct { Notice that byte one of the RequestID and ContentLength preceed byte zero. This is convenient if you are working with C where you can easily bit shift the first byte's bits with the >> operator. This is fast, but more of a hangover from ASM than a high level language implementation, which would entail simply defining a two byte WORD in place of the two bytes and reading the value directly into a structure! Since this schema is prevalent in all the FastCGI structures, if you are not writing in C/C++ and your language does not support efficient bit shifting, then I suggest using BYTE pointers for each of the two bytes of a WORD and assigning their values in reverse order. Then the value will be available directly in a WORD variable. What a concept. If your language (like VB) does not support pointers, then, well, you should probably consider working with a different language.The next item that needs to be decoded is the name/value pair of the environment variables (and later the request variables). It is not immediately obvious, but these are sent without an equal sign, and the format for each name/value pair in the sequence is: NameLengthByte ValueLengthByte NameBytes ValueBytes . . . So now you step your way through the body, using the length parameters to know how far each step is. Since a byte can hold a maximum value of 255, and some request parameters can be longer, the length parameters can occupy four bytes! You determine the length by testing the MSB (bit #7) of the length byte. If it is set (value=1), then four bytes describe the length; if not, only one byte describes the length. As you have probably realized, bit 7 cannot actually count in the value of the byte (because it is being used as a flag), so that leaves 7 out of 8 bits available to store values. We also know that any length value over 127 will need to occupy four bytes, and the 4 value bytes are in this order:nameLengthB3 nameLengthB2 nameLengthB1 nameLengthB0 As a result, you cannot just lay a 4 byte unsigned integer over them and read the value. Each byte has to be read in the correct order: @pValLen[3] = @pData[0] @pValLen[2] = @pData[1] @pValLen[1] = @pData[2] @pValLen[0] = @pData[3] (Note the reversed byte order.) But even this will give the wrong value because bit 7 of that first byte is in fact a flag! So either reset the bit before reading, or just set the first byte to zero because it is highly unlikely any name value pair will exceed 1,6777,215 characters/bytes (the maximum value of a three byte unsigned integer). That's almost 17MB. So for all practical purposes, the conversion can be stated as: @pValLen[3] = 0 @pValLen[2] = @pData[1] @pValLen[1] = @pData[2] @pValLen[0] = @pData[3] (Note the reversed byte order. ) Next we finally get to the FCGI_BEGIN_REQUEST record which contains the role and flags. I have yet to see a compelling reason for using anything but the responder role, so you can probably skip decoding these variables unless you want to write something very specific. Finally we have the FCGI_PARAMS records, which are more of the name/value pair format. Among these you will find: REQUEST_METHOD CONTENT_LENGTH QUERY_STRING If the REQUEST_METHOD was GET, then QUERY_STRING contains your name/values. If the REQUEST_METHOD was POST, then CONTENT_LENGTH contains the number of bytes you will expect to see next in the final records of a FastCGI request, FCGI_STDIN. After processing all these record types (the request), you are now ready to send a reply. 6. STDOUT  Since this is HTTP, a minimal header would be good: "Content-Type: text/html" + CRLF + CRLF + "Your text goes here" It can be tricky to get exactly the right rRecord format on your first attempt, so I'll detail it here: The STDOUT header: @pHead.version = 1 // Identifies the FastCGI protocol version. This specification documents FCGI_VERSION_1 @pHead.ContentType = FCGI_STDOUT // Identifies the FastCGI record type, i.e. the general
function that the record performs Specific record types and their functions are detailed in later sections @pHead.requestIdB1 = 0 // Reflects the
request ID sent
@pHead.requestIdB0 = 1 // Reflects the request ID sent
@pHead.contentLengthB1 = @pByte[1] // The number of bytes in the contentData
component of the record
@pHead.contentLengthB0 = @pByte[0] // Between 0 and 65535 bytes of data, interpreted according to the record type @pHead.paddingLength = PadLen // The number of bytes in the paddingData component of the record (between 0 and 255 bytes) The STDOUT body: Pad the length to a multiple of 8 bytes (practically this might not be necessary on most servers, but the spec does call for it) The STDOUT termination header: Finally, create a 16 byte end request record, basically just: Byte 001 = 01 001 Byte 002 = 03 003 Byte 003 = 00 000 Byte 004 = 00 001 Byte 005 = 00 000 Byte 006 = 10 016 Byte 007 = 00 000 Byte 008 = 00 000 The STDOUT termination body: Byte 009 = 00 000 Byte 010 = 00 000 Byte 011 = 00 000 Byte 012 = 00 000 Byte 013 = 00 000 Byte 014 = 00 000 Byte 015 = 00 000 Byte 016 = 00 000 Now return the buffer to the Web server (via the pipe) with: WriteFile() FlushFileBuffers() DisconnectNamedPipe() This completes the request/response cycle, and your FastCGI application can loop
to accept the next request.
That is about as far as the framers went. We assume they were counting on the Unix OS to handle the communication of a SIGTERM when the Web server is about to KILL the FastCGI application. On Windows, things are a little different. Although this is the end of the story, this also is where this project began.  Here, the ride gets a little bumpy. From the spec: When this signal is sent by the Web server, you should clean up, meaning close databases, release handles, etc., and exit the loop that is waiting for an incoming request. You should then close the pipe with CloseHandle() if you created it (i.e., not IIS), and exit the application. However, there is one small problem. There is no SIGTERM record! This signal is not processed on the Windows platform because it is not sent. That took a few months to get to the bottom of and lead to the development process described here. The libfcgi.dll does not provide a handler for SIGTERM (even if it existed on Windows), so there is no way to clean up before the Web server unceremoniously issues a TerminateProcess() or SIGKILL and your FastCGI application is unloaded from memory.We spent hours examining ways to add this to libfcgi.dll and finally just gave up. We decided to write a new library from scratch, around a SIGTERM handler. This turned out to be the only way to do it. Because the spec refers to a SIGTERM implemented on Unix, and because Windows does not support a SIGTERM per se, we asked the FastCGI development team at Microsoft to consider implementing something that made sense from their perspective in the Windows environment. We discussed many ideas over a couple of months, including a Windows message and a custom FasctCGI record (or at least use of an existing record type). The first idea was good since Windows messages are prevalent, but since a CGI/FastCGI process cannot have a dialog, you might wonder how that is going to help? The answer is a little trick used by console applications: a hidden window. This would allow a message pump to process messages and trap the SIGTERM message. But we decided not to do this.Then we looked at how php implements SIGTERM since php is well established. Its developers elected to use an event. This is also what the Abyss server does. So the final decision was to send a handle to a termination event in the environment variable _FCGI_SHUTDOWN_EVENT_. Next we looked at the code for libfcgi.dll which suggests that a handler does exist for a shutdown event, but as the notes suggest, its handling is not very pretty: static void ShutdownRequestThread(void * arg) { HANDLE shutdownEvent = (HANDLE) arg; WaitForSingleObject(shutdownEvent, INFINITE); shutdownPending = TRUE; if (listenType == FD_PIPE_SYNC) { // It's a hassle to get ConnectNamedPipe to return early, // so just wack the whole process. Yes, this will toast // any requests in progress, but at least it's a clean // shutdown (and better than TerminateProcess()) exit(0); } // FD_SOCKET_SYNC: When in Accept(), select() is used to poll // the shutdownPending flag. - Yeah, this isn't pretty either // but it's only one process doing it if an Accept mutex is used. // This at least buys no toasted requests. } Remember, the goal of a SIGTERM is to get a warning from the Web server that a termination (based on a timeout) is immanent, and it allow the FastCGI application to exit gracefully. Notice I said exit gracefully, not "just whack the whole process." Now in the Abyss implementation, two variables are needed: - a handle to an event signaled (user defined seconds) before the process is terminated - the name of the pipe to break the FastCGI client pipe blocking and allow clean up code to execute The ACCEPT function is essentially waiting on a connection to the pipe by the NamedPipe client (the Web server), and the web server is not going to make that connection (because there is no new request), so we need a second "client" to connect. That will allow ConnectNamedPipe() to return (stop blocking) at which point we can process our clean up code. The Abyss NamedPipe allows multiple clients, so this works by using a thread to WaitForSingleObject() which then calls CreateFile() to establish the connection. Since Abyss sends the NamedPipe name in the environment variable _FCGI_X_PIPE_, all this was possible and works. Unfortunately, IIS did not send the NamedPipe name (in versions prior to 1.5). One way to get the IIS NamedPipe name would be to just enumerate all the NamedPipes with ZwQueryDirectoryFile() and look for the one that has the string "IIS" somewhere in it. An example of a IIS Named pipe name: \\.\pipe\IISFCGI-881a7335-3b0d-415f-86bd-5f849636c4f0 However, after a little discussion, the FastCGI team agreed to just pass the pipe name in the environment variable _FCGI_X_PIPE_. (Thank you, guys.) So now we have all we need to create a real SIGTERM handler for IIS Web Servers right? Err...no. Remember the pipe server and pipe client are created behind the scenes by IIS and the pipe handle is passed to the FastCGI client. This is good news in that we do not have to do the pipe dance that we must do with the Abyss Web server, but it's bad news in that behind the scenes, DisconnectNamedPipe() and thus ConnectNamedPipe() are never called. Also, IIS is using a full duplex I/O completion ports
implementation of the NamedPipe, and they elected to leave the
connection open. Why not indeed! In this case, blocking is still possible
because ReadFile() blocks until the server writes some data to the pipe. So the
problem becomes, how do we break the blocking of ReadFile()? Even if we could use the connect trick from a separate
thread, Windows NamedPipes limit the NamedPipe to ONE client only, so doing that
will not work. After some head scratching, we turned up the Windows function CancelSynchronousIo(). Unfortunately it is supported only on Vista
and Windows Server 2008, so we did not bother to explore this function further. Next we tried calling DisconnectNamedPipe() and
CloseHandle(), but the block is not released, and that's all there is to it. It became clear that without switching to asynchronous I/O (I/O completion ports), there really is no way to unblock ReadFile() in a synchronous implementation. But this is not necessarily a problem when you consider the application is about to terminate anyway. All we would need to do is put it
in a thread so that when the SIGTERM was received, we could either terminate
the thread or just abandon it and let the ExitProcess() (SIGKILL) clean it up. This works, of course. It's not ideal, but it's appropriate under the circumstances. So now we need one thread waiting for the SIGTERM event and a second thread processing all the ReadFile() calls. The challenge then became redesigning the operational flow so that only one additional thread was needed, and then implementing this in a way that accommodates the Abyss style of blocking, all within a single logical structure. This would allow a single DLL to work for both Web server protocols. After another re-write, we eventually achieved this by reorganizing the pieces of the puzzle in a clever way and utilizing a semaphored C++ StringBuilder class for managing data streams. One benefit of using a secondary IPC method to signal SIGTERM (instead of the FastCGI protocol itself) is that it allows termination at any time, even when the FastCGI application is busy processing a request (assuming that the FastCGI process takes more time than the WebServer timeout setting allows--which could occur if a process were waiting on a lengthy database operation, for example). You would think that the FastCGI spec would specifically include a "SIGTERM" record type, but the framers did not include one, possibly because Unix, unlike Windows, does not require the FastCGI process to be inherited from the Web server process and thus does not assume "responsibility" for cleaning up a FastCGI application.  So far we have focused on the requirements for the FCGI protocol. In
fact, to use this code the focus must shift to the FastCGI application because it
drives the bus; that is, the FastCGI application must call the FCGI library, not
the other way around. So now we must encapsulate the FCGI functionality within a
class that can be called in such a way as to allow the application
code to drive the FCGI library. In the original libfcgi.dll, they elected to use an Init() and Accept() function. The Init is called once, and the Accept is called each time the Web server sends a request. They added a Finish() function to clean up (but you can do that when you re-call Accept()). When the SIGTERM is sent, a problem arises. The Accept code cannot cleanup any FCGI resources because the user may want to send a STDOUT reply before exiting. Since Finish() is defined as meaning that the request is complete, not as releasing all FCGI resources, it should probably not be re-purposed. The solution we settled on was to use DLL_PROCESS_DETACH. This approach of course requires global variables (to hold any handles to be released), but it makes the FCGI cleanup transparent to the user and creates the illusion that the application code is driving the bus. 9. General Overall, the Windows implementation is a little more precise, and the Abyss implementation a little more forgiving. IIS requires the correct ID for the returned termination header, for example, because it assigns a different ID to each request up to 255 then begins again at 1. The Abyss server just uses ID=1 for all requests because without multi-plexing, only one request at a time can be processed. With the Abyss server you could connect another client on the same pipe. (This would allow forwarding the FCGI requests, for example to a Windows service.) The Windows pipe is limited to one client and is using overlapped I/O. Both servers use a different name for the pipe each time a new instance of the FastCGI client is spawned. IIS never will directly call TerminateProcess without first calling SIGTERM if the SIGTERM event is set (by defining the SignalBeforeTerminateSeconds property in fcgiext.ini). When TerminateProcess is called, your process will be terminated immediately, and then the pipe handle is released. You might want to know the meaning of a Windows error, like 0x800700c1: FastCGI Error The FastCGI Handler was unable to process the request. Error Details: * Error Number: 193 (0x800700c1). * Error Description: Unknown Error HTTP Error 500 - Server Error. Internet Information Services (IIS) All explanations can be found here, and of course here.
 Three months later I discovered that someone else had expected the
same thing ten years ago, and it took her 3.84 months (without a SIGTERM
handler) at a cost of $20,000. Developing a library like this is not a trivial task! Special thanks to the Microsoft FastCGI team, especially Kanwal Singla, Yamini Jagadeesan, Russlan Yakushev, and Wade Hilmo for their patient help, and to Chuck Hicks, Don Dickenson, Florent Hayworth, and Peter Simmons for their contributions. |