XML for the Specification of the Euronext FIX Trading protocol

1. Introduction: Migration Challenges

Euronext emanates from the merger of the stock exchanges of Amsterdam (Amsterdam Exchanges), Brussels (Brussels Exchanges), and Paris (Paris Bourse). As one can easily appreciate, many business, functional, and technical issues need to be solved during such a large-scale merger, which implies in-depth changes to business procedures and technical systems. Among such changes was the adoption of a new protocol and message format for the trading systems, made essential by the need to merge the three proprietary protocols in use.

After careful examination of the available standards, Euronext selected the Financial Information eXchange (FIX) protocol. FIX is a messaging standard developed specifically for the real-time electronic exchange of securities transactions. FIX is a public-domain specification owned and maintained by FIX Protocol Ltd. Although XML-based standards were considered, they were not mature enough for adoption by Euronext within the required timeframe.

This standard was not used by any of the exchanges before the merger, but was selected because it was well supported by tool vendors (trading workstations, etc) and it had good coverage of the features required.

The migration involved a large number of partners. All the members (brokers, banks and traders) of the three major stock exchanges needed to evolve from a positional message format (COBOL-oriented) to the new field-delimited FIX format.

Since the timeframe was extremely short (4 to 6 month) the information needed to be communicated efficiently to all partners and the most recent versions of the specifications had to be available immediately to all parties. It was of paramount importance for the documentation to be correct and to remain consistent with the systems made available for testing. Supporting and assisting partners during their migration process was also fundamental to the success of the project.

2. The FIX Message Format

The FIX (Financial Information eXchange) protocol is a messaging standard developed specifically for the real-time electronic exchange of securities transactions. FIX is a public-domain specification that can be found on the FIX web site (http://www.fixprotocol.org/) as an MS-Word or HTML document.

The most important parts of that document are listed below.

1. The definition and specification of various important characteristics common to all messages: physical message format, data types, etc.

2. The specification of the session-level protocol, including logon, logout, error handling and recovery.

3. The specification of the messages themselves (both administrative and application messages).

4. The specification of all the fields that can appear in FIX messages. Each field has an identifier and a name, a data type and a description.

5. Various annexes.

This paper focuses on parts 3 (message specifications) and 4 (field definitions), although some other aspects could lend themselves to formal specification techniques (eg the use of state-transition diagrams to specify the session protocol or some business scenarios).

By way of illustration, the following tables show excerpts of the FIX 4.2 specification.

Figure 1: Example of FIX Message Specification

Source: The FIX Protocol Specification

The specification for any given message includes a general description (not shown here), the list of fields that can be included in the message and their cardinality, as well as a message-specific comment for each field. The actual definition of each field (eg its data type) is found in a separate part of the specification, since fields are reused in several messages (cfr. Figure 2).

Each message includes a standard message header for which the first few well-known fields are mandatory, allowing the determination of the FIX protocol version and the message type.

Each field in an actual message has the following form: <field-id>=<value><SOH>. Fields are delimited with the <SOH> character (ASCII code 1). This delimiter must be used to determine the end of field data, except for 'binary' fields for which the length is specified by a numeric field immediately preceding it.

The symbol in the specification denotes a field that is part of a repeating group. The number of occurrences in the group is determined by a numeric field immediately preceding the repeating group. The beginning of an occurrence is detected by the presence of the first field in a group, which is always mandatory.

Figure 2: Example of FIX Field Definitions

Source: The FIX Protocol Specification

Below is an example of a FIX message:

Figure 3: Example of a FIX Logon Message

3. The Process

As can be seen in the previous section, the FIX format is made of 'tagged', delimited fields and allows for repeatable field groups. Except for a few details, this format is self-describing, much like XML but less readable. However, it lacks a formal schema language: the FIX protocol specification is simply an MS-Word document.

The absence of a formal schema makes it difficult to perform a variety of tasks on FIX messages such as:

• validating messages (ensuring that mandatory fields are present, verifying data types and verifying valid values, etc);

• parsing complex messages (it is necessary to know what fields are part of repeating groups, and what fields contain binary data, in order to parse the messages correctly, since a FIX message is not completely self-describing);

• providing a programmer-friendly API to help the creation and manipulation of valid FIX messages.

As will be shown in the next sections, providing a formal schema not only is necessary to implement such tasks in a maintainable way, but also brings benefits and opportunities.

To meet those goals, it was decided to create a highly structured version of the FIX format specification in XML. It was necessary to create a customized DTD for this task, since none of the schema languages available at that time were suitable to express the specific structure of FIX messages, keeping the separation of fields and messages, with rich data types.

The MS-Word FIX specification has been converted to XML using a set of custom-made MS-Word macros, to produce:

• a set of schemas for the FIX messages, including data types and additional constraints on fields and messages such as valid values and mandatory fields;

• textual documentation for all messages and fields, linked by a table of contents and descriptive material.

The following XML excerpts show the specifications corresponding to the above FIX specifications. The first corresponds to the definition of the Logon message:

[...]
<field id="s05-398" name="EncryptMethod" mandatory="Y"> 
<comment><p>(Always unencrypted)</p></comment>
</field>
<field id="s05-3108" name="HeartBtInt" mandatory="Y"> 
<comment><p></p></comment>
</field>
<field id="s05-3141" name="ResetSeqNumFlag" mandatory="N"> 
<comment><p>Indicates both sides of a FIX session should reset 
sequence numbers</p></comment>
</field>
<field id="s05-3383" name="MaxMessageSize" mandatory="N"> 
<comment><p>Can be used to specify the maximum number of bytes 
supported for messages received</p></comment>
</field>
[...]
 

The second corresponds to the specification of the first fields:

[...]
<fielddef id="s05-31" name="Account"> <String maxlen="12"/> 
<comment><p>Account mnemonic as agreed between broker and 
institution.</p></comment>
</fielddef>
<fielddef id="s05-32" name="AdvId"> <String maxlen="16"/> 
<comment><p>Unique identifier of advertisement 
message.</p></comment>
</fielddef>
<fielddef id="s05-34" name="AdvSide"> <char> <choices> 
<choice> <value>B</value><descr>Buy</descr></choice> 
<choice> <value>S</value><descr>Sell</descr></choice> 
</choices> </char> <comment><p>Broker's side of advertised 
trade</p></comment>
</fielddef>
<fielddef id="s05-35" name="AdvTransType"> <String maxlen="1"> 
<choices> <choice> <value>G</value>
<descr>Global cancel</descr></choice> </choices> </String> 
<comment><p>Identifies advertisement message transaction type</p></comment>
</fielddef>
<fielddef id="s05-36" name="AvgPx"> <Price/> 
<comment><p>Always = 0</p></comment>
</fielddef>
[...]
 

Naturally, the XML documents match closely the original MS-Word specifications. Fortunately, the source document was relatively well structured so the conversion macros ran quite smoothly, and few manual adaptations wre necessary.

This first conversion step provided a sound basis to develop robust tools exploiting the specification. Such tools are described in the next section.

4. Exploitation of the Specification

The documentation and tools described in this section are available on the web. The exact URLs are temporary, but one can go to http://www.euronext.com/, then select 'Migration', then 'Euronext Brussels', and then look for the 'FIX Trading Interface' and 'FIX Interactive Test Tool'.

4.1. Documentation

The basic specifications extracted from the original Word document have been enriched in several ways.

• Additional documentation has been added, both at the message and field level. Moreover, architecture documents, protocol information, and various annexes have been merged with the core message specifications, in order to produce a single, complete document providing all the necessary technical information for partners willing to connect to Euronext.

• Additional constraints have been integrated in the specifications, such as field lengths, making it more complete and robust.

• Specific Euronext messages and fields have been added.

This XML documentation has been converted to HTML using XSLT transforms, in order to provide online documentation that is regularly maintained. Thanks to the online version, providing working drafts during the development phases was very easy. This web site is easy to navigate and provides many cross-references. It proves to be a valuable tool that is highly appreciated by people needing to manipulate the specification, owing to its numerous navigation facilities.

The same source has been used to produce a high quality PDF document, complete with table of contents and cross-references. This version is used for the creation of the printed manual.

4.2. Message Manipulation and Validation Mibrary

Although the creation of accurate documentation was the primary goal, Euronext soon realized that the XML schemas could be used for more sophisticated tasks. The first tool developed was a library to parse, validate, and create FIX messages, written in the Python language (http://www.python.org/).

This library loads the XML specifications at run-time and uses it to parse the messages accurately, flagging unknown fields and handling repeating group rules that were read from the schemas.

It can also validate messages, exploiting all the constraints laid down in the specifications.

Finally, it provides a simple, intuitive API to create and read FIX messages. An example is shown below.

# create and populate a Logon message (type A)
mo = fixspec.createMessage("A")
mo.MaxMessageSize = 1024
mo.HeartBeatInt = 60
# use the message writer to print it in FIX format
mw = fixspec.createMessageWriter(sys.stdout)
mw.writeMessage(mo)
 

4.3. Interactive Test Tool

The message manipulation and validation library made it possible to create an interactive test tool. This GUI application reads the specifications and dynamically builds a user interface based on them. The fields are created on-screen depending on the content of the specifications. The validation library is exploited to provide interactive feedback on the validity of entered data.

Figure 4: Typical Screen of the FIX Interactive Test Tool

The test tool configures itself dynamically from the XML specifications and is thus constantly up-to-date and immediately usable with the latest version of the documentation. It is equipped with a basic communication infrastructure, so it can exchange messages with other test tools or with other FIX engines.

This tool is used by Euronext developers to test the FIX interface of the trading engine and ensure that it matches the documented specification exactly. Moreover, it is available for member institutions to help them understand the specification better by having hands-on experience, and provide them with a tool to test their own implementation early without actually connecting to Euronext servers.

5. Benefits

Effort invested in creating XML-structured specifications for the Euronext FIX protocol is paying dividends.

• The specification and the test tool are online on the Internet and immediately available to all members.

• Developers and users appreciate the consistency and cross-referencing that are provided by the high quality HTML and PDF versions.

• The test tool enables both the partners and the Euronext developers to verify their implementations by ensuring that they match the specification, thus improving the quality of the documentation and the systems at the same time. When a new version of the specification becomes available, it can readily be installed and the test tools exploit it immediately. This allows very short release cycles for the specification, since the tools do not need to be redeployed. All partners can collaborate on improving the specification and documentation by fixing errors and testing the changes to the specification by themselves, without waiting for a new official release.

In addition, partners have access to the XML version of the specification, which they can exploit in their own applications. In fact, the availability of the XML specification boosts the adoption of the new protocol, because developers prefer to work with the formal XML specification of the new format rather than translating the plain English specification of the legacy protocol into their application code.

A longer-term benefit is the possibility to migrate very easily to an XML syntax for the messages. Indeed, it is perfectly possible to generate DTDs or XSD schemas for an alternative representation of FIX messages. A proof of concept has been realized and shows the feasibility of the approach. The following is an XML DTD produced automatically from the specification shown above. In this case, few constraints are exported in the DTDs (data types are lost and only mandatory and optional fields are indicated). Of course, more complete schemas could be produced to include data types, valid values, and documentation.

<!-- Logon message A --> 
<!ELEMENT M-A ( EncryptMethod, HeartBtInt, ResetSeqNumFlag?, MaxMessageSize?)> 
<!ELEMENT EncryptMethod (#PCDATA)>
<!ELEMENT HeartBtInt (#PCDATA)>
<!ELEMENT ResetSeqNumFlag (#PCDATA)>
<!ELEMENT MaxMessageSize (#PCDATA)>
 

Euronext can thus be confident that when the need arises, it will be able to provide an XML interface to its trading engine in record time.

6. Conclusions

XML as a message format has an enormous advantage over traditional EDI (in the broad meaning) methods: the presence of formal schema languages such as DTD or XSD. Such schemas serve as documentation, are used to validate the messages, and can be exploited in many ways by applications. This is a tremendous benefit over other methods where the specifications are informal and present only in the form of more or less structured English text that needs to be interpreted and re-coded manually by all partners.

This paper shows how this major feature of XML can be successfully applied to legacy formats. The lack of standard tools such as parsers or validators makes the process slightly more complex, but the benefits are tremendous: a single source specification is used to produce documentation and a variety of applications that range from test tools to actual validators on the production systems.

It has been shown how this approach creates a clear migration path to implement a pure XML message format, by generating DTD or XML schemas automatically.

Design and Development by deepX Ltd.