How to maintain a family of DTDs and keep them related using switchboards

5 minute intro

A family of DTDs

Who we are: (Wolters) Kluwer (Nederland) (Law, Tax ( & Professional) Publishers).
DTDs should be as tight as possible.
But not all documents fit in these tight DTDs, so we need a lot of DTDs.
We want them as similar as possible: sub and super DTDs.
We need a system to keep them related.
We want content experts, not SGML experts to adapt a DTD to a specific need.
Switchboards make that possible.

Switches

Switches are parameter entities with possible values of "INCLUDE" or "IGNORE"; they switch marked sections on and off:

<!ENTITY % DO.subtitle.IN.parablock "IGNORE">
<![ %DO.subtitle.IN.parablock; [
  <!ENTITY % HF.subtitle.IN.parablock "subtitle?, ">
]]>
<!ENTITY % HF.subtitle.IN.parablock "">
<!ENTITY % H.parablock "title, %HF.subtitle.IN.parablock">
<!ENTITY % B.parablock "para+">
<!ENTITY % T.parablock "">
<!ENTITY % C.parablock "(%H.parablock; %B.parablock;%T.parablock;)">
<!ELEMENT parablock %C.parablock;>

Switchboards

DTD x is defined in a switchfile x.SWI, that only contains switch definitions:

<!ENTITY % DO.subtitle.IN.parablock "INCLUDE">

Switches can trigger other switches in the central switchboard:

<![ %DO.subtitle.IN.parablock; [
  <!ENTITY % DO.subtitle "INCLUDE">
]]>

The only way a user can change his DTD is by modifying the switch file.

The advantages

Only foreseen and approved content models can be created.
The sub DTD is completely defined by the switches (interface).
Whether they switch element declarations or (Relax) schema documents is implementation.
It should now be 09:50

Why we think we need so many DTDs

Beacause of who we were when starting with SGML

Kluwer Legal and Tax publishers
Market leader in the Netherlands (80%)
Books since 1739
Loose leaf since 1920
Digital typesetting since 1972
On line since 1979
CD-ROM's since 1987
SGML since 1989
Highly profitable (30 % ROS)
In short: old economy

Some more data (rough estimates)

Loose leaf titles:
- Legal: 130 (50% SGML)
- Tax: 150 (30% SGML)
- 150,000 pages SGML
300 journals (5% SGML), 100 CD-ROM's
300 book series (5% SGML)
800 books a year
16,000 Laws (300 MB)
50,000 Court sentences (750 MB)
600 SGML titles (1750 MB)

Why we need some loose DTDs

Because we have legal stuff:
- The law is sacred to us. But it is not to the regulator. All mistakes, misspellings, misnumberings, misstructures must be preserved.
- Laws have long life cycles. Editorial views change. If it used to be (a, b) and changes to (b, a), both must be allowed.
- Because laws change frequently, many publications are loose leaf.
We make money by selling loose leafs:
- Page breaks occur in the middle of a word.
- Our batch pagination system can't handle tranformations very well, certainly not across page boundaries.
  Therefore, if in one publication the order is (a,b) and in the other (b,a), the elements are coded in that order, not transformed while formatting.
- If the order changes during a publication's lifecycle, both orders must be allowed.
Kluwer Law and Kluwer Professional publishers merged again.
- Lots of publications (oops, information components) now that don't care for law.
- It may be illegal publishing, but we finally do it professionally.
- More books; 170 loose leaf titles that seldomly change, just grow (no page but block replacement): easier to impose strict structures.

Tight DTDs are better

In a tight DTD, more things can be done automatically:
When the content model of a block of paragraphs on a court sentence is
```
<!ELEMENT pboc (title, para+, source)>
```
a smart editor will automatically open title and the first para when creating pboc.
The less options, the less choices and the less time spent in choosing or making wrong decisions.
Conditional style sheets are easier when there are fewer options.

Real world situation

If you encounter a new situation, that can't be fixed, the DTD must be modified (loosened) to allow it. F.i., one parablock without a title:

<!ELEMENT pboc (title?, para+,  source)>

and then one with more than one source:

<!ELEMENT pboc (title?, para+, source+)>

and one with the source as title:

<!ELEMENT pboc ((title?, para+, source+) |(source+, para+))>

The conflict

If you have one DTD for all documents, all other documents loose some control.
If you have lots of specific DTDs, it is hard to make global changes, and ensure compatibility between the DTDs and Docs.
So you have to make a system to ensure all target DTDs are subDTDs of one super(exchange) DTD.
A DTD is defined by the set of documents it allows. If the set of DTD A is a subset of the set of dox allowed by DTD B, A is a subDTD of B, and B is a super DTD of A.

Some classic techniques

Parameter entities:

<!ENTITY % text "PCDATA | emph | ref | sup | sub %OR.noterefs; %OR.local.floaters;">

External entities:

<!ENTITY % calstbls PUBLIC "-//SGML Open//DTD Exchange Table Model 19960430//EN">
%calstbls;

DocBook / TEI approach

Switches to exclude modules (groups of related element declarations) from the DTD
Parameter entities %local.stuff for adding your elements

Example of DocBook module

<!ENTITY % msgexplan.module "INCLUDE">
<![ %msgexplan.module; [
  <!ENTITY % local.msgexplan.attrib "">
  <!ENTITY % msgexplan.role.attrib "%role.attrib;">
  <!ELEMENT MsgExplan - - (Title?, (%component.mix;)+)>
  <!ATTLIST MsgExplan
    %common.attrib;
    %msgexplan.role.attrib;
    %local.msgexplan.attrib; >
]]>

Why is DocBook not good enough?

Too little granularity: you can't make the title required in or exclude it just from MsgExplan and not for other elements:
```
<!ELEMENT MsgExplan - - (Title?, (%component.mix;)+)>
```
More complex changes require rewriting parameter entities that define content models. This must be done in a local file in the beginning of the DTD, where you can't use yet content model fragment entities that are defined later in the global DocBook DTD, so if they are changed, that is not changed in your local file as well!

The solution: switchboards

All changes are made just by changing switches in a local switch file.
Then the central switchboard sets all other switches to the appropriate value.
Then parameter entities with all possible attribute values are declared.
Then external modules with all element and attribute declarations are called.
Then all references to globally ignored elements are removed and the resulting content models are normalized.

How a simple switch file looks like

<!ENTITY % DO.ugv 'INCLUDE'  -- the root element -->
<!ENTITY % DO.UGV "INCLUDE" 
-- the general switch to include UGV like structures -->

Some switchboard stuff

<!-- General settings for  XML/SGML -->
<!ENTITY % DO.XML "IGNORE">
<![ %DO.XML; [
  <!-- Tag omission: illegal in XML -->
  <!ENTITY % NoTagOmit "">
  <!ENTITY % EndTagOmit "">
  <!ENTITY % BothTagOmit "">
  <!ENTITY % StartTagOmit "">
  <!ENTITY % SDATA "">
  <!ENTITY % NAME "NMTOKEN">
  <!ENTITY % NAMES "NMTOKENS">
  <!ENTITY % NMTOKEN "NMTOKEN">
  <!ENTITY % NMTOKENS "NMTOKENS">
  <!ENTITY % NUMBER "NMTOKEN">
  <!ENTITY % NUMBERS "NMTOKENS">
  <!ENTITY % NUTOKEN "NMTOKEN">
  <!ENTITY % NUTOKENS "NMTOKENS">
  <!ENTITY % DO.Exclusions "IGNORE">
]]>
<!-- End of XML stuff; now you get the SGML stuff -->
<!ENTITY % NoTagOmit "- -">
<!ENTITY % EndTagOmit "- o">
<!ENTITY % BothTagOmit "o o">
<!ENTITY % StartTagOmit "o -">
<!ENTITY % SDATA "SDATA">
<!-- Declared attribute values that are more specific in SGML than XML
  allows; %NMTOKEN(S) is just added for convenience -->
<!ENTITY % NAME "NAME">
<!ENTITY % NAMES "NAMES">
<!ENTITY % NMTOKEN "NMTOKEN">
<!ENTITY % NMTOKENS "NMTOKENS">
<!ENTITY % NUMBER "NUMBER">
<!ENTITY % NUMBERS "NUMBERS">
<!ENTITY % NUTOKEN "NUTOKEN">
<!ENTITY % NUTOKENS "NUTOKENS">
<!ENTITY % DO.Exclusions "INCLUDE">
<!ENTITY % DO.UGV "IGNORE">
<![ %DO.UGV; [
  <!ENTITY % DO.A.RunHeadAlg "INCLUDE">
  <!ENTITY % DO.A.sX.RunHeadAlg "IGNORE" -- but not in S1-16 -->
  <!ENTITY % DO.A.contentkind "INCLUDE">
  <!ENTITY % DO.A.s1.contentkind "IGNORE" -- but not for element S1 -->
  <!ENTITY % DO.A.s2.contentkind "IGNORE" -- nor for element S2 -->
  <!ENTITY % DO.cpart "INCLUDE">
  <![ %DO.cpart; [
    <!ENTITY % DO.J.annotgpX "INCLUDE"
     -- allow skipping in annotation group levels -->
  ]]><!-- %DO.cpart; -->
  <!ENTITY % DO.J.sX "INCLUDE" -- allow skip section levels S1-16 -->
  <!ENTITY % DO.s16 "INCLUDE" -- have 16 levels of sections -->
  <!ENTITY % DO.sX.IN.lq "INCLUDE" -- and in long quotations -->
  ...
  <!ENTITY % DO.wctt.IN.sX "INCLUDE">
]]><!-- %DO.UGV -->

A simple element declaration

<!ENTITY % DO.hnar "IGNORE">
<![ %DO.hnar; [
  <!ENTITY % O.hnar "%EndTagOmit;" -- tag omission -->
  <!ENTITY % M.hnar "%MF.atext;" -- mixed content -->
  <!ENTITY % C.hnar "(%M.hnar;)*" -- content model -->
  <!ENTITY % E.hnar "" -- no exclusions -->
  <!ELEMENT hnar %O.hnar; %C.hnar; %E.hnar;>
  <!ENTITY % DO.A.hnar.nr "%DO.A.nr;">
  <![ %DO.A.hnar.nr; [
    <!ENTITY % A.hnar.nr "%A.nr;">
    <!ATTLIST hnar %A.hnar.nr;>
  ]]>
]]>

Some complex declaration stuff

<!ENTITY % DO.pboc "IGNORE">
<![ %DO.pboc; [
  <!ENTITY % O.pboc "%EndTagOmit;">
  <!ENTITY % DO.OPT.ti.IN.pboc "IGNORE">
  <![ %DO.OPT.ti.IN.pboc; [
    <!ENTITY % OC.ti.IN.pboc "?">
  ]]>
  <!ENTITY % OC.ti.IN.pboc "">
  <!ENTITY % DO.ti.IN.pboc "IGNORE">
  <![ %DO.ti.IN.pboc; [
    <!ENTITY % HF.ti.IN.pboc "ti%OC.ti.IN.pboc;, ">
  ]]>
  <!ENTITY % HF.ti.IN.pboc "">
  <!ENTITY % H.pboc "%HF.Meta; %HF.ti.IN.pboc;">
  <!ENTITY % DO.REQ.para.IN.pboc "IGNORE">
  <![ %DO.REQ.para.IN.pboc; [
    <!ENTITY % OC.para.IN.pboc "+">
  ]]>
  <!ENTITY % OC.para.IN.pboc "*">
  <!ENTITY % REQ.para.IN.pboc "+">
  <!ENTITY % DO.MULT.source.IN.pboc "IGNORE">
  <![ %DO.MULT.source.IN.pboc; [
    <!ENTITY % REQ.source.IN.pboc "+">
    <!ENTITY % FAC.source.IN.pboc "*">
  ]]>
  <!ENTITY % REQ.source.IN.pboc "">
  <!ENTITY % FAC.source.IN.pboc "?">
  <!ENTITY % DO.REQ.source.IN.pboc "IGNORE">
  <![ %DO.REQ.source.IN.pboc; [
    <!ENTITY % OC.source.IN.pboc "%REQ.source.IN.pboc;">
  ]]>
  <!ENTITY % OC.source.IN.pboc "%FAC.source.IN.pboc;">
  <!ENTITY % DO.source.IN.H.pboc "IGNORE">
  <!ENTITY % DO.source.IN.T.pboc "IGNORE">

  <![ %DO.source.IN.H.pboc; [
    <![ %DO.source.IN.T.pboc; [
      <!ENTITY % B.pboc "((source%REQ.source.IN.pboc;, para%OC.para.IN.pboc;)
                         |(para%REQ.para.IN.pboc;, source%OC.source.IN.pboc;))">
    ]]>
    <!ENTITY % B.pboc "source%OC.source.IN.pboc;, para%OC.para.IN.pboc;">
  ]]>
  <![ %DO.source.IN.T.pboc; [
    <!ENTITY % B.pboc "para%OC.para.IN.pboc;, source%OC.source.IN.pboc;">
  ]]>
  <!ENTITY % B.pboc "para%OC.para.IN.pboc;">
  <!ENTITY % T.pboc "">
  <!ENTITY % C.pboc "(%H.pboc; %B.pboc; %T.pboc;)">
  <!ENTITY % E.pboc "">
  <!ELEMENT pboc %O.pboc; %C.pboc; %E.pboc;>
]]><!-- %DO.pboc; -->

Things we can rule

We can have switches for things like:
%DO.x.IN.y: allow element x in element y
%DO.REQ.x.IN.y: is it required?
%DO.MULT.x.IN.y: multiple?
%DO.J.sX: allow jumping from s3 to s5
%DO.x.IN.H.y: x in the header of y
%DO.L.H.y: header of y is loose (f.i. (title|number|subtitle)* )
Not all elements have this kind of control! If necessary, we add a switch.

Normalizing it

We wrote a Perl script, inspired by NormDTD.exe (Richard Light), based on DTD2HTML.PL (Earl Hood), that reads the source files and creates normalized DTD.

Removes all references to undeclared elements.

Normalizes:

((,a),(|b|c)+,(,,)*,(d+|||)?,)

becomes

(a,(b|c)+,d*)

Writes alle declarations in 1 DTD file.

Why normalizing plain vanilla SGML?

Not every parser aproves this kind of parameter entities, thinks it is obfuscating.
If element x is not declared in the DTD but allowed in a content, some parsers give error messages.
Can't predict behaviour if they don't.
Speed of parsing and ease of transportation if only one file with no comments or marked sections.

An old reason to normalize

First we had empty tokens before removing undeclared elements:

(a, %b.IN.x;, c)

where %b.IN.x; could be "b" or "" empty; we now say

(a, %HF.b.IN.x; c)

where %HF.b.IN.x can be "b," or ""
So now we can parse our source DTD with NSGMLS, even before normalizing.

Why schemas don't matter

The solution is all plain vanilla SGML.
But that isn't the point. The point is, that the user changes the DTD just by flipping some switches. It will be easy to write a script that hides the user from SGML syntax, and just shows some checkboxes.
The sub DTD is completely identified by the switches (declarative) in stead of by content fragments (procedural). The switchfile is the interface.
Whether you generate a DTD or a schema isn't interesting; that's implementation!

How to schematize a switch:

If the DTD stuff looks like:

<![ %DO.OPT.title.IN.pboc; [
  <!ENTITY % OC.title.IN.pboc "?">
]]>
<!ENTITY % OC.title.IN.pboc "">
<!ENTITY % H.pboc "title%OC.title.IN.pboc;, ">

then the schema stuff in the internal subset will look like:

<![ %DO.OPT.title.IN.pboc; [
  <!ENTITY MINOC.title.IN.pboc "0">
]]>
<!ENTITY MINOC.title.IN.pboc "1">

and in the document:

<complextype name="pboc">
  <element ref="title" minoccurs="&MINOC.title.IN.pboc;"/>
  ...
</complextype>

How to RELAX a switch

If the DTD stuff still looks like:

<![ %DO.OPT.title.IN.pboc; [
  <!ENTITY % OC.title.IN.pboc "?">
]]>
<!ENTITY % OC.title.IN.pboc "">
<!ENTITY % H.pboc "title%OC.title.IN.pboc;, ">

then the RELAX stuff in the internal subset will look like:

<![ %DO.OPT.title.IN.pboc; [
  <!ENTITY % OC.title.IN.pboc "?">
]]>
<!ENTITY % OC.title.IN.pboc "">