[5 March 2008]

My evil twin Skippy continues his reaction to the issue Henry Thompson has raised against the SML specification. He has lost the first point, in which he claimed that the XML spec doesn’t actually say that XML documents are character sequences.

Well, OK. But even if only character sequences can be XML documents strictu sensu, how does referring to information sets instead of XML documents actually help? Any system which handles XML documents in their serial form can also, if it wishes to, handle other representations of the same information without any loss of conformance.

At this point, I interrupt to remind Skippy that some people, at least, believe that if a spec requires that its implementations work upon XML documents, then by that same rule it forbids them from accepting DOM objects, or SAX event streams, and working on them. This is roughly how the XSD spec came to describe an XML syntax for schema documents, while carefully denying that its input needed to be in XML.

Skippy responds:

Suppose I claim to be a conforming implementation of the XYZ spec, which defines document the way SML does, as well-formed XML, instead of as XSD does, as “an information set as defined in [XML-Infoset]”.

Actually, I interject, this isn’t offered as a definition of document, just as a characterization of the input required for schema-validity assessment. Skippy points out, in return, that “as defined in [XML-Infoset]” is more hand-waving, since in fact the Infoset spec does not offer any definition of the term information set.

Eventually, we stop squabbling and I let him continue his argument.

I allow people to define new documents through a DOM or SAX interface, and I work with those documents just as I work with documents I acquire through a sequence of characters conforming to the grammar of the XML spec.

Now, suppose someone who believes that angle brackets are the true test of XMLness decides to challenge my claim to conformance. They will claim, I believe, that I am working with things that are not really XML documents, and thus violating the XYZ spec, which says that they MUST be XML documents. My response will be to ask why they think that. “Show us the data!” they will cry. And I will provide a sequence of characters which is indisputably an XML document.

The claim that I am not conforming is necessarily based on a claim about what I am doing ‘under the covers’, rather than at my interfaces to the outside world. But precisely because it’s a question of what I do “inside”, there is no way for an outside observer to know whether I am working directly from the DOM data structure or SAX event stream, or am making a sequence of characters internally, and working from that. One reason there is so little problem in reality about the distinction between serialized XML documents and other representations, even in arguments about conformance, is that in reality, if XML is accepted at, and produced at, the interfaces when requested, there is no way to make any argument that forbids other forms from being accepted at interfaces.

I have always thought of myself as a reasonably good language lawyer, but I am not sure how to refute Skippy’s claim. Can any readers suggest ways to prove that it matters? Or is Skippy right that SML will do better to refer to the XML spec, to describe its inputs, than to the Infoset spec?

[5 March 2008]

Henry Thompson has raised an issue against the SML spec, asking why SML defines the term document as

A well-formed XML document, as defined in [XML].

Henry suggests:

‘document’ as defined in the XML spec. is a character sequence — this seems unnecessarily restrictive — surely a reference to the Infoset spec. would be more appropriate here?

At this point, a voice which I identify as that of my evil twin Skippy sounds in my head, to wonder whether it really makes a difference or not.

I disagree with the premise that the XML spec defines “an XML document” solely as a character sequence. What I remember from the XML spec about the fundamental nature of XML documents is mostly energetic hand-waving.

Hmm. What is the truth of the matter?

I think Skippy has a point here, at least about the hand-waving. Both the first edition of the XML spec, with which I am probably most familiar, and the current (fourth) edition, say “A textual object is a well-formed XML document if” certain conditions are met. Surely the phrase “textual object” begs the question.

Both versions of the spec also hyperlink the words “XML document” to what seems to be a definition of sorts:

A data object is an XML document if it is well-formed, as defined in this specification. A well-formed XML document may in addition be valid if it meets certain further constraints.

The phrase “data object” may be ever so slightly less suspect than “textual object”, but I’m feeling quite a breeze here.

Certainly this is consistent with my recollection that I actively resisted Dan Connolly’s repeated request to nail down, once and for all, what counted as an “XML document”. Most SGML users, I think, were inclined to regard the terms document and SGML document as denoting not just sequences of characters, but also any data structures with recognizable elements, attributes, parents, siblings, and children, and so forth that one might choose to represent the same information in. And I certainly planned to do the same for the term XML document. If anyone had told me that a DOM or SAX interface does not expose an XML document but something which must be religiously distinguished from XML documents, I would have said “Bullshit. The document production defines the serialization of a document, not the document.”

Some people might say, given the level of abstraction implicit in the XML WG’s usage of the term document (or at least in mine), that we regarded the term document as denoting any instance of the data model. (Of course, others will immediately chime in to say that SGML and XML don’t have data models. But as I’ve said elsewhere, at this point I in the discussion I don’t have any idea what the second group means by the words “have a data model”, so I can’t say one way or another.) Speaking for myself, I think I would have been horrified by the suggestion that once one had read a document into memory and begun working with it, one was no longer working with an XML document.

But nevertheless, the XML spec may have nailed things down in just the way I thought I had successfully resisted doing. The conditions imposed on a data object, in order for it to be an XML document, include that it be well-formed, which in turn entails that

Taken as a whole, it matches the production labeled document.

It’s embarrassing for those who like to think of XML documents in a slightly more abstract way, but it’s not clear to me how anything but a character sequence can match the ‘document’ production.

Sorry, Skippy, but while I think Henry’s note over-simplified things, he is probably more nearly right than you.

it probably won’t change by one whit the way I use the term XML document. But it’s nice to learn some new things now and then, even if what you learn is that you failed at something you attempted, ten years ago, and that as a consequence, you’ve been wrong about a related question ever since.

The recent publication of XML 1.0 Fifth Edition, and the ensuing discussion of how best to define the set of name characters, has made me think about a proposal that came up during the original development of XML 1.0.

We had taken as our task the definition of a grammar production to select all and only the Unicode characters suitable for use in identifiers. For our SGML-influenced minds, suitability essentially meant being a letter or something letter-like, by analogy with the characters A through Z and a through z allowed in SGML’s reference concrete syntax, or a numeric symbol, by analogy with 0 through 9. Now, it is an empirical fact that any such grammar production will be complicated, especially if it excludes not just unsuitable Unicode characters but unassigned code points.

As we were contemplating the unappetizing lists of ranges and properties that became productions [84] through [89] of the XML spec, someone — my memory says it was Tim Bray, but I have not tried to verify this memory against the historical record, and I don’t remember whether it was in email or on a phone call — said “Well, I realize I risk being labeled a xenophobe for suggesting this, but you know, the lexer has to have character tables for recognizing names and other tokens, and this rule is going to make those character tables just huge. We all know that in the ideal, normal case, the user is not going to be eyeballing XML source directly, you’re going to want to have some sort of presentation interface. So the element names are just codes, not utterances in natural language. They might as well be E001, E002, E003, and so on, as far as the parser is concerned. So why don’t we just do the same thing as SGML’s reference concrete syntax and restrict names to ASCII characters? That would make the tables you need for the lexical scanner a lot smaller. Any UCS character can go in the data, and the data is all the human reader needs to care about anyway. It’s only in the element and attribute names that we make the restriction, and the proposal is not really Anglo-centric since element and attribute names don’t have to be natural-language words; they shouldn’t be part of the user interface in the first place.”

I remember having a sinking heart at this point. The analysis is so plausible, and yet wrong in so many ways. One of the design goals of XML is that it should be legible to humans, and having identifiers that take the form of words in a natural language is an important tool in making markup legible. E001, E002, and so on just don’t cut it. (A librarian did suggest once in all seriousness that SGML vocabularies should have numeric identifiers, like the numeric field identifiers in MARC records. 245 means main title, everyone knows that, but unlike “main title” it doesn’t have an Anglophone bias. But as a design principle for actual vocabularies, this seemed to me too much like ensuring that all languages are treated equally by ensuring that the vocabulary is hard to understand for everyone.)

Also, while I have seen some very nice interfaces to marked up text, I’ve spent most of my writing life over the last twenty years working with SGML and XML source, mostly in emacs, and no interface that hides the markup has ever made me want to change. (For a while I did use the hide-tags mode in XMetal for rereading and copy editing, but I could never draft in it, and when I upgraded my operating system and XMetal ceased to run, I didn’t actually spend a lot of time looking for a new copy, or a new GUI.) I don’t want to hide the tags. The tags are what make things work. There is a story that a worker came into Bertolt Brecht’s apartment in Berlin once to install some curtains, and after doing so he started to put up a valance to hide the curtain rod and the hooks and strings. Brecht came into the room and made him stop and take it back down, with (as I remember it — I have not spent any time looking for the source of this story to make sure I’m getting the details right) the words “Never hide the mechanism!” When it comes to markup, at least, I’m with Brecht.

So for me, and anyone like me, the claim “no human really ever looks at this stuff” is false (and also conveys indirectly the suggestion that we counter-examples aren’t really human and thus don’t need to be accounted for). Even if everyone in the world used tag-hiding editors, the programmers who create those editors and the stylesheet authors who specify the mapping from tags to display form will spend time looking at the raw markup. And last I looked, they were all humans, too.

But none of those arguments were going to go anywhere against that brutally simple argument about the size and complexity of the scanner tables, and the seductive suggestion that it really would be inhumane to require users to use tag-visible interfaces. So for a while I was really worried that the proposal for ASCII-only identifiers was going to carry the day.

I’m not sure whether my memory of what happened next is what happened in reality, or whether I’ve just persuaded myself to accept as real a fantasy of what could have happened, if only someone had had the wit to think of it, and the nerve to say it, at the time.

The way I remember it is this. Someone (I don’t remember who) responds to the suggestion carefully, thoughtfully, and without screaming. They agree that it’s important to keep the lexical scanner simple, and that identifiers really don’t need to be used to represent arbitrary natural-language words or phrases. And they conclude by saying “So I think I’m willing to support this proposal, if we can improve it a little bit. Right now, the natural way to represent the scanner table for ASCII-only identifiers is to use a table 128 octets wide. But that’s a lot bigger than we need. So I propose that we don’t allow all the characters A through Z in both upper and lower case. If we restrict ourselves to A through O, uppercase only, then we can get by with a much smaller table. So I’ll support the proposal if we restrict identifiers not to any ASCII letter, but to A through O uppercase only, i.e. U+0041 through U+004F. And since no one wants to use natural-language words for identifiers, that’s not a problem, right?”

And then, silence. You can just about hear the proponents of ASCII-only identifiers starting to say “but then you couldn’t use ‘q” or ‘quote” or even ‘html” or ‘body” as element names,” before biting their tongue as they realize that they can’t say that without giving away the store.

Whether it happened the way I remember it or in some other less dramatic and memorable way, the end is the same: after people thought about it for a while, the proposal for ASCII-only identifiers evaporated, like dew in bright sunlight.