What’s the best archival format for your important documents? In a previous post I suggested parchment might be—but that’s impractical. All joking aside, the issue is a serious for anyone who is going to spend the better part of their life writing and needs reliable access to this material.

James King of Adobe has a blog, Inside PDF, that’s well worth checking out. Besides covering ISO 32000—the ISO standard based on PDF 1.7, James has made an interesting case for PDF/A as an archival format.

Part of the case is a case against XML based alternatives such as ODF and OOXML. One problem is false advertising: While they do contain XML subfiles, they are, in fact ZIP archives that contain, besides binary files. Not only false advertising, but false promises as well:

There is what I think is a rather technically shallow belief that XML files are easier to work with and will survive the passing of time, even great periods of time, better than other formats. The text held within XML files can usually be viewed with any generic text editor and I guess that gives people a warm feeling that it will therefore also be easier to retrieve with a program. Fair enough. But what is glossed over way too much is that that text is enveloped within XML for [something]. (See my earlier blog entry.) The envelopes (schemas) offered by ODF and OOXML are different. Different enough that a simple program cannot extract just the raw text from either. And is that all I really want from a document in the future, the raw text. Because when you get to the layout and the images and the color space definitions and the fonts, these things do not lend themselves well to XML and are often stored within the ZIP archives as binary data. So tell me again where the advantage to XML is for this purpose?

To these two criticisms, let me add a third. The structure encoded in the XML subfiles of ODF and OOXML is not the logical structure of the document but the functions of the word processor. But that’s not what needs preserving.

Rob Weir at an Antic Disposition, not surprisingly, had an alternative view. Rob observes that not all goals that one might have in archiving is well served by PDF. Reflection on these raise a number of questions, none of which are are answered by PDF:

1. What was the nature of collaboration that lead to this document? How many people worked on it? Who contributed what?
2. How did the document evolve from revision to revision?
3. In the case of a spreadsheet, what was the underlying model and assumptions? In other words, what are the formulas behind the cells?
4. In the case of a presentation, how did the document interact with embedded media such as audio, animation, video?
5. How was technology used to create this document? In what way did the technology help or impede the author’s expression? (Note that researchers in the future may be as interested in the technology behind the document as the contents of the document itself.)

Nevertheless, Rob is not blind to the attractions of PDF only sensitive to the way it offers a partial solution to the problem of archiving. In the end he entertains a hybrid approach:

An intriguing idea is whether we can have it both ways. Suppose you are in an ODF editor and you have a “Save for archiving…” option that would save your ODF document as normal, but also generate a PDF version of it and store it in the zip archive along with ODF’s XML streams. Then digitally sign the archive along with a time stamp to make it tamper-proof. You would need to define some additional access conventions, but you could end up with a single document that could be loaded in an ODF editor (in read-only mode) to allow examination of the details of spreadsheet formulas, etc., as well as loaded in a PDF reader to show exactly how it was formated.

There is a third way.

It may not be the Final Solution (especially given its current incarnation). But it has the advantages of both approaches and the deficits of neither: Structural markup of plain text files kept under version control. Plain text is the lingua franca of computers and will remain that way in the foreseeable future. Any given file will remain editable, but any given commit will be preserved. Moreover, the version control system will preserve a wealth of metadata about the development of the document, the contribution of collaborators, etc. There are choices in implementation concerning both the markup—be it LaTeX, ConTeXt, or XML variants such as DocBook—and the version control system—be it Subversion, Git, Mercurial. None are perfect. But for now, my bet is on the third way.

Over at An Antic Disposition, Rob Weir compares Micorsoft’s attempt to pass off OOXML as a standard to Diogenes the Cynic’s response to Plato’s definition of man.

Plato, teaching in the Akademia grove, defined Man as “a biped, without feathers.” This was answered by the original smart-ass, Diogenes of Sinope, aka Diogenes the Cynic, who showed up shortly after with a plucked chicken, saying, “Here is Plato’s Man.”

Was Diogenes the Cynic a Microsoft shill?

There is a petition against OOXML becoming an ISO standard here. Besides the fact that there already is an ISO document standard, ODF, a real standard is easy to implement and naturally has a variety of implementations. But there is no working implementation of OOXML and unlikely to be any produced by anyone other than Microsoft. Support open standards and sign the petition.

Nature has announced that it cannot accept OOXML documents:

We currently cannot accept files saved in Microsoft Office 2007 formats. Equations and special characters (for example, Greek letters) cannot be edited and are incompatible with Nature’s own editing and typesetting programs.

And so has Science:

Because of changes Microsoft has made in its recent Word release that are incompatible with our internal workflow, which was built around previous versions of the software, Science cannot at present accept any files in the new .docx format produced through Microsoft Word 2007, either for initial submission or for revision. Users of this release of Word should convert these files to a format compatible with Word 2003 or Word for Macintosh 2004 (or, for initial submission, to a PDF file) before submitting to Science.

Though, why anyone would be using Word for scientific document processing is beyond me. Indeed, the scientific community’s support for LaTeX during the ascendancy of word-processing—during the triumph of the Image over the Word—is heroic, and for this we are indebted to them. Even if a latter-day St Benedict of document processing has yet to arrive thus establishing editing with structural markup as the norm.

In an Ars Technica article, a Microsoft spokesperson has some harsh words for IBM:

“Microsoft has determined that it is important to shine a bright light on IBM’s activities that will have a negative impact on the IT industry and customers, including taking concrete steps to prevent customer choice, engaging in hypocrisy, and working against the industry and against customer needs,” said the spokesperson. “Microsoft will continue to be public in identifying the ways that IBM is trying to prevent customer choice.”

Wow.

Black is white. Up is down. Left is Right.

Read the article for an insightful commentary about what might be transpiring.

Some of the problems with storing data in proprietary binaries that motivated the development of ODF are familiar to readers of this blog:

• I want to own my data.
• I do not want access to my data controlled by a single commercial entity.
• I do not want to require that people go out and purchase a particular application in order to read my documents.
• I want my documents to be in a format that has long-term stability and understandability
• I want my documents to be in a format that lends itself to processing by a range of tools, both commercial and free.
• I want my documents to be a format that everyone can understand.
• I want to break out of the cycle of having to constantly upgrade my software every time my vendor decides to change formats on me.

Amen, brother, amen.

While I no longer word-process, if I did ODF would be the way to go. Almost every major word-processor now supports this open standard, and many that don’t are planning to implement it.

One of the advantages touted by Microsoft of their new XML standard is its compatibility with legacy binary formats. Rob shrewdly observes the irony of this:

So now, today, Microsoft is pushing their Office Open XML standard, “old wine in new wine skins”, not so much a new format as a new ploy. What should enrage every thoughtful person is that they are using compatibility with the legacy binary formats as the main selling point of the OOXML format. Think about it. Compatibility with the binary format that they withdrew from the public seven years ago when they cemented their monopoly, is now being touted as their unique advantage. Said differently, Microsoft is selling OOXML as the solution to an interoperability problem that they themselves created and carefully orchestrated.

…

So what prevents Microsoft from doing the same thing again?

Indeed.

Check out Rob’s post. And geeks who are writers (or at least, writers who are geeks abut the technology of writing) will enjoy An Antic Disposition’s informative and insightful discussion of document formats.