11 Oct 2000 $Id: test.xml,v 1.4 2001/01/10 10:28:31 ndw Exp $ 1.2 11 Oct 2000 ndw Documented significant changes to the conduits. 1.1 15 Aug 2000 ndw Fixed broken internal links. 1.0 15 Aug 2000 ndw First public release. 0.4 15 Aug 2000 ndw Added SyncXmlTD and SyncXmlMB 0.3 14 Aug 2000 ndw Added link table 0.2 14 Aug 2000 ndw A few revisions; spellcheck 0.1 08 Aug 2000 ndw Initial draft NormanWalsh Staff Engineer Sun Microsystems, XML Technology Center Norman Walsh, a Staff Engineer in Sun's XML Technology Center, is chair of the OASIS DocBook Technical Committee and serves on a number of W3C Working Groups, including the XML Core, XSL, and XML Schema WGs. 2000 Sun Microsystems, Inc. This article was updated on 11 Oct 2000 to reflect changes made in the version 1.1 conduits. For a quick summary of the changes in version 1.1, see ; for upgrading instructions see . If you're like me, you rely on your Palm™ organizer to keep a semblance of order in your life. Without it, I wouldn't get to meetings on time, or remember to participate in telephone conference callsOk, it's definitely a mixed blessing, I see that, but... :-), or know how to reach my colleagues when I'm on the road. Unfortunately, for all its benefits, I still have some troubles with my Palm. Among them, the fact that I can't sync my Palm address book with other information management tools that are important to me (e.g., my BBDBI have since been informed of SyncBBDB, but I'm still working on other XML-based information management tools. in Emacs) or publish the calendar on the web so that I can share my calendar with my manager and colleagues. Now, I'm sure I could have gone out and found solutions for some of these problems, for example, one of the web calendar syncing tools, but when I hear a problem described that involves open-format information exchange and multiple output formats, one answer springs immediately to mind: XML. So, what I wanted was some way to sync my Palm to my desktop machine using XML so that I could transform the XML into other formats and sync it with other formats.

This was very much a personal project, so the amount of time and energy that I had to spend on it was directly influenced by the number and priority of other projects that I had going on. Initially I looked into the possibility of reverse engineering the PDB file formats that the Palm Desktop application created when syncing my Palm. It didn't take me long to abandon that course. I'm sure it could be done, but how tedious! Using the gcc-based conduit tools, or buying one of the commercial conduit toolkits, crossed my mind. But that looked like a pretty steep learning curve, too. Next, I found the pilot-link package. This package included pilot-file, which produced a much more readable dump of a Palm database. Armed with this dump, I was able to reverse engineer the Datebook format well enough to produce an XML version. But that only solved half the problem because I couldn't see any way to rebuild the file header information that would be necessary to sync the database back into my Palm. Besides, when I looked at the pilot-file dump of the Addressbook, I was reminded of just how tedious it had been to reverse engineer the Datebook. Looking at the source for pilot-xfer convinced me that everything I needed was in there. The next logical step was to hack pilot-xfer so that it read/wrote XML instead of PDBs. That was back sometime in January or February. Before I got around to hacking on pilot-xfer, I started running Linux on my desktop and discovered PilotManager. PilotManager is essentially a Perl/TK interface wrapped around the pilot-link tools that provides a complete Palm syncing tool for Unix. More importantly, it provides an extensible framework for writing conduits of your own.

shows the PilotManager application after a sync operation. As you can see, I made a couple of changes to the Addressbook on the desktop (specifically by editing .xmlAddr) and updated my schedule on the Pilot. These changes were synced appropriately.

I should note at this point that the PilotManager application runs on Linux, Solaris, and presumably other Unixes, but does not run on Windows, as far as I know. The code that underlies PilotManager: Perl, Perl/Tk, and pilot-link have all been ported, so it's theoretically a simple matter of porting, but I haven't tried it.

The PilotManager application includes several conduits. I modeled the SyncXml Conduits off the standard SyncAB conduit. In simplest terms, the PilotManager conduit interface is a set of hashes. A sync operation reads information from the Palm and returns each record as a hash. Appropriately constructed hashes can be passed back to update the Palm. If the conduit can provide enough information about each record (its fields and what fields constitute a match, etc.), as the Addressbook and Datebook conduits can, PilotManager takes care of most of the bookkeeping for us.

If you have been using the version 1.0 SyncXML Conduits, you are strongly encouraged to upgrade. In addition to a few new features, the version 1.1 conduits contain several bug fixes. Note, in particular, that the version 1.0 SyncXmlTB conduit is fundamentally broken; it does not preserve the due date of Todo items across a sync! Before you upgrade the conduits, make sure that you make a complete backup of your Palm using the Backup conduit or whatever method you normally use for full backups. This will allow you to recover in case of catastrophic error. Perform a sync operation using the version 1.0 SyncXml Conduits. Examine the configuration of each conduit and record the location of the XML databases. Generally, these will be in ~/.xmlAddr, ~/.xmlDate, etc., but the conduits may have been configured differently. Download the SyncXml Conduits 1.1 package and unpack it somewhere. Install the version 1.1 conduits: Replace the version 1.0 SyncXml{AB,DB,MB,TB}.pm files on your system with the version 1.1 files of the same name Install the new file SyncXml.pm in the same location. Install the new DTDs as well. Now delete the XML databases. In order to do this successfully, you must delete both the XML files (stored at the locations you recorded a moment ago) and the PilotManager binary databases. The binary databases are usually ~/.pilotmgr/SyncXmlAB/addr.db, ~/.pilotmgr/SyncXmlDB/date.db, ~/.pilotmgr/SyncXmlMB/memo.db, and ~/.pilotmgr/SyncXmlTB/todo.db. Naturally, if you configured PilotManager to use another directory, they will be elsewhere. Make sure you delete both the XML files and the binary files before you continue. Do not delete the SyncXml{AB,DB,MB,TB} directories. As near as I can tell, PilotManager only creates them when a new conduit is added. Now restart PilotManager and re-sync using the SyncXML Conduits. Because the local databases do not exist, the conduits will copy all of the data off your Palm and rebuild the local databases. The version 1.1 conduits have several new configuration options; you may want to examine the configuration of each new conduit.

Before you begin, make sure that you make a complete backup of your Palm using whatever method you normally use for full backups. This will allow you to recover in case of catastrophic error. If you have not already done so, install PilotManager and make sure that it's working. This will guarantee that you have appropriate versions of Perl and other supporting tools in place. Get the Perl XML::Parser and XML::DOM modules from CPAN and install them, if you do not already have them installed. Download the SyncXml Conduits package and unpack it somewhere. Install the conduits by placing the SyncXml.pm and SyncXml{AB,DB,MB,TB}.pm files where PilotManager will find them. Exactly where that is depends on how you've installed PilotManager. On my system, that would be in /usr/lib/perl5/PilotMgr or in ~/.pilotmgr. Install the DTDs by placing them where the SyncXml conduits will find them. I recommend that you install them in lib/schema under wherever you installed the conduits, but you can also put them directly in the conduits directory or in ~/.pilotmgr/SyncXml{AB,DB,MB,TB}. When all the installation is complete, run pilot-manager. It will announce the discovery of the new conduits. Under the File/Properties... menu, add the new conduits to the list of active conduits and they'll get used the next time you sync.

The SyncXmlAB conduit syncs your Palm AddressBook with your desktop.

Configuring SyncXmlAB reveals the panel shown in :

Enable user fields If this option is enabled, the Notes field of each Addressbook record is treated as structured text according to the conventions discussed in . XML file: Identifies the location where the XML Addressbook file is to be stored. This file is used to sync with the Palm. Changes to this file will be reflected in the Palm at the next sync, and changes in the Palm will be written back to this file. Validate: Identifies the location of the DTD to which the XML Addressbook file will refer. During first-time initialization, the conduit attempts to locate the DTD. If you subsequently move the DTD, or if you have installed it somewhere that the conduit does not look, you will have to update this value; otherwise, it's unlikely that you'll ever need to change this.

Syncing your Palm with the SyncXmlAB conduit, produces a single XML file, by default ~/.xmlAddr. This file contains the complete Addressbook database. The PilotManager also keeps a binary representation of the database in ~/.pilotmgr/SyncXmlAB/addr.db. If you want to delete the XML address book and resync from the Palm, delete both files.

The root element of an Addressbook is addressbook. It identifies the version number of the XML file format and the name of the user. Attempts to sync the wrong user will generate a warning. If a future version of the conduit uses a different version number, the sync will abort so that you don't wind up with a horribly corrupted database.

The Palm stores categories, field names, and phone number labels only once in the database, essentially as arrays. Each record in the database refers to the field name by offset into the appropriate array. Each of these label arrays is stored at the top of the XML file. If you change these labels, the effects will be global within the addressbook. Modify these fields with some caution because the conduit makes no effort to assure that the changes you've made are reasonable.

Following the categories and other fields, each record appears in an <address> element. Each address is a fairly straightforward duplicate of the record from the Addressbook database. There are only a few issues to consider.

Internally, the Palm maintains a unique ID value for each record. The Palm IDs are not stored in the XML file because there would be no way to generate a new one on the desktop in order to add a record to the XML address book. Instead, a mapping is maintained between the XML ID values and the Palm unique IDs. To add a new address to the XML file, simply construct a new XML ID value that is unique within the XML address book. You do not need to consider the Palm unique IDs.

The label value on each phone number is expected to match one of the labels in the top-level <phones> list. The sync will fail if it does not. The phone number identified as primary will be the one displayed on the address book summary screen.

Labels on the custom fields are just a convenience for stylesheets and other processing tools. The fields are actually inserted into each record in the order presented, without regard to the labels.

The category value is expected to match one of the labels in the top-level <categories> list. The sync will fail if it does not.

The SyncXmlDB conduit syncs your Palm DateBook with your desktop.

Configuring SyncXmlDB reveals the panel shown in :

Enable user fields If this option is enabled, the Notes field of each Addressbook record is treated as structured text according to the conventions discussed in . Write Calendar If this option is enabled, the Calendar file will be written. The Calendar file is a write-only, linear representation of the Palm Datebook. Enable DateBk4 extensions If this option is enabled, the conduit will attempt to decode the additional information stored in each appointment by the DateBk4 application. See . XML file: Identifies the location where the XML Datebook file is to be stored. This file is used to sync with the Palm. Changes to this file will be reflected in the Palm at the next sync, and changes in the Palm will be written back to this file. Val. XML: Identifies the location of the DTD to which the XML Datebook file will refer. During first-time initialization, the conduit attempts to locate the DTD. If you subsequently move the DTD, or if you have installed it somewhere that the conduit does not look, you will have to update this value; otherwise, it's unlikely that you'll ever need to change this. Calendar: Identifies the location where the Calendar will be written, if the Write Calendar option is enabled. This file is write-only, it is updated by a sync operation, but is not consulted. Any local changes made to this file will be lost at each sync. Val. Cal.: Identifies the location of the DTD to which the XML Calendar file will refer. During first-time initialization, the conduit attempts to locate the DTD. If you subsequently move the DTD, or if you have installed it somewhere that the conduit does not look, you will have to update this value; otherwise, it's unlikely that you'll ever need to change this. Cal Weeks: Identifies the number of weeks (starting with the first week of the current month) that should be copied to the Calendar file. Use dummy days If this option is enabled, each week in the Calendar file will contain exactly seven days. So, for example, if the first of the month occurs on a Wednesday, "dummy days" will be inserted before the first to pad the week to seven days. Similarly, the last week in each month may be padded. Dummy days are only inserted in the Calendar file (not the Datebook XML file). They may simplify the task of writing stylesheets and other software designed to display the calendar.

Syncing your Palm with the SyncXmlDB conduit, produces one or two XML files. It always produces an XML version of the date book, by default ~/.xmlDate. It may also produce a calendar, by default ~/.xmlCalendar. The XML version of the date book contains the complete Datebook database. The PilotManager also keeps a binary representation of the database in ~/.pilotmgr/SyncXmlDB/date.db. If you want to delete the XML date book and resync from the Palm, delete both files. The date book is a compact representation of your schedule. Each entry in your schedule appears once in the date book, with start and end dates, repeating information, and exceptions all represented compactly as properties of the event. The events do not even appear in chronological order, necessarily, with in the date book. While this compact representation is useful for syncing, it is difficult to produce your typical month at a glance or other traditional calendar view from it. Events must be sorted into proper chronological order, repeating events must be generated (with exceptions), etc. In order to make the problem of constructing a calendar view more tractable for stylesheets, the conduit can also produce a calendar file. In this file, every month, week, and day is represented in chronological order. The XSL stylesheet, calendar.xsl, distributed with the conduits demonstrates how this file may be transformed.

The root element of a Datebook is datebook. It identifies the version number of the XML file format, the name of the user, the day of the week that the user identifies as the start of the week, and the date that that the file was produced. Attempts to sync the wrong user will generate a warning. If a future version of the conduit uses a different version number, the sync will abort so that you don't wind up with a horribly corrupted database.

Each record appears in an <appointment> element. Each appointment is a fairly straightforward duplicate of the record from the Addressbook database. There are only a few issues to consider.

Internally, the Palm maintains a unique ID value for each record. The Palm IDs are not stored in the XML file because there would be no way to generate a new one on the desktop in order to add a record to the XML date book. Instead, a mapping is maintained between the XML ID values and the Palm unique IDs. To add a new appointment to the XML file, simply construct a new XML ID value that is unique within the XML date book. You do not need to consider the Palm unique IDs.

The builtin date book on the Palm (or at least, on some of the Palm devices) does not support categories on appointments. But in fact the database does contain them. Some applications allow you to edit the category values on the Palm. This is reflected in a category on each appointment. The category value is expected to match one of the labels in the top-level <categories> list. The sync will fail if it does not.

The DateBk4 application from Pimlico Software adds several useful features to the Palm datebook. In order to maintain complete compatibility with the built-in datebook database, these additional features are encoded in the Note field of each appointment. If you enable DateBk4 extensions, this information is decoded in the XML Datebook and Calendar files. This allows you to observe the icons, time zones, and other DateBk4 extensions. Note, however, that these are "read only" values in the XML file; if you change them, the changes will not be synced to the Palm.

The SyncXmlMB conduit syncs your Palm Memos with your desktop.

Configuring SyncXmlMB reveals the panel shown in :

XML file: Identifies the location where the XML Memo file is to be stored. This file is used to sync with the Palm. Changes to this file will be reflected in the Palm at the next sync, and changes in the Palm will be written back to this file. Validate: Identifies the location of the DTD to which the XML Memo file will refer. During first-time initialization, the conduit attempts to locate the DTD. If you subsequently move the DTD, or if you have installed it somewhere that the conduit does not look, you will have to update this value; otherwise, it's unlikely that you'll ever need to change this.

Syncing your Palm with the SyncXmlMB conduit, produces a single XML file, by default ~/.xmlMemo. This file contains the complete Memo database. The PilotManager also keeps a binary representation of the database in ~/.pilotmgr/SyncXmlMB/memo.db. If you want to delete the XML Memo list and resync from the Palm, delete both files.

The root element of an Memo list is memobook. It identifies the version number of the XML file format and the name of the user. Attempts to sync the wrong user will generate a warning. If a future version of the conduit uses a different version number, the sync will abort so that you don't wind up with a horribly corrupted database.

The Palm stores categories only once in the database, essentially as an array. Each record in the database refers to the category name by offset into this single array. The category label array is stored at the top of the XML file. If you change these labels, the effects will be global within the Memo list. Modify these fields with some caution because the conduit makes no effort to assure that the changes you've made are reasonable.

Following the categories, each record appears in a <memo> element. Each memo is a fairly straightforward duplicate of the record from the Memo database. There are only a few issues to consider.

Internally, the Palm maintains a unique ID value for each record. The Palm IDs are not stored in the XML file because there would be no way to generate a new one on the desktop in order to add a record to the XML memo list. Instead, a mapping is maintained between the XML ID values and the Palm unique IDs. To add a new memo to the XML file, simply construct a new XML ID value that is unique within the XML memobook. You do not need to consider the Palm unique IDs.

The category value is expected to match one of the labels in the top-level <categories> list. The sync will fail if it does not.

The SyncXmlTB conduit syncs your Palm Todo list with your desktop.

Configuring SyncXmlTB reveals the panel shown in :

Enable user fields If this option is enabled, the Notes field of each Todo record is treated as structured text according to the conventions discussed in . Enable DateBk4 extensions If this option is enabled, the conduit will attempt to decode the additional information stored in each appointment by the DateBk4 application. See . XML file: Identifies the location where the XML Memo file is to be stored. This file is used to sync with the Palm. Changes to this file will be reflected in the Palm at the next sync, and changes in the Palm will be written back to this file. Validate: Identifies the location of the DTD to which the XML Todo file will refer. During first-time initialization, the conduit attempts to locate the DTD. If you subsequently move the DTD, or if you have installed it somewhere that the conduit does not look, you will have to update this value; otherwise, it's unlikely that you'll ever need to change this.

Syncing your Palm with the SyncXmlTB conduit, produces a single XML file, by default ~/.xmlTodo. This file contains the complete Todo database. The PilotManager also keeps a binary representation of the database in ~/.pilotmgr/SyncXmlTB/todo.db. If you want to delete the XML Todo list and resync from the Palm, delete both files.

The root element of an Todo list is todobook. It identifies the version number of the XML file format and the name of the user. Attempts to sync the wrong user will generate a warning. If a future version of the conduit uses a different version number, the sync will abort so that you don't wind up with a horribly corrupted database.

The Palm stores categories only once in the database, essentially as an array. Each record in the database refers to the category name by offset into this single array. The category label array is stored at the top of the XML file. If you change these labels, the effects will be global within the Todo list. Modify these fields with some caution because the conduit makes no effort to assure that the changes you've made are reasonable.

Following the categories, each record appears in a <todo> element. Each memo is a fairly straightforward duplicate of the record from the Memo database. There are only a few issues to consider.

Internally, the Palm maintains a unique ID value for each record. The Palm IDs are not stored in the XML file because there would be no way to generate a new one on the desktop in order to add a record to the XML todo list. Instead, a mapping is maintained between the XML ID values and the Palm unique IDs. To add a new todo to the XML file, simply construct a new XML ID value that is unique within the XML todobook. You do not need to consider the Palm unique IDs.

The category value is expected to match one of the labels in the top-level <categories> list. The sync will fail if it does not.

The DateBk4 application from Pimlico Software adds several useful features to the Palm todo list. In order to maintain complete compatibility with the built-in todo database, these additional features are encoded in the Note field of each todo entry. If you enable DateBk4 extensions, this information is decoded in the XML Todo file. This allows you to observe the icons, alarm times, and other DateBk4 extensions. Note, however, that these are "read only" values in the XML file; if you change them, the changes will not be synced to the Palm.

In order to store additional information in my Palm in a quasi-structured fashion, I've adopted a convention for the 'Note' field. If the Note field contains the line user-fields:, then all of the following lines that begin field-name: are user fields. If you enable user fields, these are parsed into the XML. For example, a DateBook entry with the following Note: user-fields: loc: Bay Area url: http://.../ type: business other notes will be parsed into the XML as: Bay Area http://.../ business user-fields: loc: Bay Area url: http://.../ type: business other notes ]]> Note that the user-fields are write-only in the database. If you want to change them on the desktop side, edit the value of the Note field directly. This is necessary because some applications, such as the Pimlico Software DateBk* applications, also use the notes field in semantically important ways. Leading and trailing whitespace associated with removing the user-fields from the Note, often resulted in apparent changes that caused the Palm to appear out-of-sync in ways that made the syncing unreliable.

The enhanced DateBook application included in the HandSpring Visor™ is actually a licensed subset of Pimlico's DateBk3 application. This application, and its successor DateBk4, use structured values at the very beginning of the Notes field to hold information about semantics not included in the database proper (floating events, time zones, etc.). On the Palm, these values are invisible, but if you look in the XML database, you'll see them at the beginning of the Note of each augmented appointment. In order to preserve these characteristics, it is vital that you do not edit or delete these values, or insert notes before them.

One of the benefits I touted at the beginning of this article was multiple output formats, for example the ability to publish my schedule for others to see. Included in the SyncXml Conduits package is an XSL stylesheet that produces an HTML calendar from the .xmlCalendar file. After syncing, I run: xt .xmlCalendar /projects/pilot/conduits/calendar.xsl sched.html This produces a calendar like the one shown in .

The calendar.xsl stylesheet supports several user fields: The class and type user fields determine foreground and background colors on the month-view of the calendar. (Specifically, appointments with an undecided class are highlighted and appointments with types of business, personal, and holiday are colored differently. If the user field url is set, then the appointment will include a link to that URL. If the user field loc is set, then the location will appear parenthetically after the description. If the user field locurl is set, then the location will be a hypertext link to that URL. In addition, a few user fields are usually ignored. You can make them significant by setting the stylesheet parameter ignoreuserfields to 0. With these additional fields: Weekly appointments with the user field type set to personal are not displayed. (This prevents my weekly reminder to take out the garbage on Monday mornings from appearing on my web calendar!) Appointments that do not have a user field of type are suppressed. This effectively makes untyped appointments personal by default.

I'm by no means done hacking my XML address book and schedule. I still have to write the XML BBDB/Palm merging tool and I may decided to write a stylesheet for putting my address book online. I hope some of you find the SyncXml Conduits useful and feel inspired to introduce XML into your applications. Let me know!

The following significant changes were introduced in version 1.1 of the SyncXml conduits: Fixed a significant bug in SyncXmlTB. The version 1.0 conduit does not properly sync dated Todo items. The due date is lost! Fixed a significant bug in SyncXmlDB. The version 1.0 conduit does not properly sync items that repeat MonthlyByDay or MonthlyByDate. Fixed a significant bug in SyncXmlAB. The version 1.0 conduit sometimes syncs the "show in list" phone number incorrectly. Improved the encoding of international and XML markup characters. There were a few places in the version 1.0 conduits where markup characters were not escaped, leading to XML files that could not be loaded. Added support for DateBk4 extensions to the Datebook and Todo databases. Provided user-interface configuration for previously "hidden" preferences. Added support for categories in the Datebook. Added sortByCompany and country attributes to the Addressbook. Added sortByAlpha attribute to the Memo file. Added sortByPriority attribute to the Todo file. Added support for DateBk4 icons to the Calendar stylesheet. The distribution also includes a simple-minded Perl script that can construct XPM icon files from the DateBk4 icon memo.