Normalization of ODF RNG files & ODF TC Tools

[Svante Schubert <svante.schubert@gmail.com>]

Dear TC members,

Michael pointed out the normalized (alphabetical sorted) ODF 1.2 RNG files in the OASIS TC SVN repository to be usedÂfor text comparison with the normalized ODF 1.3 RNG files.
For example https://tools.oasis-open.org/version-control/browse/wsvn/office/branches/v1.2/v1.2/OpenDocument-schema-v1.2.rng

As I wasn't sure if this version was directly sorted from the ODF 1.2, I have created a quick similar XSLT sort.

Currently, we have three files that we are using for text comparing:

1. Latest ODF 1.2: OpenDocument-v1.2-os-schema.rng
   http://docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-os-schema.rng
2. A sorted ODF 1.2 RNG file can be found at:
   http://tools.oasis-open.org/version-control/browse/wsvn/office/trunk/v1.2/OpenDocument-schema-v1.2.rng
3. Latest ODF 1.3: OpenDocument-v1.3-cs01-schema.rng
   http://docs.oasis-open.org/office/OpenDocument/v1.3/cs01/schemas/OpenDocument-schema-v1.3.rng

Obviously the sorted ODF 1.2 RNG above was sorted (normalized) by the Java Utility called OdfHistory:
https://gitlab.com/odfplugfest/odfhistory
(to build apply my patch: https://gitlab.com/odfplugfest/odfhistory/-/merge_requests/2)
Run the command line command:
"java -jar <input_RNG> <output_RNG>

To verify that the sorted SVN file has the same feature set as the original ODF 1.2, a very similar XSL transformation was created:
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/xslt/rng/rng2normalization.xslÂÂ

The following steps were done to successfully validate that the SVN file as the same feature set as the original ODF 1.2 schema file:

Step 0: The original ODF 1.2 RNG schema file was sorted by OdfHistory to
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/odf12/rng-normalization/OpenDocument-v1.2-os-schema_sorted_by_odfhistory.rng

Step 1: Before the XSLT was triggered some whitespace noise was manually removed:
Removing 18 empty lines within the ODF 1.2 RNG schema, which would be removed from OdfHistory as well:
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/odf12/rng-normalization/OpenDocument-v1.2-os-schema_original_but_manual_space_adoption.rng

Step 2: Running the XSLT with command line in the xslt-runner project: "mvn clean install"
Creating:
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/target/generated-resources/xml/xslt/OpenDocument-v1.2-os-schema_original_but_manual_space_adoption.xml
This should be similar as the existing:
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/odf12/rng-normalization/OpenDocument-v1.2-os-schema_original_but_manual_space_adoption_XSLT-OUPUT_SORTED.rng

Step 3: Both identical files from Step 2 show a lot of changes to the sorted file by OdfHistory, as the latter is adding the rng: prefix.
Therefore all "<rng:" element prefixes of the file in step 0 created by OdfHistory are being replaced with "<", resulting into the file:
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/odf12/rng-normalization/OpenDocument-v1.2-os-schema_sorted_by_odfhistory_manualAdopted.rng

Step 4: Comparing
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/odf12/rng-normalization/OpenDocument-v1.2-os-schema_sorted_by_odfhistory_manualAdopted.rng
with
https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/xslt-runner/src/test/resources/odf12/rng-normalization/OpenDocument-v1.2-os-schema_original_but_manual_space_adoption_XSLT-OUPUT_SORTED.rng
show only differences in the namespace declarations, which were sorted by OdfHistory as well.

It took some time, but at least it could be shown thatÂOdfHistory worked well for our largest RNG schema file.

Perhaps an older version of OdfHistory has caused the Manifest RNG regular _expression_ issue that Michael had found.

It is necessary that we are adding automated regression tests on our ODF TC GitHub.

@Patrick DurusauÂ

Have we asked alreadyÂfor a TC GitHub? I would volunteer to assist in configuring the GitHub System to speed things up.

We really need to gather our tooling at one central safe place.Â

Do you also have the scripts to generate the content with the ODT specification? What else do we have?

Best regards,

Svante

PS: Next step for any volunteer:

1. Testing all ODF 1.2 RNG files, I have tested alreadyÂbriefly the manifest RNG and the regular _expression_ is no longer corrupted (if at all) by OdfHistory.
2. How does OdfHistory behave with comments (there had been many added to the ODF 1.3 schema), is it still usable?
3. What new features do exist in ODF 1.3 and how does the ODF XML 1.2 to 1.3 changes relate to these features (see the appendix of each new ODF 1.3 spec with referenced JIRA tasks).