Submission Policy
Introduction
Since the World Wide Web relies on technological interoperability, the need arises, both for
vendors and for product users, for testing product conformance to the W3C specifications. The
objective of the OASIS XSLT/XPath Conformance Committee ("Committee") is to develop a test suite
for use in assessing the conformance of XSLT processors to the technical specifications contained
in the Recommendations of the W3C (called the "Specification" in this document). The full text of
this Submission Policy and its companion, the Review Policy, are available online at
http://www.oasis-open.org/committees/xslt. The
Committee welcomes submissions of test cases from all vendors and other interested parties. Tests
will be considered for inclusion in its test suite (according to the Review Policy) on a
case-by-case basis. The Committee will will work toward thorough coverage by accumulating
submitted tests. The quality and comprehensiveness of these test submissions will determine how
robust the test suite will be.
The Committee encourages all test submissions. The purpose of these Guidelines is to inform
Submitters of what the test suite is meant to do and which tests are more likely to be included
in the test catalog, given these design criteria. The Committee also encourages Submitters to
prepare follow-up submissions, including repairs to individual tests and significant test expansions.
Submission Guidelines
The first seven Guidelines define the scope of the test suite.
1. Submitters' tests should test only a single simple requirement in the
Specification.
In a comprehensive test suite, each testable assertion in the Specification should be tested
independently of each other assertion, to the extent possible. These assertions are often the
result of taking one or two key sentences from the Specification, but considering effects and
conditions described at various other places in the Specification. Citations (see Guideline 3)
point to portions of text that may not be as precise as testable assertions, while a
purpose statement for the test (see Guideline 2) can say exactly what is being singled out in
that test case.
If a test follows the first Guideline above, one failure points out a singular instance where the
processor does not conform to the Specifications (called "non-conformance" in this document).
Failure of several cases may point out a single failure if one can readily identify the common
element of all the failing cases. One non-conformance may cause the failure of dozens of tests that
involve various invocations of the non-conforming situation.
2. Submitters' tests must be accompanied by a statement of the purpose of the
test.
Each test must have just one purpose, and, ideally, that purpose should be unique
within the test suite. The purpose statement should state the requirement in the Specification,
whether this requirement is found in a simple assertion using one citation pointer to the
Specification or a compound assertion, using two or more citation pointers to statements and
inferences within the Specification. The purpose statement helps Reviewers and test labs to
understand the Submitter's intention in submitting a particular test.
3. Submitters' tests must include at least one citation pointer to the
Specification.
Recommendation citations are in the form of XPath expressions to statements in the Specification.
There should be one citation pointer for each statement in the Specification that, with any other
relevant statements, make up a requirement. Requirements that are found in two (or more) sentences
in the Specification should include two (or more) citation pointers. Submitters may look inside the
file xsltcfg.xml for data about the types of pointers the Committee recognizes. Citations can be in
one or more of the formats. The more detail included, the better.
4. The tests should target specific XSLT/XPath language issues.
The tests should be aimed at the language features and versions that are included in the
Specification. Issues that cause parser errors or that involve other W3C specifications that are
out of scope for the current test suite should not be included. If submitted, the Committee may
not run or include tests involving parser issues or errata of the Specification. Tests aimed at
revealing mistakes on parsing the input or on serializing the output should be excluded. The
method for comparing results will overcome irrelevant differences in serialization.
5. The tests should target "must" provisions of the Specification, not "should"
provisions.
The Specification contains some assertions (or requirements) that are mandatory ("must") and some
that are optional ("should"). For the version 1.0 of the test suite, the Committee is concerned with
"must" requirements. "Should" provisions are the discretion of the implementer. While the Committee
welcomes submissions of all kinds, those testing "should" provisions may not be included in final
test results.
6. A test should target only explicit processor choices, not unspecified
areas of the Specification.
There are areas of the Specification that do not specify what a processor needs to do, so it is
impossible to test for what they actually do. In other areas the processor is given a choice
regarding how it behaves. The remaining areas are unconditional required behaviors.
The suite will differentiate test cases based on the discretionary choice(s) that the Submitter
indicates as relevant, if any. The Reviewers need to know if a test corresponds to a particular
choice made available to the processor. (These will be enumerated in the information included with
the catalogue document model). The completed test suite will test that portion of the Catalog of
Discretion that is deemed "testable" and where a question or two can clearly elicit the choice made
by the developer.
7. Later versions of the test suite may allow a wider range of tests.
Later versions of the test suite may include a broader scope of issues and may choose to include
a wider range of tests.
8. The Committee reserves the right to exclude any test submitted.
Tests submitted to Version 1.0 of the test suite may be rejected if they do not comply with these
guidelines.
Please see the Review Policy for a full description on how the Committee will judge eligibility of a
test (http://www.oasis-open.org/committees/xslt).
9. In those instances where a Submitter has a test or tests within its overall
submission whose creator(s) will be making a separate submission, the
Submitter should filter out those tests so they are not submitted twice.
The Submitter should send the tests it created, plus any tests others created
that are both 1) free and clear for such use and 2) that the Submitter doesn't
believe the Committee will have already.
10. The tests will become public. No royalties will be associated with their
use.
The Committee intends to retain the personal names of Submitters, if provided, so they may get
public credit for their work.