Review Policy

Reviewers should refer to the submission guidelines in the Submission Policy, available online at http://www.oasis-open.org/committees/xslt. The tests in version 1.0 of the Conformance Test Suite should fail when the processor is non-conformant with a single "must" provision in scope. (See Submission Policy Guidelines 1-7 for more details on scope.) All accepted tests are intended to test conformance to the Specification on the basis of output. To the extent possible, Committee Reviewers should remove tests whose reference result constitutes interpretation of the Specification, unless the test is cataloged with a Committee-approved gray-area designation.. This will result in equal application of Review Policy criteria by all involved, thus producing a consistent and quality work product. Differences between Submitter and Reviewer output will be examined by the Committee, which will reach consensus to 1) accept the test, 2) reject the test or 3) defer deciding on the test while the issue is forwarded to the W3C for clarification. (See Guideline 7 below for more details.)

Review Procedures

1. At least two Reviewers will check off on each test. Only the assessment of a single member is required for the test to be included in the draft release.

2. Ineligible tests (by definition) should be rejected.
Eligibility is the quality by which a candidate test submitted by a Submitter is judged to determine whether it ends up in the test suite as published by the committee.

3. Eligibility should be judged by the following:

3.1 The accuracy of the test.
Accuracy of a test is determined by a judgement by the Reviewer. Accuracy is defined as the extent to which the test case actually tests what the Submitter states the test case tests. Accuracy is measured against the baseline of the cited parts of the Specification. If it does not match, or only partially matches, the test should be considered inaccurate. This determination is made by the Reviewer's interpretation of the Recommendation, and if necessary, the opinion of the Committee as a whole, and if necessary, the definitive assessment by the W3C Working Group.

3.2 The scope of the test.
See the Submission Policy for a definition of the scope of the test suite.

3.3 The clarity of the test.
Clarity of a test is a determination of whether the aspect being tested is clearly described with the anticipated results acceptably explained.

3.4 The clarity of aspect of the Specification being tested.
The Test Suite aims to test parts of the Specification and errata that aren't vague.

3.5 Should/shall use in the Specification.
This is the same as "must" and "should", discussed in the Submission Policy. The test must clearly address a requirement in the Specification that is a "shall" (or "must") requirement and not a "should" requirement.

3.6 Determination of whether a test is testing a discretionary item.
The Committee has developed a Catalogue of Discretion, which includes a listing of all options given to developers of the technology in the Specification. See the webster for a list of discretionary items ( http://www.oasis-open.org/committees/xslt). Not all discretionary items are testable.

3.7 The simple or compound nature of the test.
Simple and compound tests are described in the Submission Policy.

4. Judge each eligible test through a process.

5. Run each test through multiple processors.
Although there is no reference implementation, the Committee will form consensus on which of the prominent processors to use. The baseline is unanimity of their results, as reduced to infoset-equivalence.

6. Differences between infoset-equivalence of Submitter and Reviewer output will trigger examination by the Committee.

7. The Committee will then reach consensus opinion to accept the test, reject the test, or defer deciding on the test while the issue is forwarded to the W3C for clarification.
A test can be rejected by Reviewers even if all prominent processors give the same result when the test is not a true conformance test. When Reviewers think a test is good, but two or more reliable XSLT processors give different results when they run it, the discrepancy may result from different interpretations of Specification verbiage, processor bugs, or other causes which are not immediately apparent. There are several possible (non-exclusive) actions Reviewers can take:

7.1 Reject the test and update errata and errata exclusion.
The test would then be excluded from the published collection. The Test Suite control file dictating which submitted tests are excluded from the published collection is updated. Furthermore, issuance of an erratum actually gives the Committee a way to include the test case, subject to filtering out at the final stage of "rendering" a suite for a particular processor.

7.2 Reject the test with advice to go to W3C.
In this case, the Submitter thinks the test is accurate and the Committee agrees the test is not accurate and the Recommendation is clear enough that we needn't bother the W3C with an interpretation issue. Rejection requires consensus of the Committee. This scenario begins when the Submitter looks at the Committee report and sees that a particular case submitted was excluded and writes to ask why. The Reviewer will respond to explain. The response includes reference to the W3C's mail alias for questions about the Specification.

7.3 The test case is forwarded to W3C for clarification.
If the above options do not avail, the Committee can forward the test to the W3C for clarification.

7.4 Additionally, the Committee may wish to accommodate external comment from the community at large.

7.5 The Committee will publish a consensus opinion of response to comment with justification from Recommendation (not just precedence of how a processor has acted).

8. During the testing process, Reviewers will do the following:

8.1 A Reviewer will report to the list the hierarchy of tests undertaken for comparison with multiple processors.

8.2 A tally of tests will be tracked on a visible web page for the Committee.

8.3 Reviewers report that a batch of tests in a given hierarchy has been examined, including a summary of findings of tests not to be included in the resulting suite.

8.4 A given hierarchy is not considered complete for a final release until reports from at least two members have been submitted.
A given hierarchy may be included in a draft only after at least one member's report is submitted.

9. During the testing process, the Committee will invite public review:

9.1 An initial suite of a very small set of files will be used to test procedures and scripts and stylesheets.

9.2 The Committee will publish draft work periodically, starting with very small set.

9.3 The Committee will solicit comments on usability of the product.

9.4 The Committee will publish a disposition of comments.

9.5 The Reviewers will continue reviewing test cases until all categories are covered.