[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Subject: Final Release of Review & Submission Policy docs
All, Please find first release final Submission Policy and Review Policy documents attached. Before these documents are distributed outside the workgroup, it will be necessary to make sure that copies are placed at www.oasis-open.org/committees/xslt; otherwise, the references to this url in both documents will be in error. (Note: both the .html documents and the .css document must be transferred to the website in the same directory in order for the files to work properly.) If this is not the right directory, or if the directory changes, please notify me so these documents can be updated to the correct url. Many thanks. It has been a pleasure to serve you. CrisTitle: Submission Policy
Submission Policy
Introduction
Since the World Wide Web relies on technological interoperability, the need arises, both for vendors and for product users, for testing product conformance to the W3C specifications. The objective of the OASIS XSLT/XPath Conformance Committee ("Committee") is to develop a test suite for use in assessing the conformance of XSLT processors to the technical specifications contained in the Recommendations of the W3C (called the "Specification" in this document). The full text of this Submission Policy and its companion, the Review Policy, are available online at www.oasis-open.org/committees/xslt. The Committee welcomes submissions of test cases from all vendors and other interested parties. Tests will be considered for inclusion in its test suite (according to the Review Policy) on a case-by-case basis. The Committee will will work toward thorough coverage by accumulating submitted tests. The quality and comprehensiveness of these test submissions will determine how robust the test suite will be. The Committee encourages all test submissions. The purpose of these Guidelines is to inform Submitters of what the test suite is meant to do and which tests are more likely to be included in the test catalog, given these design criteria. The Committee also encourages Submitters to prepare follow-up submissions, including repairs to individual tests and significant test expansions.Submission Guidelines
The first seven Guidelines define the scope of the test suite. 1. Submitters' tests should test only a single simple requirement in the Specification.Review Policy
Reviewers should refer to the submission guidelines in the Submission Policy, available online at www.oasis-open.org/committees/xslt. The tests in version 1.0 of the Conformance Test Suite should fail when the processor is non-conformant with a single "must" provision in scope. (See Submission Policy Guidelines 1-7 for more details on scope.) All accepted tests are intended to test conformance to the Specification on the basis of output. To the extent possible, Committee Reviewers should remove tests whose reference result constitutes interpretation of the Specification, unless the test is cataloged with a Committee-approved gray-area designation.. This will result in equal application of Review Policy criteria by all involved, thus producing a consistent and quality work product. Differences between Submitter and Reviewer output will be examined by the Committee, which will reach consensus to 1) accept the test, 2) reject the test or 3) defer deciding on the test while the issue is forwarded to the W3C for clarification. (See Guideline 7 below for more details.)Review Procedures
1. At least two Reviewers will check off on each test. Only the assessment of a single member is required for the test to be included in the draft release. 2. Ineligible tests (by definition) should be rejected.3.1 The accuracy of the test.
Accuracy of a test is determined by a judgement by the Reviewer. Accuracy is defined
as the extent to which the test case actually tests what the Submitter states the test
case tests. Accuracy is measured against the baseline of the cited parts of the
Specification. If it does not match, or only partially matches, the test should
be considered inaccurate.
This determination is made by the Reviewer's interpretation of the Recommendation, and
if necessary, the opinion of the Committee as a whole, and if necessary, the definitive
assessment by the W3C Working Group.
3.2 The scope of the test.
See the Submission Policy for a definition of the scope of the test suite.
3.3 The clarity of the test.
Clarity of a test is a determination of whether the aspect being tested is clearly
described with the anticipated results acceptably explained.
3.4 The clarity of aspect of the Specification being tested.
The Test Suite aims to test parts of the Specification and errata that aren't vague.
3.5 Should/shall use in the Specification.
This is the same as "must" and "should", discussed in the Submission Policy. The test
must clearly address a requirement in the Specification that is a "shall" (or "must") requirement and
not a "should" requirement.
3.6 Determination of whether a test is testing a discretionary item.
The Committee has developed a Catalogue of Discretion, which includes
a listing of all options given to developers of the technology in the Specification. See the
webster for a list of discretionary items (
www.oasis-open.org/committees/xslt). Not all discretionary items are testable.
3.7 The simple or compound nature of the test.
Simple and compound tests are described in the Submission Policy.
7.1 Reject the test and update errata and errata exclusion.
The test would then be excluded from the published collection. The Test Suite
control file dictating which submitted tests are excluded from the published collection is updated.
Furthermore, issuance of an erratum actually gives the Committee a way to include the test case,
subject to filtering out at the final stage of "rendering" a suite for a particular processor.
7.2 Reject the test with advice to go to W3C.
In this case, the Submitter thinks the test is accurate and the Committee agrees
the test is not accurate and the Recommendation is clear enough that we needn't bother the W3C with
an interpretation issue. Rejection requires consensus of the Committee.
This scenario begins when the Submitter looks at the Committee report and sees that a particular
case submitted was excluded and writes to ask why. The Reviewer will respond to explain. The
response includes reference to the W3C's mail alias for questions about the Specification.
7.3 The test case is forwarded to W3C for clarification.
If the above options do not avail, the Committee can forward the test to the W3C
for clarification.
7.4 Additionally, the Committee may wish to accommodate external comment from the community at large.
7.5 The Committee will publish a consensus opinion of response to comment with justification from Recommendation (not just precedence of how a processor has acted).
8. During the testing process, Reviewers will do the following:8.1 A Reviewer will report to the list the hierarchy of tests undertaken for comparison with multiple processors.
8.2 A tally of tests will be tracked on a visible web page for the Committee.
8.3 Reviewers report that a batch of tests in a given hierarchy has been examined, including a summary of findings of tests not to be included in the resulting suite.
8.4 A given hierarchy is not considered complete for a final
release until reports from at least two members have been
submitted.
A given hierarchy may be included in a draft only
after at least one member's report is submitted.
9.1 An initial suite of a very small set of files will be used to test procedures and scripts and stylesheets.
9.2 The Committee will publish draft work periodically, starting with very small set.
9.3 The Committee will solicit comments on usability of the product.
9.4 The Committee will publish a disposition of comments.
9.5 The Reviewers will continue reviewing test cases until all categories are covered.
body {background-color: #FFFFFF; font-family: "Times New Roman"; font-size: 12pt; color: #000000; margin: 0px; } a:link {color: #0000FF } a:visited {color: #3333FF } a:hover {color: #FFCC00 } a:active {color: #FF0000 } p.title {color: #000000; padding-top: 50px; padding-bottom: 10px; font-family: "Times New Roman"; font-size: 18pt; text-align: center } p.subtitle {color: #000000; padding-top: 18px; padding-bottom: 0px; font-family: "Times New Roman"; font-size: 14pt; text-align: left } span.section {color: #000000; font-family: "Times New Roman"; font-size: 12pt; font-weight: bold; text-align: left } p.subsection {color: #000000; padding-top: 0px; padding-bottom: 0px; padding-left: 20px; font-family: "Times New Roman"; font-size: 12pt; font-weight: normal; font-style: italic; text-align: left } span.text {color: #000000; font-family: "Times New Roman"; font-size: 12pt; font-weight: normal; font-style: normal; text-align: left }
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Powered by eList eXpress LLC