Also see Testing report – Year 1 (deliverable D3.1, published on September 2011).
Towards a generic testing framework
As part of their work on developing Web standards, W3C working groups create test suites to ensure interoperability between implementations as well as to improve the overall quality of the specification itself. This testing effort has long been done on a working group per working group basis, using ad-hoc testing frameworks, servers and methods to measure the coverage of a specification in terms of tests. With the advent of Web applications in general, and mobile Web applications in particular, interoperability has become a fundamental key for the success of the Web as an application platform. This platform typically relies on W3C specifications such as HTML5, CSS, SVG, device APIs, XMLHttpRequest, widgets, etc.
Through MobiWebApp, the W3C proposes a new approach, reviewing common testing requirements to develop a generic testing framework that would be used across working groups. This generic testing framework would also encourage reuse of W3C test suites within other organizations that depend on Web standards, e.g. for certification purpose of widgets runtimes.
W3C Test Server
A typical test case for the Web technologies that are relevant to the mobile Web Application platform consists of a short Web page that uses the functionality under test and, as much as possible and practical, uses only the feature under test. In other words, typical test cases are atomic, checking one particular testable assertion in a given section of a given specification. Depending on the feature under test, test cases may be of different types, for instance:
- JavaScript may be used to test and report the result to the user.
- Reference tests pass when the rendering of the test page matches that of a reference page. In turn, this comparison could be done on an automated basis or on a manual basis.
- DOM tree comparison tests pass when the DOM tree generated by the test page matches that of a reference page. Such tests are useful for parsing tests.
- Self describing tests ask users to indicate whether the test has passed or failed, e.g. « please check that the input field below is disabled ». Ideally, for scalability reasons, such test cases should be avoided when possible.
The role of the testing framework is to run test cases in order and collect results, automatically when possible or manually through user interaction when not. The testing framework sits as a higher level layer on top of test cases. Test cases should not depend on the this upper layer so that the framework may be updated or adapted to other environments without impacting test cases.
A Web server dedicated to serving these test cases was deployed at: http://w3c-test.org/
The diagram below describes how the current setup works.
Practically speaking, test cases are stored in a Mercurial repository (distributed version control system). Whenever the repository is updated, static test files (HTML, CSS, JavaScript) are automatically pushed to the Web server and thus immediately become available. For security reasons, dynamic files (PHP files) are put on hold until someone from W3C team approves the files. Once approved, the PHP files are deployed to the Web server.
Documentation
We even have documentation to get you started. For test writing, you’ll want to read up on using-testharness.js. If you’re creating a test suite, you might find the metadata format description useful, and if you’re managing a test suite as part of the framework, there’s also documentation on preparing a suite for inclusion. Finally, developers wishing to reuse the open data in the framework can find out all they need from our JSON/REST API documentation.
Get Involved!
To get involved or simply follow discussions on the generic testing framework, please join the public-test-infra@w3.org mailing-list (with public archives).
The state of the mobile Web applications landscape (Standards for Web Applications on Mobile – September 2011) lists the specifications that are particularly relevant to the development of mobile Web applications and tries to assess the current coverage for each specification. To contribute test cases, please get in touch with the groups that are responsible for the specifications you are interested in.
