Solid Conformance Test Suite
In this post we describe an approach and proposed new architecture for a conformance test suite for the Solid Protocol and associated specifications.
Interoperability is one of the core underlying requirements for a thriving open Solid ecosystem. This is achieved through the Solid Protocol and the associated specifications. The Solid server, upon which all new and exciting interoperable apps will depend, is one of the central components of the Solid ecosystem. Once a Solid server conforms with the Solid specifications we can be confident that it will interoperate as expected in the distributed ecosystem. But how do we know that a server is conformant with the Solid specifications?
Enter the need for a high quality conformance test suite. Test suites drive interoperability and are a key part of making sure standards are implemented correctly and consistently.
Like everything else on the web, we did not start this from a blank page. Much work has been done to date on investigating how to approach a test suite for the Solid protocol and there are even two initial implementations that provide some tests today. We have taken many learnings from these efforts and would like to acknowledge and thank Kjetil Kjernsmo, Michiel de Jong and everybody else who committed to the repositories and issues, for their open source contributions and work on Solid test suites.
Initially we spent some time considering what would be required of a good test suite. The following sections outline the main requirements we ended up with. The existing implementations don’t meet these requirements so we would like to close that gap by taking a different approach to the test suite. We found it is relatively easy to take the existing tests from those implementations and make them available in the proposed implementation.
Traceability to specifications
It must be possible to automate the linking of significant units of information in specifications, such as requirements as well as security, privacy, accessibility and internationalisation considerations. In order to achieve this, we support the proposals on embedding RDFa in specifications and test suite description documents. This has been done with other specifications and is documented by Sarven Capadisli in Linked Specifications, Test Suites, and Implementation Reports.
Declarative and easy to understand
If tests are difficult to understand, overly complex, or require high levels of expertise then the risk of incorrect tests will be high. Server implementations written to pass an incorrect test suite may drive interoperability on incorrect behaviour, which would have widespread, negative consequences for the entire Solid ecosystem. Therefore we should strive to make tests declarative and easy to understand to avoid incorrect tests in the test suite. This has the added benefit of making the test suite understandable to a larger number of people in the community. This should lead to both a larger number of contributors and a larger number of reviewers.
Specification requirement levels
The Solid specifications contain requirements of different levels using keywords as defined by BCP14. The test suite must be capable of determining the combinations of tests to execute based on the specifications and the capabilities of a given Solid server. This ensures that tests for optional features are only executed against servers that claim they implement those features.
Extensive work has already been carried out on reporting formats by a working group and the resulting language, EARL, should be used for reports from conformance test suites. This will ensure that the data in the reports can be used by many different tools.
The reports generated by the test suite should provide high level pass/fail results, but it should also be possible to drill down to fine levels of detail to determine the exact tests that failed and why they failed. Of course, it must be easy to automatically make those reports available on the web.
Developers writing Solid servers will benefit from detailed information associated with failing tests; therefore the request and response payloads between the client and server for each test executed should be accessible by the developer. This will make it easier for developers to debug their server code.
The test suite needs to evolve with the specifications as they mature. Therefore it is imperative that the system be easily maintainable, as it is likely to be modified and maintained by many different people over the years.
The system used to execute the tests should be decoupled from the tests themselves. This will ensure that most of the complexity can be encapsulated in a test harness while the tests themselves can be as simple as possible.
Obviously the test harness must be well architected and of a high quality to avoid the problems of brittle software and technical debt. This will ensure the test harness remains maintainable by the community for years to come. We propose that;
- The architecture for the test suite should be reviewed by the specification team and the community
- The code should conform to a published set of coding standards
- The code for the test suite should be accompanied by automated tests providing 100% branch coverage
As the Solid specifications evolve and as the Solid ecosystem matures there will be a need to add many more tests and different types of tests. It should be simple to extend the test suite with new tests for existing specifications and for new specifications without having to add more code to the test harness. This will reduce the risk of breaking existing tests and will enable more people in the community to contribute to the test suite.
Flexibility and ease of use
Different types of people will want to use the test suite for different purposes. For example, Solid server developers may want to include it in their continuous integration pipeline. This type of developer may want the suite to execute as fast as possible so will benefit from the ability to run many tests in parallel. Those deciding which Solid server to use may want to execute the test suite from the command line when experimenting with a server. Others may want to provide web pages with the results of running the test suite against multiple servers.
The test suite will also be used in many different environments such as different cloud environments, or an individual developer’s workstation. With all these considerations in mind the test suite should make minimal assumptions about its environment and should be easy to configure to use in many different ways.
The following is an illustration of the proposed high level architecture of the test suite.
The test harness controls the overall execution of a test suite. It is responsible for loading the test suite, locating the tests, creating and controlling test executors and generating test suite compliance reports.
The harness will provide different interfaces to the test suite such as a REST API and a command line interface, and will be platform agnostic.
The test executor is responsible for executing tests against a Solid server. The harness can create multiple test executors where each is responsible for executing a set of tests. Test executors do not need to run on the same host as the test harness.
Test suite description document
This is an RDF document containing metadata about the test suite. The metadata includes;
- Prerequisites (e.g. dependencies)
- Requirements level from the specification (e.g., MUST, SHOULD, MAY, etc.)
- References to the appropriate section in the specification
- References to the actual tests
The test suite at any point in time may not cover all of the tests required for the specifications. The test harness will use the linkage between the test suite document and the RDF in the specifications to determine how much of the specifications are covered by the test suite.
The test cases will be contained in a repository and may be grouped in various ways to enable logical sets of tests to be grouped cohesively.
Conformance reports will be generated using EARL thereby making them available for consumption by many different tools.
The following is an illustration of the component architecture for the test harness and executor.
We have already mentioned that we believe tests should be declarative and easy to understand. We are proposing to use Karate as the underlying test framework and test language. Tests are written in Gherkin, but with built-in support for HTTP interactions. We will add RDF format conversions and assertions. Gherkin has a successful history with a rich set of integrations and the Karate extensions are well suited to the needs of a Solid conformance test suite.
The following is a generic Karate example:
Now for an example of Solid tests in Karate:
There is even the possibility of creating the Solid tests in RDF:
We have already completed a proof of concept using the proposed architecture. The test harness was written in Java and dockerised, while all of the tests were written in Karate. We are now ready to bring the proposal to the Solid specification team and proceed with a production quality implementation.
Of course this will be an open source contribution to the Solid ecosystem and we would be delighted to get feedback and contributions from members of the community. The public repository will contain all of the source code and links to all the relevant material.
Thank you for taking the time to read this post and we look forward to working with you to ensure we all enjoy an interoperable Solid ecosystem.