Inasmuch as a suite of Schematron tests contains many contexts where a bug in a document will make a Schematron assert fail or a report succeed, it follows that for any new test suite and any reasonably sized but buggy document set, there will straight away be many assert and report messages produced by the tests. When that happens, how can you be sure your Schematron tests all worked as expected? How can you separate the expected results from the unexpected? What’s needed is a way to characterise the Schematron tests before you start as reporting only what they should, no more, and no less.

stf (https://github.com/MenteaXML/stf) is a XProc pipeline that runs a Schematron test suite on test documents (that you create) and winnows out the expected results and report just the unexpected. stf uses a processing instruction (PI) in each of a set of (typically, small) test documents to indicate the test’s expected asserts and reports: the expected results are ignored, and all you see is what’s extra or missing. And when you have no more unexpected results from your test documents, you’re ready to use the Schematron on your real documents.

<?stf?> Processing Instruction

The format of the PI is:

<?stf \s+ ( '#NONE' | ROLE ':' COUNT ( \s+ ROLE ':' COUNT )* ) ?>

where:

stf: PI target
#NONE: No faild assert or successful report expected. Use with ‘go’ tests that should not produce any assert or report messages. If running Schematron on the test produces any asserts or reports, they are reported as an error.
ROLE: Token corresponding to @role value of an assert or a report in the Schematron. Schematron allows @role to be an arbitrary string, but restricting it to a single token makes it easier to deal with the PI using regular expressions rather than having to parse roles that may contain spaces.
COUNT: Integer number of expected occurrences of failed asserts or successful reports with @role value matching ROLE. A mismatch between the expected and actual count is reported as an error. A ROLE starting with # does not have its count checked.
\s: Whitespace character

Examples

<?stf ERROR_FOO:2 ERROR_BAR:1 ?>

A failed assert or successful report with role="ERROR_FOO" is expected twice in the SVRL from the test document, and either with role="ERROR_BAR" is expected once.

<?stf ERROR_FOO:2 #ERROR_BAR:1 ?>

A failed assert or successful report with role="ERROR_FOO" is expected twice in the SVRL, and no assert or report with role="ERROR_BAR" is expected since # precedes ERROR_BAR.

<?stf #NONE ?>

No assert or report are expected for the current document.

Usage

stf requires Ant and Calabash. Calabash (as used here) requires Saxon. All three require Java.

Set the properties in properties.xml to match your local setup.
Write the tests, including a <?stf?> processing instruction in each.
One practice is to use a tests directory containing a go subdirectory for tests that are expected to produce no Schematron assert or report messages and a nogo subdirectory for tests that are expected to have errors, but you can organise them any way you like.
Run Ant
You can run the test.schematron from build.xml directly:
```
ant -f /path/to/stf/build.xml test.schematron
```
or you can import the stf build.xml into your local build.xml:
```
<property name="stf.dir" location="/path/to/stf" />
<import file="${stf.dir}/build.xml" />
```
and run the test.schematron target, or you can import the stf build.xml and use the <test.schematron/> macro in a target in your local build.xml:
```
<target name="test">
  <test.schematron />
  <xspec xspec.xml="tests.xspec" />
</target>
```

Ant Properties

${schematron}: Schematron file to test.
${tests.dir}: Directory containing test files.
${calabash.jar}: Location of Calabash jar.
${saxon.jar}: Location of Saxon 9.2 (or later) jar.
${resolver.jar}: Location of XML catalog resolver library.
${resolver.class}: Class name of XML catalog resolver. Default is org.apache.xml.resolver.tools.CatalogResolver.

XProc Processor

The pipeline currently depends on Calabash extensions.

The version of Calabash used in testing depended on Saxon 9.2 or later.

Running (Not Testing) Schematron

Ant build file also includes schematron macro and run.schematron target to make it easy to run Schematron on real files once you’re sure it works on your tests.

License

Licensed under the terms of a BSD license.