Pytest-Working with non-python tests

A basic example for specifying tests in Yaml files {#yaml plugin}

Here is an example conftest.py (extracted from Ali Afshar's special
purpose
pytest-yamlwsgi
plugin). This conftest.py will collect test*.yaml files and will
execute the yaml-formatted content as custom tests:

# content of conftest.py
import pytest


def pytest_collect_file(parent, path):
        return YamlFile.from_parent(parent, fspath=path)


class YamlFile(pytest.File):
    def collect(self):
        # We need a yaml parser, e.g. PyYAML.
        import yaml

        raw = yaml.safe_load(self.fspath.open())
        for name, spec in sorted(raw.items()):
            yield YamlItem.from_parent(self, name=name, spec=spec)


class YamlItem(pytest.Item):
    def __init__(self, name, parent, spec):
        super().__init__(name, parent)
        self.spec = spec

    def runtest(self):
        for name, value in sorted(self.spec.items()):
            # Some custom test execution (dumb example follows).
            if name != value:
                raise YamlException(self, name, value)

    def repr_failure(self, excinfo):
        """Called when self.runtest() raises an exception."""
        if isinstance(excinfo.value, YamlException):
            return "\n".join(
                [
                    "usecase execution failed",
                    "   spec failed: {1!r}: {2!r}".format(*excinfo.value.args),
                    "   no further details known at this point.",
                ]
            )

    def reportinfo(self):
        return self.fspath, 0, f"usecase: {self.name}"


class YamlException(Exception):
    """Custom exception for error reporting."""

You can create a simple example file:

# test_simple.yaml
ok:
    sub1: sub1

hello:
    world: world
    some: other

and if you installed PyYAML or a
compatible YAML-parser you can now execute the test specification:

nonpython $ pytest test_simple.yaml
platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/nonpython
collected 2 items

test_simple.yaml F.                                                  [100%]

______________________________ usecase: hello ______________________________
usecase execution failed
   spec failed: 'some': 'other'
   no further details known at this point.
FAILED test_simple.yaml::hello

You get one dot for the passing sub1: sub1 check and one failure.
Obviously in the above conftest.py you'll want to implement a more
interesting interpretation of the yaml-values. You can easily write your
own domain specific testing language this way.

::: {.note}
::: {.title}
Note
:::

repr_failure(excinfo) is called for representing test failures. If you
create custom collection nodes you can return an error representation
string of your choice. It will be reported as a (red) string.
:::

reportinfo() is used for representing the test location and is also
consulted when reporting in verbose mode:

nonpython $ pytest -v
platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/nonpython
collecting ... collected 2 items

test_simple.yaml::hello FAILED                                       [ 50%]
test_simple.yaml::ok PASSED                                          [100%]

______________________________ usecase: hello ______________________________
usecase execution failed
   spec failed: 'some': 'other'
   no further details known at this point.
FAILED test_simple.yaml::hello

While developing your custom test collection and execution it's also
interesting to just look at the collection tree:

nonpython $ pytest --collect-only
platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/nonpython
collected 2 items

<Package nonpython>
  <YamlFile test_simple.yaml>
    <YamlItem hello>
    <YamlItem ok>