LifeMonitor

Logo

A testing and monitoring service for scientific workflows

View the Project on GitHub crs4/life_monitor

Workflow Testing RO-Crate

Please note this is a draft, subject to change. You can leave any comments here.

Workflow Testing RO-Crate is a specialization of Workflow RO-Crate that supports additional metadata related to the testing of computational workflows. LifeMonitor uses this as an exchange format that allows RO-Crate authors to describe test suites associated with workflows.

Introduction

The LifeMonitor service aims to provide two main functionalities:

Concepts

This section uses terminology from the RO-Crate 1.1 specification, which is the basis for the Workflow RO-Crate spec.

The context used by Workflow Testing RO-Crates is an extension of the RO-Crate 1.1 context that includes test-specific classes and properties defined in the test RO-Terms vocabulary:

[
    "https://w3id.org/ro/crate/1.1/context",
    {
        "TestSuite": "https://w3id.org/ro/terms/test#TestSuite",
        "TestInstance": "https://w3id.org/ro/terms/test#TestInstance",
        "TestService": "https://w3id.org/ro/terms/test#TestService",
        "TestDefinition": "https://w3id.org/ro/terms/test#TestDefinition",
        "PlanemoEngine": "https://w3id.org/ro/terms/test#PlanemoEngine",
        "JenkinsService": "https://w3id.org/ro/terms/test#JenkinsService",
        "TravisService": "https://w3id.org/ro/terms/test#TravisService",
        "GithubService": "https://w3id.org/ro/terms/test#GithubService",
        "instance": "https://w3id.org/ro/terms/test#instance",
        "runsOn": "https://w3id.org/ro/terms/test#runsOn",
        "resource": "https://w3id.org/ro/terms/test#resource",
        "definition": "https://w3id.org/ro/terms/test#definition",
        "engineVersion": "https://w3id.org/ro/terms/test#engineVersion"
    }
]

The most recent version of the context is available from https://github.com/ResearchObject/ro-terms/blob/master/test.

A Workflow Testing RO-Crate MUST be a valid Workflow RO-Crate (e.g., it has to contain a Main Workflow). In addition, it COULD refer to one or more test suites from the root data entity via the mentions property:

{
    "@id": "./",
    "@type": "Dataset",
    "mentions": [
        {
            "@id": "#test1"
        },
        {
            "@id": "#test2"
        }
    ],
    ...
}

Test suite

A test suite describes a set of tests for a computational workflow. It is represented by a context entity of type "TestSuite". A test suite MUST refer either to one or more test instances (via the instance property) or to a test definition (via the definition property) or both. Additionally, a test suite SHOULD refer to the tested workflow via mainEntity.

{
    "@id": "#test1",
    "@type": "TestSuite",
    "mainEntity": {"@id": "sort-and-change-case.ga"},
    "instance": [
        {"@id": "#test1_1"}
    ],
    "definition": {"@id": "test/test1/sort-and-change-case-test.yml"}
}

Test instance

A test instance is a specific job that executes a test suite on a test service. It is represented by a context entity of type "TestInstance". A test instance MUST refer to: a test service via the runsOn property; the base URL of the specific test service deployment where it runs via the url property; the relative URL of the test project via the resource property:

{
    "@id": "#test1_1",
    "@type": "TestInstance",
    "runsOn": {"@id": "https://w3id.org/ro/terms/test#JenkinsService"},
    "url": "http://example.org/jenkins",
    "resource": "job/tests/"
}

Test service

A test service is a software service where tests can be run. It is represented by a context entity of type "TestService":

{
    "@id": "https://w3id.org/ro/terms/test#JenkinsService",
    "@type": "TestService",
    "name": "Jenkins",
    "url": {"@id": "https://www.jenkins.io"}
}

Test definition

A test definition is a set of metadata that describes how to run a test suite. It is represented by a data entity of type ["File", "TestDefinition"]. A test definition MUST refer to the test engine it is written for via conformsTo and to the engine’s version via engineVersion:

{
    "@id": "test/test1/my-test.yml",
    "@type": [
        "File",
        "TestDefinition"
    ],
    "conformsTo": {"@id": "https://w3id.org/ro/terms/test#PlanemoEngine"},
    "engineVersion": ">=0.70"
},

Test engine

A test engine is a software application that runs workflow tests according to a definition. It is represented by a context entity of type "SoftwareApplication":

{
    "@id": "https://w3id.org/ro/terms/test#PlanemoEngine",
    "@type": "SoftwareApplication",
    "name": "Planemo",
    "url": {"@id": "https://github.com/galaxyproject/planemo"}
}

Example

An example of Workflow Testing RO-Crate metadata.

{
    "@context": [
        "https://w3id.org/ro/crate/1.1/context",
        {
            "TestSuite": "https://w3id.org/ro/terms/test#TestSuite",
            "TestInstance": "https://w3id.org/ro/terms/test#TestInstance",
            "TestService": "https://w3id.org/ro/terms/test#TestService",
            "TestDefinition": "https://w3id.org/ro/terms/test#TestDefinition",
            "PlanemoEngine": "https://w3id.org/ro/terms/test#PlanemoEngine",
            "JenkinsService": "https://w3id.org/ro/terms/test#JenkinsService",
            "TravisService": "https://w3id.org/ro/terms/test#TravisService",
            "GithubService": "https://w3id.org/ro/terms/test#GithubService",
            "instance": "https://w3id.org/ro/terms/test#instance",
            "runsOn": "https://w3id.org/ro/terms/test#runsOn",
            "resource": "https://w3id.org/ro/terms/test#resource",
            "definition": "https://w3id.org/ro/terms/test#definition",
            "engineVersion": "https://w3id.org/ro/terms/test#engineVersion"
        }
    ],
    "@graph": [
        {
            "@id": "ro-crate-metadata.json",
            "@type": "CreativeWork",
            "about": {
                "@id": "./"
            },
            "conformsTo": {
                "@id": "https://w3id.org/ro/crate/1.1"
            }
        },
        {
            "@id": "./",
            "@type": "Dataset",
            "name": "sort-and-change-case",
            "description": "sort lines and change text to upper case",
            "license": "Apache-2.0",
            "mainEntity": {
                "@id": "sort-and-change-case.ga"
            },
            "hasPart": [
                {
                    "@id": "sort-and-change-case.ga"
                },
                {
                    "@id": "LICENSE"
                },
                {
                    "@id": "README.md"
                },
                {
                    "@id": "test/test1/sort-and-change-case-test.yml"
                }
            ],
            "mentions": [
                {
                    "@id": "#test1"
                }
            ]
        },
        {
            "@id": "sort-and-change-case.ga",
            "@type": [
                "File",
                "SoftwareSourceCode",
                "ComputationalWorkflow"
            ],
            "programmingLanguage": {
                "@id": "#galaxy"
            },
            "name": "sort-and-change-case"
        },
        {
            "@id": "LICENSE",
            "@type": "File"
        },
        {
            "@id": "README.md",
            "@type": "File"
        },
        {
            "@id": "#galaxy",
            "@type": "ComputerLanguage",
            "name": "Galaxy",
            "identifier": {
                "@id": "https://galaxyproject.org/"
            },
            "url": {
                "@id": "https://galaxyproject.org/"
            }
        },
        {
            "@id": "#test1",
            "name": "test1",
            "@type": "TestSuite",
            "mainEntity": {
                "@id": "sort-and-change-case.ga"
            },
            "instance": [
                {"@id": "#test1_1"}
            ],
            "definition": {"@id": "test/test1/sort-and-change-case-test.yml"}
        },
        {
            "@id": "#test1_1",
            "name": "test1_1",
            "@type": "TestInstance",
            "runsOn": {"@id": "https://w3id.org/ro/terms/test#JenkinsService"},
            "url": "http://example.org/jenkins",
            "resource": "job/tests/"
        },
        {
            "@id": "test/test1/sort-and-change-case-test.yml",
            "@type": [
                "File",
                "TestDefinition"
            ],
            "conformsTo": {"@id": "https://w3id.org/ro/terms/test#PlanemoEngine"},
            "engineVersion": ">=0.70"
        },
        {
            "@id": "https://w3id.org/ro/terms/test#JenkinsService",
            "@type": "TestService",
            "name": "Jenkins",
            "url": {"@id": "https://www.jenkins.io"}
        },
        {
            "@id": "https://w3id.org/ro/terms/test#PlanemoEngine",
            "@type": "SoftwareApplication",
            "name": "Planemo",
            "url": {"@id": "https://github.com/galaxyproject/planemo"}
        }
    ]
}