Collecting and Reporting Pytest Results

retrieve pytest results programmatically when run via pytest.main()

Write a small plugin that collects and stores reports for each test. Example:

import time
import pytest


class ResultsCollector:
    def __init__(self):
        self.reports = []
        self.collected = 0
        self.exitcode = 0
        self.passed = 0
        self.failed = 0
        self.xfailed = 0
        self.skipped = 0
        self.total_duration = 0

    @pytest.hookimpl(hookwrapper=True)
    def pytest_runtest_makereport(self, item, call):
        outcome = yield
        report = outcome.get_result()
        if report.when == 'call':
            self.reports.append(report)

    def pytest_collection_modifyitems(self, items):
        self.collected = len(items)

    def pytest_terminal_summary(self, terminalreporter, exitstatus):
        print(exitstatus, dir(exitstatus))
        self.exitcode = exitstatus.value
        self.passed = len(terminalreporter.stats.get('passed', []))
        self.failed = len(terminalreporter.stats.get('failed', []))
        self.xfailed = len(terminalreporter.stats.get('xfailed', []))
        self.skipped = len(terminalreporter.stats.get('skipped', []))

        self.total_duration = time.time() - terminalreporter._sessionstarttime

def run():
    collector = ResultsCollector()
    pytest.main(plugins=[collector])
    for report in collector.reports:
        print('id:', report.nodeid, 'outcome:', report.outcome)  # etc
    print('exit code:', collector.exitcode)
    print('passed:', collector.passed, 'failed:', collector.failed, 'xfailed:', collector.xfailed, 'skipped:', collector.skipped)
    print('total duration:', collector.total_duration)


if __name__ == '__main__':
    run()

Collecting pytest results in pandas dataframe

You could implement a custom makereport hook in your conftest.py. A simple example:

import pytest
import pandas as pd


df = pd.DataFrame(columns=('failed', 'nodeid', ))


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    global df
    outcome = yield
    rep = outcome.get_result()

    if rep.when == 'call':
        df = df.append({'failed': rep.failed, 'nodeid': rep.nodeid}, ignore_index=True)

How to collect data results in pytest using junitxml?

The shipped junitxml plugin does not have hooks to add such data
you can print it to stdout though, since that gets added to the junitxml data.

So as long as you print out logs you will at least be able to know the data.

How can I access the overall test result of a pytest test run during runtime?

I could not find a suitable pytest hook to access the overall test result yet.

You don't need one; just collect the test results yourself. This is the blueprint I usually use when in need of accessing the test results in batch:

# conftest.py
import pytest


def pytest_sessionstart(session):
    session.results = dict()


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    result = outcome.get_result()

    if result.when == 'call':
        item.session.results[item] = result

Now all test results are stored under session.results dict; example usage:

# conftest.py (continued)

def pytest_sessionfinish(session, exitstatus):
    print()
    print('run status code:', exitstatus)
    passed_amount = sum(1 for result in session.results.values() if result.passed)
    failed_amount = sum(1 for result in session.results.values() if result.failed)
    print(f'there are {passed_amount} passed and {failed_amount} failed tests')

Running the tests will yield:

$ pytest -sv
================================== test session starts ====================================
platform darwin -- Python 3.6.4, pytest-3.7.1, py-1.5.3, pluggy-0.7.1 -- /Users/hoefling/.virtualenvs/stackoverflow/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/hoefling/projects/private/stackoverflow/so-51711988, inifile:
collected 3 items

test_spam.py::test_spam PASSED
test_spam.py::test_eggs PASSED
test_spam.py::test_fail FAILED
run status code: 1
there are 2 passed and 1 failed tests


======================================== FAILURES =========================================
_______________________________________ test_fail _________________________________________

    def test_fail():
>       assert False
E       assert False

test_spam.py:10: AssertionError
=========================== 1 failed, 2 passed in 0.05 seconds ============================

In case the overall pytest exit code (exitstatus) is sufficient info (info about # passed, # failed, etc. not required) use the following:

# conftest.py

def pytest_sessionfinish(session, exitstatus):
    print()
    print('run status code:', exitstatus)

Accessing the error

You can access the error details from the call.excinfo object:

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    result = outcome.get_result()
    if result.outcome == "failed":
        exception = call.excinfo.value
        exception_class = call.excinfo.type
        exception_class_name = call.excinfo.typename
        exception_type_and_message_formatted = call.excinfo.exconly()
        exception_traceback = call.excinfo.traceback

---

Related Topics