Open source · MIT licensed

The Python test runner built for precision.

▸ Zero plugins. Full arsenal.

Get Started View on GitHub
Python 3.9 – 3.14 Zero dependencies MIT licensed
LOCK AZM 270° RNG 4.7m TEST JUNKIE
The Essentials

Everything you'd expect

The building blocks every framework has — clean, decorator-based, zero boilerplate.

@Suite & @test

Organize tests in plain Python classes. No base class to extend — your IDE, linter, and debugger work as-is.

Lifecycle Hooks

@beforeClass, @afterClass, @beforeTest, @afterTest — each fires at the right scope, declared directly on the class.

Automatic Retries

retry=N on any test re-runs it on failure. Extendable with exception-type control when you need more precision.

Built-in Reports

HTML, XML, and JSON reports generated after every run. No plugins, no config files, no post-processing step.

Only in Test Junkie

What sets it apart

Capabilities built into the core — not available through plugins in other frameworks.

GroupRules

Shared Lifecycle

Define driver setup, teardown, and hooks once and attach them to many suites — no inheritance, no copy-paste, no conftest proliferation.

  • A Rules subclass defines before_class, after_class, before_test, and after_test hooks; any suite that passes rules= inherits them — one change updates every suite.
  • Suites keep their own @beforeClass and @afterClass — the Rules class wraps around them, not replacing them.
Explore shared lifecycle → 
from test_junkie.rules import Rules

class BrowserRules(Rules):
    def after_test(self, **kwargs):
        logs = Browser.get_log("browser")
        errors = [m["message"] for m in logs]
        if errors:
            raise AssertionError(str(errors))
        Browser.shutdown()

@Suite(rules=BrowserRules)
class CheckoutSuite: ...

@Suite(rules=BrowserRules)
class LoginSuite: ...

Multi-layer Parametrization

Coverage

Parametrize at the suite level and the test level independently — write one test, run it across every combination of environments, browsers, and user roles.

  • Suite params cross-multiply with test params: 2 environments × 3 browsers = 6 independent runs from a single test method.
  • Each variant is tracked, retried, and reported separately — test_checkout[prod][safari] fails independently from test_checkout[staging][chrome].
Explore parametrization → 
@Suite(parameters=[
    {"env": "staging"},
    {"env": "prod"},
])
class CheckoutSuite:
    @test(parameters=[
        {"browser": "chrome"},
        {"browser": "firefox"},
        {"browser": "safari"},
    ])
    def test_checkout(self, parameter, suite_parameter):
        ...  # 6 variants, 1 method

Built-in Parallel Execution

Performance

No plugin, no external orchestrator — run suites and tests simultaneously at the thread level with independent concurrency limits at each tier.

  • -T 10 -S 2 runs 10 suites concurrently, each with 2 parallel tests — suite and test thread limits are controlled separately.
  • Opt-in per test and per suite — non-thread-safe tests run single-threaded while everything else scales out.
Explore parallelism → 
# tj run -s checkout.py -T 10 -S 2

@Suite(parallelized=True)
class CheckoutSuite:

    @test(parallelized=True)
    def test_add_to_cart(self):
        ...

    @test(parallelized=False — db writes)
    def test_place_order(self):
        ...

Exception-aware Retries

Reliability

Retry only on specific exception types — not blindly on any failure. Stop masking real bugs with policies that retry everything.

  • retry_on=[ConnectionError, TimeoutError] silently retries infrastructure noise; an AssertionError always surfaces immediately.
  • no_retry_on is an explicit block list — security and data-integrity assertions can always fail fast, regardless of the retry count.
Explore retries → 
@test(
    retry=3,
    retry_on=[
        ConnectionError,
        TimeoutError,
    ],
    no_retry_on=[
        AssertionError,
    ],
)
def test_payment_gateway(self):
    ...
# infra flakiness retries quietly
# real failures always surface

Structured Metadata & CI Targeting

Observability

First-class owner, component, and tags on every test for CI filtering; priority= controls execution order — lower number runs first.

  • tj run --tags smoke in CI, full suite locally — no test selection scripts to maintain as the repo grows.
  • Use meta(bug="TJ-42") inside a running test to attach a ticket to the failure dynamically — no post-run correlation needed.
Explore metadata → 
@test(
    owner="checkout-team",
    component="payment",
    priority=1,
    tags=["smoke", "regression"],
)
def test_checkout(self):
    meta(bug="TJ-123")
    ...

# tj run --tags smoke

Accessible Objects & Listeners

Programmatic

Run results are plain Python objects and per-suite listeners fire live — build Slack alerts, Jira tickets, or custom CI gates without touching a plugin system.

  • runner.summary.suites gives typed objects with exception text, timing, retry count, and owner — everything to build your own reporting on top.
  • Listener(on_failure=my_fn) fires the moment a test finishes — your notification pipeline runs live alongside the suite, not after it.
Explore objects → 
runner = Runner([CheckoutSuite])
runner.run()

for suite in runner.summary.suites:
    for test in suite.tests:
        if test.is_failed():
            slack.post(
                test.get_exception())
# your reporting, not ours

As simple or as complex as your use case

Minimal test file to full production setup. CLI command to fully programmatic. It's all the same API — add what you need, leave out what you don't.

Define tests
Getting started Minimal — fully functional 
from test_junkie.runner import Runner
from test_junkie.decorators import Suite, test

@Suite()
class LoginSuite:

    @test()
    def test_valid_login(self):
        assert login("admin", "pass")

    @test()
    def test_invalid_password(self):
        assert login("admin", "wrong") is False

Runner([LoginSuite]).run()
Full power All features active 
@Suite(
    parameters=[{"env": "staging"}, {"env": "prod"}],
    parallelized=True, listener=AlertListener,
)
class CheckoutSuite:
    @beforeClass()
    def setup(self, suite_parameter):
        self.driver = new_driver(suite_parameter["env"])

    @test(
        owner="checkout-team", component="payment",
        priority=1, tags=["smoke"], retry=3,
        retry_on=[ConnectionError],
        no_retry_on=[AssertionError],
        parameters=[{"browser": "chrome"},
                    {"browser": "safari"}],
        parallelized=True,
    )
    def test_checkout(self, parameter, suite_parameter): ...
Run it
CLI tj run — targeting, parallelism, filtering 
# Run all tests in a directory
$ tj run -s tests/

# CI: smoke tests only
$ tj run -s tests/ --tags smoke

# Parallel: 10 suite threads, 2 test threads
$ tj run -s tests/ -T 10 -S 2

# Filter by owner and component
$ tj run -s tests/ --owners checkout-team --components payment
Programmatic Runner API — run, inspect, report 
from test_junkie.runner import Runner
from test_junkie.constants import TestCategory

runner = Runner([CheckoutSuite, LoginSuite])
runner.run()

for suite in runner.get_executed_suites():
    for test_obj in suite.get_test_objects():
        metrics = test_obj.metrics.get_metrics()
        for _, by_param in metrics.items():
            for _, data in by_param.items():
                if data["status"] == TestCategory.FAIL:
                    slack.notify(data["exceptions"][-1])

runner.get_html_report("./report.html")
runner.get_xml_report("./report.xml")

How it stacks up

Feature-by-feature against pytest, unittest, and Robot Framework — what ships in the box versus what requires plugins, workarounds, or simply isn't possible.

✓ Built-in— native, zero plugins ~ Plugin / Partial— possible but not included ✗ Not supported— no viable path
Feature test_junkie pytest unittest Robot Framework
Execution & Resilience
Parallel execution ✓ Built-in Thread-based; suite concurrency and test concurrency set independently; parallelized=False opts any test out without touching others ~ pytest-xdist Process-based workers; distributes test collection across processes; no dual-level thread limits; no per-test opt-out ✗ Not supported Sequential only; unittest-parallel exists as a third-party tool ~ Plugin (pabot) Subprocess-based via robotframework-pabot; suite-level by default, test-level with --testlevelsplit; PabotLib for inter-process resource locking
Test retries ✓ Built-in retry=N at test or suite level; on parametrized tests only the failing variants retry — not the full parameter set ~ pytest-rerunfailures --reruns N; retries on any failure; entire parametrized test reruns if any variant fails ✗ Not supported ~ Plugin robotframework-retryfailed adds tag-driven per-test retry counts; --rerunfailed + rebot --merge for multi-pass reruns; no built-in retry decorator
Exception-type retry controlUnique ✓ Built-in retry_on=[ExceptionClass] retries only on matched exception types; no_retry_on=[ExceptionClass] blocks specific classes from ever retrying ✗ Not supported pytest-rerunfailures --only-rerun REGEX matches on exception message text, not exception class; approximate and easily wrong ✗ Not supported ✗ Not supported Neither native retry nor any plugin filters retries by exception type; no equivalent exists in the RF ecosystem
Test Design
Parametrized execution ✓ Built-in Dict-based params via @test(parameters=[...]); accepts a callable as provider for lazy/dynamic generation ✓ Built-in @pytest.mark.parametrize; mature and widely used; stacking decorators produces a cartesian product at test level ~ Partial self.subTest() context manager records sub-results but lifecycle hooks don't re-run; no decorator; variants can't be retried independently ~ Partial Test Templates with inline data rows or FOR loops; robotframework-datadriver drives templates from CSV/Excel; no dict-based params or callable provider
Suite-level × test-level cross-multiplyUnique ✓ Built-in @Suite(parameters=[...]) × @test(parameters=[...]) cross-multiply; 2 environments × 3 browsers = 6 independent variants, each tracked and retried separately ✗ Not supported No class-level parametrize decorator; indirect fixture parametrization via conftest.py is possible but has no cross-multiply model and no per-variant retry ✗ Not supported ✗ Not supported No native suite-level × test-level Cartesian product; robotframework-datadriver operates at test level only; all variants must be written or generated explicitly
Shared lifecycle across suitesUnique ✓ Built-in @GroupRules attaches shared before/after hooks to a named set of suites across any files; includes @beforeGroup / @afterGroup events ~ Partial conftest.py fixtures scoped to session / module / class; scope is directory-based, not a named class set; no @beforeGroup equivalent ~ Partial setUpModule() / tearDownModule() at module level only; no cross-file class grouping without inheritance ~ Partial __init__.robot init files apply Suite/Test Setup to all suites in a directory tree; Resource files share keywords across suites; directory-scoped, not attachable to a named cross-directory set
Metadata & Targeting
Structured test metadata ✓ Built-in First-class owner, component, feature, priority, tags; priority also controls execution order in parallel runs ~ Partial @pytest.mark.X free-form string markers; no first-class owner / component / priority; markers must be registered in pytest.ini; allure-pytest plugin adds structure ✗ Not supported Tests identified by class + method name only; no metadata or tagging system ~ Partial Built-in [Tags] and suite-wide Test Tags; [Metadata] for free name-value pairs; owner / component / feature / priority encoded as tag strings by convention (owner:john), not structured first-class fields
Execution priority order ✓ Built-in priority= on @Suite and @test; lower number runs first; three tiers: priority-set (ascending) → parallel / no priority → sequential / no priority ~ Plugin pytest-ordering; @pytest.mark.order(N) or --randomly-seed; no built-in priority= field ✗ Not supported Alphabetical order by default; sortTestMethodsUsing for custom sort; no priority field ✗ Not supported File / alphabetical order by default; --randomize for randomization; no built-in priority field
CLI targeting & run filters ✓ Built-in --tags, --owners, --components, --features; AND / OR match modes; same filters available in Runner.run() programmatically; priority= controls execution order, not filtering ~ Partial -m "marker" + -k "expression"; no built-in --owner / --component; multi-dimension filtering needs plugins or custom conftest ~ Partial Fully-qualified name selection only (python -m unittest Suite.test_name); no metadata-based filtering ~ Partial robot --include TAG, --exclude TAG, --test NAME, --suite NAME; AND/OR/NOT tag logic; because owner/component/priority are not structured fields, no dedicated flags exist for them
Runtime metadata updatesUnique ✓ Built-in Meta.update() inside a running test can change owner, tags, component, linked bug ticket, or expected result dynamically mid-run ✗ Not supported request.node.user_properties.append(...) is append-only; markers cannot be added or changed once the test is running ✗ Not supported ~ Partial Set Tags / Remove Tags BuiltIn keywords modify a test's tags during execution; no equivalent for attaching arbitrary key-value pairs to the result record mid-run
Results & Observability
HTML / XML / JSON reports ✓ Built-in All three formats included; HTML dashboard includes CPU & memory graphs per run; zero configuration, zero plugins ~ Partial JUnit XML built-in (--junitxml); HTML requires pytest-html; JSON requires pytest-json-report; no system resource graphs ✗ Text only TextTestRunner terminal output; xmlrunner third-party for XML; no HTML or JSON built-in ~ Partial Native log.html + report.html + output.xml every run; XUnit XML via --xunit; no native JSON format; no resource graphs
Per-suite event listeners ✓ Built-in Listener class attached per suite; 21 named events covering test and suite lifecycle; fires live during the run — not post-process ~ Partial conftest.py hooks (pytest_runtest_logreport etc.); global scope, not per-class; requires understanding the plugin protocol ~ Partial Subclass TestResult with addSuccess / addFailure / addError; no named event system; requires custom runner wiring ✓ Built-in Listener API v2/v3 in core; events: start_suite, end_suite, start_test, end_test, start_keyword, and more; v3 passes live model objects; attached via --listener MyListener
Programmatic result access ✓ Built-in runner.get_executed_suites() returns typed SuiteObject / TestObject with timing, retry count, exception data, owner, tags, and parameter sets ✗ Not in core Primary interface is CLI; pytest-json-report plugin writes a JSON file; no typed result objects in core ~ Partial TestResult.failures / .errors / .skipped — list of (TestCase, str) tuples; raw strings, no timing or structured exception ✓ Built-in robot.api.ExecutionResult('output.xml') + ResultVisitor in core; full access to suite/test status, timing, messages; live access during run via v3 listener API model objects
Project Health
Maintenance status ✓ Active MIT; Python 3.9–3.14; zero runtime dependencies ✓ Active BSD; Python 3.8+; the most widely used Python testing framework; 1,000+ community plugins ✓ Stdlib Part of CPython; zero external dependencies; stable but feature-frozen since subTest in Python 3.4 ✓ Active Apache 2.0; RF 7.x; large enterprise ecosystem and standard library; keyword-driven DSL — tests written in custom syntax, not Python; extensive community library support