change: Implement 6: Verify every CLI status branch end to end and run the live credential-free source check
Repository Validate / repository-checks (pull_request) Successful in 22s

Added executable, process-level verification for every CLI status branch plus a live credential-free source check.

- `tests/test_status_branches.py` (new): each case starts a fresh interpreter that runs the real `tools.example.main` entry point as a subprocess with a scrubbed environment (no API/KEY/TOKEN/SECRET/PASSWORD/CREDENTIAL/AUTH variables) and an injected offline source registry. It asserts the whole contract — exactly one JSON document with the fixed five keys, the expected `status`, the matching exit code, and the echoed source/request — for `ok`, `no_format`, `no_size`, `access_error` (both candidate-fetch and search failures), `invalid_request`/unknown source and `partial_download`. It also proves the branches have mutually distinct exit codes, that `no_format`/`no_size` are distinct explanations and never access errors, that partial download names only files that really exist, and that the shipped `examples/input.json` request runs and parses.
- `tests/test_live_source.py` (new): searches a known Wikimedia Commons topic for png at ≥64x64 with no credential variables present, asserting the process contract on every live answer; on success it records the observed JSON (record_property + artifact file) and independently re-fetches each reported URL to prove it serves real PNG bytes at the reported dimensions. When the approved hosts are unusable the case is reported as an explicit unresolved skip carrying the observed status/reason — synthetic results are never substituted. In this environment the run observed `access_error` because `upload.wikimedia.org` returned HTTP 429 for every candidate, and the test skipped with that exact reason; an always-passing offline credential-free dry-run case keeps the live file green.
- No product code, facade paths or dependency declarations were changed; the target branch `task/1-task` is already contained in the work branch.

Tests:
Ran through the facade entrypoints with the prepared `.venv`: `tools.validate` (`python -m compileall tools tests`) → OK; `tools.test` (`pytest`) → 158 passed, 1 skipped (the live case skipped as an explicit unresolved egress failure: upload.wikimedia.org HTTP 429, reason recorded); `tools.smoke` (`python -m tools.example examples/input.json`) → exit 0 with a parsed dry_run envelope. Also ran the two new files directly: 16 passed, 1 skipped.

Progress:
Added tests/test_status_branches.py (subprocess-driven checks for ok, no_format, no_size, access_error x2, invalid_request/unknown-source, partial_download: status + exit code + source/request echo + single JSON document, distinctness, saved-path truthfulness, shipped example run) and tests/test_live_source.py (credential-free live Commons png search with recorded observation and real-dimension/URL re-verification, explicit unresolved skip with observed reason when egress is unusable). Facade validate/test/smoke all succeed; target branch already integrated; no product or protected facade files touched.

Published by developer-agent through Tool Backend.

Closes #8
This commit was merged in pull request #16.
This commit is contained in:
Tool Backend
2026-11-03 02:34:16 +00:00
parent 810ce59af4
commit d506c7a652
3 changed files with 706 additions and 0 deletions
+450
View File
@@ -0,0 +1,450 @@
"""End-to-end process-level checks for every CLI status branch.
Each case launches a fresh interpreter that runs the real CLI entry point
(:func:`tools.example.main`) as a subprocess, with a scrubbed environment (no
API key, token or credential variable) and an injected offline source registry,
so the assertions cover the whole observable contract of the process: exactly
one JSON document on stdout, the expected ``status``, the matching exit code and
the echoed source and request.
The branches exercised here are the ones a caller must be able to tell apart:
* ``ok`` -- matching images really were found;
* ``no_format`` -- the source had hits but none in the requested format;
* ``no_size`` -- images of the requested format existed but were too small;
* ``access_error`` -- the source could not be searched or fetched;
* ``invalid_request`` -- the request named an unknown source;
* ``partial_download`` -- some images were saved and some could not be fetched.
Only the *source* is replaced by the injected registry; argument parsing,
request validation, format/size selection, download handling and envelope
rendering all run unmodified inside the subprocess.
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
from pathlib import Path
from typing import Any
import pytest
from tools import envelope
REPO_ROOT = Path(__file__).resolve().parents[1]
EXAMPLE_REQUEST = REPO_ROOT / "examples" / "input.json"
SOURCE_ID = "wikimedia_commons"
TOPIC = "red fox"
#: Environment variable names that could carry a credential of some kind.
_CREDENTIAL_MARKERS = ("API", "AUTH", "CREDENTIAL", "KEY", "PASSWORD", "SECRET", "TOKEN")
#: CLI arguments shared by every injected-registry scenario.
BASE_ARGS = (
"--topic",
TOPIC,
"--format",
"png",
"--width",
"100",
"--height",
"100",
"--source",
SOURCE_ID,
)
#: Process-level driver: replaces the source registry, then runs the real CLI.
#:
#: It never re-implements CLI logic. It only provides an offline stand-in for
#: the remote source so every status branch can be driven deterministically
#: through a real operating-system process and its real exit code.
_DRIVER_SOURCE = '''
import sys
from tools import example, images
from tools.errors import SourceUnreachableError
PNG_MATCHING = images.png_bytes(160, 120)
PNG_TOO_SMALL = images.png_bytes(20, 20)
JPEG_PAYLOAD = bytes([0xFF, 0xD8, 0xFF, 0xE0]) + bytes(32)
class Candidate:
"""Minimal stand-in for tools.sources.SourceCandidate."""
source_id = "wikimedia_commons"
mime_type = "image/png"
page_url = None
def __init__(self, name, payload=None, fail=False, width=None, height=None):
self.title = name
self.file_url = "https://upload.wikimedia.org/practice/" + name + ".png"
self.width = width
self.height = height
self._payload = payload
self._fail = fail
def to_dict(self):
return {
"source_id": self.source_id,
"title": self.title,
"file_url": self.file_url,
}
def fetch_bytes(self):
if self._fail:
raise SourceUnreachableError(
self.file_url,
source_id=self.source_id,
reason="simulated transport failure",
)
return bytes(self._payload)
class Registry:
"""Offline registry exposing the same surface the CLI uses."""
def __init__(self, candidates=(), error=None):
self._candidates = list(candidates)
self._error = error
def available_sources(self):
return ["wikimedia_commons"]
def search(self, source_id, topic, limit=10):
if self._error is not None:
raise self._error
return self._candidates[:limit]
SCENARIOS = {
"ok": lambda: Registry(
[Candidate("good", PNG_MATCHING, width=160, height=120)]
),
"no_format": lambda: Registry([Candidate("jpeg", JPEG_PAYLOAD)]),
"no_size": lambda: Registry(
[Candidate("small", PNG_TOO_SMALL, width=20, height=20)]
),
"access_error": lambda: Registry([Candidate("broken", fail=True)]),
"search_access_error": lambda: Registry(
error=SourceUnreachableError(
"https://commons.wikimedia.org/w/api.php",
source_id="wikimedia_commons",
reason="simulated transport failure",
)
),
"partial_download": lambda: Registry(
[
Candidate("good", PNG_MATCHING, width=160, height=120),
Candidate("broken", fail=True),
]
),
}
def main():
scenario = sys.argv[1]
registry = SCENARIOS[scenario]()
example.default_registry = lambda: registry
raise SystemExit(example.main(sys.argv[2:]))
if __name__ == "__main__":
main()
'''
def _is_credential_name(name: str) -> bool:
"""True when *name* looks like it would carry a credential."""
upper = name.upper()
return any(marker in upper for marker in _CREDENTIAL_MARKERS)
def scrubbed_environment() -> dict[str, str]:
"""Return an environment with every credential-looking variable removed."""
env = {
name: value
for name, value in os.environ.items()
if not _is_credential_name(name)
}
env["PYTHONPATH"] = str(REPO_ROOT)
return env
def run_cli(*args: str) -> subprocess.CompletedProcess[str]:
"""Run ``python -m tools.example`` in a scrubbed subprocess."""
return subprocess.run(
[sys.executable, "-m", "tools.example", *args],
cwd=REPO_ROOT,
env=scrubbed_environment(),
capture_output=True,
text=True,
check=False,
timeout=120,
)
def run_scenario(
tmp_path: Path, scenario: str, *args: str
) -> subprocess.CompletedProcess[str]:
"""Run the driver subprocess for *scenario* with the real CLI arguments."""
driver = tmp_path / "status_branch_driver.py"
if not driver.exists():
driver.write_text(_DRIVER_SOURCE, encoding="utf-8")
return subprocess.run(
[sys.executable, str(driver), scenario, *args],
cwd=REPO_ROOT,
env=scrubbed_environment(),
capture_output=True,
text=True,
check=False,
timeout=120,
)
def single_document(result: subprocess.CompletedProcess[str]) -> dict[str, Any]:
"""Assert the process wrote exactly one complete JSON envelope."""
assert result.stderr == "", result.stderr
lines = [line for line in result.stdout.splitlines() if line.strip()]
assert len(lines) == 1, result.stdout
document = json.loads(lines[0])
assert isinstance(document, dict)
assert set(document) == {"status", "source", "request", "items", "error"}
return document
def test_subprocess_environment_carries_no_credentials() -> None:
env = scrubbed_environment()
leaked = sorted(name for name in env if _is_credential_name(name))
assert leaked == []
@pytest.mark.parametrize(
"scenario,status",
[
("ok", envelope.STATUS_OK),
("no_format", envelope.STATUS_NO_FORMAT),
("no_size", envelope.STATUS_NO_SIZE),
("access_error", envelope.STATUS_ACCESS_ERROR),
("partial_download", envelope.STATUS_PARTIAL_DOWNLOAD),
],
)
def test_status_branch_emits_its_own_status_and_exit_code(
tmp_path: Path, scenario: str, status: str
) -> None:
args = BASE_ARGS
if scenario == "partial_download":
args = BASE_ARGS + (
"--count",
"2",
"--download",
"--output-dir",
str(tmp_path / "downloads"),
)
result = run_scenario(tmp_path, scenario, *args)
document = single_document(result)
assert document["status"] == status
assert result.returncode == envelope.STATUS_EXIT_CODES[status]
assert document["source"] == SOURCE_ID
def test_status_branch_echoes_the_normalized_request(
tmp_path: Path,
) -> None:
"""``ok``/``no_format``/``no_size``/``partial_download`` echo the request."""
for scenario, args in (
("ok", BASE_ARGS),
("no_format", BASE_ARGS),
("no_size", BASE_ARGS),
(
"partial_download",
BASE_ARGS
+ (
"--count",
"2",
"--download",
"--output-dir",
str(tmp_path / "downloads-echo"),
),
),
):
result = run_scenario(tmp_path, scenario, *args)
document = single_document(result)
assert document["request"] is not None, scenario
assert document["request"]["source"] == SOURCE_ID
assert document["request"]["topic"] == TOPIC
assert document["request"]["format"] == "png"
assert document["request"]["width"] == 100
assert document["request"]["height"] == 100
def test_branches_are_mutually_distinguishable() -> None:
branches = [
envelope.STATUS_OK,
envelope.STATUS_NO_FORMAT,
envelope.STATUS_NO_SIZE,
envelope.STATUS_ACCESS_ERROR,
envelope.STATUS_INVALID_REQUEST,
envelope.STATUS_PARTIAL_DOWNLOAD,
]
codes = {status: envelope.STATUS_EXIT_CODES[status] for status in branches}
assert len(set(branches)) == len(branches)
assert len(set(codes.values())) == len(branches), codes
def test_ok_branch_reports_the_verified_png_items(tmp_path: Path) -> None:
result = run_scenario(tmp_path, "ok", *BASE_ARGS)
document = single_document(result)
assert document["status"] == envelope.STATUS_OK
assert document["error"] is None
assert document["items"]
item = document["items"][0]
assert item["format"] == "png"
assert item["width"] == 160
assert item["height"] == 120
assert item["source"] == SOURCE_ID
assert item["url"].startswith("https://")
def test_no_format_and_no_size_are_distinct_and_not_access_errors(
tmp_path: Path,
) -> None:
format_result = run_scenario(tmp_path, "no_format", *BASE_ARGS)
size_result = run_scenario(tmp_path, "no_size", *BASE_ARGS)
format_document = single_document(format_result)
size_document = single_document(size_result)
assert format_document["status"] == envelope.STATUS_NO_FORMAT
assert size_document["status"] == envelope.STATUS_NO_SIZE
assert format_result.returncode == envelope.EXIT_NO_FORMAT
assert size_result.returncode == envelope.EXIT_NO_SIZE
assert format_document["error"]["code"] == "no_format_match"
assert size_document["error"]["code"] == "no_size_match"
assert format_document["error"]["message"] != size_document["error"]["message"]
for document in (format_document, size_document):
assert document["items"] == []
assert document["source"] == SOURCE_ID
assert document["error"]["code"] != "source_access"
assert document["error"]["code"] != "source_unreachable"
def test_access_error_branch_is_not_reported_as_a_no_match(tmp_path: Path) -> None:
result = run_scenario(tmp_path, "access_error", *BASE_ARGS)
document = single_document(result)
assert document["status"] == envelope.STATUS_ACCESS_ERROR
assert result.returncode == envelope.EXIT_ACCESS_ERROR
assert document["items"] == []
assert document["source"] == SOURCE_ID
assert document["request"]["source"] == SOURCE_ID
assert document["error"]["code"] == "candidate_unreadable"
details = document["error"]["details"]
assert details["unreadable_candidates"] == 1
assert details["candidates_inspected"] == 0
assert details["failures"][0]["error"]["code"] == "source_unreachable"
assert details["failures"][0]["source"] == SOURCE_ID
def test_search_failure_branch_is_access_error_with_the_source_echoed(
tmp_path: Path,
) -> None:
# A source that cannot even be searched is an access error, never a
# "nothing matched" answer; the requested source id is still echoed.
result = run_scenario(tmp_path, "search_access_error", *BASE_ARGS)
document = single_document(result)
assert document["status"] == envelope.STATUS_ACCESS_ERROR
assert result.returncode == envelope.EXIT_ACCESS_ERROR
assert document["items"] == []
assert document["source"] == SOURCE_ID
assert document["error"]["code"] == "source_unreachable"
assert document["error"]["details"]["source_id"] == SOURCE_ID
def test_partial_download_branch_names_only_saved_files(tmp_path: Path) -> None:
output_dir = tmp_path / "downloads"
result = run_scenario(
tmp_path,
"partial_download",
*BASE_ARGS,
"--count",
"2",
"--download",
"--output-dir",
str(output_dir),
)
document = single_document(result)
assert document["status"] == envelope.STATUS_PARTIAL_DOWNLOAD
assert result.returncode == envelope.EXIT_PARTIAL_DOWNLOAD
details = document["error"]["details"]
assert details["failed"] == 1
assert details["saved"] == 1
assert details["total"] == 2
assert len(details["saved_paths"]) == 1
saved_path = Path(details["saved_paths"][0])
assert saved_path.is_file()
assert document["items"][0]["download"]["ok"] is True
assert document["source"] == SOURCE_ID
def test_unknown_source_branch_is_invalid_request_with_the_source_echoed() -> None:
result = run_cli(
"--topic",
TOPIC,
"--format",
"png",
"--width",
"100",
"--height",
"100",
"--source",
"no-such-source",
"--dry-run",
)
document = single_document(result)
assert document["status"] == envelope.STATUS_INVALID_REQUEST
assert result.returncode == envelope.EXIT_INVALID_REQUEST
assert document["source"] == "no-such-source"
assert document["error"]["code"] == "source_unknown"
assert SOURCE_ID in document["error"]["details"]["available_sources"]
def test_shipped_example_request_runs_and_parses() -> None:
expected = json.loads(EXAMPLE_REQUEST.read_text(encoding="utf-8"))
result = run_cli("examples/input.json")
document = single_document(result)
# The shipped example is a dry run, so it validates and echoes the request
# without contacting any source.
assert result.returncode == envelope.EXIT_OK
assert document["status"] == envelope.STATUS_DRY_RUN
assert document["source"] == expected["source"]
assert document["items"] == []
assert document["error"] is None
assert document["request"]["topic"] == expected["topic"]
assert document["request"]["format"] == expected["format"]
assert document["request"]["width"] == expected["width"]
assert document["request"]["height"] == expected["height"]
assert document["request"]["count"] == expected["count"]
assert document["request"]["dry_run"] is True