change: Implement 2: Add the pluggable credential-free image source registry and HTTPS fetch layer #12
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: projects/product-54354138-1#12
Reference in New Issue
Block a user
Added tools/errors.py (typed error hierarchy: ToolkitError -> SourceError -> UnknownSourceError/DuplicateSourceError/SearchRequestError/SourceAccessError with SourceUnreachableError/SourceHTTPError/SourcePayloadError/HostNotAllowedError, each with a stable code, details payload and to_dict()) and tools/sources.py (credential-free image source layer). tools/sources.py provides: an HTTPSClient restricted to the approved hosts commons.wikimedia.org, upload.wikimedia.org and api.openverse.org over HTTPS/443, sending an identifying User-Agent, with bounded exponential retry/backoff (capped) on 429/5xx and typed SourceUnreachableError/SourceHTTPError/SourcePayloadError instead of empty results; a SourceAdapter protocol; WikimediaCommonsAdapter (MediaWiki API generator=search + imageinfo) and OpenverseAdapter (/v1/images/) that pass the caller topic and result limit and normalize results into SourceCandidate records carrying source_id, file_url, reported mime type, dimensions, page URL and a byte-fetch callable; and a SourceRegistry that reports available source ids, rejects duplicate registration and raises UnknownSourceError listing the available ids. No adapter or registry path reads API keys, tokens, registrations or secrets from arguments, environment or config. Added tests/test_sources.py covering registry lookup, duplicate registration, unknown-source errors listing ids, request-URL construction with topic/limit, 429/5xx retry and exhaustion, non-retryable statuses, unreachable hosts, malformed payloads, host allowlist rejection without network contact, candidate normalization and the credential-free guarantee.
Tests:
prepare_environment restore of the tools component (locked env); tools.validate
python -m compileall tools testssucceeded; tools.testpytestsucceeded (54 passed, incl. 41 in tests/test_sources.py); tools.smokepython -m tools.example examples/input.jsonreturned status ok.Progress:
Implemented tools/errors.py, tools/sources.py and tests/test_sources.py for the credential-free source registry and HTTPS fetch layer. Verified with the facade commands tools.validate, tools.test (54 passed) and tools.smoke, all green on the integrated branch (target task/1-task was already at HEAD). No protected facade paths touched.
Published by developer-agent through Tool Backend.
Closes #4
Developer-agent completed implementation and CI passed, but updating the PR title/body failed.
Intended title: change: Implement 2: Add the pluggable credential-free image source registry and HTTPS fetch layer
Commit:
1f929c5d38CI: CI passed.
Intended body:
Added tools/errors.py (typed error hierarchy: ToolkitError -> SourceError -> UnknownSourceError/DuplicateSourceError/SearchRequestError/SourceAccessError with SourceUnreachableError/SourceHTTPError/SourcePayloadError/HostNotAllowedError, each with a stable code, details payload and to_dict()) and tools/sources.py (credential-free image source layer). tools/sources.py provides: an HTTPSClient restricted to the approved hosts commons.wikimedia.org, upload.wikimedia.org and api.openverse.org over HTTPS/443, sending an identifying User-Agent, with bounded exponential retry/backoff (capped) on 429/5xx and typed SourceUnreachableError/SourceHTTPError/SourcePayloadError instead of empty results; a SourceAdapter protocol; WikimediaCommonsAdapter (MediaWiki API generator=search + imageinfo) and OpenverseAdapter (/v1/images/) that pass the caller topic and result limit and normalize results into SourceCandidate records carrying source_id, file_url, reported mime type, dimensions, page URL and a byte-fetch callable; and a SourceRegistry that reports available source ids, rejects duplicate registration and raises UnknownSourceError listing the available ids. No adapter or registry path reads API keys, tokens, registrations or secrets from arguments, environment or config. Added tests/test_sources.py covering registry lookup, duplicate registration, unknown-source errors listing ids, request-URL construction with topic/limit, 429/5xx retry and exhaustion, non-retryable statuses, unreachable hosts, malformed payloads, host allowlist rejection without network contact, candidate normalization and the credential-free guarantee.
Tests:
prepare_environment restore of the tools component (locked env); tools.validate
python -m compileall tools testssucceeded; tools.testpytestsucceeded (54 passed, incl. 41 in tests/test_sources.py); tools.smokepython -m tools.example examples/input.jsonreturned status ok.Published by developer-agent through Tool Backend.
Approved. tools/errors.py and tools/sources.py implement the typed, credential-free source registry and HTTPS layer; tests/test_sources.py exercises registry/duplicate/unknown-source behavior, request construction, retries, host allowlisting, normalization, and credential-free operation. Exact-head CI ran the standard
pytestentrypoint with 54 passed (41 in test_sources.py) and the compile/validate/smoke stages passed. No protected facade paths changed.Findings:
urllib_transportfollows redirects without re-validating the redirect target againstAPPROVED_HOSTS; a later hardening change could disable or validate redirects. This is not part of this task's acceptance checks and does not block merging.Acceptance evidence for
1f929c5d38f2ba30057b8e9bbfa4245453707d30:tools/sources.pyexists at the head SHA and contains the complete HTTPS client, approved-host allowlist, SourceAdapter protocol, Wikimedia/Openverse adapters, SourceCandidate normalization, and SourceRegistry. The exact-headtools.validatejob compiledtools/sources.pyand exited 0. (tools/sources.py:1-160; tools/sources.py:460-578; CI job 3:315-396)tests/test_sources.pycovers registry lookup, duplicate registration, unknown-source errors listing ids, URL construction with topic/limit, 429/5xx retry and exhaustion, non-retryable statuses, unreachable hosts, malformed payloads, host allowlist rejection without network contact, normalization, and the credential-free guarantee. Exact-head CI ran it via the standardpytestentrypoint: 54 passed total, including the 41 tests intests/test_sources.py. (tests/test_sources.py:1-160; tests/test_sources.py:300-452; tests/test_sources.py:160-319; CI job 3:315-396)tools.testaspytest, and the exact-head CI log containsAGENT_CI_RESULT command_id=tools.test ... exit_code=0 ... status=passedafter collecting and passing 54 tests. (.agent/project.yaml:1-160; CI job 3:315-396)source_registrycapability is provided by the implementedSourceRegistryplusWikimediaCommonsAdapter/OpenverseAdapterintools/sources.py, covered by the passing adapter/registry tests, with the requiredtools.testCI result and successful locked-environment restore. (tools/sources.py:1-160; tools/sources.py:460-578; tests/test_sources.py:300-452; CI job 3:315-396)