|
|
|
@@ -1,3 +1,318 @@
|
|
|
|
|
# product-54354138-1
|
|
|
|
|
# product-54354138-1 image toolkit
|
|
|
|
|
|
|
|
|
|
agent-intake-source: platform/product-requests#1
|
|
|
|
|
A plain native-process command-line tool for a calling AI agent. One invocation
|
|
|
|
|
accepts a request, searches an approved free image source, verifies every
|
|
|
|
|
candidate against its *downloaded bytes* and writes **exactly one
|
|
|
|
|
machine-readable JSON document** to stdout. The process exit code mirrors the
|
|
|
|
|
document's `status`, so success, "nothing matched", a broken request and an
|
|
|
|
|
unusable source are distinguishable without parsing prose.
|
|
|
|
|
|
|
|
|
|
There is no Docker, no container image and no packaging step: the tool is run
|
|
|
|
|
directly as an executable process with the interpreter that is already present.
|
|
|
|
|
|
|
|
|
|
## Contents
|
|
|
|
|
|
|
|
|
|
- [Requirements](#requirements)
|
|
|
|
|
- [Running the tool](#running-the-tool)
|
|
|
|
|
- [Request parameters](#request-parameters)
|
|
|
|
|
- [JSON request document](#json-request-document)
|
|
|
|
|
- [JSON response document](#json-response-document)
|
|
|
|
|
- [Status vocabulary and exit codes](#status-vocabulary-and-exit-codes)
|
|
|
|
|
- [Image sources](#image-sources)
|
|
|
|
|
- [Download and resize behaviour](#download-and-resize-behaviour)
|
|
|
|
|
- [Tests and checks](#tests-and-checks)
|
|
|
|
|
|
|
|
|
|
## Requirements
|
|
|
|
|
|
|
|
|
|
- Python >= 3.11 (declared in `pyproject.toml`).
|
|
|
|
|
- No runtime third-party dependencies: the HTTPS client, PNG/ICO container
|
|
|
|
|
parsing, decoding, resampling and re-encoding all live in the standard
|
|
|
|
|
library and in `tools/`.
|
|
|
|
|
- No API keys, tokens or registration for any source (see
|
|
|
|
|
[Image sources](#image-sources)).
|
|
|
|
|
|
|
|
|
|
There is nothing to install to *use* the tool from a checkout. Installing the
|
|
|
|
|
project (`pip install .`, `uv sync`, ...) only matters for the optional test
|
|
|
|
|
tooling.
|
|
|
|
|
|
|
|
|
|
## Running the tool
|
|
|
|
|
|
|
|
|
|
The tool is always invoked as a module of the working tree:
|
|
|
|
|
|
|
|
|
|
```console
|
|
|
|
|
python -m tools.example --topic "red fox" --width 640 --height 480 --format png
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
A JSON request document can be passed instead:
|
|
|
|
|
|
|
|
|
|
```console
|
|
|
|
|
python -m tools.example examples/input.json
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`--help` is the authoritative parameter list; the table below is a summary:
|
|
|
|
|
|
|
|
|
|
```console
|
|
|
|
|
python -m tools.example --help
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Rules for the two invocation styles:
|
|
|
|
|
|
|
|
|
|
- Either a request document **or** flags must be given. With neither, the run
|
|
|
|
|
fails with `invalid_request` (exit code 2).
|
|
|
|
|
- A request document cannot be combined with any flag; doing so is
|
|
|
|
|
`invalid_request` (exit code 2).
|
|
|
|
|
|
|
|
|
|
## Request parameters
|
|
|
|
|
|
|
|
|
|
| Parameter | Kind | Meaning |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `REQUEST.json` | positional | Path to a JSON request document holding the same fields as the flags. It cannot be combined with any of the flags. |
|
|
|
|
|
| `-t`, `--topic TEXT` | text | Search topic (required unless a request document is given). |
|
|
|
|
|
| `--width PX` | positive integer | Target width in pixels; only images at least this wide are accepted. |
|
|
|
|
|
| `--height PX` | positive integer | Target height in pixels; only images at least this tall are accepted. |
|
|
|
|
|
| `-f`, `--format FMT` | `png` or `ico` | Image format; required. Any other value is `invalid_request`. |
|
|
|
|
|
| `-s`, `--source ID` | source id | Image source id (default: `wikimedia_commons`); unknown ids are rejected with the list of available sources. |
|
|
|
|
|
| `-c`, `--count N` | integer 1-50 | How many candidates to request (default: `10`). |
|
|
|
|
|
| `--download` | flag | Download every matching image into the output directory. |
|
|
|
|
|
| `-o`, `--output-dir DIR` | path | Caller-specified download directory; required with `--download`. |
|
|
|
|
|
| `--resize` | flag | Downscale every downloaded image that is larger than the request in either dimension to exactly the requested width and height, re-encoded in the requested format; images are never scaled up. Without it the original bytes and dimensions are written. |
|
|
|
|
|
| `--dry-run` | flag | Validate the request and emit the envelope without any network access. |
|
|
|
|
|
|
|
|
|
|
`--width` and `--height` are a **minimum**, not an exact match: an image that
|
|
|
|
|
is at least that large is usable because it can be downscaled, while a smaller
|
|
|
|
|
image would have to be scaled up and is therefore rejected. `--resize` decides
|
|
|
|
|
whether the *stored* file is downscaled; it never changes this filter.
|
|
|
|
|
|
|
|
|
|
`--count` bounds how many matching images are reported. Because nothing a
|
|
|
|
|
source reports about a hit is trusted, the tool inspects a bounded window of up
|
|
|
|
|
to `count * 4` candidates (capped at 50) and stops as soon as `count` verified
|
|
|
|
|
images have been found.
|
|
|
|
|
|
|
|
|
|
Short flags: `-t`/`--topic`, `-f`/`--format`, `-s`/`--source`, `-c`/`--count`,
|
|
|
|
|
`-o`/`--output-dir`, `-h`/`--help`. Every other parameter is long-form only.
|
|
|
|
|
|
|
|
|
|
## JSON request document
|
|
|
|
|
|
|
|
|
|
The document accepted as the positional argument (see `examples/input.json`)
|
|
|
|
|
is a single JSON **object** using the same names as the flags, in
|
|
|
|
|
`snake_case`:
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
{
|
|
|
|
|
"version": 1,
|
|
|
|
|
"topic": "red fox in the snow",
|
|
|
|
|
"width": 640,
|
|
|
|
|
"height": 480,
|
|
|
|
|
"format": "png",
|
|
|
|
|
"source": "wikimedia_commons",
|
|
|
|
|
"count": 5,
|
|
|
|
|
"download": false,
|
|
|
|
|
"output_dir": "downloads",
|
|
|
|
|
"resize": false,
|
|
|
|
|
"dry_run": true
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
| Field | Type | Required | Notes |
|
|
|
|
|
| --- | --- | --- | --- |
|
|
|
|
|
| `topic` | string, non-empty | yes | Same as `--topic`. |
|
|
|
|
|
| `width` | integer > 0 or `null` | no | Minimum width; omitted/`null` means unconstrained. |
|
|
|
|
|
| `height` | integer > 0 or `null` | no | Minimum height; omitted/`null` means unconstrained. |
|
|
|
|
|
| `format` | `"png"` or `"ico"` | yes | Same as `--format`. |
|
|
|
|
|
| `source` | string | no | Defaults to `wikimedia_commons`. |
|
|
|
|
|
| `count` | integer 1-50 | no | Defaults to `10`. |
|
|
|
|
|
| `download` | boolean | no | Defaults to `false`. |
|
|
|
|
|
| `output_dir` | string, non-empty | required when `download` is `true` | Download directory. |
|
|
|
|
|
| `resize` | boolean | no | Defaults to `false`. |
|
|
|
|
|
| `dry_run` | boolean | no | Defaults to `false`. |
|
|
|
|
|
| `version` | integer | no | Optional document metadata; accepted and otherwise ignored. |
|
|
|
|
|
|
|
|
|
|
Document rules that produce `invalid_request` (exit code 2):
|
|
|
|
|
|
|
|
|
|
- the file cannot be read, is not valid JSON, or is not a JSON object;
|
|
|
|
|
- an unknown field is present (`error.code = "request_field"`, with
|
|
|
|
|
`details.unknown_fields` and `details.accepted_fields`);
|
|
|
|
|
- a field has the wrong type, is empty, or is out of range;
|
|
|
|
|
- `format` is outside `png|ico`;
|
|
|
|
|
- `download` is `true` while `output_dir` is absent or empty.
|
|
|
|
|
|
|
|
|
|
## JSON response document
|
|
|
|
|
|
|
|
|
|
Every run - success or failure - prints exactly one JSON document (one line) to
|
|
|
|
|
stdout, with this fixed set of top-level keys:
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
{
|
|
|
|
|
"status": "ok",
|
|
|
|
|
"source": "wikimedia_commons",
|
|
|
|
|
"request": {"topic": "red fox", "width": 640, "height": 480, "format": "png", "source": "wikimedia_commons", "count": 1, "download": false, "output_dir": null, "resize": false, "dry_run": false},
|
|
|
|
|
"items": [
|
|
|
|
|
{
|
|
|
|
|
"source": "wikimedia_commons",
|
|
|
|
|
"url": "https://upload.wikimedia.org/wikipedia/commons/.../Fox.png",
|
|
|
|
|
"file_url": "https://upload.wikimedia.org/wikipedia/commons/.../Fox.png",
|
|
|
|
|
"format": "png",
|
|
|
|
|
"width": 1280,
|
|
|
|
|
"height": 853,
|
|
|
|
|
"size_bytes": 1489231,
|
|
|
|
|
"title": "File:Fox.png",
|
|
|
|
|
"page_url": "https://commons.wikimedia.org/wiki/File:Fox.png",
|
|
|
|
|
"mime_type": "image/png"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"error": null
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
| Key | Type | Meaning |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `status` | string | One value of the fixed vocabulary below. |
|
|
|
|
|
| `source` | string or `null` | Source id the request selected, when known. |
|
|
|
|
|
| `request` | object or `null` | The validated request (every field, defaults filled in), `null` when the request could not be validated. |
|
|
|
|
|
| `items` | array | Verified images (empty unless `status` is `ok` or `partial_download`). |
|
|
|
|
|
| `error` | object or `null` | `{"code", "message", "details"}`; `details` is optional. |
|
|
|
|
|
|
|
|
|
|
An item is only reported after its bytes were downloaded and parsed, so its
|
|
|
|
|
`format`, `width` and `height` are the **actual** values read from the payload
|
|
|
|
|
container, never what the source claimed. `url` and `file_url` are the same
|
|
|
|
|
direct HTTPS location; `file_url` is kept as an alias. `title`, `page_url` and
|
|
|
|
|
`mime_type` are present only when the source provides them. `size_bytes` is the
|
|
|
|
|
number of stored/fetched bytes.
|
|
|
|
|
|
|
|
|
|
With `download` enabled every item additionally carries a `download` record:
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
{"ok": true, "path": "downloads/001-file-fox.png", "bytes": 1489231, "format": "png", "width": 1280, "height": 853}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
With `resize` enabled the record also carries a `resize` fragment:
|
|
|
|
|
`{"requested": {...}, "from": {...}, "to": {...}, "format": "png", "applied": true}`
|
|
|
|
|
(`"applied": false, "reason": "already_fits"` when nothing had to change).
|
|
|
|
|
|
|
|
|
|
A file whose stored bytes do not read back as the requested format at the
|
|
|
|
|
expected size is deleted and reported as `{"ok": false, "target": "...",
|
|
|
|
|
"error": {...}}` instead of being named as saved.
|
|
|
|
|
|
|
|
|
|
## Status vocabulary and exit codes
|
|
|
|
|
|
|
|
|
|
The status vocabulary is fixed and every branch reuses it; the exit code is
|
|
|
|
|
derived from the status.
|
|
|
|
|
|
|
|
|
|
| `status` | Exit code | Meaning |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `ok` | 0 | At least one image really is the requested format and at least the requested width and height (all downloads completed when requested). |
|
|
|
|
|
| `dry_run` | 0 | The request was validated and echoed without contacting a source. |
|
|
|
|
|
| `invalid_request` | 2 | The request itself is unusable: bad or missing flags, unreadable/invalid document, unknown field, unsupported format, unknown source, or `download` without `output_dir`. |
|
|
|
|
|
| `no_format` | 3 | The source answered but offers nothing in the requested format for the topic. |
|
|
|
|
|
| `no_size` | 4 | The source has images of the requested format, but all of them are smaller than requested. |
|
|
|
|
|
| `access_error` | 5 | The source could not be searched, or not a single candidate could be downloaded; never reported as "no match". |
|
|
|
|
|
| `partial_download` | 6 | A download run where some images were saved and some could not be (see below). |
|
|
|
|
|
|
|
|
|
|
The distinction is intentional and stable:
|
|
|
|
|
|
|
|
|
|
- `no_format` / `no_size` describe the *topic*, not an outage, so they are not
|
|
|
|
|
errors of the source.
|
|
|
|
|
- A candidate whose bytes turn out to be a different format (JPEG, SVG, TIFF,
|
|
|
|
|
an HTML error page, ...) is a format rejection that feeds `no_format`, not an
|
|
|
|
|
access error.
|
|
|
|
|
- `access_error` means the source could not be used at all; its `error.code`
|
|
|
|
|
comes from the typed error (`source_unreachable`, `source_http_error`,
|
|
|
|
|
`source_invalid_response`, `source_host_not_allowed`, `candidate_unreadable`,
|
|
|
|
|
`access_error`).
|
|
|
|
|
- `invalid_request` always carries a machine-readable `error.code`
|
|
|
|
|
(`request_invalid`, `request_document`, `request_field`,
|
|
|
|
|
`request_format_unsupported`, `source_unknown`) plus `error.details`.
|
|
|
|
|
|
|
|
|
|
## Image sources
|
|
|
|
|
|
|
|
|
|
Two working free sources ship with the tool. Neither needs an API key, a token,
|
|
|
|
|
registration or a personal account; both are queried anonymously over HTTPS
|
|
|
|
|
with a stable identifying `User-Agent`.
|
|
|
|
|
|
|
|
|
|
| Source id | Service | Approved hosts |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `wikimedia_commons` (default) | Wikimedia Commons | `commons.wikimedia.org`, `upload.wikimedia.org` |
|
|
|
|
|
| `openverse` | Openverse | `api.openverse.org` |
|
|
|
|
|
|
|
|
|
|
Only those hosts on `https`/port 443 are contacted; any other URL is refused
|
|
|
|
|
with `source_host_not_allowed` rather than fetched. HTTP 429 and 5xx responses
|
|
|
|
|
are retried a bounded number of times with capped exponential backoff and then
|
|
|
|
|
reported as `access_error`; a failed or unknown source is never silently
|
|
|
|
|
degraded into an empty result.
|
|
|
|
|
|
|
|
|
|
An unknown `--source` value fails with `invalid_request` and lists the
|
|
|
|
|
available ids:
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
{"error": {"code": "source_unknown", "message": "unknown image source 'nope'; available sources: openverse, wikimedia_commons", "details": {"source_id": "nope", "available_sources": ["openverse", "wikimedia_commons"]}}}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Adding another source
|
|
|
|
|
|
|
|
|
|
The registry in `tools/sources.py` is the extension point:
|
|
|
|
|
|
|
|
|
|
- Implement the `SourceAdapter` protocol: a non-empty `source_id` string and
|
|
|
|
|
`search(topic, limit) -> list[SourceCandidate]`.
|
|
|
|
|
- Build `SourceCandidate` values whose `file_url` is an approved HTTPS URL and
|
|
|
|
|
whose `fetch_bytes()` returns the image bytes through `HTTPSClient`.
|
|
|
|
|
- Register the adapter with `SourceRegistry.register(adapter)` (a duplicate id
|
|
|
|
|
raises `DuplicateSourceError`; `unregister(source_id)` removes one) and wire
|
|
|
|
|
it into `build_registry()` / `default_registry()`, which is what the CLI uses.
|
|
|
|
|
`HTTPSClient(allowed_hosts=...)` and `validate_source_url()` extend the
|
|
|
|
|
approved-host allowlist for a new provider.
|
|
|
|
|
- The new id becomes selectable through `--source` / `"source"` immediately;
|
|
|
|
|
`SourceRegistry.get()` raises `UnknownSourceError` listing all registered ids
|
|
|
|
|
for anything else.
|
|
|
|
|
|
|
|
|
|
## Download and resize behaviour
|
|
|
|
|
|
|
|
|
|
Two product decisions are fixed and documented here:
|
|
|
|
|
|
|
|
|
|
1. **`--download` without `--output-dir` is an invalid request.** There is no
|
|
|
|
|
hidden default directory; a download run without a caller-supplied
|
|
|
|
|
directory could only guess where files should go. Such a run fails with
|
|
|
|
|
`status: "invalid_request"`, exit code 2, and
|
|
|
|
|
`error.code: "request_field"`, `error.details.field: "output_dir"`.
|
|
|
|
|
The same rule applies to a request document with `"download": true` and no
|
|
|
|
|
`output_dir`.
|
|
|
|
|
2. **Partial saves are reported as `partial_download`.** When a download run
|
|
|
|
|
saves some images but fails to fetch or verify others, the status is
|
|
|
|
|
`partial_download` (exit code 6) - never `ok`. The document still lists every
|
|
|
|
|
successfully saved item, and `error.code: "download_incomplete"` carries
|
|
|
|
|
`details.saved`, `details.failed`, `details.total`, `details.saved_paths`
|
|
|
|
|
and `details.failures`. A run in which every image was saved is `ok`
|
|
|
|
|
(exit code 0); a run in which nothing could be fetched at all is
|
|
|
|
|
`access_error` (exit code 5).
|
|
|
|
|
|
|
|
|
|
Files are written into `--output-dir` as `NNN-<slug>.<format>`, created with
|
|
|
|
|
`mkdir -p` semantics. Without `--resize` the verified original bytes are
|
|
|
|
|
stored unchanged. With `--resize` each stored image is downscaled to exactly
|
|
|
|
|
the requested width and height (a dimension left open keeps its source value)
|
|
|
|
|
and re-encoded in the requested container format; an image already at or below
|
|
|
|
|
the requested size is written unchanged and the resize fragment reports
|
|
|
|
|
`"applied": false`. `--resize` never enlarges an image. A search without
|
|
|
|
|
`--download` never touches the filesystem.
|
|
|
|
|
|
|
|
|
|
## Tests and checks
|
|
|
|
|
|
|
|
|
|
The test suite entry point is `pytest` from the repository root:
|
|
|
|
|
|
|
|
|
|
```console
|
|
|
|
|
pytest
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Other repository checks used by CI:
|
|
|
|
|
|
|
|
|
|
```console
|
|
|
|
|
python -m compileall tools tests
|
|
|
|
|
python -m tools.example --help
|
|
|
|
|
python -m tools.example examples/input.json
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`examples/input.json` is a `--dry-run` request, so the smoke command validates
|
|
|
|
|
the contract and prints the envelope without any network access.
|
|
|
|
|
|
|
|
|
|
Running the tests may require the declared dev dependency (`pytest>=8.0`) to be
|
|
|
|
|
installed, for example `pip install -e '.[dev]'` or `uv sync`. That install is
|
|
|
|
|
only for development; the tool itself runs as a plain process with no Docker or
|
|
|
|
|
container packaging at any stage.
|
|
|
|
|