human-experience-filter-art/docs/USER_GUIDE.md

Human Experience Filter — User Guide

Scope (current build state). Two pieces are built: the catalog + selection core, and the tools/ ingest & review pipeline (sub-project 2). You can populate the catalog two ways — hand-author records (below) or assisted ingest that fetches from public-domain archives, mechanically tags, drafts coordinates, and lets you review them to approved (see Ingesting & reviewing media). The room player is a separate, not-yet-built sub-project.

---

What "configuring media" means

The installation plays media chosen by where a viewer sets the control knobs. Every piece of media is one record in a catalog file. A record says what the file is, where it lives on the drive, its license, and — the important part — its coordinate in the experience-space. At run time the player will pick the record nearest the knob position; for now, you build and validate that catalog.

The catalog is a single file: catalog/library.jsonl. It is JSON Lines — one JSON object per line, one line per piece of media. Blank lines are ignored.

---

Prerequisites

The commands in this guide need Python 3.11+ and must be run from the repo root, where import hef resolves with no install step. Examples use .venv/bin/python; if you made the project virtualenv they work as-is, otherwise substitute your own python3.

Creating the virtualenv is optional but recommended:

python3 -m venv .venv

---

The record schema

Each line is a JSON object. Twelve fields are required; the rest have defaults and may be omitted.

Required fields

Field	Type	Meaning
id	string	Stable unique id (non-empty). Your choice; keep it unique.
title	string	Human title.
source_url	string	Where the piece came from.
source_archive	string	Origin label, e.g. internet_archive, musopen, librivox.
license	string	One of: public_domain, cc0, cc_by, cc_by_nc.
mode	string	One of: audio, video, av.
left	int	Analytical / verbal axis, 0–4 (see rubric).
right	int	Artistic / abstract axis, 0–4.
dark	int	Somber / heavy axis, 0–4.
light	int	Uplifting / serene axis, 0–4.
duration_s	int	Segment length in seconds (≥ 0).
file_path	string	Path to the media file on the player's drive.

Optional fields (defaults shown)

Field	Default	Meaning
review_status	"proposed"	proposed or approved — whether a human has blessed it.
attribution	""	Required text when license is cc_by or cc_by_nc.
resolution	""	e.g. 1920x1080 (video).
dominant_color	""	Hex color. Optional / opt-in — computed only with --dominant-color.
rationale	""	One-line note on why you chose the coordinate.
reviewed_at	null	Timestamp when approved.
notes	""	Free text.

Validation rules (enforced when you load/save)

• id must be non-empty.
• mode, license, review_status must be one of the allowed values above.
• left, right, dark, light must be integers in 0..4 (booleans are rejected).
• duration_s must be ≥ 0.
• cc_by and cc_by_nc require a non-empty attribution.
• Unknown fields and missing required fields are rejected.

The file referenced by file_path is not checked for existence yet — only the record's structure is validated.

---

The coordinate rubric

The four coordinates are the curatorial heart of the work. Each is a felt judgment on a 0–4 scale. The two pairs are independent knobs, not opposite ends of one slider — a piece can be high on both left and right ("whole brain"), or high on both dark and light ("bittersweet").

Axis	0 means…	4 means…
left	not analytical	strongly analytical / verbal / structured — narration, language, logic, sequence, instruction, documentary, spoken word
right	not artistic	strongly artistic / emotional / abstract — music, abstract visuals, dance, nature, awe, non-verbal, dreamlike
dark	not somber	strongly somber / heavy — ominous, melancholic, night, decay, minor-key, storms, noir
light	not light	strongly uplifting / serene — joyful, hopeful, bright, sunrise, gardens, major-key, calm

mode describes the media's channels: audio (sound only), video (image only), av (both). There is no none record — "None" is a control state the player handles by going dark, not a kind of media.

License stance: prefer public_domain / cc0; cc_by is fine if you record attribution; cc_by_nc only while the installation stays non-commercial.

---

Adding media — two ways

Option A: edit the JSONL file by hand

Append one line per piece to catalog/library.jsonl. A complete example record (formatted across lines here for readability — it must be a single line in the file):

{"id": "nasa-earthrise", "title": "Earthrise", "source_url": "https://archive.org/details/earthrise", "source_archive": "internet_archive", "license": "public_domain", "mode": "video", "left": 0, "right": 4, "dark": 1, "light": 3, "duration_s": 600, "file_path": "/media/nasa-earthrise.mp4", "resolution": "1920x1080", "rationale": "wordless awe, bright Earth on black — strongly right-brain, gently light"}

Option B: use the Python API (validates before writing)

.venv/bin/python -c '
from hef.catalog import Record, append_record
r = Record(
    id="librivox-meditations",
    title="Meditations, Book II (excerpt)",
    source_url="https://librivox.org/meditations/",
    source_archive="librivox",
    license="public_domain",
    mode="audio",
    left=4, right=1, dark=2, light=2,
    duration_s=720,
    file_path="/media/librivox-meditations-ii.mp3",
    rationale="spoken philosophy, verbal and reflective — strongly left-brain",
)
append_record(r, "catalog/library.jsonl")
print("added", r.id)
'

append_record validates the record and raises CatalogError (printing what is wrong) before it writes, so a bad record never lands in the file.

Option C: assisted ingest (fetch + tag + draft, then review)

See the next section.

---

Ingesting & reviewing media

Instead of hand-authoring every record, the tools/ pipeline can fetch a piece from a public-domain archive, fill in the mechanical fields for you, draft a coordinate, and write the record as review_status: proposed. You then walk the proposed records and bless each one to approved. The four coordinates are still a human act — the tool only drafts a starting point; nothing is selectable until you approve it.

Prerequisites

• Python 3.11+, run from the repo root (as everywhere in this guide).
• ffmpeg and ffprobe on your PATH — used to read media properties (mode/duration_s/resolution), render review previews, and (opt-in) compute dominant_color. Install via your package manager (e.g. brew install ffmpeg). The unit tests don't need them; the live ingest/review do.

Where downloaded media goes (--media-root)

Ingest downloads each file to <media-root>/<archive>/<id>.<ext> and records a file_path relative to the media root (e.g. nasa/nasa-earthrise.mp4). Relative paths are portable: the tagging workstation and the Pi's drive store the same tree under different mounts, and the player joins file_path with its own mount. The media root is --media-root DIR (or HEF_MEDIA_ROOT), default ./media/, which is gitignored — media never enters the repo, only metadata.

Note on file_path style. Hand-authored records (Option A/B above) often use absolute paths like /media/earthrise.mp4; ingested records use archive-relative paths. Both load and validate fine (validate() does not constrain the format). Keep a catalog internally consistent where you can; the player spec will define how mounts are resolved.

First-ship archives

Three keyless, clean-license archives are wired today:

archive arg	Pool	License
librivox	Public-domain audiobook recordings	public_domain
nasa	NASA imagery / video	public_domain
internet_archive	Internet Archive / Prelinger	per item (PD / CC)

musopen, fma, and freesound are defined but deferred (auth or API-stability tax) — invoking them exits with a "deferred" message. Freesound will need an API token supplied via the FREESOUND_API_TOKEN environment variable (a secret — never put it on the command line or into a record).

Running ingest

.venv/bin/python -m tools.ingest_cli nasa --query "earthrise" --limit 5
.venv/bin/python -m tools.ingest_cli internet_archive --resolve prelinger_blast
.venv/bin/python -m tools.ingest_cli librivox --query "meditations" --media-root ./media

Flags: --query (search) or --resolve <identifier> (one item); --limit N; --catalog (default catalog/library.jsonl); --media-root (default ./media, or HEF_MEDIA_ROOT); --dominant-color to compute dominant_color for video/av records (off by default — its only consumer, the procedural side walls, was dropped in the single-panoramic-projector design change). Re-running is idempotent: a candidate whose id already exists is skipped.

Reviewing proposed records

.venv/bin/python -m tools.review_cli --catalog catalog/library.jsonl --media-root ./media

For each proposed record it prints the id/title/source/license, the mechanical fields, the drafted coordinates + rationale, and opens a preview frame (video/av) or waveform image (audio). Then it prompts:

• a — accept the drafted coordinates and mark approved.
• e — edit left/right/dark/light (enter blank to keep a value), then approve.
• s — skip; leave it proposed.
• q — save and quit.

Each approval stamps reviewed_at and is written immediately (full rewrite), so an interrupted session keeps its progress. How far from done is just the count of records still proposed. Add --no-preview to skip frame rendering.

---

Validating the whole catalog

Loading the catalog validates every record. Run this after editing by hand:

.venv/bin/python -c "from hef.catalog import load_catalog; print(len(load_catalog('catalog/library.jsonl')), 'records OK')"

• Prints e.g. 12 records OK if everything is valid.
• Raises CatalogError describing the problem if not. A validation error names the field and reason, e.g. coordinate left=9 out of range 0..4; malformed JSON additionally names the line, e.g. line 4: invalid JSON: ....

---

Sanity-checking selection (optional)

To feel how tagging drives playback, you can ask the selection core which record a given knob position would pick. Coordinates are Coordinate(left, right, dark, light); the second argument is the content-mode selector (audio / video / av / none).

.venv/bin/python -c "
from hef.catalog import load_catalog
from hef.selection import Coordinate, select
lib = load_catalog('catalog/library.jsonl')
pick = select(lib, Coordinate(left=0, right=4, dark=1, light=3), 'video')
print(pick.id if pick else 'nothing')
"

Notes that shape how you tag:

• Selection is nearest-match — every knob position resolves to the closest record, so a sparse catalog still works; you do not need to fill every coordinate.
• select(..., 'none') always returns nothing (the void state).
• Pass approved_only=True to restrict to records whose review_status is approved.

---

Where the media files go

Records point at files via file_path. Those files live on the player's drive, not in this git repo (the repo holds only the catalog metadata). Keep your file_path values consistent with wherever you mount that drive on the machine that will eventually run the room.

Playing with the simulator (alteration preview)

The simulator is a web stand-in for the installation's control panel. It runs the real player.alteration engine and alters a neutral base clip toward the knob state in the browser, so you can tune the look of the filter before any hardware exists. (The earlier selection-era "curator's X-ray" view was retired when the piece moved from selecting clips to altering them.)

One-time setup — populate the sample footage (look-tuning only; not shipped content). The ring uses real strict-PD footage in a rotating pool per scale (docs/content-candidate-pool.md); regenerate the manifest + new transition placeholders with:

python simulator/build_pool_manifest.py --media     # write the pool manifest + coast-edge transitions

build_pool_manifest.py holds the hand-authored label/affect content and emits simulator/sample_media/manifest.json (the 19-clip pool baseline). The pool clips themselves are sourced via the content pipeline (tools/pipeline/run.py process_clip); the .mp4 binaries are gitignored. (setup_scales_media.py is the older one-base-per-scale generator, now superseded.)

Run it (Docker):

make sim

then open http://localhost:8000.

Run it (no Docker):

pip install -e ".[sim]"
make sim-local

What you see and can do:

• Content dial — picks audio/video channel; "off" and audio-only positions go to black walls.
• Scale ring (endless encoder) — ⊖ out / in ⊕ (or scroll the stage) walk a closed ring of neutral "scales of nature" — cosmos → orbit → coast → reef → abyss and back around (diving past the smallest wraps to the largest). Each scale is a rotating pool of vetted clips: landing on a scale plays a random pool member, so the ring feels fresh each pass. It is relative (an endless encoder), distinct from the absolute 0–4 knobs: each step plays a short placeholder zoom/warp transition, then settles, with the current alteration still applied. The current scale + chosen member is named beside the buttons (scale · member (i/N · pool N)). A fast spin (≥3 detents at once) collapses to one quick blended pass instead of grinding through every transition.
• Three experience knobs:
  • Mood (−4 dark .. 0 .. +4 light) — a live runtime color grade (cool/dark ↔ warm/bright; 0 = the raw footage).
  • Right (dreamlike, 0–4) — a deterministic real-time painterly dream (a WebGL Kuwahara restyle of the live frames; holds still across the loop, goes trippy at max). It also drives progressive emotion tiers: when both knobs are up, the affect words escalate from basic to compound as Right rises.
  • Left (analytical, 0–4) — a live HUD of labelled boxes from the clip's authored annotation track. Labels use progressive detail tiers: low Left shows fewer, more general labels (high-salience objects only); raising Left brings in more objects and escalates each label general → specific → scientific → +fact. Time-windowed tracked labels appear only while their subject is on screen and follow it.
• RenderPlan readout — always shows the exact numbers the engine produced (the project's honesty "X-ray," over the alteration model).

The base clips, the Left annotation tracks (with tiers + appear/disappear windows), affect anchors, and per-tier string tables all come from simulator/sample_media/manifest.json.

Authoring labels — the author mode

Open http://localhost:8000/author.html to author a clip's labels by eye (content-pipeline §11.5). Pick a pool clip, scrub to where a subject is on screen, drag a box over it, set the label key + salience + the four detail tiers, then Run tracker to propagate the box into a keyframed track (classical OpenCV optical flow) with an appear/disappear window — or Add as static box for a fixed label. Shift-click the stage to place an affect anchor and type its emotion tiers. Save to manifest writes the entry back via the pipeline's idempotent upsert (keeping the clip's media + provenance, replacing only the authored content). The tracker only propagates box geometry — every label string, scientific name, and fact is hand-authored, because a generic detector can't produce "Gymnothorax" or a lifespan.