human-experience-filter-art/docs/USER_GUIDE.md

Human Experience Filter — User Guide

Scope (current build state). Two pieces are built: the catalog + selection core, and the tools/ ingest & review pipeline (sub-project 2). You can populate the catalog two ways — hand-author records (below) or assisted ingest that fetches from public-domain archives, mechanically tags, drafts coordinates, and lets you review them to approved (see Ingesting & reviewing media). The room player is a separate, not-yet-built sub-project.

---

What "configuring media" means

The installation plays media chosen by where a viewer sets the control knobs. Every piece of media is one record in a catalog file. A record says what the file is, where it lives on the drive, its license, and — the important part — its coordinate in the experience-space. At run time the player will pick the record nearest the knob position; for now, you build and validate that catalog.

The catalog is a single file: catalog/library.jsonl. It is JSON Lines — one JSON object per line, one line per piece of media. Blank lines are ignored.

---

Prerequisites

The commands in this guide need Python 3.11+ and must be run from the repo root, where import hef resolves with no install step. Examples use .venv/bin/python; if you made the project virtualenv they work as-is, otherwise substitute your own python3.

Creating the virtualenv is optional but recommended:

python3 -m venv .venv

---

The record schema

Each line is a JSON object. Twelve fields are required; the rest have defaults and may be omitted.

Required fields

Field	Type	Meaning
id	string	Stable unique id (non-empty). Your choice; keep it unique.
title	string	Human title.
source_url	string	Where the piece came from.
source_archive	string	Origin label, e.g. internet_archive, musopen, librivox.
license	string	One of: public_domain, cc0, cc_by, cc_by_nc.
mode	string	One of: audio, video, av.
left	int	Analytical / verbal axis, 0–4 (see rubric).
right	int	Artistic / abstract axis, 0–4.
dark	int	Somber / heavy axis, 0–4.
light	int	Uplifting / serene axis, 0–4.
duration_s	int	Segment length in seconds (≥ 0).
file_path	string	Path to the media file on the player's drive.

Optional fields (defaults shown)

Field	Default	Meaning
review_status	"proposed"	proposed or approved — whether a human has blessed it.
attribution	""	Required text when license is cc_by or cc_by_nc.
resolution	""	e.g. 1920x1080 (video).
dominant_color	""	Hex color. Optional / opt-in — computed only with --dominant-color.
rationale	""	One-line note on why you chose the coordinate.
reviewed_at	null	Timestamp when approved.
notes	""	Free text.

Validation rules (enforced when you load/save)

• id must be non-empty.
• mode, license, review_status must be one of the allowed values above.
• left, right, dark, light must be integers in 0..4 (booleans are rejected).
• duration_s must be ≥ 0.
• cc_by and cc_by_nc require a non-empty attribution.
• Unknown fields and missing required fields are rejected.

The file referenced by file_path is not checked for existence yet — only the record's structure is validated.

---

The coordinate rubric

The four coordinates are the curatorial heart of the work. Each is a felt judgment on a 0–4 scale. The two pairs are independent knobs, not opposite ends of one slider — a piece can be high on both left and right ("whole brain"), or high on both dark and light ("bittersweet").

Axis	0 means…	4 means…
left	not analytical	strongly analytical / verbal / structured — narration, language, logic, sequence, instruction, documentary, spoken word
right	not artistic	strongly artistic / emotional / abstract — music, abstract visuals, dance, nature, awe, non-verbal, dreamlike
dark	not somber	strongly somber / heavy — ominous, melancholic, night, decay, minor-key, storms, noir
light	not light	strongly uplifting / serene — joyful, hopeful, bright, sunrise, gardens, major-key, calm

mode describes the media's channels: audio (sound only), video (image only), av (both). There is no none record — "None" is a control state the player handles by going dark, not a kind of media.

License stance: prefer public_domain / cc0; cc_by is fine if you record attribution; cc_by_nc only while the installation stays non-commercial.

---

Adding media — two ways

Option A: edit the JSONL file by hand

Append one line per piece to catalog/library.jsonl. A complete example record (formatted across lines here for readability — it must be a single line in the file):

{"id": "nasa-earthrise", "title": "Earthrise", "source_url": "https://archive.org/details/earthrise", "source_archive": "internet_archive", "license": "public_domain", "mode": "video", "left": 0, "right": 4, "dark": 1, "light": 3, "duration_s": 600, "file_path": "/media/nasa-earthrise.mp4", "resolution": "1920x1080", "rationale": "wordless awe, bright Earth on black — strongly right-brain, gently light"}

Option B: use the Python API (validates before writing)

.venv/bin/python -c '
from hef.catalog import Record, append_record
r = Record(
    id="librivox-meditations",
    title="Meditations, Book II (excerpt)",
    source_url="https://librivox.org/meditations/",
    source_archive="librivox",
    license="public_domain",
    mode="audio",
    left=4, right=1, dark=2, light=2,
    duration_s=720,
    file_path="/media/librivox-meditations-ii.mp3",
    rationale="spoken philosophy, verbal and reflective — strongly left-brain",
)
append_record(r, "catalog/library.jsonl")
print("added", r.id)
'

append_record validates the record and raises CatalogError (printing what is wrong) before it writes, so a bad record never lands in the file.

Option C: assisted ingest (fetch + tag + draft, then review)

See the next section.

---

Ingesting & reviewing media

Instead of hand-authoring every record, the tools/ pipeline can fetch a piece from a public-domain archive, fill in the mechanical fields for you, draft a coordinate, and write the record as review_status: proposed. You then walk the proposed records and bless each one to approved. The four coordinates are still a human act — the tool only drafts a starting point; nothing is selectable until you approve it.

Prerequisites

• Python 3.11+, run from the repo root (as everywhere in this guide).
• ffmpeg and ffprobe on your PATH — used to read media properties (mode/duration_s/resolution), render review previews, and (opt-in) compute dominant_color. Install via your package manager (e.g. brew install ffmpeg). The unit tests don't need them; the live ingest/review do.

Where downloaded media goes (--media-root)

Ingest downloads each file to <media-root>/<archive>/<id>.<ext> and records a file_path relative to the media root (e.g. nasa/nasa-earthrise.mp4). Relative paths are portable: the tagging workstation and the Pi's drive store the same tree under different mounts, and the player joins file_path with its own mount. The media root is --media-root DIR (or HEF_MEDIA_ROOT), default ./media/, which is gitignored — media never enters the repo, only metadata.

Note on file_path style. Hand-authored records (Option A/B above) often use absolute paths like /media/earthrise.mp4; ingested records use archive-relative paths. Both load and validate fine (validate() does not constrain the format). Keep a catalog internally consistent where you can; the player spec will define how mounts are resolved.

First-ship archives

Three keyless, clean-license archives are wired today:

archive arg	Pool	License
librivox	Public-domain audiobook recordings	public_domain
nasa	NASA imagery / video	public_domain
internet_archive	Internet Archive / Prelinger	per item (PD / CC)

musopen, fma, and freesound are defined but deferred (auth or API-stability tax) — invoking them exits with a "deferred" message. Freesound will need an API token supplied via the FREESOUND_API_TOKEN environment variable (a secret — never put it on the command line or into a record).

Running ingest

.venv/bin/python -m tools.ingest_cli nasa --query "earthrise" --limit 5
.venv/bin/python -m tools.ingest_cli internet_archive --resolve prelinger_blast
.venv/bin/python -m tools.ingest_cli librivox --query "meditations" --media-root ./media

Flags: --query (search) or --resolve <identifier> (one item); --limit N; --catalog (default catalog/library.jsonl); --media-root (default ./media, or HEF_MEDIA_ROOT); --dominant-color to compute dominant_color for video/av records (off by default — its only consumer, the procedural side walls, was dropped in the single-panoramic-projector design change). Re-running is idempotent: a candidate whose id already exists is skipped.

Reviewing proposed records

.venv/bin/python -m tools.review_cli --catalog catalog/library.jsonl --media-root ./media

For each proposed record it prints the id/title/source/license, the mechanical fields, the drafted coordinates + rationale, and opens a preview frame (video/av) or waveform image (audio). Then it prompts:

• a — accept the drafted coordinates and mark approved.
• e — edit left/right/dark/light (enter blank to keep a value), then approve.
• s — skip; leave it proposed.
• q — save and quit.

Each approval stamps reviewed_at and is written immediately (full rewrite), so an interrupted session keeps its progress. How far from done is just the count of records still proposed. Add --no-preview to skip frame rendering.

---

Validating the whole catalog

Loading the catalog validates every record. Run this after editing by hand:

.venv/bin/python -c "from hef.catalog import load_catalog; print(len(load_catalog('catalog/library.jsonl')), 'records OK')"

• Prints e.g. 12 records OK if everything is valid.
• Raises CatalogError describing the problem if not. A validation error names the field and reason, e.g. coordinate left=9 out of range 0..4; malformed JSON additionally names the line, e.g. line 4: invalid JSON: ....

---

Sanity-checking selection (optional)

To feel how tagging drives playback, you can ask the selection core which record a given knob position would pick. Coordinates are Coordinate(left, right, dark, light); the second argument is the content-mode selector (audio / video / av / none).

.venv/bin/python -c "
from hef.catalog import load_catalog
from hef.selection import Coordinate, select
lib = load_catalog('catalog/library.jsonl')
pick = select(lib, Coordinate(left=0, right=4, dark=1, light=3), 'video')
print(pick.id if pick else 'nothing')
"

Notes that shape how you tag:

• Selection is nearest-match — every knob position resolves to the closest record, so a sparse catalog still works; you do not need to fill every coordinate.
• select(..., 'none') always returns nothing (the void state).
• Pass approved_only=True to restrict to records whose review_status is approved.

---

Where the media files go

Records point at files via file_path. Those files live on the player's drive, not in this git repo (the repo holds only the catalog metadata). Keep your file_path values consistent with wherever you mount that drive on the machine that will eventually run the room.

Playing with the simulator (alteration preview)

The simulator is a web stand-in for the installation's control panel. It runs the real player.alteration engine and alters a neutral base clip toward the knob state in the browser, so you can tune the look of the filter before any hardware exists. (The earlier selection-era "curator's X-ray" view was retired when the piece moved from selecting clips to altering them.)

One-time setup — populate the sample footage (look-tuning only; not shipped content):

python simulator/setup_sample_media.py             # stage the forest neutral base
python -m simulator.bake_right_variants --scale forest   # real Right variants (SD on MPS, ~11 min)
python simulator/setup_scales_media.py             # the cosmos + abyss scales + ring transitions

setup_sample_media.py stages the forest neutral base from the session-0008 POC artifacts (~/hef-poc/out/neutral.mp4). bake_right_variants.py then produces the real flow-stabilized painterly Right variants (strengths 1–4) for the forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS, the temporally-calm POC algorithm, productionized (scales design §1/§4). The keyframe-strength ramp per Right level is by-eye tunable in simulator/bake_right_variants.py. setup_scales_media.py makes the scale ring demonstrable: cheap synthetic placeholder bases for the two true-PD scales (cosmos = NASA/Hubble, abyss = NOAA Ocean Exploration) plus the per-edge zoom/warp transition clips — these scales carry a raw base only (no real Right variants yet; the baker can extend to them once their real footage is sourced). The .mp4 binaries are gitignored.

Run it (Docker):

make sim

then open http://localhost:8000.

Run it (no Docker):

pip install -e ".[sim]"
make sim-local

What you see and can do:

• Content dial — picks audio/video channel; "off" and audio-only positions go to black walls.
• Scale ring (endless encoder) — ⊖ out / in ⊕ (or scroll the stage) walk a closed ring of neutral "scales of nature" clips — cosmos → forest → abyss and back around (diving past the smallest wraps to the largest). It is relative (an endless encoder), distinct from the absolute 0–4 knobs: each step plays a short placeholder zoom/warp transition, then settles on the next scale, with the current knob alteration still applied. The current scale is named beside the buttons (name (i/N)). A fast spin (scroll several detents at once, ≥3) collapses to a single quick blended pass straight to the destination scale instead of grinding through every transition (scales design §3); slow single steps still chain one full transition per scale crossed.
• Four experience knobs (0–4):
  • Dark / Light — a live runtime color grade (cool/dark ↔ warm/bright; equal or zero = the raw footage).
  • Right (dreamlike) — selects a discrete pre-baked, flow-stabilized restyle variant and crossfades to it (strength 0 = raw base).
  • Left (analytical) — a live overlay: labelled boxes from the clip's authored annotation track, with more annotations appearing at higher levels. Text is shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
• Calibration sliders — adjust the grade/overlay gain curves live; once a look is liked, bake the values into DEFAULT_CALIBRATION in player/alteration.py.
• RenderPlan readout — always shows the exact numbers the engine produced (the project's honesty "X-ray," now over the alteration model).

The base clips, Right variants, Left annotation track, and string tables come from simulator/sample_media/manifest.json.