# Annotation protocol v0.1

Purpose: explore task understanding, temporal segmentation, and semantic detail
on the five supplied 6thsense videos. This is an assistant-authored draft for
review, not human ground truth or a benchmark of an installed vision model.

The protocol was drafted after inspecting bagging, fitting_board, and folding,
before inspecting bin_picking and soldering. The latter two are transfer checks
for the protocol, not statistically independent test data. All five will be
inspected in this exploration; a future benchmark needs new untouched videos.

## Units and hierarchy

- Episode: the full source recording, with an observed activity summary.
- Subtask: a meaningful step or cycle. Include preparation, transitions, and
  pauses rather than silently dropping them.
- Action: an observable manipulation within a subtask. Use a verb, stable
  within-clip object IDs, and hand roles where visually supported.
- Object-state change: include only the before/after state supported by evidence.
  'Not visible' and 'unknown' are valid states.
- Overlap is allowed, especially for holding with one hand while the other acts.
  Do not force all hand activity into one exclusive label track.

## Evidence and provenance

- Metadata and source presentation timestamps come from ffprobe.
- Samples are every 30 decoded frames, plus the final frame. The actual rate is
  approximately 30.048 fps, so these are not exact one-second samples.
- Baseline interpretation uses the left half of each side-by-side frame.
  Full paired frames are retained. This does not verify stereo calibration,
  frame synchronization, metric scale, or the sensor's capture clock.
- Semantic annotations are `assistant_visual_review`, based on timestamped
  images in this conversation. They were not produced by a locally installed
  semantic model. No separate API, model download, or cloud service is used.
- Every segment references source frame indices and actual presentation times.
  References locate supporting images; they do not prove every instant in a span.
- Sampled review cannot establish that no brief events occurred between samples.
- Filenames are descriptive hints, not ground truth. Activity summaries must
  follow the imagery. No assigned instructions were supplied.

## Time boundaries

All times are seconds relative to the first presented source video frame.
Intervals use [start, end). The last segment may end at the media duration.
The initial action boundaries are deliberately approximate, usually with a
one-second uncertainty on each side, clipped to the video endpoints. These
ranges describe review precision, not statistically calibrated confidence.
Selected transitions get denser frame inspection; only explicitly recorded
refinements may claim narrower observed brackets. Do not equate exact frame PTS
with exact semantic boundaries. Boundary starts/ends inherited from the clip
edge are truncations, not observed action starts/ends.

## Label rules

1. Describe visible motion before assigning intent: 'tip held against board'
   may support 'apparent soldering', but not a sound electrical joint.
2. Keep task inference, visible facts, and unknowns in separate fields.
3. Retain 'apparent' for contact or fastening that is not independently measured.
4. Use left/right anatomically under an unmirrored-view assumption; record that
   assumption. Use both/unknown if identity is ambiguous.
5. Do not infer force, torque, temperature, conductivity, product readiness,
   successful fastening, or exact grip type from these frames.
6. Use distinct IDs for distinguishable garments/bags; use a group ID for
   visually similar loose parts when persistent identity is not established.
7. Do not turn repeated regrips or corrections into failures without evidence.
8. Use 'no visible manipulation' for a pause. Its purpose may be unknown.
9. Require external human review for acceptance. Do not use model confidence as
   an accuracy measurement. All initial labels have review_status=pending.

## Initial reusable verbs

reach, grasp, lift, transport, place, release, hold, open, gather, manipulate,
align, insert, press, fold, unfold, smooth, rotate, operate_tool, withdraw,
adjust, inspect, pause, gesture, transition.

These are exploratory labels, not a closed industrial task taxonomy. Object
names and subtask labels retain plain-language detail alongside reusable verbs.

## Evaluation once references exist

Review the task summary, object identity, left/right role, unsupported claims,
missed visible actions, and start/end boundaries separately. Compare temporal
overlap and boundary error only after agreeing on action granularity. Count
human edits and review time. Record actual inference cost only when inference
runs exist. These five videos alone do not provide an accuracy estimate.