# OpenRobot full-rig sample sessions (2026-09-11)

Two short recordings from the wearable full rig: one **head stereo camera** and two **wrist mono cameras**,
all three exposed on the same hardware trigger pulse at 30 fps, recorded by a Raspberry Pi 5 running
`openrobot-multicam`. This folder is the raw capture, losslessly repackaged, plus one alignment table
(`sync.csv`) that tells you which three frames belong together.

| # | Session | Start | Length | Frames head / wrist L / wrist R | Rig frames with all three |
|---|---|---|---|---|---|
| 1 | `full-rig_20260911_071612` | 2026-09-11 07:16:14 IST | 6m20s | 11,342 / 11,406 / 11,330 | 11,322 (99.3 %) |
| 2 | `full-rig_20260911_072454` | 2026-09-11 07:24:56 IST | 4m52s | 8,710 / 8,759 / 8,726 | 8,702 (99.3 %) |

## Rig

| Camera | Sensor | Resolution | Role on the trigger line | IMU | Exposure (fixed) |
|---|---|---|---|---|---|
| `head` | ZXCZ AR0234 stereo (two eyes side by side, one H.264 stream) | 3840 × 1200 @ 30 fps | follower | 6-axis, ~500 Hz | 2000 µs |
| `wrist_left` | ZXCZ AR0234 mono | 1920 × 1200 @ 30 fps | **master** (generates the 30 Hz pulse) | 6-axis, ~500 Hz | 2000 µs |
| `wrist_right` | ZXCZ AR0234 mono | 1920 × 1200 @ 30 fps | follower | 6-axis, ~500 Hz | 2000 µs |

The pulse is fanned out by a small sync board. Every camera opens its shutter on the same rising edge, so
"frame N of the rig" is one instant in the world. The stereo frame contains both eyes: left eye is
x 0–1919, right eye is x 1920–3839. The `wrist_left` / `wrist_right` names are the recorder's names for
the two cuffs; which physical wrist each was worn on has not been re-verified for these two sessions.

**Orientation.** The `wrist_right` camera is mounted upside down on its cuff, rolled 180° about its
optical axis relative to `wrist_left`. Its `video.mp4` carries a 180° display-rotation flag in the MP4
header, so video players show it upright. The stored H.264 bytes are unchanged, and every table
(`frames.csv`, `imu.csv`, `mag.csv`) stays in that camera's own sensor frame. Tools that ignore MP4
rotation metadata (for example `ffmpeg -noautorotate`, or a decoder reading the elementary stream) return
upside-down images: rotate them 180° after decoding. The IMU is rigidly attached to the camera, so its
axes are rolled 180° about the optical axis in the same way; account for that before comparing the two wrists.

No microphone audio is present (the streams are silent, see below). The head camera reports **no factory
stereo calibration blob** in these recordings; intrinsics/extrinsics have to come from a separate calibration.

## Layout

```
README.md                     this file
<session>/
  manifest.json               recorder manifest: rig, cameras, start/end, config
  verify.json                 recorder's own post-run integrity check (per camera)
  sync.csv                    rig-level frame alignment table (see below)
  sync_report.json            the sync statistics quoted in this README
  session.log  health.csv     recorder log and 10 s health samples (temps, RAM, throttle flags)
  checksums.sha256            sha256 of every file in the session
  <camera>/
    video.mp4                 the camera's H.264 stream, unchanged bytes, in an MP4 container, 30 fps CFR
                              (wrist_right also carries a 180° display rotation, see Orientation)
    frames.csv                one row per video frame: host arrival time + the camera's own per-frame metadata
    imu.csv                   gyro + accelerometer samples (~500 Hz), raw integers
    mag.csv                   magnetometer samples (~100 Hz), raw integers
    audio.wav                 16 kHz mono PCM from the camera's mic input (silent, not time-calibrated)
    device.json               USB identity, format, control values, sensor scale factors, run stats
```

## Timing model (read this before using timestamps)

Three clocks appear in the data:

1. **`host_ns`** (frames.csv): the Pi's monotonic clock at the moment the USB frame arrived. Common to all
   cameras, but it includes USB transfer latency (a few ms, larger for the 3840-wide stereo frames). Use it
   to *pair* frames across cameras, not as the exposure instant.
2. **Camera clock, `start_line_pts_us`** (frames.csv), and **`pts_us`** (imu.csv, mag.csv): microseconds on
   that camera's own clock. Video frames and IMU samples of the *same camera* share this clock, so IMU ↔ video
   alignment inside a camera is exact. Each camera's clock has its own epoch and its own crystal; do not
   compare `pts_us` across cameras directly.
3. **Trigger index, `trigger_index`** (frames.csv): the camera's count of trigger edges seen. On the master
   it increments by exactly one per frame; on the followers it can over-count (line ringing) so it is not a
   safe join key. It is kept for diagnostics.

**Rule: join cameras through `sync.csv`, join sensors to frames inside a camera through `pts_us`.**

### sync.csv

One row per master frame (`wrist_left`). Columns:

| column | meaning |
|---|---|
| `rig_frame` | 0-based rig frame index, equals the master's frame index |
| `t_s` | seconds since the master's first frame (host clock) |
| `master_host_ns` | master frame arrival, host monotonic ns |
| `wrist_left_frame`, `head_frame`, `wrist_right_frame` | row number in that camera's `frames.csv` = frame number in that camera's `video.mp4`; empty if that camera has no frame for this pulse |
| `*_cam_us` | that frame's `start_line_pts_us` on its own camera clock (use for IMU lookup) |
| `head_host_offset_ms`, `wrist_right_host_offset_ms` | arrival delay of the follower frame relative to the master frame |
| `all_three` | 1 when all three cameras have a frame on this pulse |

Pairing was done by nearest host arrival within ±16.6 ms (half a frame period), and every follower frame was
used at most once. Master frames without a follower frame are almost all in the first ~1.5 s (followers are
started after the master so their IMU stream initialises against a running pulse) plus a handful of skipped
exposures listed in `verify.json`.

### Evidence that the three cameras are hardware-locked

| # | follower | matched frames | cam period µs | host offset ms (mean ± sd) | offset, first → last 10 % of run | cam clock rate vs master, ppm | long periods |
|---|---|---|---|---|---|---|---|
| 1 | head | 11,340 / 11,406 | 33331 | +4.56 ± 3.30 | +4.04 → +4.67 | -53.0 | 0 |
| 1 | wrist_right | 11,327 / 11,406 | 33333 | +0.04 ± 0.53 | +0.06 → +0.09 | -0.6 | 8 |
| 2 | head | 8,704 / 8,759 | 33331 | +4.38 ± 3.31 | +4.32 → +4.89 | -53.0 | 1 |
| 2 | wrist_right | 8,725 / 8,759 | 33333 | +0.07 ± 0.46 | +0.08 → +0.14 | -0.4 | 3 |

Read it as: the arrival offset of each follower relative to the master **does not drift** over the run
(a free-running camera would walk through a full frame period every ~20 s at 50 ppm), the per-frame period
of every camera is 33 333 µs to the microsecond, and the followers' own clocks advance in lockstep with the
master's apart from a constant crystal-tolerance slope. The stereo's slightly larger host jitter is USB
transfer time for its 4.6 MP frames, not exposure jitter.

## Column reference

**frames.csv**: `frame` (0-based), `segment` (recorder restart count, 0 throughout), `v4l2_seq`, `host_ns`,
`offset`/`size` (byte range in the original raw stream; frame `n` here is frame `n` of `video.mp4`),
`trigger_index`, `start_line_pts_us`, `exposure_start_us`, `exposure_us`, `pwm_rise_pts_us` (the camera's
timestamp of the trigger edge), `valid_flags` (0x1f = all timing fields valid), `gyro_n`/`acc_n`/`mag_n`
(sensor samples carried in that frame's metadata), plus vendor SEI bookkeeping fields. Columns ending in
`_r` are the stereo's right eye; identical to the left for a locked pair.

**imu.csv**: `frame` (the video frame whose metadata carried the sample), `sensor` (`gyro`|`acc`), `x y z`
(raw signed 16-bit), `temp` (milli-°C), `pts_us` (camera clock).
Scale: gyro × 0.000532632 → rad/s (±1000 dps full scale);
accel × 0.0011971 → m/s² (±4 g).
Values are raw sensor integers; no axis rotation, bias or temperature correction applied. IMU part: TDK ICM-42688-P.

**mag.csv**: `frame`, `x y z` (raw signed integers, physical scale not established), `tout_raw` (vendor
register byte), `temp_mC` (0 = not read), `pts_us` (camera clock).

**audio.wav**: 16 kHz, mono, 16-bit, concatenated from the recorder's 5 s chunks. Not calibrated to the video
clock (only a process-launch host timestamp exists in the raw capture) and effectively silent in both sessions.

## Known limitations of these two sessions

- Followers over-count trigger edges on the monos (line ringing on the sync board). No frames were lost
  because of it, but `trigger_index` on the followers is not monotonic by exactly one.
- A few skipped exposures per camera (see `verify.json` → `problems`), visible in `sync.csv` as empty cells.
- The Pi logged an under-voltage event during session 1 (bench power supply); no capture impact.
- The first head frame is an oversized IDR that the recorder truncated at 1 MiB; decoders may drop frame 0.
- No stereo calibration blob on the camera; no camera-to-IMU extrinsics supplied.

Generated from the raw captures on the Pi `six-cam-receiver-one`; recorder: openrobot-multicam, session schema v2.