Add drone ingest pipeline + local D1 dev config

- pipeline/: DJI SRT telemetry parser, ffmpeg frame extraction, detector
  interface (stub + ONNX hook), pinhole/flat-ground georeferencer, ingest client
- Verified end-to-end: synthetic flight → 319 georeferenced detections → D1 → dashboard
- wrangler.dev.toml enables local D1 (live deploy stays simulator-only until D1 token)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
maverick 2026-07-21 21:41:59 +10:00
parent 63393c052f
commit 328b3c53df
10 changed files with 555 additions and 1 deletions

View file

@ -49,7 +49,15 @@ Open `/` for the dashboard. Point it anywhere:
The georeferencing step (frame + drone GPS + gimbal angle + terrain → lat/lon) lives
in the drone pipeline, not here — this app just needs the resulting detections.
## Drone ingest pipeline
See [`pipeline/`](pipeline/) — turns real drone footage + DJI telemetry into
georeferenced detections and POSTs them here. Runs end-to-end today via a
synthetic flight + stub detector; drop in a trained YOLO model for real thermal.
## Roadmap
- [ ] Drone ingest pipeline (RTMP/SRT frames → YOLO thermal detector → georeferencer)
- [x] Drone ingest pipeline (frames + SRT telemetry → detector → georeferencer → ingest)
- [ ] Wire a trained thermal-wildlife YOLO model into the ONNX detector
- [ ] Remote D1 (needs a D1-capable Cloudflare token; live site runs the simulator until then)
- [ ] Live mode (stream detections during flight, not just post-survey)
- [ ] Track clustering (merge repeat sightings of the same animal across passes)
- [ ] DEM-based georeferencing (ray/terrain intersection instead of flat ground)

68
pipeline/README.md Normal file
View file

@ -0,0 +1,68 @@
# CritterScope drone ingest pipeline
Turns drone footage + telemetry into georeferenced detections and pushes them to
a CritterScope instance.
```
video / RTMP frames + telemetry (DJI .SRT)
│ │
extractFrames parseDjiSrt
└──────────┬─────────────┘
detector (stub | ONNX YOLO) → bbox per frame
pixelToGround → lat/lon per detection
POST /api/ingest → D1
view at /?survey=<id>
```
## Quick start (no drone needed)
Runs a synthetic lawnmower flight through the stub detector and prints what it
would send — proves the whole chain:
```bash
cd pipeline
node src/run.js --synthetic --dry-run
```
Push it into a running local CritterScope (with local D1, see root README):
```bash
node src/run.js --synthetic --name "test-survey" --base-url http://localhost:8787
# → View it: http://localhost:8787/?survey=sv-test-survey-...
```
## Real footage
```bash
node src/run.js \
--source flight.MP4 --srt flight.SRT \
--detector onnx --model yolo-thermal.onnx \
--width 1920 --height 1080 \
--drone "DJI Mavic 2 Enterprise Advanced" \
--base-url https://critterscope.theradicalparty.com
```
## Live RTMP (DJI Fly app livestream)
Point the DJI Fly app's RTMP stream at a local server, then:
```bash
node src/run.js --source rtmp://localhost/live/stream --srt live.SRT --fps 1
```
## The three stages you can swap
- **Telemetry** (`src/telemetry.js`) — DJI SRT parser + synthetic flight. Add other
drones by mapping their telemetry into `{lat, lon, altAGL, heading, gimbalPitch, hfovDeg}`.
- **Detector** (`src/detector.js`) — `stub` (works now) and `onnx` (wire your trained
YOLO: JPEG decode → letterbox → run → NMS → map class ids to species). Everything
downstream is model-agnostic.
- **Georeferencer** (`src/georef.js`) — pinhole camera + flat-ground projection.
Verified: centre pixel at nadir → drone position; right-edge offset matches
`altitude·tan(hfov/2)`; near-horizon rays correctly return no ground hit.
Next refinement: intersect the ray with a DEM instead of flat ground.
## Requirements
- Node 18+ (uses global `fetch`)
- `ffmpeg` on PATH (only for `--source`)
- `onnxruntime-node` (optional, only for `--detector onnx`)
## What's real vs. stubbed
- ✅ telemetry parse, frame extraction, georeferencing, survey assembly, ingest, map render
- 🔩 the ONNX detector's tensor plumbing is left for when you have a trained thermal
wildlife model — the stub detector exercises the identical downstream path today.

15
pipeline/package.json Normal file
View file

@ -0,0 +1,15 @@
{
"name": "critterscope-pipeline",
"version": "0.1.0",
"description": "Drone footage + telemetry → georeferenced detections → CritterScope",
"private": true,
"type": "module",
"bin": { "critterscope-ingest": "src/run.js" },
"scripts": {
"demo": "node src/run.js --synthetic --dry-run",
"demo:ingest": "node src/run.js --synthetic --base-url http://localhost:8787"
},
"optionalDependencies": {
"onnxruntime-node": "^1.19.0"
}
}

81
pipeline/src/detector.js Normal file
View file

@ -0,0 +1,81 @@
// Detector interface.
//
// A detector takes a frame (path to a JPEG + width/height) and returns an array
// of detections: { type, confidence, bbox:[x,y,w,h] } in pixel coordinates.
//
// Two backends:
// - onnx: real YOLO via onnxruntime-node (needs a model + the optional dep)
// - stub: deterministic synthetic detections so the whole pipeline runs and
// can be validated end-to-end without a model or real footage.
//
// The class list is COCO-ish remapped to the CritterScope species vocabulary.
// For thermal you'd train/fine-tune on thermal wildlife imagery; the interface
// is identical.
const SPECIES = ["kangaroo", "deer", "rabbit", "fox", "boar", "person", "vehicle", "unknown"];
function mulberry32(seed) {
let a = seed >>> 0;
return () => {
a = (a + 0x6d2b79f5) | 0;
let t = Math.imul(a ^ (a >>> 15), 1 | a);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
export function createStubDetector({ W = 1920, H = 1080, seed = 99 } = {}) {
return {
name: "stub",
async detect(frame) {
// seed by frame index so results are stable & reproducible per run
const rng = mulberry32(seed + (frame.index | 0) * 2654435761);
const n = Math.floor(rng() * 3); // 0..2 animals per frame
const out = [];
for (let i = 0; i < n; i++) {
const type = SPECIES[Math.floor(rng() * SPECIES.length)];
const w = 30 + rng() * 90;
const h = 30 + rng() * 90;
// keep detections toward frame centre (better georef geometry)
const x = W * (0.2 + rng() * 0.6) - w / 2;
const y = H * (0.2 + rng() * 0.6) - h / 2;
out.push({
type,
confidence: Number((0.4 + rng() * 0.58).toFixed(2)),
bbox: [x, y, w, h],
});
}
return out;
},
};
}
// Real YOLO backend. Lazily requires onnxruntime-node so the stub path has no
// heavy dependency. Provide a model exported to ONNX and a labels→species map.
export async function createOnnxDetector({ modelPath, labelMap = {}, W, H, confThreshold = 0.35 }) {
let ort;
try {
ort = await import("onnxruntime-node");
} catch (e) {
throw new Error(
"onnxruntime-node not installed. Run `npm i onnxruntime-node` in pipeline/ to use the ONNX detector."
);
}
const session = await ort.InferenceSession.create(modelPath);
return {
name: "onnx",
session,
async detect(frame) {
// Intentionally a documented stub of the tensor plumbing: decode JPEG →
// letterbox to model input → run → NMS → map class ids to species.
// Wire this once you have a trained model; the georef/ingest stages are
// already model-agnostic.
throw new Error(
"ONNX detect() not wired — plug in JPEG decode + preprocessing for your model. " +
"Everything downstream (georef, ingest) already works via the stub detector."
);
},
};
}
export const SPECIES_LIST = SPECIES;

37
pipeline/src/frames.js Normal file
View file

@ -0,0 +1,37 @@
// Frame extraction via ffmpeg. Works on a video file or a live RTMP/RTSP URL.
// Samples at a fixed fps into a temp dir and returns the frame file list.
// (DJI Fly livestreams RTMP; point --source at rtmp://<your-server>/live/stream.)
import { spawn } from "node:child_process";
import { mkdtempSync, readdirSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
export function extractFrames({ source, fps = 1, maxFrames = 0 } = {}) {
return new Promise((resolve, reject) => {
const dir = mkdtempSync(join(tmpdir(), "critter-frames-"));
const args = [
"-hide_banner", "-loglevel", "error",
...(source.startsWith("rtmp") || source.startsWith("rtsp") ? ["-rtsp_transport", "tcp"] : []),
"-i", source,
"-vf", `fps=${fps}`,
...(maxFrames ? ["-frames:v", String(maxFrames)] : []),
"-q:v", "3",
join(dir, "f_%06d.jpg"),
];
const ff = spawn("ffmpeg", args);
let err = "";
ff.stderr.on("data", (d) => (err += d));
ff.on("error", (e) =>
reject(new Error(`ffmpeg not found or failed to start: ${e.message}. Install ffmpeg.`))
);
ff.on("close", (code) => {
if (code !== 0) return reject(new Error(`ffmpeg exited ${code}: ${err.slice(0, 400)}`));
const files = readdirSync(dir)
.filter((f) => f.endsWith(".jpg"))
.sort()
.map((f, i) => ({ index: i, path: join(dir, f) }));
resolve({ dir, files });
});
});
}

87
pipeline/src/georef.js Normal file
View file

@ -0,0 +1,87 @@
// Pixel → ground georeferencing.
//
// Given a detection's pixel in a drone frame plus the drone's telemetry
// (position, altitude above ground, camera heading + gimbal pitch, and the
// camera field of view), project that pixel onto flat ground and return the
// lat/lon it corresponds to.
//
// Model: pinhole camera + flat-earth ground plane. This is the standard
// first-order approach and is accurate to a few metres for typical survey
// altitudes over gently sloping terrain. (A DEM-based intersection would remove
// the flat-ground assumption; that's a later refinement.)
//
// Frame convention: world ENU (x=East, y=North, z=Up). Drone at (0,0,h).
// Image: +u right, +v down, principal point at centre.
const D2R = Math.PI / 180;
function norm(v) {
const m = Math.hypot(v[0], v[1], v[2]);
return [v[0] / m, v[1] / m, v[2] / m];
}
const cross = (a, b) => [
a[1] * b[2] - a[2] * b[1],
a[2] * b[0] - a[0] * b[2],
a[0] * b[1] - a[1] * b[0],
];
/**
* @param {object} p
* @param {number} p.u pixel x (0..W)
* @param {number} p.v pixel y (0..H)
* @param {number} p.W image width px
* @param {number} p.H image height px
* @param {number} p.hfovDeg horizontal field of view (deg)
* @param {number} [p.vfovDeg] vertical FOV (deg); derived from aspect if omitted
* @param {number} p.lat drone latitude
* @param {number} p.lon drone longitude
* @param {number} p.altAGL drone height above ground (m)
* @param {number} p.headingDeg camera azimuth, clockwise from North (deg)
* @param {number} p.gimbalPitchDeg gimbal pitch, 0=horizon, -90=straight down
* @returns {{lat:number, lon:number, slantRange:number}|null}
*/
export function pixelToGround(p) {
const {
u, v, W, H, hfovDeg, lat, lon, altAGL, headingDeg, gimbalPitchDeg,
} = p;
const vfovDeg = p.vfovDeg ?? (2 * Math.atan(Math.tan((hfovDeg * D2R) / 2) * (H / W))) / D2R;
const fx = W / 2 / Math.tan((hfovDeg * D2R) / 2);
const fy = H / 2 / Math.tan((vfovDeg * D2R) / 2);
const cx = W / 2;
const cy = H / 2;
// camera-axis (forward) direction in world ENU
const a = headingDeg * D2R;
const pit = gimbalPitchDeg * D2R;
const forward = norm([
Math.cos(pit) * Math.sin(a), // E
Math.cos(pit) * Math.cos(a), // N
Math.sin(pit), // U (negative when looking down)
]);
// right = horizontal, 90° clockwise from heading (roll assumed 0)
const right = [Math.cos(a), -Math.sin(a), 0];
// image-down axis completes the right-handed camera frame
const imgDown = norm(cross(forward, right));
// ray for pixel (u,v)
const nx = (u - cx) / fx;
const ny = (v - cy) / fy;
const dir = norm([
forward[0] + nx * right[0] + ny * imgDown[0],
forward[1] + nx * right[1] + ny * imgDown[1],
forward[2] + nx * right[2] + ny * imgDown[2],
]);
if (dir[2] >= -1e-6) return null; // ray points at/above horizon → no ground hit
const t = altAGL / -dir[2]; // drone at z=altAGL, ground z=0
const east = t * dir[0];
const north = t * dir[1];
const slantRange = t;
const dLat = north / 111320;
const dLon = east / (111320 * Math.cos(lat * D2R));
return { lat: lat + dLat, lon: lon + dLon, slantRange };
}

16
pipeline/src/ingest.js Normal file
View file

@ -0,0 +1,16 @@
// POST an assembled survey to a CritterScope instance.
export async function postSurvey({ baseUrl, survey, flight, detections }) {
const res = await fetch(`${baseUrl.replace(/\/$/, "")}/api/ingest`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ survey, flight, detections }),
});
const text = await res.text();
let body;
try { body = JSON.parse(text); } catch { body = text; }
if (!res.ok) {
throw new Error(`ingest failed ${res.status}: ${typeof body === "string" ? body : JSON.stringify(body)}`);
}
return body;
}

142
pipeline/src/run.js Normal file
View file

@ -0,0 +1,142 @@
#!/usr/bin/env node
// CritterScope drone ingest pipeline.
//
// video/RTMP frames + telemetry (DJI SRT) → detector → georeferencer
// → survey JSON → POST /api/ingest → view at /?survey=<id>
//
// Examples
// # end-to-end test, no drone needed (synthetic flight + stub detector):
// node src/run.js --synthetic --base-url http://localhost:8787
//
// # real footage:
// node src/run.js --source flight.MP4 --srt flight.SRT --detector onnx \
// --model yolo-thermal.onnx --base-url https://critterscope.theradicalparty.com
//
// # live RTMP stream from the DJI Fly app:
// node src/run.js --source rtmp://localhost/live/stream --srt live.SRT --fps 1
import { parseDjiSrt, syntheticFlight } from "./telemetry.js";
import { extractFrames } from "./frames.js";
import { createStubDetector, createOnnxDetector } from "./detector.js";
import { pixelToGround } from "./georef.js";
import { postSurvey } from "./ingest.js";
function parseArgs(argv) {
const a = { fps: 1, detector: "stub", baseUrl: "http://localhost:8787", width: 1920, height: 1080, radius: 1000 };
for (let i = 2; i < argv.length; i++) {
const k = argv[i];
const nx = () => argv[++i];
switch (k) {
case "--source": a.source = nx(); break;
case "--srt": a.srt = nx(); break;
case "--synthetic": a.synthetic = true; break;
case "--detector": a.detector = nx(); break;
case "--model": a.model = nx(); break;
case "--base-url": a.baseUrl = nx(); break;
case "--fps": a.fps = parseFloat(nx()); break;
case "--width": a.width = parseInt(nx(), 10); break;
case "--height": a.height = parseInt(nx(), 10); break;
case "--center": { const [lon, lat] = nx().split(",").map(Number); a.center = [lon, lat]; break; }
case "--radius": a.radius = parseFloat(nx()); break;
case "--name": a.name = nx(); break;
case "--drone": a.drone = nx(); break;
case "--dry-run": a.dryRun = true; break;
default: console.warn("unknown arg", k);
}
}
return a;
}
async function main() {
const a = parseArgs(process.argv);
// 1. telemetry
let samples, meta;
if (a.srt) {
samples = parseDjiSrt(a.srt);
if (!samples.length) throw new Error(`no telemetry parsed from ${a.srt}`);
meta = null;
} else if (a.synthetic) {
({ samples, meta } = syntheticFlight({ center: a.center, radiusM: a.radius }));
} else {
throw new Error("provide --srt <file> or --synthetic");
}
console.log(`telemetry: ${samples.length} samples`);
// 2. frames (optional in synthetic+stub mode; required for real detection)
let frames = null, tmpDir = null;
if (a.source) {
const r = await extractFrames({ source: a.source, fps: a.fps });
frames = r.files; tmpDir = r.dir;
console.log(`frames: ${frames.length} extracted → ${tmpDir}`);
}
// 3. detector
const detector =
a.detector === "onnx"
? await createOnnxDetector({ modelPath: a.model, W: a.width, H: a.height })
: createStubDetector({ W: a.width, H: a.height });
console.log(`detector: ${detector.name}`);
// Decide the unit of work: real frames if we have them, else one per sample.
const units = frames
? frames.map((f, i) => ({ frame: f, sample: samples[Math.round((i / Math.max(1, frames.length - 1)) * (samples.length - 1))] }))
: samples.map((s) => ({ frame: { index: s.index, path: null }, sample: s }));
// 4. detect + georeference
const detections = [];
let seq = 0;
for (const { frame, sample } of units) {
const dets = await detector.detect({ index: frame.index, path: frame.path, W: a.width, H: a.height });
for (const d of dets) {
const [x, y, w, h] = d.bbox;
const u = x + w / 2;
const v = y + h / 2;
const g = pixelToGround({
u, v, W: a.width, H: a.height,
hfovDeg: sample.hfovDeg ?? 73,
lat: sample.lat, lon: sample.lon, altAGL: sample.altAGL,
headingDeg: sample.heading, gimbalPitchDeg: sample.gimbalPitch,
});
if (!g) continue;
detections.push({
id: `det-${seq++}`,
type: d.type,
lat: g.lat, lon: g.lon,
confidence: d.confidence,
temp_c: d.temp_c ?? null,
t: sample.t ?? null,
frame: frame.index,
});
}
}
console.log(`detections: ${detections.length} georeferenced`);
// 5. assemble survey
const flight = samples.map((s, i) => ({ seq: i, lon: s.lon, lat: s.lat, agl_m: s.altAGL, t: s.t ?? null }));
const centerLon = meta?.center_lon ?? flight.reduce((a2, p) => a2 + p.lon, 0) / flight.length;
const centerLat = meta?.center_lat ?? flight.reduce((a2, p) => a2 + p.lat, 0) / flight.length;
const survey = {
id: `sv-${a.name ? a.name.replace(/\W+/g, "-") : "run"}-${flight[0]?.t?.slice(0, 19) || "t0"}`,
name: a.name ?? "Drone survey",
center_lon: centerLon,
center_lat: centerLat,
radius_m: meta?.radius_m ?? a.radius,
sensor: a.detector === "onnx" ? "thermal+rgb" : "rgb",
drone: a.drone ?? (a.synthetic ? "SYNTHETIC" : "unknown"),
started_at: flight[0]?.t ?? null,
ended_at: flight[flight.length - 1]?.t ?? null,
};
if (a.dryRun) {
console.log(JSON.stringify({ survey, flightPoints: flight.length, detections }, null, 2).slice(0, 4000));
console.log(`\n[dry-run] would POST ${detections.length} detections to ${a.baseUrl}/api/ingest`);
return;
}
const res = await postSurvey({ baseUrl: a.baseUrl, survey, flight, detections });
console.log("ingest:", res);
console.log(`\nView it: ${a.baseUrl}/?survey=${encodeURIComponent(survey.id)}`);
}
main().catch((e) => { console.error("pipeline error:", e.message); process.exit(1); });

78
pipeline/src/telemetry.js Normal file
View file

@ -0,0 +1,78 @@
// Telemetry providers: per-frame drone pose (lat, lon, altAGL, heading, gimbal
// pitch, timestamp). Two sources:
// - DJI .SRT sidecar (what DJI drones write alongside video)
// - synthetic lawnmower flight (for testing without footage)
import { readFileSync } from "node:fs";
// Parse a DJI SRT subtitle telemetry file into an array of samples.
// DJI encodes fields like [latitude: -33.7301] [longitude: 150.31] [rel_alt: 118.5]
// [gb_yaw: 12.3] [gb_pitch: -89.0] plus a per-block timecode.
export function parseDjiSrt(path) {
const raw = readFileSync(path, "utf8");
const blocks = raw.split(/\r?\n\r?\n/).filter((b) => b.trim());
const samples = [];
const num = (s, key) => {
const m = s.match(new RegExp(`\\[${key}\\s*:\\s*(-?\\d+(?:\\.\\d+)?)`, "i"));
return m ? parseFloat(m[1]) : undefined;
};
for (let i = 0; i < blocks.length; i++) {
const b = blocks[i];
const lat = num(b, "latitude") ?? num(b, "GPS.*?lat");
const lon = num(b, "longitude") ?? num(b, "GPS.*?lon");
if (lat === undefined || lon === undefined) continue;
samples.push({
index: i,
lat,
lon,
altAGL: num(b, "rel_alt") ?? num(b, "altitude") ?? 100,
heading: num(b, "gb_yaw") ?? num(b, "yaw") ?? 0,
gimbalPitch: num(b, "gb_pitch") ?? -90,
hfovDeg: 73, // DJI Mini-class wide FOV default; override via CLI if known
});
}
return samples;
}
// Synthetic lawnmower flight for testing. Returns { samples, meta }.
export function syntheticFlight({
center = [150.3119, -33.73],
radiusM = 1000,
legs = 8,
altAGL = 120,
hfovDeg = 73,
startISO = "2026-07-21T06:20:00.000Z",
} = {}) {
const [lon, lat] = center;
const dLat = radiusM / 111320;
const dLon = radiusM / (111320 * Math.cos((lat * Math.PI) / 180));
const samples = [];
const start = new Date(startISO).getTime();
const dt = 2; // seconds per sample
let idx = 0;
for (let i = 0; i < legs; i++) {
const x = -dLon + (2 * dLon * i) / (legs - 1);
const north = i % 2 === 0;
const heading = north ? 0 : 180;
const steps = 40;
for (let j = 0; j <= steps; j++) {
const frac = j / steps;
const y = north ? -dLat + 2 * dLat * frac : dLat - 2 * dLat * frac;
samples.push({
index: idx,
t: new Date(start + idx * dt * 1000).toISOString(),
lat: lat + y,
lon: lon + x,
altAGL,
heading,
gimbalPitch: -90, // nadir survey
hfovDeg,
});
idx++;
}
}
return {
samples,
meta: { center_lon: lon, center_lat: lat, radius_m: radiusM },
};
}

22
wrangler.dev.toml Normal file
View file

@ -0,0 +1,22 @@
# Local-dev config ONLY. Enables a LOCAL D1 so `POST /api/ingest` and stored
# surveys work end-to-end on your machine, without needing remote D1 perms.
# wrangler d1 execute critterscope --local --config wrangler.dev.toml --file schema.sql
# wrangler dev --local --config wrangler.dev.toml
# The live deploy uses wrangler.toml (simulator; no D1) until a D1-capable
# Cloudflare token is available — then move this binding into wrangler.toml
# with the real database_id.
name = "critterscope"
main = "src/index.js"
compatibility_date = "2024-11-01"
compatibility_flags = ["nodejs_compat"]
[vars]
BASE_URL = "http://localhost:8787"
DEFAULT_LON = "150.3119"
DEFAULT_LAT = "-33.7300"
DEFAULT_RADIUS_M = "1000"
[[d1_databases]]
binding = "DB"
database_name = "critterscope"
database_id = "local-critterscope"