Add drone ingest pipeline + local D1 dev config
- pipeline/: DJI SRT telemetry parser, ffmpeg frame extraction, detector interface (stub + ONNX hook), pinhole/flat-ground georeferencer, ingest client - Verified end-to-end: synthetic flight → 319 georeferenced detections → D1 → dashboard - wrangler.dev.toml enables local D1 (live deploy stays simulator-only until D1 token) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
63393c052f
commit
328b3c53df
10 changed files with 555 additions and 1 deletions
10
README.md
10
README.md
|
|
@ -49,7 +49,15 @@ Open `/` for the dashboard. Point it anywhere:
|
||||||
The georeferencing step (frame + drone GPS + gimbal angle + terrain → lat/lon) lives
|
The georeferencing step (frame + drone GPS + gimbal angle + terrain → lat/lon) lives
|
||||||
in the drone pipeline, not here — this app just needs the resulting detections.
|
in the drone pipeline, not here — this app just needs the resulting detections.
|
||||||
|
|
||||||
|
## Drone ingest pipeline
|
||||||
|
See [`pipeline/`](pipeline/) — turns real drone footage + DJI telemetry into
|
||||||
|
georeferenced detections and POSTs them here. Runs end-to-end today via a
|
||||||
|
synthetic flight + stub detector; drop in a trained YOLO model for real thermal.
|
||||||
|
|
||||||
## Roadmap
|
## Roadmap
|
||||||
- [ ] Drone ingest pipeline (RTMP/SRT frames → YOLO thermal detector → georeferencer)
|
- [x] Drone ingest pipeline (frames + SRT telemetry → detector → georeferencer → ingest)
|
||||||
|
- [ ] Wire a trained thermal-wildlife YOLO model into the ONNX detector
|
||||||
|
- [ ] Remote D1 (needs a D1-capable Cloudflare token; live site runs the simulator until then)
|
||||||
- [ ] Live mode (stream detections during flight, not just post-survey)
|
- [ ] Live mode (stream detections during flight, not just post-survey)
|
||||||
- [ ] Track clustering (merge repeat sightings of the same animal across passes)
|
- [ ] Track clustering (merge repeat sightings of the same animal across passes)
|
||||||
|
- [ ] DEM-based georeferencing (ray/terrain intersection instead of flat ground)
|
||||||
|
|
|
||||||
68
pipeline/README.md
Normal file
68
pipeline/README.md
Normal file
|
|
@ -0,0 +1,68 @@
|
||||||
|
# CritterScope drone ingest pipeline
|
||||||
|
|
||||||
|
Turns drone footage + telemetry into georeferenced detections and pushes them to
|
||||||
|
a CritterScope instance.
|
||||||
|
|
||||||
|
```
|
||||||
|
video / RTMP frames + telemetry (DJI .SRT)
|
||||||
|
│ │
|
||||||
|
extractFrames parseDjiSrt
|
||||||
|
└──────────┬─────────────┘
|
||||||
|
detector (stub | ONNX YOLO) → bbox per frame
|
||||||
|
│
|
||||||
|
pixelToGround → lat/lon per detection
|
||||||
|
│
|
||||||
|
POST /api/ingest → D1
|
||||||
|
│
|
||||||
|
view at /?survey=<id>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick start (no drone needed)
|
||||||
|
Runs a synthetic lawnmower flight through the stub detector and prints what it
|
||||||
|
would send — proves the whole chain:
|
||||||
|
```bash
|
||||||
|
cd pipeline
|
||||||
|
node src/run.js --synthetic --dry-run
|
||||||
|
```
|
||||||
|
Push it into a running local CritterScope (with local D1, see root README):
|
||||||
|
```bash
|
||||||
|
node src/run.js --synthetic --name "test-survey" --base-url http://localhost:8787
|
||||||
|
# → View it: http://localhost:8787/?survey=sv-test-survey-...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Real footage
|
||||||
|
```bash
|
||||||
|
node src/run.js \
|
||||||
|
--source flight.MP4 --srt flight.SRT \
|
||||||
|
--detector onnx --model yolo-thermal.onnx \
|
||||||
|
--width 1920 --height 1080 \
|
||||||
|
--drone "DJI Mavic 2 Enterprise Advanced" \
|
||||||
|
--base-url https://critterscope.theradicalparty.com
|
||||||
|
```
|
||||||
|
|
||||||
|
## Live RTMP (DJI Fly app livestream)
|
||||||
|
Point the DJI Fly app's RTMP stream at a local server, then:
|
||||||
|
```bash
|
||||||
|
node src/run.js --source rtmp://localhost/live/stream --srt live.SRT --fps 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## The three stages you can swap
|
||||||
|
- **Telemetry** (`src/telemetry.js`) — DJI SRT parser + synthetic flight. Add other
|
||||||
|
drones by mapping their telemetry into `{lat, lon, altAGL, heading, gimbalPitch, hfovDeg}`.
|
||||||
|
- **Detector** (`src/detector.js`) — `stub` (works now) and `onnx` (wire your trained
|
||||||
|
YOLO: JPEG decode → letterbox → run → NMS → map class ids to species). Everything
|
||||||
|
downstream is model-agnostic.
|
||||||
|
- **Georeferencer** (`src/georef.js`) — pinhole camera + flat-ground projection.
|
||||||
|
Verified: centre pixel at nadir → drone position; right-edge offset matches
|
||||||
|
`altitude·tan(hfov/2)`; near-horizon rays correctly return no ground hit.
|
||||||
|
Next refinement: intersect the ray with a DEM instead of flat ground.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
- Node 18+ (uses global `fetch`)
|
||||||
|
- `ffmpeg` on PATH (only for `--source`)
|
||||||
|
- `onnxruntime-node` (optional, only for `--detector onnx`)
|
||||||
|
|
||||||
|
## What's real vs. stubbed
|
||||||
|
- ✅ telemetry parse, frame extraction, georeferencing, survey assembly, ingest, map render
|
||||||
|
- 🔩 the ONNX detector's tensor plumbing is left for when you have a trained thermal
|
||||||
|
wildlife model — the stub detector exercises the identical downstream path today.
|
||||||
15
pipeline/package.json
Normal file
15
pipeline/package.json
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
{
|
||||||
|
"name": "critterscope-pipeline",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"description": "Drone footage + telemetry → georeferenced detections → CritterScope",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"bin": { "critterscope-ingest": "src/run.js" },
|
||||||
|
"scripts": {
|
||||||
|
"demo": "node src/run.js --synthetic --dry-run",
|
||||||
|
"demo:ingest": "node src/run.js --synthetic --base-url http://localhost:8787"
|
||||||
|
},
|
||||||
|
"optionalDependencies": {
|
||||||
|
"onnxruntime-node": "^1.19.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
81
pipeline/src/detector.js
Normal file
81
pipeline/src/detector.js
Normal file
|
|
@ -0,0 +1,81 @@
|
||||||
|
// Detector interface.
|
||||||
|
//
|
||||||
|
// A detector takes a frame (path to a JPEG + width/height) and returns an array
|
||||||
|
// of detections: { type, confidence, bbox:[x,y,w,h] } in pixel coordinates.
|
||||||
|
//
|
||||||
|
// Two backends:
|
||||||
|
// - onnx: real YOLO via onnxruntime-node (needs a model + the optional dep)
|
||||||
|
// - stub: deterministic synthetic detections so the whole pipeline runs and
|
||||||
|
// can be validated end-to-end without a model or real footage.
|
||||||
|
//
|
||||||
|
// The class list is COCO-ish remapped to the CritterScope species vocabulary.
|
||||||
|
// For thermal you'd train/fine-tune on thermal wildlife imagery; the interface
|
||||||
|
// is identical.
|
||||||
|
|
||||||
|
const SPECIES = ["kangaroo", "deer", "rabbit", "fox", "boar", "person", "vehicle", "unknown"];
|
||||||
|
|
||||||
|
function mulberry32(seed) {
|
||||||
|
let a = seed >>> 0;
|
||||||
|
return () => {
|
||||||
|
a = (a + 0x6d2b79f5) | 0;
|
||||||
|
let t = Math.imul(a ^ (a >>> 15), 1 | a);
|
||||||
|
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
||||||
|
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createStubDetector({ W = 1920, H = 1080, seed = 99 } = {}) {
|
||||||
|
return {
|
||||||
|
name: "stub",
|
||||||
|
async detect(frame) {
|
||||||
|
// seed by frame index so results are stable & reproducible per run
|
||||||
|
const rng = mulberry32(seed + (frame.index | 0) * 2654435761);
|
||||||
|
const n = Math.floor(rng() * 3); // 0..2 animals per frame
|
||||||
|
const out = [];
|
||||||
|
for (let i = 0; i < n; i++) {
|
||||||
|
const type = SPECIES[Math.floor(rng() * SPECIES.length)];
|
||||||
|
const w = 30 + rng() * 90;
|
||||||
|
const h = 30 + rng() * 90;
|
||||||
|
// keep detections toward frame centre (better georef geometry)
|
||||||
|
const x = W * (0.2 + rng() * 0.6) - w / 2;
|
||||||
|
const y = H * (0.2 + rng() * 0.6) - h / 2;
|
||||||
|
out.push({
|
||||||
|
type,
|
||||||
|
confidence: Number((0.4 + rng() * 0.58).toFixed(2)),
|
||||||
|
bbox: [x, y, w, h],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Real YOLO backend. Lazily requires onnxruntime-node so the stub path has no
|
||||||
|
// heavy dependency. Provide a model exported to ONNX and a labels→species map.
|
||||||
|
export async function createOnnxDetector({ modelPath, labelMap = {}, W, H, confThreshold = 0.35 }) {
|
||||||
|
let ort;
|
||||||
|
try {
|
||||||
|
ort = await import("onnxruntime-node");
|
||||||
|
} catch (e) {
|
||||||
|
throw new Error(
|
||||||
|
"onnxruntime-node not installed. Run `npm i onnxruntime-node` in pipeline/ to use the ONNX detector."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const session = await ort.InferenceSession.create(modelPath);
|
||||||
|
return {
|
||||||
|
name: "onnx",
|
||||||
|
session,
|
||||||
|
async detect(frame) {
|
||||||
|
// Intentionally a documented stub of the tensor plumbing: decode JPEG →
|
||||||
|
// letterbox to model input → run → NMS → map class ids to species.
|
||||||
|
// Wire this once you have a trained model; the georef/ingest stages are
|
||||||
|
// already model-agnostic.
|
||||||
|
throw new Error(
|
||||||
|
"ONNX detect() not wired — plug in JPEG decode + preprocessing for your model. " +
|
||||||
|
"Everything downstream (georef, ingest) already works via the stub detector."
|
||||||
|
);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export const SPECIES_LIST = SPECIES;
|
||||||
37
pipeline/src/frames.js
Normal file
37
pipeline/src/frames.js
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
// Frame extraction via ffmpeg. Works on a video file or a live RTMP/RTSP URL.
|
||||||
|
// Samples at a fixed fps into a temp dir and returns the frame file list.
|
||||||
|
// (DJI Fly livestreams RTMP; point --source at rtmp://<your-server>/live/stream.)
|
||||||
|
|
||||||
|
import { spawn } from "node:child_process";
|
||||||
|
import { mkdtempSync, readdirSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
|
||||||
|
export function extractFrames({ source, fps = 1, maxFrames = 0 } = {}) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "critter-frames-"));
|
||||||
|
const args = [
|
||||||
|
"-hide_banner", "-loglevel", "error",
|
||||||
|
...(source.startsWith("rtmp") || source.startsWith("rtsp") ? ["-rtsp_transport", "tcp"] : []),
|
||||||
|
"-i", source,
|
||||||
|
"-vf", `fps=${fps}`,
|
||||||
|
...(maxFrames ? ["-frames:v", String(maxFrames)] : []),
|
||||||
|
"-q:v", "3",
|
||||||
|
join(dir, "f_%06d.jpg"),
|
||||||
|
];
|
||||||
|
const ff = spawn("ffmpeg", args);
|
||||||
|
let err = "";
|
||||||
|
ff.stderr.on("data", (d) => (err += d));
|
||||||
|
ff.on("error", (e) =>
|
||||||
|
reject(new Error(`ffmpeg not found or failed to start: ${e.message}. Install ffmpeg.`))
|
||||||
|
);
|
||||||
|
ff.on("close", (code) => {
|
||||||
|
if (code !== 0) return reject(new Error(`ffmpeg exited ${code}: ${err.slice(0, 400)}`));
|
||||||
|
const files = readdirSync(dir)
|
||||||
|
.filter((f) => f.endsWith(".jpg"))
|
||||||
|
.sort()
|
||||||
|
.map((f, i) => ({ index: i, path: join(dir, f) }));
|
||||||
|
resolve({ dir, files });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
87
pipeline/src/georef.js
Normal file
87
pipeline/src/georef.js
Normal file
|
|
@ -0,0 +1,87 @@
|
||||||
|
// Pixel → ground georeferencing.
|
||||||
|
//
|
||||||
|
// Given a detection's pixel in a drone frame plus the drone's telemetry
|
||||||
|
// (position, altitude above ground, camera heading + gimbal pitch, and the
|
||||||
|
// camera field of view), project that pixel onto flat ground and return the
|
||||||
|
// lat/lon it corresponds to.
|
||||||
|
//
|
||||||
|
// Model: pinhole camera + flat-earth ground plane. This is the standard
|
||||||
|
// first-order approach and is accurate to a few metres for typical survey
|
||||||
|
// altitudes over gently sloping terrain. (A DEM-based intersection would remove
|
||||||
|
// the flat-ground assumption; that's a later refinement.)
|
||||||
|
//
|
||||||
|
// Frame convention: world ENU (x=East, y=North, z=Up). Drone at (0,0,h).
|
||||||
|
// Image: +u right, +v down, principal point at centre.
|
||||||
|
|
||||||
|
const D2R = Math.PI / 180;
|
||||||
|
|
||||||
|
function norm(v) {
|
||||||
|
const m = Math.hypot(v[0], v[1], v[2]);
|
||||||
|
return [v[0] / m, v[1] / m, v[2] / m];
|
||||||
|
}
|
||||||
|
const cross = (a, b) => [
|
||||||
|
a[1] * b[2] - a[2] * b[1],
|
||||||
|
a[2] * b[0] - a[0] * b[2],
|
||||||
|
a[0] * b[1] - a[1] * b[0],
|
||||||
|
];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param {object} p
|
||||||
|
* @param {number} p.u pixel x (0..W)
|
||||||
|
* @param {number} p.v pixel y (0..H)
|
||||||
|
* @param {number} p.W image width px
|
||||||
|
* @param {number} p.H image height px
|
||||||
|
* @param {number} p.hfovDeg horizontal field of view (deg)
|
||||||
|
* @param {number} [p.vfovDeg] vertical FOV (deg); derived from aspect if omitted
|
||||||
|
* @param {number} p.lat drone latitude
|
||||||
|
* @param {number} p.lon drone longitude
|
||||||
|
* @param {number} p.altAGL drone height above ground (m)
|
||||||
|
* @param {number} p.headingDeg camera azimuth, clockwise from North (deg)
|
||||||
|
* @param {number} p.gimbalPitchDeg gimbal pitch, 0=horizon, -90=straight down
|
||||||
|
* @returns {{lat:number, lon:number, slantRange:number}|null}
|
||||||
|
*/
|
||||||
|
export function pixelToGround(p) {
|
||||||
|
const {
|
||||||
|
u, v, W, H, hfovDeg, lat, lon, altAGL, headingDeg, gimbalPitchDeg,
|
||||||
|
} = p;
|
||||||
|
const vfovDeg = p.vfovDeg ?? (2 * Math.atan(Math.tan((hfovDeg * D2R) / 2) * (H / W))) / D2R;
|
||||||
|
|
||||||
|
const fx = W / 2 / Math.tan((hfovDeg * D2R) / 2);
|
||||||
|
const fy = H / 2 / Math.tan((vfovDeg * D2R) / 2);
|
||||||
|
const cx = W / 2;
|
||||||
|
const cy = H / 2;
|
||||||
|
|
||||||
|
// camera-axis (forward) direction in world ENU
|
||||||
|
const a = headingDeg * D2R;
|
||||||
|
const pit = gimbalPitchDeg * D2R;
|
||||||
|
const forward = norm([
|
||||||
|
Math.cos(pit) * Math.sin(a), // E
|
||||||
|
Math.cos(pit) * Math.cos(a), // N
|
||||||
|
Math.sin(pit), // U (negative when looking down)
|
||||||
|
]);
|
||||||
|
|
||||||
|
// right = horizontal, 90° clockwise from heading (roll assumed 0)
|
||||||
|
const right = [Math.cos(a), -Math.sin(a), 0];
|
||||||
|
// image-down axis completes the right-handed camera frame
|
||||||
|
const imgDown = norm(cross(forward, right));
|
||||||
|
|
||||||
|
// ray for pixel (u,v)
|
||||||
|
const nx = (u - cx) / fx;
|
||||||
|
const ny = (v - cy) / fy;
|
||||||
|
const dir = norm([
|
||||||
|
forward[0] + nx * right[0] + ny * imgDown[0],
|
||||||
|
forward[1] + nx * right[1] + ny * imgDown[1],
|
||||||
|
forward[2] + nx * right[2] + ny * imgDown[2],
|
||||||
|
]);
|
||||||
|
|
||||||
|
if (dir[2] >= -1e-6) return null; // ray points at/above horizon → no ground hit
|
||||||
|
|
||||||
|
const t = altAGL / -dir[2]; // drone at z=altAGL, ground z=0
|
||||||
|
const east = t * dir[0];
|
||||||
|
const north = t * dir[1];
|
||||||
|
const slantRange = t;
|
||||||
|
|
||||||
|
const dLat = north / 111320;
|
||||||
|
const dLon = east / (111320 * Math.cos(lat * D2R));
|
||||||
|
return { lat: lat + dLat, lon: lon + dLon, slantRange };
|
||||||
|
}
|
||||||
16
pipeline/src/ingest.js
Normal file
16
pipeline/src/ingest.js
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
// POST an assembled survey to a CritterScope instance.
|
||||||
|
|
||||||
|
export async function postSurvey({ baseUrl, survey, flight, detections }) {
|
||||||
|
const res = await fetch(`${baseUrl.replace(/\/$/, "")}/api/ingest`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ survey, flight, detections }),
|
||||||
|
});
|
||||||
|
const text = await res.text();
|
||||||
|
let body;
|
||||||
|
try { body = JSON.parse(text); } catch { body = text; }
|
||||||
|
if (!res.ok) {
|
||||||
|
throw new Error(`ingest failed ${res.status}: ${typeof body === "string" ? body : JSON.stringify(body)}`);
|
||||||
|
}
|
||||||
|
return body;
|
||||||
|
}
|
||||||
142
pipeline/src/run.js
Normal file
142
pipeline/src/run.js
Normal file
|
|
@ -0,0 +1,142 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// CritterScope drone ingest pipeline.
|
||||||
|
//
|
||||||
|
// video/RTMP frames + telemetry (DJI SRT) → detector → georeferencer
|
||||||
|
// → survey JSON → POST /api/ingest → view at /?survey=<id>
|
||||||
|
//
|
||||||
|
// Examples
|
||||||
|
// # end-to-end test, no drone needed (synthetic flight + stub detector):
|
||||||
|
// node src/run.js --synthetic --base-url http://localhost:8787
|
||||||
|
//
|
||||||
|
// # real footage:
|
||||||
|
// node src/run.js --source flight.MP4 --srt flight.SRT --detector onnx \
|
||||||
|
// --model yolo-thermal.onnx --base-url https://critterscope.theradicalparty.com
|
||||||
|
//
|
||||||
|
// # live RTMP stream from the DJI Fly app:
|
||||||
|
// node src/run.js --source rtmp://localhost/live/stream --srt live.SRT --fps 1
|
||||||
|
|
||||||
|
import { parseDjiSrt, syntheticFlight } from "./telemetry.js";
|
||||||
|
import { extractFrames } from "./frames.js";
|
||||||
|
import { createStubDetector, createOnnxDetector } from "./detector.js";
|
||||||
|
import { pixelToGround } from "./georef.js";
|
||||||
|
import { postSurvey } from "./ingest.js";
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const a = { fps: 1, detector: "stub", baseUrl: "http://localhost:8787", width: 1920, height: 1080, radius: 1000 };
|
||||||
|
for (let i = 2; i < argv.length; i++) {
|
||||||
|
const k = argv[i];
|
||||||
|
const nx = () => argv[++i];
|
||||||
|
switch (k) {
|
||||||
|
case "--source": a.source = nx(); break;
|
||||||
|
case "--srt": a.srt = nx(); break;
|
||||||
|
case "--synthetic": a.synthetic = true; break;
|
||||||
|
case "--detector": a.detector = nx(); break;
|
||||||
|
case "--model": a.model = nx(); break;
|
||||||
|
case "--base-url": a.baseUrl = nx(); break;
|
||||||
|
case "--fps": a.fps = parseFloat(nx()); break;
|
||||||
|
case "--width": a.width = parseInt(nx(), 10); break;
|
||||||
|
case "--height": a.height = parseInt(nx(), 10); break;
|
||||||
|
case "--center": { const [lon, lat] = nx().split(",").map(Number); a.center = [lon, lat]; break; }
|
||||||
|
case "--radius": a.radius = parseFloat(nx()); break;
|
||||||
|
case "--name": a.name = nx(); break;
|
||||||
|
case "--drone": a.drone = nx(); break;
|
||||||
|
case "--dry-run": a.dryRun = true; break;
|
||||||
|
default: console.warn("unknown arg", k);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return a;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const a = parseArgs(process.argv);
|
||||||
|
|
||||||
|
// 1. telemetry
|
||||||
|
let samples, meta;
|
||||||
|
if (a.srt) {
|
||||||
|
samples = parseDjiSrt(a.srt);
|
||||||
|
if (!samples.length) throw new Error(`no telemetry parsed from ${a.srt}`);
|
||||||
|
meta = null;
|
||||||
|
} else if (a.synthetic) {
|
||||||
|
({ samples, meta } = syntheticFlight({ center: a.center, radiusM: a.radius }));
|
||||||
|
} else {
|
||||||
|
throw new Error("provide --srt <file> or --synthetic");
|
||||||
|
}
|
||||||
|
console.log(`telemetry: ${samples.length} samples`);
|
||||||
|
|
||||||
|
// 2. frames (optional in synthetic+stub mode; required for real detection)
|
||||||
|
let frames = null, tmpDir = null;
|
||||||
|
if (a.source) {
|
||||||
|
const r = await extractFrames({ source: a.source, fps: a.fps });
|
||||||
|
frames = r.files; tmpDir = r.dir;
|
||||||
|
console.log(`frames: ${frames.length} extracted → ${tmpDir}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. detector
|
||||||
|
const detector =
|
||||||
|
a.detector === "onnx"
|
||||||
|
? await createOnnxDetector({ modelPath: a.model, W: a.width, H: a.height })
|
||||||
|
: createStubDetector({ W: a.width, H: a.height });
|
||||||
|
console.log(`detector: ${detector.name}`);
|
||||||
|
|
||||||
|
// Decide the unit of work: real frames if we have them, else one per sample.
|
||||||
|
const units = frames
|
||||||
|
? frames.map((f, i) => ({ frame: f, sample: samples[Math.round((i / Math.max(1, frames.length - 1)) * (samples.length - 1))] }))
|
||||||
|
: samples.map((s) => ({ frame: { index: s.index, path: null }, sample: s }));
|
||||||
|
|
||||||
|
// 4. detect + georeference
|
||||||
|
const detections = [];
|
||||||
|
let seq = 0;
|
||||||
|
for (const { frame, sample } of units) {
|
||||||
|
const dets = await detector.detect({ index: frame.index, path: frame.path, W: a.width, H: a.height });
|
||||||
|
for (const d of dets) {
|
||||||
|
const [x, y, w, h] = d.bbox;
|
||||||
|
const u = x + w / 2;
|
||||||
|
const v = y + h / 2;
|
||||||
|
const g = pixelToGround({
|
||||||
|
u, v, W: a.width, H: a.height,
|
||||||
|
hfovDeg: sample.hfovDeg ?? 73,
|
||||||
|
lat: sample.lat, lon: sample.lon, altAGL: sample.altAGL,
|
||||||
|
headingDeg: sample.heading, gimbalPitchDeg: sample.gimbalPitch,
|
||||||
|
});
|
||||||
|
if (!g) continue;
|
||||||
|
detections.push({
|
||||||
|
id: `det-${seq++}`,
|
||||||
|
type: d.type,
|
||||||
|
lat: g.lat, lon: g.lon,
|
||||||
|
confidence: d.confidence,
|
||||||
|
temp_c: d.temp_c ?? null,
|
||||||
|
t: sample.t ?? null,
|
||||||
|
frame: frame.index,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log(`detections: ${detections.length} georeferenced`);
|
||||||
|
|
||||||
|
// 5. assemble survey
|
||||||
|
const flight = samples.map((s, i) => ({ seq: i, lon: s.lon, lat: s.lat, agl_m: s.altAGL, t: s.t ?? null }));
|
||||||
|
const centerLon = meta?.center_lon ?? flight.reduce((a2, p) => a2 + p.lon, 0) / flight.length;
|
||||||
|
const centerLat = meta?.center_lat ?? flight.reduce((a2, p) => a2 + p.lat, 0) / flight.length;
|
||||||
|
const survey = {
|
||||||
|
id: `sv-${a.name ? a.name.replace(/\W+/g, "-") : "run"}-${flight[0]?.t?.slice(0, 19) || "t0"}`,
|
||||||
|
name: a.name ?? "Drone survey",
|
||||||
|
center_lon: centerLon,
|
||||||
|
center_lat: centerLat,
|
||||||
|
radius_m: meta?.radius_m ?? a.radius,
|
||||||
|
sensor: a.detector === "onnx" ? "thermal+rgb" : "rgb",
|
||||||
|
drone: a.drone ?? (a.synthetic ? "SYNTHETIC" : "unknown"),
|
||||||
|
started_at: flight[0]?.t ?? null,
|
||||||
|
ended_at: flight[flight.length - 1]?.t ?? null,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (a.dryRun) {
|
||||||
|
console.log(JSON.stringify({ survey, flightPoints: flight.length, detections }, null, 2).slice(0, 4000));
|
||||||
|
console.log(`\n[dry-run] would POST ${detections.length} detections to ${a.baseUrl}/api/ingest`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const res = await postSurvey({ baseUrl: a.baseUrl, survey, flight, detections });
|
||||||
|
console.log("ingest:", res);
|
||||||
|
console.log(`\nView it: ${a.baseUrl}/?survey=${encodeURIComponent(survey.id)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((e) => { console.error("pipeline error:", e.message); process.exit(1); });
|
||||||
78
pipeline/src/telemetry.js
Normal file
78
pipeline/src/telemetry.js
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
// Telemetry providers: per-frame drone pose (lat, lon, altAGL, heading, gimbal
|
||||||
|
// pitch, timestamp). Two sources:
|
||||||
|
// - DJI .SRT sidecar (what DJI drones write alongside video)
|
||||||
|
// - synthetic lawnmower flight (for testing without footage)
|
||||||
|
|
||||||
|
import { readFileSync } from "node:fs";
|
||||||
|
|
||||||
|
// Parse a DJI SRT subtitle telemetry file into an array of samples.
|
||||||
|
// DJI encodes fields like [latitude: -33.7301] [longitude: 150.31] [rel_alt: 118.5]
|
||||||
|
// [gb_yaw: 12.3] [gb_pitch: -89.0] plus a per-block timecode.
|
||||||
|
export function parseDjiSrt(path) {
|
||||||
|
const raw = readFileSync(path, "utf8");
|
||||||
|
const blocks = raw.split(/\r?\n\r?\n/).filter((b) => b.trim());
|
||||||
|
const samples = [];
|
||||||
|
const num = (s, key) => {
|
||||||
|
const m = s.match(new RegExp(`\\[${key}\\s*:\\s*(-?\\d+(?:\\.\\d+)?)`, "i"));
|
||||||
|
return m ? parseFloat(m[1]) : undefined;
|
||||||
|
};
|
||||||
|
for (let i = 0; i < blocks.length; i++) {
|
||||||
|
const b = blocks[i];
|
||||||
|
const lat = num(b, "latitude") ?? num(b, "GPS.*?lat");
|
||||||
|
const lon = num(b, "longitude") ?? num(b, "GPS.*?lon");
|
||||||
|
if (lat === undefined || lon === undefined) continue;
|
||||||
|
samples.push({
|
||||||
|
index: i,
|
||||||
|
lat,
|
||||||
|
lon,
|
||||||
|
altAGL: num(b, "rel_alt") ?? num(b, "altitude") ?? 100,
|
||||||
|
heading: num(b, "gb_yaw") ?? num(b, "yaw") ?? 0,
|
||||||
|
gimbalPitch: num(b, "gb_pitch") ?? -90,
|
||||||
|
hfovDeg: 73, // DJI Mini-class wide FOV default; override via CLI if known
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return samples;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Synthetic lawnmower flight for testing. Returns { samples, meta }.
|
||||||
|
export function syntheticFlight({
|
||||||
|
center = [150.3119, -33.73],
|
||||||
|
radiusM = 1000,
|
||||||
|
legs = 8,
|
||||||
|
altAGL = 120,
|
||||||
|
hfovDeg = 73,
|
||||||
|
startISO = "2026-07-21T06:20:00.000Z",
|
||||||
|
} = {}) {
|
||||||
|
const [lon, lat] = center;
|
||||||
|
const dLat = radiusM / 111320;
|
||||||
|
const dLon = radiusM / (111320 * Math.cos((lat * Math.PI) / 180));
|
||||||
|
const samples = [];
|
||||||
|
const start = new Date(startISO).getTime();
|
||||||
|
const dt = 2; // seconds per sample
|
||||||
|
let idx = 0;
|
||||||
|
for (let i = 0; i < legs; i++) {
|
||||||
|
const x = -dLon + (2 * dLon * i) / (legs - 1);
|
||||||
|
const north = i % 2 === 0;
|
||||||
|
const heading = north ? 0 : 180;
|
||||||
|
const steps = 40;
|
||||||
|
for (let j = 0; j <= steps; j++) {
|
||||||
|
const frac = j / steps;
|
||||||
|
const y = north ? -dLat + 2 * dLat * frac : dLat - 2 * dLat * frac;
|
||||||
|
samples.push({
|
||||||
|
index: idx,
|
||||||
|
t: new Date(start + idx * dt * 1000).toISOString(),
|
||||||
|
lat: lat + y,
|
||||||
|
lon: lon + x,
|
||||||
|
altAGL,
|
||||||
|
heading,
|
||||||
|
gimbalPitch: -90, // nadir survey
|
||||||
|
hfovDeg,
|
||||||
|
});
|
||||||
|
idx++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
samples,
|
||||||
|
meta: { center_lon: lon, center_lat: lat, radius_m: radiusM },
|
||||||
|
};
|
||||||
|
}
|
||||||
22
wrangler.dev.toml
Normal file
22
wrangler.dev.toml
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
# Local-dev config ONLY. Enables a LOCAL D1 so `POST /api/ingest` and stored
|
||||||
|
# surveys work end-to-end on your machine, without needing remote D1 perms.
|
||||||
|
# wrangler d1 execute critterscope --local --config wrangler.dev.toml --file schema.sql
|
||||||
|
# wrangler dev --local --config wrangler.dev.toml
|
||||||
|
# The live deploy uses wrangler.toml (simulator; no D1) until a D1-capable
|
||||||
|
# Cloudflare token is available — then move this binding into wrangler.toml
|
||||||
|
# with the real database_id.
|
||||||
|
name = "critterscope"
|
||||||
|
main = "src/index.js"
|
||||||
|
compatibility_date = "2024-11-01"
|
||||||
|
compatibility_flags = ["nodejs_compat"]
|
||||||
|
|
||||||
|
[vars]
|
||||||
|
BASE_URL = "http://localhost:8787"
|
||||||
|
DEFAULT_LON = "150.3119"
|
||||||
|
DEFAULT_LAT = "-33.7300"
|
||||||
|
DEFAULT_RADIUS_M = "1000"
|
||||||
|
|
||||||
|
[[d1_databases]]
|
||||||
|
binding = "DB"
|
||||||
|
database_name = "critterscope"
|
||||||
|
database_id = "local-critterscope"
|
||||||
Loading…
Add table
Reference in a new issue