Skip to content

Problems

This project documents its problems in the open, on purpose. The brief's Task 1 explicitly grades understanding the limits of the systems we use — and nothing demonstrates those limits more sharply than the moments they bit us. Almost every non-obvious thing we learned about indoor GPS-denied flight, this flight controller, and this sensor stack, we learned by hitting a wall first. So the failures are not hidden in a lab notebook; they are collected here as first-class documentation, because they are the most instructive part of the whole project.

Each problem below is written up where it is most relevant (the sensor page, the simulation guide, the incident analysis, …). This page is the catalogue that links them together. Every entry lists the symptom as it actually appeared, the resolution or current status, and a link to the full write-up — and every link target is a page that exists in this site.

The catalogue

Problem Symptom as it appeared Resolution / status Documented in
Full-throttle climb into the ceiling (2026-08-21) Manual Stabilize flight went to 100 % throttle at ~1 m and hit the ceiling at 4.8 g; a leftover fence forced LAND while the EKF altitude read −1070 m on the floor. Resolved in software & parameters. All three causal links identified from the logs; each got a fix (fence off by default with crash-proof restore, ground-drift preflight check, arming checks restored, rangefinder-trusting takeover gate). Crash & recovery, full incident analysis
"Destroyed" barometer / bad compass after the crash Stock 4.6.3 halted at boot with Config Error: Baro; a custom no-baro build then showed Bad Compass Health; all telemetry read zero. Resolved, zero-cost. The FlywooF745's single I2C bus was hung by bent GPS-connector pins — every device unreachable, no dead chips. Pins straightened (2026-08-23); stock 4.6.3 re-flashed (2026-08-24) → barometer detected and working again. Crash & recovery
Leftover geofence / parameter persistence A FENCE_ALT_MAX = 4 m, FENCE_ACTION = 2 (always-LAND) fence from a days-old companion run breached on every takeoff (baro downwash spikes to 4–6.7 m) and forced LAND. Resolved. FC parameters outlive the process that wrote them; the companion now saves every parameter before changing it, restores on every exit, and mirrors originals to disk so even a killed run is cleaned up. Fence off by default indoors. Full incident analysis
EKF height fusion diverges when the rangefinder input freezes (POSZ = 2) With EK3_SRC1_POSZ = 2 (rangefinder as the EKF height source), a rangefinder that stops delivering fresh data leaves the EKF integrating the accelerometer alone: parked with no height reference at all on 2026-08-21 it diverged to −1070 m over hours; on real milestone-2 test flights on 2026-09-22 the sensor's live data froze mid-climb and the estimate drifted 0.8–1.7 m in seconds, a smooth, plausible-looking climb that never happened. Software blind spot closed (2026-09-22): the companion now cross-checks the rangefinder against the EKF altitude during the climb itself and aborts on divergence, not just once afterwards. Hardware root cause of the 2026-09-22 sensor freeze still open — two parameter changes tried live (RNGFND1_GNDCLEAR, EK3_RNG_M_NSE) did not fix it. Full incident analysis, Rangefinder dropout mid-flight
Disabled arming checks masked everything ARMING_CHECK = 0 let the aircraft arm with a 1000 m vertical estimate error and a stale fence — nothing refused. Resolved. ARMING_CHECK = 786390 (everything except the GPS lock that can never pass indoors), documented with its reason. Full incident analysis
Firmware flash wiped all parameters Flashing the custom 4.8.0-dev build reset every parameter to defaults, including accel calibration and the MTF-01P serial setup. Mitigated. The full pre-crash state was recovered from the crash log's own parameter records into fc_baseline_463_20260821.parm (+ fc_safe_overrides.parm), kept under version control and reloaded after any flash. Full incident analysis
Arm: LAND mode not armable confusion After a fence-forced landing the vehicle stayed in LAND mode; re-arming was refused with Arm: LAND mode not armable, which was misread as a flat battery for two test days. Resolved (understanding). The refusal was literally true — read the mode first. The companion now mirrors every autopilot STATUSTEXT into the mission log so the reason is on record. Firmware limitations
Optical-flow bootstrap deadlock on the ground main.py --milestone 2 hung forever waiting for EKF_POS_HORIZ_REL: flow needs height, height needs takeoff, takeoff needs a position estimate. Resolved by design. The --takeover mode lets a safety pilot fly the first metre by hand (Stabilize needs no position); the companion takes over once the EKF converges in the air. Optical flow
MTF-01P shipped in MSP mode, never configured On first bring-up the FC saw nothing: no OPTICAL_FLOW, no RANGEFINDER, no error — the sensor speaks MSP out of the box, the FC listens for MAVLink. Resolved. Switch the sensor to MAVLink in the MicroAir assistant (once per sensor), then configure the FC serial/flow/rangefinder parameters. MTF-01P configuration
4.6 vs 4.7 parameter rename, silently ignored RNGFND1_MIN_CM/_MAX_CM (cm) became RNGFND1_MIN/_MAX (m) in 4.7; ArduPilot silently ignores unknown parameter names, so the wrong-version name "sets" without error and changes nothing. Mitigated. Check the firmware banner first, use the _CM names on 4.6.3, and read every value back after writing it; the companion writes version-fallback names and verifies by read-back. MTF-01P configuration
SITL SIM_TERRAIN makes the rangefinder read 0.00 m With --custom-location and terrain enabled (the default), SITL measures the rangefinder against a terrain model far below the modelled ground, so it reports a constant 0.00 m at every altitude. Resolved. Set SIM_TERRAIN 0 for indoor tests; a rangefinder that reads 0.00 m at every altitude is this bug, not the driver. SITL setup
pigpio removed in Debian 13 (trixie) Older servo guides say apt install pigpio; on the trixie-based Pi OS this fails with "has no installation candidate", and forcing the pigpio pin factory breaks the drop servo. Resolved. Use the gpiozero + lgpio stack instead of pigpio for the drop mechanism. Servo mechanism
Loiter drifts in the hall, compass unreliable in the building (2026-08-25, reproduced 2026-09-21) In the hall, Loiter drifted and the pilot abandoned it within seconds on all three attempts; AltHold in the same flight was clean (36 s stable, 0.12 m position excursion). A repeat session on 2026-09-21, flown in Stabilize to stay on the safe side, still shows the identical signature: three sudden 30–44° EKF yaw jumps, each logged by the autopilot as ground mag anomaly, yaw re-aligned. Root cause understood, flyable with a mitigation. The hall's magnetic field is distorted by the building structure (rebar/steel): magnitude varies 195–565 mGauss across the flight volume, and a fixed-spot measurement found a vertical gradient from ~360 mGauss on the floor to ~850 mGauss at ~2 m, larger than the entire Earth field, which points at the building rather than the vehicle. EKF3's own consistency check saw the bad readings as plausible (innovation ratio 0.56, under its rejection threshold) and fused them instead of rejecting them, producing a discrete yaw reset that misdirects the optical-flow position controller. Minimising ground dwell before climbing ("arm and go") measurably reduces this, and both AltHold and Loiter now fly on the real aircraft — Loiter good in the lab, still imperfect but flyable in the hall. Compassless flight was refused arming in SITL, so the compass stays mandatory; a cleaner fix (stable takeoff zone, damped field-learning rate) is still open. Loiter drifts in the hall

The catalogue — landing-pad detector

The detector had its own run of instructive failures. Several of them were only visible because something was measured that the standard metric does not cover.

Problem Symptom as it appeared Resolution / status Documented in
A 0.94 mAP that measured memorisation The detector scored 0.94 mAP on validation and looked finished. Resolved. The 295-image export comes from only 123 source photos: Roboflow had generated three augmented copies of each, and our own photos were additionally shot as near-continuous bursts. A random split therefore trained on one copy and validated on another. Re-split into contiguous blocks with adjacent frames dropped — 175/25/16, leak-free. Dataset
The detector reported landing pads on a rucksack, a wall poster and a chair back Live webcam in an office, confidences up to 0.87 — higher than many true detections, so no threshold could separate them. Resolved, and it was the single most valuable change in the project. Not one of the 175 training images was a true negative, so "there is no pad here" was never a supported answer. 76 pad-free hard negatives (30 % background) took false positives from 0.33 to 0.02 per image at unchanged recall. Training
mAP could not tell six recipes apart Every training run scored mAP50 0.995 on 16 test photos of one high-contrast pad on a clean floor. Resolved by measuring something else. Yaw / altitude / capture-path stress probes on the polygon labels; they separated the runs immediately — worst-case altitude recall ranged 0.25 to 0.75 across recipes that all "scored 0.995". Evaluation
IMX export dies with cannot import name 'runtime_version' The export aborts partway through, in model_compression_toolkit. Resolved. Sony's converter pins protobuf==4.25.5, TensorFlow needs 5.x, and MCT imports TF merely because it is installed. Export from a virtualenv with no TensorFlow in it. Deployment
PulpError: Cannot multiply variables with NaN/inf values Quantisation aborts in the mixed-precision solver, with no reference to images. Resolved. MCT normalises float-vs-quantised outputs by their norm; on an image with no pad the confidence output is ~0 and it divides by ~0. The hard negatives are essential for training and harmful for calibration — filter them out, 197 calibration images remain. Deployment
No x86 machine can produce a .rpk imx500-package cannot be installed on macOS, Colab or any cloud VM. Resolved. imx500-tools ships no amd64/all package and its packager is an aarch64 ELF binary — verifiable, not just documented. A free ubuntu-24.04-arm GitHub Actions runner does the step, so no Pi has to be powered on. Deployment
rpicam-apps shows no detections from our model The bundled imx500_object_detection post-processing stage runs but displays nothing useful. Resolved. Its parsers match the output layouts Sony ships models in, not Ultralytics' YOLO layout. Read the tensors directly via picamera2/IMX500 or Sony's modlib. Deployment
The camera floods the terminal with Failed to queue buffer After an aborted run the next start filled the screen with V4L2 errors and the camera stopped responding. Resolved. The camera pipeline is left half-configured by the aborted run; sudo reboot on the Pi clears it and nothing needs reinstalling. Deployment
Training the two-class model crashes on Apple MPS RuntimeError: shape mismatch ... (utils/tal.py) in the task-aligned assigner — but only for the two-class set, never for the pad-only runs. Resolved. An MPS boolean-mask assignment bug that fires once images carry many objects: the pad runs had ~1 object per image, the person images up to 48. Ultralytics 8.4.90 with torch 2.12.1 runs it. Training
The .rpk and the flight code disagree about box order Not yet observed in the air — found by reading both sides against the sensor output, then confirmed on hardware. Open, one setting. The sensor emits (x0, y0, x1, y1) in input-tensor pixels. Pixel values are now normalised automatically and the input size reads back correctly as 320, but Pi-Code's cam_box_order still defaults to "yxyx", which reads the tensor transposed — it must be set to "xyxy". Shows up as swapped dx/dy in one milestone-2 hover. Flight-code integration
An offline measurement recommended the wrong confidence threshold The documented threshold for the .rpk was 0.3, justified by a distant-pad recall figure that the source table does not support. Resolved. Two errors, one correction: the quantised numbers come from an ONNX simulation, not from the sensor, and the penalty at 0.5 is one image in sixteen (recall at ×0.3 zoom 0.94 → 0.88), not the claimed 0.94 → 0.75. On the sensor, 0.3 additionally admits a whole quantisation step of junk. Threshold corrected to 0.5, and the measurement provenance is now labelled. Evaluation
One missed annotation poisoned six training images Three training images carried an empty annotation file, and three of the 76 "pad-free" hard negatives turned out to be close-ups of the pad. Open, minutes to fix. Photo 1776 was never annotated in Roboflow — the only such mistake in the 295-image export. The augmentation multiplied it into three images, and then negatives.py, which only avoids labelled pads, cut three crops straight out of it and saved them as background. 2.4 % of the training set therefore teaches the model that a clear pad is nothing. Annotate 1776, re-export, re-run negatives.py. The held-out false-positive benchmark is unaffected. Dataset
A double annotation in the validation set Photo 1761 carries two overlapping outlines of the same single pad. No impact, kept for the record. Ground truth would have counted one correct detection as a miss. The image sits in the original export's validation split and was dropped entirely by our leak-free re-split, so it never reached any run or any measurement. Dataset
The sensor's confidence output is discrete Live confidences arrived only as 0.32, 0.38, 0.44, 0.50, 0.56, 0.62, 0.68, 0.73, 0.78 — never between. Understood, not a fault. The quantised score output has roughly 0.06 resolution, so the threshold has about nine usable positions and any value between two steps behaves identically. No offline measurement showed this. Evaluation
Implausible single-frame detections at the lowest step Boxes glued to the frame edge, aspect ratios of 1:2.5 and worse, all at exactly 0.32, each lasting one frame — while the real pad held 0.50–0.78 for dozens of frames. Mitigated by threshold, properly fixed by persistence. Raising to 0.5 removes them. The durable fix is that no landing controller should act on a single frame: require the pad in 3 of 4 consecutive frames and reject geometrically impossible boxes. Not yet implemented. Flight-code integration

Read on