LBP.Robot 0.5.0

LBP.Robot

Bonsai package for the Little Brain Project robot hardware I/O. Provides Source nodes that stream sensor data from the physical robot (cameras, raw UDP bytes) into a Bonsai workflow.

Objective

Bridge the LBP robot hardware to Bonsai. The robot carries two cameras (ESP32-CAM and Raspberry Pi Zero) and communicates over WiFi via UDP. This package wraps those streams into reactive IObservable<T> sequences so Bonsai workflows can consume camera frames and raw sensor bytes the same way they would consume any other Bonsai source.

Summary

How It Works

Camera streams

ESP32-CAM (UdpJpegStream) — The ESP32 breaks each JPEG frame into multiple UDP packets. Each packet header carries [frameId, totalChunks, chunkIndex]. The node reassembles all chunks for a given frameId into a complete JPEG byte array, decodes it with OpenCV CV.DecodeImage, and emits a FrameData (image + frame number). A "HELLO" datagram is sent to the ESP32 on startup to trigger streaming.

Raspberry Pi Zero (RpiJpegStream) — The RPi sends complete JPEG frames as single UDP packets (identified by 0xFF 0xD8 SOI magic bytes). A background task periodically sends a "hello" keepalive to maintain the stream. Each received frame is decoded with OpenCV and emitted as IplImage.

BMP HTTP stream (BmpStream / BmpVideoSource) — Polls an HTTP URL every 100 ms, downloads a BMP image, and fires NewFrame events through the AForge IVideoSource interface. BmpStream wraps this in a Bonsai VideoCapture so it integrates with standard Bonsai image processing nodes.

Generic UDP (ReceiveBytes / ToUInt16Array)

On subscribe, sends an optional connection message to the remote device, then opens a UDP listener. Every incoming datagram is emitted as byte[]. This is the generic channel for OSC-framed sensor data (/bumpers, /wencoders, /analogs, /gyro) — pair it with Bonsai.Osc.ReceiveMessage nodes downstream to decode the OSC payload.

Code Structure

LBP.Robot/
├── LBP.Robot.csproj          # net472; deps: Bonsai.Core, OpenCV.Net, AForge, Bonsai.Video
├── UdpJpegStream.cs          # Source<FrameData>   — ESP32-CAM chunked JPEG receiver
├── RpiJpegStream.cs          # Source<IplImage>    — RPi Zero JPEG receiver + keepalive
├── BmpStream.cs              # VideoCapture        — HTTP BMP stream Bonsai node
├── BmpVideoSource.cs         # IVideoSource        — AForge polling worker thread
├── ReceiveBytes.cs           # Source<byte[]>      — generic UDP datagram listener
├── ToUInt16Array.cs          # Source<byte[]>      — UDP listener (uint16 payloads)
└── FrameMetadata.cs          # data class          — carries FrameNumber per frame

Dependencies

Integration with the rest of LBP

• Camera → LBP.Torch: RpiJpegStream or UdpJpegStream produce IplImage frames. Feed them through Bonsai.Vision transforms (resize, convert, normalize) and then into LBP.Torch.ToTensor to produce a Tensor for the neural network forward pass.
• Sensor bytes → OSC → brain: ReceiveBytes feeds raw bytes to Bonsai.Osc.ReceiveMessage. Decoded OSC values (/analogs, /bumpers) can be assembled into tensors via LBP.Torch.ToTensor or scalar expressions and routed to the network.
• Motor commands → robot: Motor outputs from the neural network are sent back to the robot via standard Bonsai SendMessage UDP nodes using the /wheels OSC address.
• Simulation drop-in: When using the Python 2D or MuJoCo 3D simulator instead of the real robot, the workflow does not use LBP.Robot nodes at all — OSC messages are sent/received over the same ports.