PicnicPlace — Specification (Draft 0.1)¶

Status: Draft. Sections marked [OPEN] need a decision before work in that area starts. Last updated: 2026-11-07

1. Summary¶

PicnicPlace is an open source pick-and-place (PnP) machine controller in the spirit of OpenPnP. It differs in one structural way: it is network first. A headless core process owns the machine (motion, vision, feeders, job execution), and every user interface is a client that talks to the core over a documented network API. The flagship interface is a modern GUI with a visual style inspired by OrcaSlicer.

The first-class hardware target is the Opulo LumenPnP.

2. Goals¶

1. Separation of control and interface. The core runs headless on the machine's host computer (a PC, mini PC, or SBC). GUIs, CLIs, scripts, and automation connect over the network. Closing or crashing a GUI never stops or corrupts a running job.
2. A documented, versioned API that third parties can build against (custom UIs, MES integration, Home Assistant/OctoPrint-style dashboards, scripted calibration).
3. A modern, attractive GUI with sensible defaults, guided calibration, and clear visual feedback, while still exposing advanced settings.
4. Strong vision built on OpenCV: fiducials, top-camera part and feeder vision, and bottom-camera part alignment.
5. Linux first. The 1.0 core targets Linux. The core avoids Linux-only assumptions outside of driver and camera backends, so macOS and Windows can be added later (see §9).
6. Approachable for contributors. A popular language, conventional tooling, a simulator so most work needs no hardware, and good tests.

3. Non-goals (for 1.0)¶

• Real-time motion planning on the host. Step generation, acceleration, and kinematics stay in machine firmware (Marlin, Klipper, grblHAL, etc.). The core sends motion commands; it does not generate step pulses.
• Configuration compatibility with OpenPnP's machine.xml. An importer for OpenPnP configs and part/package libraries is a desirable later feature, not a constraint on the design.
• Multi-machine fleet management. The API should not preclude it, but 1.0 targets one core per machine.
• PCB design or CAM features beyond importing placement data.

4. Language and technology choice¶

4.1 Requirements¶

4.2 Candidates¶

4.3 Decision: Python for the core, TypeScript for the GUI¶

Core (machine control + vision): Python 3.12+.

• PnP vision workloads are light: a handful of 1–5 MP frames per placement, processed with classic algorithms (thresholding, contours, template matching, minimum-area rectangles, circle detection). OpenCV does the heavy lifting in native code, so Python's overhead is not the bottleneck; camera latency and machine motion are.
• The real-time-sensitive work (step generation) lives in firmware. The host's job is orchestration: send G-code, wait for acknowledgments, read sensors, grab frames. Python's asyncio fits this well.
• Python is the most common language for OpenCV examples, tutorials, and research, and the most likely language for a hobbyist contributor to already know. It is also the natural language for user scripting, which OpenPnP supports through JavaScript/Jython.
• Acceleration: the standard opencv-python wheels support OpenCL through the T-API, which covers integrated and discrete GPUs on all three OSes. Users with NVIDIA hardware can install a CUDA-enabled OpenCV build. The vision layer will be written so either path works without code changes (see §7.5).
• Escape hatch: if a specific pipeline stage proves too slow, it can be moved to a C++ extension (via nanobind/pybind11) without changing the architecture.

GUI: TypeScript web app (see §8). Because of the network split, the GUI language is independent of the core language. A web front end gives the widest pool of UI contributors, the richest set of modern design tools, and lets the same GUI run in a browser on a tablet or laptop or as a packaged desktop app.

Considered and rejected for now: C++ as the core language. It would give the best performance and acceleration, but it raises the barrier for contributors more than the performance gain is worth for this workload. Revisit only if profiling on real hardware shows Python cannot hold target throughput.

4.4 Core stack¶

5. Architecture¶

 ┌──────────────────────────┐   ┌─────────────┐   ┌───────────────┐
 │  PicnicPlace Studio (GUI)│   │  CLI (picnic)│   │ Scripts / 3rd │
 │  browser or desktop shell│   │              │   │ party clients │
 └────────────┬─────────────┘   └──────┬──────┘   └───────┬───────┘
              │   HTTPS REST + WebSocket events + video   │
              └──────────────────┬────────────────────────┘
                                 │
 ┌───────────────────────────────▼───────────────────────────────┐
 │                       PicnicPlace Core (picnicd)               │
 │  ┌──────────┐ ┌───────────┐ ┌───────────┐ ┌────────────────┐   │
 │  │ API layer│ │ Job engine│ │ Vision    │ │ Config & data  │   │
 │  │ auth,    │ │ planner,  │ │ pipelines,│ │ machine, parts,│   │
 │  │ sessions │ │ executor  │ │ cameras   │ │ packages, jobs │   │
 │  └────┬─────┘ └─────┬─────┘ └─────┬─────┘ └────────────────┘   │
 │       └────────┬────┴─────────────┘                            │
 │        ┌───────▼────────┐                                      │
 │        │ Machine model  │  heads, nozzles, axes, cameras,      │
 │        │                │  feeders, actuators, sensors         │
 │        └───────┬────────┘                                      │
 │        ┌───────▼────────┐                                      │
 │        │ Drivers        │  Marlin (LumenPnP), Klipper, grblHAL,│
 │        │                │  Simulator; Photon feeder bus        │
 │        └───────┬────────┘                                      │
 └────────────────┼──────────────────────────────────────────────┘
                  │ USB serial / USB video
          ┌───────▼────────┐
          │  Machine       │
          └────────────────┘

5.1 Process model¶

• picnicd is a single long-running process per machine. It can run as a systemd service, a launchd agent, a Windows service, or in a terminal for development.
• It serves the GUI's static files, so a user can browse to http://<host>:<port> with no install on the client.
• All machine access goes through one owner task per device (serial port, camera) to avoid interleaved commands.

5.2 Safety principles¶

• The core owns safety. Soft limits, collision-safe Z, vacuum checks, and motion interlocks are enforced in the core, never only in a client.
• Motion requires a connected client. The core enables motion only while at least one client holds a live event-socket connection. See §5.3.
• Emergency stop is a dedicated, highest-priority API call and WebSocket message that bypasses the command queue and sends the firmware's immediate stop (e.g. M112 or realtime reset). A physical E-stop remains the primary safety device; the software stop is a convenience.
• Safe-Z interlock. A machine profile may declare a physical safe-Z input (a switch that reads true when every nozzle is at or above safe-travel height). When present it is authoritative:
• X/Y motion is refused unless the input reads safe, whatever the computed Z position.
• On power-up or reconnect, if it reads not safe, Z must be raised or homed before any X/Y motion, including X/Y homing.
• If the computed Z position says safe but the input says not safe, Z has lost steps. The core stops motion, reports a fault, and requires re-homing.
• Machines without the input use a computed safe-Z state, and clients label it as computed rather than sensed.
• The core can only check the input before sending a move. A firmware-level interlock (firmware refuses or halts X/Y motion while the input is unsafe) is the stronger guarantee and is recommended where firmware supports it.
• Moves travel at safe Z. A move that changes X or Y first raises every nozzle to travel height (if any is below safe Z), then moves X, Y, and rotation, then lowers Z to its target. Jogs never move Z on their own: an X or Y jog with a nozzle below safe Z is refused, so the operator decides when Z moves.
• Control lease. Many clients may observe; only one holds the control lease at a time and can move the machine or switch outputs. Leases can be requested, handed over, or forcibly taken with confirmation.
• A client that commands motion while no one holds the lease takes it implicitly, so a single operator never has to think about it.
• Taking the lease from another client needs an explicit force, which clients confirm with the user first.
• Motion requests name the sending client (the ID from the event socket's hello, in the X-PicnicPlace-Client header). The client must be connected; one-shot requests from nowhere cannot move the machine.
• The emergency stop needs no lease: any client can stop the machine.

5.3 Client presence rules¶

• Presence means at least one authenticated client has an open WebSocket on /api/v1/events and is answering heartbeats. One-shot REST calls do not count. Observer clients count; the point is that a person can see the machine.
• Manual motion (jog, move, actuators) is allowed only from the lease holder while it is connected. If the lease holder disconnects, manual motion stops immediately and the lease is released.
• Jobs survive disconnects but pause. If no clients are present, a running job pauses. The job is not ended and its progress is kept.
• Grace period. The pause triggers after the last client has been gone for a short, configurable time (default 5 s), so a browser reload or a Wi-Fi blip does not interrupt a job.
• Safe pause point. The job pauses at the next safe point: the current move finishes, the head goes to safe Z, and vacuum stays on so a part already picked is held on the nozzle. A part on the nozzle is not placed before pausing; the job continues with it on resume. (A vacuum check runs on resume in case the part was lost while paused.)
• Resume is manual. A job paused for lost presence resumes only when a client reconnects and an operator chooses Resume. It never resumes automatically.
• The CLI's run command stays connected while the job runs (it follows progress), so it counts as presence. Detaching from a job leaves it to pause unless another client is connected.

6. Network API¶

6.1 Transport¶

Alternatives considered: gRPC (strong typing, streaming) was rejected as the primary API because browsers cannot call it directly. MQTT may be offered later as an optional bridge for factory/home automation.

6.2 Principles¶

• Versioned under /api/v1/…. Breaking changes require a new version.
• Schema first. The OpenAPI document is generated from Pydantic models and published. The GUI's TypeScript client is generated from it, so the GUI cannot drift from the API.
• Long-running commands are operations. Commands such as home, calibrate, or run job return an operation ID immediately; progress and results stream over the event socket and are also available at /api/v1/operations/{id}.
• Everything the GUI can do, the API can do. The GUI uses no private endpoints.
• Units: millimeters, degrees, seconds, in all API payloads and config files.

6.3 Security¶

• Default bind is 127.0.0.1. Binding to the network is an explicit setting.
• When network-bound: token authentication (API tokens for scripts; login for the GUI), and TLS with a self-signed certificate generated on first run, or a user-supplied one.
• 1.0: single shared secret. One token/password grants full access. This fits the common setup of one operator with a GUI running at the machine.
• Built for more later. Every request is resolved internally to a Principal (identity, client kind, and permission set), and every endpoint declares the permission it needs (view, motion, job, config). In 1.0 the one principal has all permissions, so adding the features below does not change the API.

Planned next: local-only motion¶

Expected soon after 1.0: remote clients can watch, but anything that causes motion must come from a client flagged as local, meaning physically at the machine.

• Local clients:
• Connections over loopback (a GUI running on the machine's host computer) are local automatically.
• Other devices, such as an ESP32 touchscreen mounted on the machine, become local by pairing at the machine: the device shows a short code that is confirmed on an already-local client (or the reverse). Pairing issues that device its own token flagged local, which can be revoked.
• IP address or subnet alone is not used to decide "local," since it is easy to spoof on a LAN.
• Remote clients get view (and optionally job actions that do not move the machine, such as editing a job), but not motion.
• The control lease (§5.2) can only be held by a local client.
• Presence (§5.3) may later be tightened to require a local client, not just any client.

Later: user roles¶

observer (view only), operator (run jobs, jog), and admin (change config, manage users), layered on the same permission model.

6.4 Sketch of resources¶

GET    /api/v1/machine                 machine description and live state
POST   /api/v1/machine/home
POST   /api/v1/machine/estop
POST   /api/v1/machine/jog             (also available over the WebSocket)
POST   /api/v1/machine/move
GET    /api/v1/cameras                 list; /cameras/{id}/stream for video
POST   /api/v1/cameras/{id}/snapshot
POST   /api/v1/vision/run              run a pipeline against a camera, return results + debug images
GET/PUT /api/v1/config/...             machine configuration
CRUD   /api/v1/parts, /packages, /feeders
CRUD   /api/v1/jobs                    import boards, edit placements
POST   /api/v1/jobs/{id}/run | pause | resume | stop | step
GET    /api/v1/operations/{id}
POST   /api/v1/lease                   request/release control lease

6.5 Video streaming¶

Live video is for people only. Vision always runs inside the core on raw, uncompressed frames, so stream compression never affects placement accuracy.

The core offers two transports from one capture pipeline per camera:

Requirements:

• MJPEG stays a permanent, supported endpoint. Clients can request resolution, frame rate, and JPEG quality (e.g. ?width=320&fps=10&quality=70) so small displays such as ESP32-based control panels receive a stream they can decode. A single-frame JPEG snapshot endpoint is also provided.
• Embedded clients do not need video. An ESP32 control panel should work with the REST and WebSocket API alone; video is optional for it.
• WebRTC is negotiated over the existing authenticated API (no separate signaling server), LAN-only by default. Studio uses WebRTC when available and falls back to MJPEG.
• Overlays are data, not pixels. Crosshairs, detected features, and vision results are sent as vector data over the event socket and drawn by the client, so they stay sharp at any stream quality and work with both transports.
• Frame timestamps are attached to frames on both transports so clients can line up video with machine position and overlay data.
• When no client is watching a stream, the core stops encoding it.
• Endpoints (built in M1): GET /api/v1/cameras, .../{id}/snapshot.jpg, and .../{id}/stream.mjpg with width, fps, quality, and frames parameters. Each JPEG carries X-Timestamp and X-Frame headers. Media endpoints also accept the token as a token query parameter, since <img> tags cannot send headers. Each camera captures once however many clients watch, and stops a few seconds after the last leaves. The simulator renders synthetic views from the machine state (bed, a demo board with fiducials, feeder tape, nozzles and held parts) that match the machine geometry, so vision can be tested against known truth.

7. Core subsystems¶

7.1 Machine model¶

A declarative, data-driven model, loaded from config:

• Axes (X, Y, Z per nozzle or shared, rotation per nozzle), with limits, safe Z, and backlash compensation settings.
• Head containing nozzles, cameras (top/"down-looking"), and actuators.
• Fixed cameras (bottom/"up-looking").
• Nozzle tips with sizes, vacuum thresholds, and tool-changer positions.
• Feeders (see §7.4).
• Actuators and sensors: vacuum pumps, valves, lights, vacuum pressure sensors.
• Transforms: camera-to-nozzle offsets, camera pixel scale (units per pixel at a given Z), lens distortion calibration, nonsquareness/skew compensation.

7.2 Drivers¶

A MotionDriver interface with implementations:

Driver responsibilities: connect/handshake, queue commands with flow control, report position and status, wait for motion completion, read sensors, issue immediate stop.

7.3 Job engine¶

• Import placement (centroid) files from KiCad, Eagle/Fusion, Altium, and generic CSV with column mapping. Import BOMs to link placements to parts.
• Board setup: board origin and rotation via fiducials (2+ fiducials; affine fit with 3+), panels/arrays, and bottom-side placements.
• Planner orders placements to minimize travel and nozzle-tip changes, and to use multiple nozzles in parallel (pick N, align N, place N).
• Executor runs the plan as a state machine with pause, resume, single step, skip, and retry. Every step emits events. Errors (mis-pick, vacuum failure, vision failure) stop or retry according to policy and ask the operator through the API when needed.
• History: each run records placements, timings, vision results, and errors in SQLite.

7.4 Feeders¶

A Feeder interface: prepare(), feed(), pick_location(), plus optional vision correction. Implementations:

• Photon feeders (LumenPnP): addressed over RS-485 through the motherboard. Support discovery, address assignment, feed, and status. 1.0.
• Strip feeders (cut tape on a fixture) with vision-located sprocket holes. 1.0.
• Tray feeders (grid of parts). 1.0.
• Push-pull/drag feeders and tube feeders. Post-1.0.

7.5 Vision¶

• Pipelines are defined as data (a list of stages with parameters), editable in the GUI with live preview, and stored per part/package/feeder. Users can also write a pipeline stage in Python.
• Built-in pipelines: fiducial locate, bottom-vision part alignment (rectangle/ body/lead fit), sprocket hole detection, nozzle-tip runout calibration, camera calibration (checkerboard/ChArUco).
• Acceleration abstraction: stages operate on an image type that maps to cv2.UMat (OpenCL) when available and to NumPy arrays otherwise. CUDA support is a later, optional backend. CPU must always work and is the reference for tests.
• Debuggability: every pipeline run can return intermediate images and numeric results over the API, so the GUI can show each stage.
• Test corpus: a collection of real LumenPnP camera images with expected results, used in CI to catch regressions.

7.6 Configuration and data¶

• Machine config: human-readable text files under a config directory, validated by Pydantic, versioned with automatic migrations. Easy to diff and back up.
• Part, package, and feeder libraries: SQLite, with import/export to text.
• Calibration results are stored separately from hand-written config so recalibration does not rewrite user edits.

8. GUI — PicnicPlace Studio¶

8.1 Technology¶

• TypeScript, React, built with Vite.
• 2D board/machine view on canvas or WebGL (e.g. PixiJS); 3D view optional later.
• Served by the core for browser use. Optionally packaged as a desktop app with Tauri, which also gives a native window, menus, and file dialogs.
• API client generated from the core's OpenAPI schema.

8.2 Design direction¶

Inspired by OrcaSlicer: a clean, dense-but-calm layout, dark and light themes, a single accent color, flat icons, and a large central canvas with settings in a side panel.

Proposed top-level workspaces (tabs across the top, as in OrcaSlicer's Prepare / Preview / Device):

Design principles:

• Guided first, advanced available. Calibration and setup are wizards with explanations and pictures; raw settings are behind an "Advanced" toggle, as OrcaSlicer does with its simple/advanced modes.
• Live state everywhere. Machine position, connection state, lease holder, and E-stop are always visible in a status bar.
• Touch friendly on the Run and Machine workspaces, for tablets mounted near the machine.
• Design tokens (color, spacing, type) in one place so themes are easy to adjust.

8.3 Machine view¶

The center of the Machine workspace is the machine view: a scale overhead map of the machine, with a head inset in its lower-right corner. Live camera feeds sit in the sidebar beside it, and the axis positions are in the status bar.

Overhead view¶

• Drawn to scale in machine coordinates (X right, Y up), with zoom and pan. The machine coordinates under the cursor are shown.
• Shows:
• the travel limits (soft limits) and machine bed outline
• boards, with their placements as markers colored by status
• feeders (Photon slots on the LumenPnP) and their pick locations
• the bottom camera, with its field of view
• the homing fiducial
• nozzle tip holders and the discard location
• The head is a crosshair at the current position, with the top camera's field-of-view rectangle and each nozzle's offset drawn around it. Its live position comes from the machine.state events.
• Click to move, only while motion is enabled. A modifier key or a confirm step is required so a stray click or tap cannot move the head. Moves to the target at safe Z.

Head inset¶

A side view of a generic head, floating in the lower-right corner of the overhead view. It can be collapsed.

• Each nozzle's height, drawn to scale against the safe-Z line. On the LumenPnP v4 both nozzles share one Z motor and move in opposite directions, and the drawing shows that.
• Each nozzle's rotation as a small dial, the fitted nozzle tip, and whether a part is held.
• Status lights along the edge, generated from the machine profile rather than hard coded, so the panel matches each machine's actual I/O:
• Inputs are read-only lights: endstops/home sensors, safe Z, and others the profile declares.
• Safe Z is a solid light when sensed by a physical input (§5.2) and an outlined light when computed.
• Vacuum per nozzle is a small bar showing the pressure reading with the pick threshold marked, plus a lit state. This makes weak picks visible before they fail.
• Outputs (pump, valves, ring lights) look different from inputs and can be toggled with a click in manual mode, when the client holds the control lease.

Core support needed¶

• A machine description endpoint: bed size and limits, camera and fiducial positions, nozzles and their offsets, tip holders, feeders, and the list of inputs and outputs with their kinds (digital, analog, sensed or computed).
• Input and output states in the machine.state events.
• The simulator provides all of the above with simulated sensor values, so the machine view can be developed and tested without hardware.

8.4 CLI¶

A picnic command line client for scripting and headless use: connect, status, home, jog, run a job, export logs. Uses the same API.

9. Platform support¶

Decided: the 1.0 core supports Linux only.

• Supported: tested in CI and on real hardware before release.
• Development only: the core runs against the simulator so contributors can develop on a Mac, but real hardware is not supported or tested.
• The browser GUI works on any OS, since it only needs a modern browser.

Camera capture and serial port naming are kept behind interfaces, which is where most porting work will be when macOS and Windows support is added.

10. LumenPnP target¶

To verify against the actual machine and Opulo's published configuration:

• Firmware: Marlin-based on the LumenPnP motherboard, G-code over USB serial.
• Two nozzles on a shared head, each with Z and rotation.
• Top camera on the head; bottom camera fixed under the head path. Both USB UVC cameras.
• Vacuum pump(s), valves, and vacuum pressure sensors per nozzle.
• Photon feeders on an RS-485 bus via the motherboard.
• Ring lights on both cameras.

Likely second target: Quad 4c. An older industrial machine the project lead plans to convert. It has a physical safe-Z input. Driver and machine-profile interfaces must not assume LumenPnP-only behavior.

PicnicPlace should ship a LumenPnP machine profile so a new user selects their machine version and goes straight to guided calibration.

Reference machine: LumenPnP v4.1. The first machine profile targets v4.1 (PCB staging plates, bottom camera at the center). v4.2 (metal plates, camera near one end) needs its own profile. Other versions are community supported. The test machine was built by Opulo around October 2026, so it should have recent stock firmware.

The Marlin driver reads the firmware name and version (M115) on connect, records it in the machine state, and warns when the version is not in the known-tested list instead of refusing to connect.

11. Milestones¶

12. Repository layout¶

picnicplace/
  core/            Python package: picnicd, drivers, vision, job engine, CLI, and the
                   built-in machine profiles (core/src/picnicplace/profiles/)
  studio/          TypeScript GUI
  docs/            Specs, ADRs, user docs, generated OpenAPI document (docs/api/)
  testdata/        Vision image corpus, sample boards

13. Open questions¶

1. ~~License~~ Decided: MPL-2.0 (see §14).
2. ~~Config format~~ Decided: TOML.
3. ~~GUI framework~~ Decided: React.
4. ~~Video transport~~ Decided: MJPEG and WebRTC (see §6.5).
5. ~~Auth~~ Decided: single shared secret for 1.0; local-only motion planned next; roles later (§6.3).
6. ~~Client disconnect~~ Decided: motion requires a connected client; jobs pause, not end, when none are connected (§5.3).
7. Exact LumenPnP v4 firmware version: read it from the machine (M115) at M1.
8. ~~Platforms~~ Decided: Linux only for 1.0 (§9).
9. ~~Part on nozzle at pause~~ Decided: hold it (§5.3).

14. Licensing (decided)¶

License: Mozilla Public License 2.0 (MPL-2.0) for all PicnicPlace code.

Goals: commercial users can adopt and integrate PicnicPlace without fear, while improvements to PicnicPlace itself stay open.

• File-level copyleft. Anyone who distributes a modified version of a PicnicPlace source file must publish that file's source under MPL-2.0. Closed-source edits to the core are not allowed.
• Proprietary additions are allowed in separate files. A vendor can ship its own driver, feeder, or vision stage in its own files under its own license. This is an intentional tradeoff to keep commercial users comfortable.
• Network clients are independent works. Software that only talks to the core over the API (custom GUIs, MES or ERP integrations) can use any license.
• Patent grant protects users and contributors.
• GPL compatible. MPL-2.0 code can be combined into GPL projects.

Supporting policies:

• Trademark. The PicnicPlace name and logo are reserved by a written trademark policy. Forks may use the code but not the name, so a fork cannot present itself as the official project.
• Contributions use the Developer Certificate of Origin (Signed-off-by on commits), not a CLA. Copyright stays with the contributors, which keeps any later relicensing to a closed license impractical.
• No OpenPnP code. OpenPnP is GPL-3.0. Its code must not be copied or ported line by line. Reimplementing published algorithms and reading its file formats is fine.

Alternatives considered: