Plasmon: Telemetry Architecture Designed for 100K+ Edge Devices

1. Context

Plasmon was engineered for continuous monitoring and remote setpoint orchestration of industrial chillers and HVAC units in harsh factory environments. Designed for 100K+ edge devices, the architecture covers the full pipeline from physical sensor interfaces up to centralized broker ingestion.

2. Problem

Deployments faced two fundamental hurdles:

1. Unreliable Cellular Coverage: Field sites experienced frequent cellular/4G dropouts and base-station reboots. Conventional HTTP polling resulted in RAM memory exhaustion and data gaps.
2. Bi-Directional Command & Control: Edge units required not only passive reporting, but also dynamic setpoint adjustments, Modbus coil/holding register writing, and secure over-the-air (OTA) firmware delivery.

3. Architecture & Data Flow

 [ Sensor & Chiller Controller ]
               │ (RS-485 / Modbus RTU)
               ▼
 ┌────────────────────────────────────────────────────────┐
 │ ESP32 Edge Gateway (FreeRTOS)                         │
 │ ┌───────────────────┐        ┌───────────────────────┐ │
 │ │ Modbus Poller     │───────►│ SPIFFS Ring-Buffer    │ │
 │ │ (Core 0, 1000ms)  │        │ (Flash buffer offline)│ │
 │ └───────────────────┘        └──────────┬────────────┘ │
 │                                         │              │
 │ ┌───────────────────┐                   │ On Reconnect │
 │ │ MQTT Client (TLS) │◄──────────────────┘              │
 │ └─────────┬─────────┘                                  │
 └───────────┼────────────────────────────────────────────┘
             │ MQTT/TLS (QoS 1) - GSM 4G / Wi-Fi
             ▼
 ┌────────────────────────────────────────────────────────┐
 │ Ingestion & Broker Cluster                             │
 │ ┌───────────────────┐        ┌───────────────────────┐ │
 │ │ EMQX / Mosquitto  │───────►│ RabbitMQ Telemetry    │ │
 │ │ TLS Broker        │        │ Topic Exchanges (DLX) │ │
 │ └───────────────────┘        └──────────┬────────────┘ │
 └─────────────────────────────────────────┼──────────────┘
                                           ▼
                                [ TimescaleDB & WebFlux ]

4. Decisions & Rejected Alternatives

• ESP32 + C (FreeRTOS) chosen, Embedded Linux rejected: In industrial cabinets exceeding 70°C, single-board computers (Raspberry Pi) suffered SD card corruptions during abrupt power outages and drew 3W+. The ESP32 with hardware watchdog and static memory allocations provided deterministic, 24/7 reliability.
• MQTT/TLS (QoS 1) chosen, HTTP REST rejected: Opening TCP/TLS handshakes for every payload consumed prohibitive cellular data. A persistent MQTT socket allowed compact streaming with sub-second disconnection detection via Last Will & Testament (LWT).
• RabbitMQ chosen for ingestion, Kafka deferred: Per-device RPC request-reply patterns, dynamic routing keys, and lower operational complexity made RabbitMQ ideal for this workload.

5. Production Code: Flash Ring-Buffer Core

#include "esp_spiffs.h"
#include "esp_log.h"

#define TELEMETRY_RING_FILE "/spiffs/ring_buf.bin"
#define MAX_BUFFER_ENTRIES   2048

typedef struct __attribute__((packed)) {
    uint32_t timestamp;
    int16_t  temp_chiller_x10;
    int16_t  temp_ambient_x10;
    uint16_t pressure_bar_x100;
    uint8_t  compressor_state;
    uint8_t  crc8;
} TelemetryFrame;

esp_err_t ring_buffer_push(const TelemetryFrame *frame) {
    FILE *f = fopen(TELEMETRY_RING_FILE, "a+b");
    if (!f) {
        ESP_LOGE("RING", "Failed to open flash ring buffer");
        return ESP_FAIL;
    }
    
    long size = ftell(f);
    if (size >= (long)(MAX_BUFFER_ENTRIES * sizeof(TelemetryFrame))) {
        // Circular wrap-around on boundary exhaustion
        fseek(f, 0, SEEK_SET);
    }
    
    fwrite(frame, sizeof(TelemetryFrame), 1, f);
    fflush(f);
    fclose(f);
    return ESP_OK;
}

6. Outcome & Measurable Results

• Design Capacity: Sized for 100K+ edge units concurrently publishing telemetry.
• Verified Test Scale: 10,000+ active sensor load simulation; p95 < 22ms broker ingest latency.
• Cellular Outage Resilience: During carrier downtime, the ~320KB internal flash ring-buffer maintained approximately ~48 hours of continuous frames without heap overflows, draining to RabbitMQ upon reconnect with 0 packet loss.

7. What I Would Do Differently

• Initially, JSON was used for device telemetry payloads. At ~180 bytes per frame, JSON caused unnecessary cellular data costs. Transitioning to Protobuf or packed binary frames from day one would have reduced bandwidth consumption by 65% upfront.