Skip to content

ADR 0008: IoT Data Pipeline — MQTT → TDengine → COS Archive

Status

Accepted

Date

2026-07-24

Context

OVES IoT data flows from globally distributed devices (battery sensors, charging stations, swap cabinets, vehicle GPS/BMS, environmental sensors) through MQTT into a time-series database, with dashboards for real-time monitoring, analytics for pattern detection, and long-term archival for compliance and historical analysis.

The existing pipeline used:

  • MQTT brokers at edge sites
  • InfluxDB on AWS EC2 for time-series storage and query
  • AWS S3 for archival
  • Grafana for dashboards (retained)

This architecture incurred cross-cloud egress costs, EC2 fixed costs, and operational overhead from managing InfluxDB upgrades.

Decision

The IoT pipeline is fully re-platformed onto Tencent Cloud: EMQX MQTT broker → Edge TDengine → Cloud TDengine (Tencent CVM) → COS Deep Archive for long-term storage.

Pipeline Architecture

Device Layer: Battery, Charger, Swap Cabinet, Vehicle, Environmental sensors
     │ MQTT / CoAP / HTTP
     ▼
Edge Layer: Edge TDengine (ARM binary, local 7d buffer, pre-aggregation)
     │ Auto-sync on reconnect
     ▼
Cloud Layer (Tencent Cloud, Guangzhou):
  ├── EMQX MQTT Broker (same CVM as TDengine)
  │     └── MQTT → InfluxDB line protocol → taosAdapter (:6041)
  ├── TDengine (4C/16G CVM): Real-time storage and query
  │     ├── 30d hot (SSD): 1-second raw data
  │     └── 90d warm (HDD): 1-minute aggregated data
  ├── Grafana: Real-time dashboards (TDengine plugin)
  ├── Python analytics: Battery degradation, anomaly detection
  └── COS Deep Archive: >90d CSV dumps, queryable via MetaInsight

Data Model (SuperTable)

CREATE STABLE battery_telemetry (
    ts TIMESTAMP, voltage FLOAT, current FLOAT,
    temperature FLOAT, soc FLOAT, soh FLOAT
) TAGS (device_id, site, model)

One SuperTable per device class, one sub-table per device instance.

Data Lifecycle

Retention Storage Resolution Purpose
0–30 days TDengine SSD 1s raw Real-time monitoring, alerting
30–90 days TDengine HDD 1min aggregate Trend analysis, dashboard history
90+ days COS Deep Archive Raw CSV Compliance, historical research, MetaInsight indexing

Archival Automation

Daily cron at 02:00: export 90-day-old data as CSV → upload to COS Deep Archive → delete from TDengine. Script included in IoT Data Pipeline Strategy document.

Migration Checklist (10 tasks, ~2 weeks)

  1. Provision Tencent CVM (Guangzhou, 4C/16G) — 1 day
  2. Install TDengine + taosAdapter — 2 hours
  3. Install EMQX MQTT Broker — 2 hours
  4. Configure MQTT → TDengine bridge — 4 hours
  5. Create SuperTable data models — 1 day
  6. Migrate InfluxDB historical data — 2 days
  7. Switch device MQTT endpoints — 1 day
  8. Build Grafana dashboards — 2 days
  9. Configure daily archive cron — 1 day
  10. Shutdown AWS EC2 (InfluxDB) — 1 hour

Consequences

Benefits

  • Zero cross-cloud egress: device → edge → cloud → archive all within Tencent ecosystem
  • 10× write performance improvement (InfluxDB → TDengine)
  • 75% storage reduction (TDengine columnar compression + delta encoding)
  • Fixed, predictable compute cost (CVM ¥500–800/month vs. EC2 $200–500/month)
  • Edge-cloud sync enables offline-capable IoT with automatic reconciliation
  • Archival cost: ¥0.01/GB/month (Deep Archive); ~¥36/year for 1,000 devices

Trade-Offs

  • Self-managed TDengine requires CVM provisioning and maintenance
  • TDengine ecosystem is smaller than InfluxDB (fewer Grafana templates, no native Telegraf plugin library — mitigated by taosAdapter)
  • Edge TDengine deployment requires ARM-capable edge hardware
  • InfluxQL → SQL query migration: ~1 week developer effort

Cost Impact

Item AWS (old) Tencent (new) Savings
Compute $200–500/month ¥500–800/month ~60%
Egress $0.09/GB × queries Free (intra-region) 100%
Archive S3 Glacier ($0.004/GB) COS Deep Archive (¥0.01/GB) ~50%
Monthly total $400–700 ¥600–1,000 ~$300–500

Supersedes

None.

Superseded By

None.