Add extended vehicle telemetry and exports

.hound.yml

@@ -1,3 +1,3 @@
 jshint:
-  enabled: true
+  enabled: false
   config_file: .jshintrc

.jshintrc

@@ -1,7 +1,7 @@
 {
     "undef": true,
     "unused": true,
-    "esversion": 6,
+    "esversion": 8,
     "node": true,
     "mocha": true,
     "multistr": true,

README.md

@@ -13,6 +13,54 @@ Backend for [EVNotify](https://github.com/EVNotify/EVNotify).
 - Optional outbound webhooks for sync and notification events
 - Error tracking through Rollbar
 
+## Extended vehicle telemetry
+
+`POST /extended` accepts optional measurements for any vehicle whose client can supply them. The backend does not decode vehicle data or restrict these fields to the Ioniq. Use JSON numbers, with voltages in **volts**, odometer in **whole kilometers**, and fan speed as a **raw numeric level** (not RPM or percent; levels may differ between vehicles).
+
+| POST field | GET `/extended` and history field | Storage and accepted range |
+| --- | --- | --- |
+| `batteryCellVoltageMin` | `battery_cell_voltage_min` | `DECIMAL(12,6)`, 0–999999.999999 V |
+| `batteryCellVoltageMax` | `battery_cell_voltage_max` | `DECIMAL(12,6)`, 0–999999.999999 V |
+| `batteryCellVoltageDelta` | `battery_cell_voltage_delta` | `DECIMAL(12,6)`, 0–999999.999999 V |
+| `batteryFanSpeed` | `battery_fan_speed` | `FLOAT`, nonnegative and finite when represented as a 32-bit float |
+| `odo` | `odo` | Existing signed `INT`, integer 0–2147483647 km |
+
+All five fields also accept `null`. An omitted field retains its current value; an explicit `null` clears it. Partial updates preserve omitted values across **all** extended telemetry fields. Zero is a measurement and is stored unchanged. New installations and migrated schemas default unknown odometer and new battery measurements to `NULL`; existing stored odometer zeros are retained.
+
+Voltages are rounded to six decimal places before storage; fan levels are normalized to 32-bit float precision. Delta is supplied by the client and remains in volts; the backend does not calculate it from retained minimum/maximum values. Fractional odometers, negative values, strings, booleans, other nonnumeric types, and out-of-range numbers in these five fields reject the entire telemetry submission with HTTP 400 and `INVALID_PARAMETERS`. Existing authentication and database-error responses are retained.
+
+```json
+{
+  "akey": "ABC123",
+  "token": "your-api-token",
+  "batteryCellVoltageMin": 3.78,
+  "batteryCellVoltageMax": 3.8,
+  "batteryCellVoltageDelta": 0.02,
+  "batteryFanSpeed": 0,
+  "odo": 123456
+}
+```
+
+`GET /extended?akey=ABC123&token=your-api-token` returns these fields as JSON numbers or `null`, using the snake_case names above. A later POST with only `batteryCellVoltageDelta: 0.01` updates that measurement and retains the other current values. Sending `batteryFanSpeed: null` clears only the fan measurement.
+
+History remains sparse: statistics record only measurements supplied in that submission, subject to the existing five-second history throttle. Retained current values are not copied into history. `/logdetail` exposes the new columns in live and archived statistics. Both XLSX report paths include an extended telemetry sheet with numeric measurements, voltage headers in V, odometer in km, a raw fan-level header, and blank unknown cells. English and German labels are available.
+
+### Database migration
+
+Fresh installations use `modules/db/db_template.sql`. Existing installations must run the manual, rerunnable [migration](modules/db/migrations/extended_vehicle_telemetry.sql) **before deploying the updated backend**:
+
+1. Back up the selected database, including archive tables. Pause both `archiveLogs` and `archiveStandaloneLogs` jobs and wait for active runs to finish. Schedule a maintenance window for ALTER TABLE locks; MySQL DDL commits implicitly.
+2. Use an account with `SELECT` and `ALTER` on the target tables and `CREATE ROUTINE`, `ALTER ROUTINE`, and `EXECUTE` privileges on the database. Run the following with the MySQL CLI, selecting the actual database:
+
+   ```bash
+   mysql -h DB_HOST -u DB_USER -p DB_NAME < modules/db/migrations/extended_vehicle_telemetry.sql
+   ```
+
+3. Verify that `sync`, `statistics`, existing `statistics_MONTH_YEAR` archives, and `statistics_log_ID` archives each contain the four nullable battery columns with the types listed above, and that `sync.odo` defaults to `NULL`. Check that existing row counts and odometer values are unchanged. The migration adds only missing columns; it does not repair conflicting preexisting definitions. If interrupted, rerun it after resolving the error.
+4. Deploy the backend, check a POST/GET round trip, and resume archive jobs. Future archives inherit the updated statistics schema. Monitor API database errors and archive-job failures.
+
+The migration supports MySQL 5.7 and 8.0. It leaves historical unknowns `NULL`, preserves the existing odometer type and values, and does not backfill measurements. To roll back application code, retain the additive columns and restored defaults rather than dropping recorded telemetry.
+
 ## Webhooks
 
 Webhooks are configured by an administrator on request, directly in the user's `settings.webhook` database column. `GET /settings` returns the configured URL. `PUT /settings` ignores any submitted `webhook` value and omits it from its response; users cannot change or disable delivery through the API. To disable delivery, an administrator sets the database value to `NULL` or an empty string.
@@ -53,13 +101,15 @@ Delivery happens in the background with a 10-second request timeout and no retri
 
 Install dependencies with `npm install`. GitHub Actions runs the full test suite in Docker using Node 8.17.0 and MySQL 5.7.34, with a disposable test database.
 
-Run the isolated settings, webhook, sync, and notification tests without a backend or database:
+Hound's older JavaScript parser is disabled. The three existing report/export TODOs have individual ESLint suppressions; other CodeFactor checks remain enabled.
+
+Run the isolated settings, webhook, sync, notification, and XLSX tests without a backend or database:
 
 ```bash
-./node_modules/.bin/mocha tests/settings/index.js tests/webhook/index.js tests/sync/index.js tests/notification/index.js
+./node_modules/.bin/mocha tests/settings tests/webhook tests/sync tests/notification tests/report
 ```
 
-The full suite also includes account integration tests, which require a running backend at `TEST_URL` (or the configured local port) and a local test database configured in `srv_config.json`. Its setup creates the database and imports the schema:
+The full suite also includes account integration tests, which require a running backend at `TEST_URL` (or the configured local port) and a local test database configured in `srv_config.json`. Its setup creates the database and imports the schema. Telemetry integration tests additionally create and remove uniquely named disposable databases using the same connection credentials; the test user needs database creation/deletion and routine privileges. They test migration reruns, both archive formats, HTTP round trips, and account isolation:
 
 ```bash
 npm test

modules/db/db_template.sql

@@ -49,7 +49,11 @@ CREATE TABLE IF NOT EXISTS `sync` (
     `battery_max_temperature` FLOAT DEFAULT 0,
     `battery_inlet_temperature` FLOAT DEFAULT 0,
     `external_temperature` FLOAT DEFAULT 0,
-    `odo` INT(6) DEFAULT 0,
+    `battery_cell_voltage_min` DECIMAL(12,6) DEFAULT NULL,
+    `battery_cell_voltage_max` DECIMAL(12,6) DEFAULT NULL,
+    `battery_cell_voltage_delta` DECIMAL(12,6) DEFAULT NULL,
+    `battery_fan_speed` FLOAT DEFAULT NULL,
+    `odo` INT(6) DEFAULT NULL,
     `latitude` DECIMAL(10, 8) DEFAULT 0,
     `longitude` DECIMAL(11, 8) DEFAULT 0,
     `gps_speed` FLOAT DEFAULT 0,
@@ -85,6 +89,10 @@ CREATE TABLE IF NOT EXISTS `statistics` (
     `battery_max_temperature` FLOAT DEFAULT NULL,
     `battery_inlet_temperature` FLOAT DEFAULT NULL,
     `external_temperature` FLOAT DEFAULT NULL,
+    `battery_cell_voltage_min` DECIMAL(12,6) DEFAULT NULL,
+    `battery_cell_voltage_max` DECIMAL(12,6) DEFAULT NULL,
+    `battery_cell_voltage_delta` DECIMAL(12,6) DEFAULT NULL,
+    `battery_fan_speed` FLOAT DEFAULT NULL,
     `odo` INT(6) DEFAULT NULL,
     `latitude` DECIMAL(10, 8) DEFAULT NULL,
     `longitude` DECIMAL(11, 8) DEFAULT NULL,

modules/db/migrations/extended_vehicle_telemetry.sql

@@ -0,0 +1,59 @@
+-- Run with a selected database using the mysql CLI (DELIMITER is a CLI command).
+-- Pause both archive jobs first. DDL commits implicitly; take a backup beforehand.
+-- Adds missing columns only; existing measurements are never backfilled.
+DROP PROCEDURE IF EXISTS evnotify_extend_vehicle_telemetry;
+DELIMITER //
+CREATE PROCEDURE evnotify_extend_vehicle_telemetry()
+BEGIN
+    DECLARE finished INT DEFAULT 0;
+    DECLARE target_table VARCHAR(64);
+    DECLARE field_index INT;
+    DECLARE field_name VARCHAR(64);
+    DECLARE field_type VARCHAR(32);
+    DECLARE column_exists INT;
+    DECLARE base_tables INT;
+    DECLARE targets CURSOR FOR
+        SELECT TABLE_NAME FROM information_schema.TABLES
+        WHERE TABLE_SCHEMA = DATABASE() AND TABLE_TYPE = 'BASE TABLE'
+          AND TABLE_NAME REGEXP '^(sync|statistics|statistics_[0-9]{1,2}_[0-9]{4}|statistics_log_[0-9]+)$';
+    DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1;
+
+    SELECT COUNT(*) INTO base_tables FROM information_schema.TABLES
+    WHERE TABLE_SCHEMA = DATABASE() AND TABLE_TYPE = 'BASE TABLE'
+      AND TABLE_NAME IN ('sync', 'statistics');
+    IF base_tables <> 2 THEN
+        SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Select an EVNotify database containing sync and statistics';
+    END IF;
+
+    OPEN targets;
+    table_loop: LOOP
+        FETCH targets INTO target_table;
+        IF finished THEN LEAVE table_loop; END IF;
+        SET field_index = 0;
+        WHILE field_index < 4 DO
+            SET field_name = CASE field_index
+                WHEN 0 THEN 'battery_cell_voltage_min'
+                WHEN 1 THEN 'battery_cell_voltage_max'
+                WHEN 2 THEN 'battery_cell_voltage_delta'
+                ELSE 'battery_fan_speed' END;
+            SET field_type = IF(field_index = 3, 'FLOAT', 'DECIMAL(12,6)');
+            SELECT COUNT(*) INTO column_exists FROM information_schema.COLUMNS
+            WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = target_table AND COLUMN_NAME = field_name;
+            IF column_exists = 0 THEN
+                SET @telemetry_ddl = CONCAT('ALTER TABLE `', target_table, '` ADD COLUMN `',
+                    field_name, '` ', field_type, ' NULL DEFAULT NULL');
+                PREPARE telemetry_statement FROM @telemetry_ddl;
+                EXECUTE telemetry_statement;
+                DEALLOCATE PREPARE telemetry_statement;
+            END IF;
+            SET field_index = field_index + 1;
+        END WHILE;
+    END LOOP;
+    CLOSE targets;
+
+    -- Retain odometer type and all stored values, including existing zeros.
+    ALTER TABLE sync ALTER COLUMN odo SET DEFAULT NULL;
+END//
+DELIMITER ;
+CALL evnotify_extend_vehicle_telemetry();
+DROP PROCEDURE evnotify_extend_vehicle_telemetry;

modules/logs/index.js

@@ -121,7 +121,7 @@ const deleteLog = async (akey, id) => {
  * @param {String} email the email where export will be sent to
  */
 const exportLog = async (akey, id, email) => {
-    const log = (await query('SELECT start, end, archived, autogenerated FROM logs where akey=? and id=?', [akey, id]))[0];
+    const log = (await query('SELECT start, end, archived, standalone, autogenerated FROM logs where akey=? and id=?', [akey, id]))[0];
 
     if (log != null) {
         if (!log.autogenerated) throw new Error(srv_errors.LOG_EXPORT_UNAVAILABLE.message);
@@ -137,6 +137,7 @@ const exportLog = async (akey, id, email) => {
         const statistics = (await query(`SELECT * FROM ${table} WHERE akey=? AND timestamp >= ? AND timestamp <= ?`, [akey, log.start, log.end]));
 
         return report.logExportReport(statistics).then((file) => {
+            // eslint-disable-next-line no-warning-comments
             // TODO localize
             mail.simpleSend(email, 'Log-Export', 'Log-Export', [{
                 filename: 'Export.xlsx',

modules/report/index.js

@@ -71,6 +71,18 @@ const worksheets = {
                     "excel_name": "SPEED"
                 },
             ],
+        },
+        {
+            "name": "EXTENDED_TELEMETRY_HISTORY",
+            "condition": ["battery_cell_voltage_min", "battery_cell_voltage_max", "battery_cell_voltage_delta", "battery_fan_speed", "odo"],
+            "sql_columns": [
+                { "name": "timestamp", "type": "date", "excel_name": "DATE_TIME" },
+                { "name": "battery_cell_voltage_min", "type": "number", "excel_name": "BATTERY_CELL_VOLTAGE_MIN_V" },
+                { "name": "battery_cell_voltage_max", "type": "number", "excel_name": "BATTERY_CELL_VOLTAGE_MAX_V" },
+                { "name": "battery_cell_voltage_delta", "type": "number", "excel_name": "BATTERY_CELL_VOLTAGE_DELTA_V" },
+                { "name": "battery_fan_speed", "type": "number", "excel_name": "BATTERY_FAN_LEVEL_RAW" },
+                { "name": "odo", "type": "number", "excel_name": "ODOMETER_KM" }
+            ]
         }
     ]
 };
@@ -126,6 +138,7 @@ const createReport = (akey, token, callback) => {
                             let wb = new excel.Workbook(),
                                 wss = worksheets.log.map(e => new Worksheet(wb, lng, e));
                             dataRes.forEach(e => wss.forEach(w => w.processRow(e)));
+                            // eslint-disable-next-line no-warning-comments
                             // TODO dynamic filename (tmp file which need to be unlinked and streamed to user..)
                             wb.write('Excel.xlsx');
                             callback(null, 'Excel.xlsx');
@@ -141,6 +154,7 @@ const logExportReport = (data) => {
     return new Promise((resolve, reject) => {
         const randomFile = 'tmp_logexportreport_' + crypto.randomBytes(20).toString('hex') + '.xlsx';
         try {
+            // eslint-disable-next-line no-warning-comments
             // TODO lng..
             const wb = new excel.Workbook({
                 dateFormat: 'm/d/yy hh:mm:ss',
@@ -189,4 +203,4 @@ module.exports = {
         });
     },
     logExportReport
-};
+};

modules/sync/index.js

@@ -8,7 +8,8 @@ const srv_config = require('./../../srv_config.json'),
     db = require('./../db'),
     abrp = require('./../integrations/abrp'),
     webhook = require('./../webhook'),
-    token = require('./../token');
+    token = require('./../token'),
+    telemetry = require('./telemetry');
 
 const updatedRecently = (now, dbTimestamp) => now - 5 < dbTimestamp;
 
@@ -65,32 +66,26 @@ const getSoC = (akey, callback) => {
  */
 const postExtended = (akey, extendedObj, callback) => {
     const now = parseInt(new Date() / 1000);
+    const supplied = telemetry.supplied(extendedObj);
+    const columns = supplied.map(field => telemetry.fields[field]);
+    const values = supplied.map(field => telemetry.storedValue(extendedObj, field));
 
-    db.query('SELECT last_extended FROM sync WHERE akey=?', [
-        akey,
-    ], (err, queryRes) => {
-        if (!err && queryRes) {
-            const lastExtended = queryRes[0] ? queryRes[0].last_extended : 0;
+    db.query('SELECT last_extended FROM sync WHERE akey=?', [akey], (err, queryRes) => {
+        if (err || !queryRes) return callback(err);
+        const lastExtended = queryRes[0] ? queryRes[0].last_extended : 0;
+        const assignments = columns.concat(['last_extended']).map(column => column + '=?');
 
-            db.query('UPDATE sync SET soh=?, charging=?, rapid_charge_port=?, normal_charge_port=?, slow_charge_port=?, aux_battery_voltage=?, dc_battery_voltage=?, dc_battery_current=?, dc_battery_power=?,\
-                cumulative_energy_charged=?, cumulative_energy_discharged=?, battery_min_temperature=?, battery_max_temperature=?, battery_inlet_temperature=?, external_temperature=?, odo=?, last_extended=? WHERE akey=?', [
-                extendedObj.soh, extendedObj.charging, extendedObj.rapidChargePort, extendedObj.normalChargePort, extendedObj.slowChargePort, extendedObj.auxBatteryVoltage, extendedObj.dcBatteryVoltage, extendedObj.dcBatteryCurrent,
-                extendedObj.dcBatteryPower, extendedObj.cumulativeEnergyCharged, extendedObj.cumulativeEnergyDischarged, extendedObj.batteryMinTemperature, extendedObj.batteryMaxTemperature, extendedObj.batteryInletTemperature, extendedObj.externalTemperature, extendedObj.odo, now, akey
-            ], err => {
-                if (!err) {
-                    if (updatedRecently(now, lastExtended)) {
-                        callback(null, true);
-                        return;
-                    }
+        db.query('UPDATE sync SET ' + assignments.join(', ') + ' WHERE akey=?',
+            values.concat([now, akey]), err => {
+                if (err) return callback(err);
+                if (updatedRecently(now, lastExtended)) return callback(null, true);
 
-                    db.query('INSERT INTO statistics (soh, charging, rapid_charge_port, normal_charge_port, slow_charge_port, aux_battery_voltage, dc_battery_voltage, \
-                    dc_battery_current, dc_battery_power, cumulative_energy_charged, cumulative_energy_discharged, battery_min_temperature, battery_max_temperature, battery_inlet_temperature, external_temperature, odo, timestamp, akey) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [
-                        extendedObj.soh, extendedObj.charging, extendedObj.rapidChargePort, extendedObj.normalChargePort, extendedObj.slowChargePort, extendedObj.auxBatteryVoltage, extendedObj.dcBatteryVoltage, extendedObj.dcBatteryCurrent,
-                        extendedObj.dcBatteryPower, extendedObj.cumulativeEnergyCharged, extendedObj.cumulativeEnergyDischarged, extendedObj.batteryMinTemperature, extendedObj.batteryMaxTemperature, extendedObj.batteryInletTemperature, extendedObj.externalTemperature, extendedObj.odo, now, akey
-                    ], (err, dbRes) => callback(err, (!err && dbRes)));
-                } else callback(err);
+                // History contains only this submission, never retained current values.
+                const historyColumns = columns.concat(['timestamp', 'akey']);
+                db.query('INSERT INTO statistics (' + historyColumns.join(', ') + ') VALUES (' +
+                    historyColumns.map(() => '?').join(', ') + ')', values.concat([now, akey]),
+                    (err, dbRes) => callback(err, (!err && dbRes)));
             });
-        } else callback(err);
     });
 };
 
@@ -100,10 +95,9 @@ const postExtended = (akey, extendedObj, callback) => {
  * @param {Function} callback callback function
  */
 const getExtended = (akey, callback) => {
-    db.query('SELECT soh, charging, rapid_charge_port, normal_charge_port, slow_charge_port, aux_battery_voltage, dc_battery_voltage, dc_battery_current, dc_battery_power, \
-    cumulative_energy_charged, cumulative_energy_discharged, battery_min_temperature, battery_max_temperature, battery_inlet_temperature, external_temperature, odo, last_extended FROM sync WHERE akey=?', [
-        akey
-    ], (err, queryRes) => callback(err, ((!err && queryRes) ? queryRes[0] : null)));
+    const columns = Object.keys(telemetry.fields).map(field => telemetry.fields[field]).concat(['last_extended']);
+    db.query('SELECT ' + columns.join(', ') + ' FROM sync WHERE akey=?', [akey],
+        (err, queryRes) => callback(err, ((!err && queryRes) ? telemetry.serialize(queryRes[0]) : null)));
 };
 
 /**
@@ -241,8 +235,8 @@ module.exports = {
         });
     },
     postExtended: (req, res) => {
-        // check required params
-        if (!req.body.akey || !req.body.token) {
+        // Validate the entire submission before any telemetry writes.
+        if (!req.body.akey || !req.body.token || !telemetry.valid(req.body)) {
             return res.status(400).json({
                 error: srv_errors.INVALID_PARAMETERS
             });