From 20da1d9b01d5c37a44b5500db83f596e4cd64f25 Mon Sep 17 00:00:00 2001 From: GPlay97 Date: Sun, 25 Oct 2026 22:17:04 +0100 Subject: [PATCH 1/4] Add extended vehicle telemetry and exports --- README.md | 54 ++++- modules/db/db_template.sql | 10 +- .../migrations/extended_vehicle_telemetry.sql | 59 +++++ modules/logs/index.js | 2 +- modules/report/index.js | 12 + modules/sync/index.js | 52 ++-- modules/sync/telemetry.js | 58 +++++ modules/translation/lng/de.json | 6 + modules/translation/lng/en.json | 6 + tests/report/index.js | 89 +++++++ tests/sync/telemetry.js | 132 +++++++++++ tests/telemetry/index.js | 222 ++++++++++++++++++ tests/telemetry/legacy-schema.sql | 91 +++++++ 13 files changed, 759 insertions(+), 34 deletions(-) create mode 100644 modules/db/migrations/extended_vehicle_telemetry.sql create mode 100644 modules/sync/telemetry.js create mode 100644 tests/report/index.js create mode 100644 tests/sync/telemetry.js create mode 100644 tests/telemetry/index.js create mode 100644 tests/telemetry/legacy-schema.sql diff --git a/README.md b/README.md index dec522d..ac14884 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,54 @@ Backend for [EVNotify](https://github.com/EVNotify/EVNotify). - Optional outbound webhooks for sync and notification events - Error tracking through Rollbar +## Extended vehicle telemetry + +`POST /extended` accepts optional measurements for any vehicle whose client can supply them. The backend does not decode vehicle data or restrict these fields to the Ioniq. Use JSON numbers, with voltages in **volts**, odometer in **whole kilometers**, and fan speed as a **raw numeric level** (not RPM or percent; levels may differ between vehicles). + +| POST field | GET `/extended` and history field | Storage and accepted range | +| --- | --- | --- | +| `batteryCellVoltageMin` | `battery_cell_voltage_min` | `DECIMAL(12,6)`, 0–999999.999999 V | +| `batteryCellVoltageMax` | `battery_cell_voltage_max` | `DECIMAL(12,6)`, 0–999999.999999 V | +| `batteryCellVoltageDelta` | `battery_cell_voltage_delta` | `DECIMAL(12,6)`, 0–999999.999999 V | +| `batteryFanSpeed` | `battery_fan_speed` | `FLOAT`, nonnegative and finite when represented as a 32-bit float | +| `odo` | `odo` | Existing signed `INT`, integer 0–2147483647 km | + +All five fields also accept `null`. An omitted field retains its current value; an explicit `null` clears it. Partial updates preserve omitted values across **all** extended telemetry fields. Zero is a measurement and is stored unchanged. New installations and migrated schemas default unknown odometer and new battery measurements to `NULL`; existing stored odometer zeros are retained. + +Voltages are rounded to six decimal places before storage; fan levels are normalized to 32-bit float precision. Delta is supplied by the client and remains in volts; the backend does not calculate it from retained minimum/maximum values. Fractional odometers, negative values, strings, booleans, other nonnumeric types, and out-of-range numbers in these five fields reject the entire telemetry submission with HTTP 400 and `INVALID_PARAMETERS`. Existing authentication and database-error responses are retained. + +```json +{ + "akey": "ABC123", + "token": "your-api-token", + "batteryCellVoltageMin": 3.78, + "batteryCellVoltageMax": 3.8, + "batteryCellVoltageDelta": 0.02, + "batteryFanSpeed": 0, + "odo": 123456 +} +``` + +`GET /extended?akey=ABC123&token=your-api-token` returns these fields as JSON numbers or `null`, using the snake_case names above. A later POST with only `batteryCellVoltageDelta: 0.01` updates that measurement and retains the other current values. Sending `batteryFanSpeed: null` clears only the fan measurement. + +History remains sparse: statistics record only measurements supplied in that submission, subject to the existing five-second history throttle. Retained current values are not copied into history. `/logdetail` exposes the new columns in live and archived statistics. Both XLSX report paths include an extended telemetry sheet with numeric measurements, voltage headers in V, odometer in km, a raw fan-level header, and blank unknown cells. English and German labels are available. + +### Database migration + +Fresh installations use `modules/db/db_template.sql`. Existing installations must run the manual, rerunnable [migration](modules/db/migrations/extended_vehicle_telemetry.sql) **before deploying the updated backend**: + +1. Back up the selected database, including archive tables. Pause both `archiveLogs` and `archiveStandaloneLogs` jobs and wait for active runs to finish. Schedule a maintenance window for ALTER TABLE locks; MySQL DDL commits implicitly. +2. Use an account with `SELECT` and `ALTER` on the target tables and `CREATE ROUTINE`, `ALTER ROUTINE`, and `EXECUTE` privileges on the database. Run the following with the MySQL CLI, selecting the actual database: + + ```bash + mysql -h DB_HOST -u DB_USER -p DB_NAME < modules/db/migrations/extended_vehicle_telemetry.sql + ``` + +3. Verify that `sync`, `statistics`, existing `statistics_MONTH_YEAR` archives, and `statistics_log_ID` archives each contain the four nullable battery columns with the types listed above, and that `sync.odo` defaults to `NULL`. Check that existing row counts and odometer values are unchanged. The migration adds only missing columns; it does not repair conflicting preexisting definitions. If interrupted, rerun it after resolving the error. +4. Deploy the backend, check a POST/GET round trip, and resume archive jobs. Future archives inherit the updated statistics schema. Monitor API database errors and archive-job failures. + +The migration supports MySQL 5.7 and 8.0. It leaves historical unknowns `NULL`, preserves the existing odometer type and values, and does not backfill measurements. To roll back application code, retain the additive columns and restored defaults rather than dropping recorded telemetry. + ## Webhooks Webhooks are configured by an administrator on request, directly in the user's `settings.webhook` database column. `GET /settings` returns the configured URL. `PUT /settings` ignores any submitted `webhook` value and omits it from its response; users cannot change or disable delivery through the API. To disable delivery, an administrator sets the database value to `NULL` or an empty string. @@ -53,13 +101,13 @@ Delivery happens in the background with a 10-second request timeout and no retri Install dependencies with `npm install`. GitHub Actions runs the full test suite in Docker using Node 8.17.0 and MySQL 5.7.34, with a disposable test database. -Run the isolated settings, webhook, sync, and notification tests without a backend or database: +Run the isolated settings, webhook, sync, notification, and XLSX tests without a backend or database: ```bash -./node_modules/.bin/mocha tests/settings/index.js tests/webhook/index.js tests/sync/index.js tests/notification/index.js +./node_modules/.bin/mocha tests/settings tests/webhook tests/sync tests/notification tests/report ``` -The full suite also includes account integration tests, which require a running backend at `TEST_URL` (or the configured local port) and a local test database configured in `srv_config.json`. Its setup creates the database and imports the schema: +The full suite also includes account integration tests, which require a running backend at `TEST_URL` (or the configured local port) and a local test database configured in `srv_config.json`. Its setup creates the database and imports the schema. Telemetry integration tests additionally create and remove uniquely named disposable databases using the same connection credentials; the test user needs database creation/deletion and routine privileges. They test migration reruns, both archive formats, HTTP round trips, and account isolation: ```bash npm test diff --git a/modules/db/db_template.sql b/modules/db/db_template.sql index bc3d748..75f29fa 100644 --- a/modules/db/db_template.sql +++ b/modules/db/db_template.sql @@ -49,7 +49,11 @@ CREATE TABLE IF NOT EXISTS `sync` ( `battery_max_temperature` FLOAT DEFAULT 0, `battery_inlet_temperature` FLOAT DEFAULT 0, `external_temperature` FLOAT DEFAULT 0, - `odo` INT(6) DEFAULT 0, + `battery_cell_voltage_min` DECIMAL(12,6) DEFAULT NULL, + `battery_cell_voltage_max` DECIMAL(12,6) DEFAULT NULL, + `battery_cell_voltage_delta` DECIMAL(12,6) DEFAULT NULL, + `battery_fan_speed` FLOAT DEFAULT NULL, + `odo` INT(6) DEFAULT NULL, `latitude` DECIMAL(10, 8) DEFAULT 0, `longitude` DECIMAL(11, 8) DEFAULT 0, `gps_speed` FLOAT DEFAULT 0, @@ -85,6 +89,10 @@ CREATE TABLE IF NOT EXISTS `statistics` ( `battery_max_temperature` FLOAT DEFAULT NULL, `battery_inlet_temperature` FLOAT DEFAULT NULL, `external_temperature` FLOAT DEFAULT NULL, + `battery_cell_voltage_min` DECIMAL(12,6) DEFAULT NULL, + `battery_cell_voltage_max` DECIMAL(12,6) DEFAULT NULL, + `battery_cell_voltage_delta` DECIMAL(12,6) DEFAULT NULL, + `battery_fan_speed` FLOAT DEFAULT NULL, `odo` INT(6) DEFAULT NULL, `latitude` DECIMAL(10, 8) DEFAULT NULL, `longitude` DECIMAL(11, 8) DEFAULT NULL, diff --git a/modules/db/migrations/extended_vehicle_telemetry.sql b/modules/db/migrations/extended_vehicle_telemetry.sql new file mode 100644 index 0000000..002bbca --- /dev/null +++ b/modules/db/migrations/extended_vehicle_telemetry.sql @@ -0,0 +1,59 @@ +-- Run with a selected database using the mysql CLI (DELIMITER is a CLI command). +-- Pause both archive jobs first. DDL commits implicitly; take a backup beforehand. +-- Adds missing columns only; existing measurements are never backfilled. +DROP PROCEDURE IF EXISTS evnotify_extend_vehicle_telemetry; +DELIMITER // +CREATE PROCEDURE evnotify_extend_vehicle_telemetry() +BEGIN + DECLARE finished INT DEFAULT 0; + DECLARE target_table VARCHAR(64); + DECLARE field_index INT; + DECLARE field_name VARCHAR(64); + DECLARE field_type VARCHAR(32); + DECLARE column_exists INT; + DECLARE base_tables INT; + DECLARE targets CURSOR FOR + SELECT TABLE_NAME FROM information_schema.TABLES + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_TYPE = 'BASE TABLE' + AND TABLE_NAME REGEXP '^(sync|statistics|statistics_[0-9]{1,2}_[0-9]{4}|statistics_log_[0-9]+)$'; + DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1; + + SELECT COUNT(*) INTO base_tables FROM information_schema.TABLES + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_TYPE = 'BASE TABLE' + AND TABLE_NAME IN ('sync', 'statistics'); + IF base_tables <> 2 THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Select an EVNotify database containing sync and statistics'; + END IF; + + OPEN targets; + table_loop: LOOP + FETCH targets INTO target_table; + IF finished THEN LEAVE table_loop; END IF; + SET field_index = 0; + WHILE field_index < 4 DO + SET field_name = CASE field_index + WHEN 0 THEN 'battery_cell_voltage_min' + WHEN 1 THEN 'battery_cell_voltage_max' + WHEN 2 THEN 'battery_cell_voltage_delta' + ELSE 'battery_fan_speed' END; + SET field_type = IF(field_index = 3, 'FLOAT', 'DECIMAL(12,6)'); + SELECT COUNT(*) INTO column_exists FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = target_table AND COLUMN_NAME = field_name; + IF column_exists = 0 THEN + SET @telemetry_ddl = CONCAT('ALTER TABLE `', target_table, '` ADD COLUMN `', + field_name, '` ', field_type, ' NULL DEFAULT NULL'); + PREPARE telemetry_statement FROM @telemetry_ddl; + EXECUTE telemetry_statement; + DEALLOCATE PREPARE telemetry_statement; + END IF; + SET field_index = field_index + 1; + END WHILE; + END LOOP; + CLOSE targets; + + -- Retain odometer type and all stored values, including existing zeros. + ALTER TABLE sync ALTER COLUMN odo SET DEFAULT NULL; +END// +DELIMITER ; +CALL evnotify_extend_vehicle_telemetry(); +DROP PROCEDURE evnotify_extend_vehicle_telemetry; diff --git a/modules/logs/index.js b/modules/logs/index.js index d00ca88..99c3436 100644 --- a/modules/logs/index.js +++ b/modules/logs/index.js @@ -121,7 +121,7 @@ const deleteLog = async (akey, id) => { * @param {String} email the email where export will be sent to */ const exportLog = async (akey, id, email) => { - const log = (await query('SELECT start, end, archived, autogenerated FROM logs where akey=? and id=?', [akey, id]))[0]; + const log = (await query('SELECT start, end, archived, standalone, autogenerated FROM logs where akey=? and id=?', [akey, id]))[0]; if (log != null) { if (!log.autogenerated) throw new Error(srv_errors.LOG_EXPORT_UNAVAILABLE.message); diff --git a/modules/report/index.js b/modules/report/index.js index a8e485b..9fb5332 100644 --- a/modules/report/index.js +++ b/modules/report/index.js @@ -71,6 +71,18 @@ const worksheets = { "excel_name": "SPEED" }, ], + }, + { + "name": "EXTENDED_TELEMETRY_HISTORY", + "condition": ["battery_cell_voltage_min", "battery_cell_voltage_max", "battery_cell_voltage_delta", "battery_fan_speed", "odo"], + "sql_columns": [ + { "name": "timestamp", "type": "date", "excel_name": "DATE_TIME" }, + { "name": "battery_cell_voltage_min", "type": "number", "excel_name": "BATTERY_CELL_VOLTAGE_MIN_V" }, + { "name": "battery_cell_voltage_max", "type": "number", "excel_name": "BATTERY_CELL_VOLTAGE_MAX_V" }, + { "name": "battery_cell_voltage_delta", "type": "number", "excel_name": "BATTERY_CELL_VOLTAGE_DELTA_V" }, + { "name": "battery_fan_speed", "type": "number", "excel_name": "BATTERY_FAN_LEVEL_RAW" }, + { "name": "odo", "type": "number", "excel_name": "ODOMETER_KM" } + ] } ] }; diff --git a/modules/sync/index.js b/modules/sync/index.js index 042e405..539786f 100644 --- a/modules/sync/index.js +++ b/modules/sync/index.js @@ -8,7 +8,8 @@ const srv_config = require('./../../srv_config.json'), db = require('./../db'), abrp = require('./../integrations/abrp'), webhook = require('./../webhook'), - token = require('./../token'); + token = require('./../token'), + telemetry = require('./telemetry'); const updatedRecently = (now, dbTimestamp) => now - 5 < dbTimestamp; @@ -65,32 +66,26 @@ const getSoC = (akey, callback) => { */ const postExtended = (akey, extendedObj, callback) => { const now = parseInt(new Date() / 1000); + const supplied = telemetry.supplied(extendedObj); + const columns = supplied.map(field => telemetry.fields[field]); + const values = supplied.map(field => telemetry.storedValue(extendedObj, field)); - db.query('SELECT last_extended FROM sync WHERE akey=?', [ - akey, - ], (err, queryRes) => { - if (!err && queryRes) { - const lastExtended = queryRes[0] ? queryRes[0].last_extended : 0; + db.query('SELECT last_extended FROM sync WHERE akey=?', [akey], (err, queryRes) => { + if (err || !queryRes) return callback(err); + const lastExtended = queryRes[0] ? queryRes[0].last_extended : 0; + const assignments = columns.concat(['last_extended']).map(column => column + '=?'); - db.query('UPDATE sync SET soh=?, charging=?, rapid_charge_port=?, normal_charge_port=?, slow_charge_port=?, aux_battery_voltage=?, dc_battery_voltage=?, dc_battery_current=?, dc_battery_power=?,\ - cumulative_energy_charged=?, cumulative_energy_discharged=?, battery_min_temperature=?, battery_max_temperature=?, battery_inlet_temperature=?, external_temperature=?, odo=?, last_extended=? WHERE akey=?', [ - extendedObj.soh, extendedObj.charging, extendedObj.rapidChargePort, extendedObj.normalChargePort, extendedObj.slowChargePort, extendedObj.auxBatteryVoltage, extendedObj.dcBatteryVoltage, extendedObj.dcBatteryCurrent, - extendedObj.dcBatteryPower, extendedObj.cumulativeEnergyCharged, extendedObj.cumulativeEnergyDischarged, extendedObj.batteryMinTemperature, extendedObj.batteryMaxTemperature, extendedObj.batteryInletTemperature, extendedObj.externalTemperature, extendedObj.odo, now, akey - ], err => { - if (!err) { - if (updatedRecently(now, lastExtended)) { - callback(null, true); - return; - } + db.query('UPDATE sync SET ' + assignments.join(', ') + ' WHERE akey=?', + values.concat([now, akey]), err => { + if (err) return callback(err); + if (updatedRecently(now, lastExtended)) return callback(null, true); - db.query('INSERT INTO statistics (soh, charging, rapid_charge_port, normal_charge_port, slow_charge_port, aux_battery_voltage, dc_battery_voltage, \ - dc_battery_current, dc_battery_power, cumulative_energy_charged, cumulative_energy_discharged, battery_min_temperature, battery_max_temperature, battery_inlet_temperature, external_temperature, odo, timestamp, akey) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [ - extendedObj.soh, extendedObj.charging, extendedObj.rapidChargePort, extendedObj.normalChargePort, extendedObj.slowChargePort, extendedObj.auxBatteryVoltage, extendedObj.dcBatteryVoltage, extendedObj.dcBatteryCurrent, - extendedObj.dcBatteryPower, extendedObj.cumulativeEnergyCharged, extendedObj.cumulativeEnergyDischarged, extendedObj.batteryMinTemperature, extendedObj.batteryMaxTemperature, extendedObj.batteryInletTemperature, extendedObj.externalTemperature, extendedObj.odo, now, akey - ], (err, dbRes) => callback(err, (!err && dbRes))); - } else callback(err); + // History contains only this submission, never retained current values. + const historyColumns = columns.concat(['timestamp', 'akey']); + db.query('INSERT INTO statistics (' + historyColumns.join(', ') + ') VALUES (' + + historyColumns.map(() => '?').join(', ') + ')', values.concat([now, akey]), + (err, dbRes) => callback(err, (!err && dbRes))); }); - } else callback(err); }); }; @@ -100,10 +95,9 @@ const postExtended = (akey, extendedObj, callback) => { * @param {Function} callback callback function */ const getExtended = (akey, callback) => { - db.query('SELECT soh, charging, rapid_charge_port, normal_charge_port, slow_charge_port, aux_battery_voltage, dc_battery_voltage, dc_battery_current, dc_battery_power, \ - cumulative_energy_charged, cumulative_energy_discharged, battery_min_temperature, battery_max_temperature, battery_inlet_temperature, external_temperature, odo, last_extended FROM sync WHERE akey=?', [ - akey - ], (err, queryRes) => callback(err, ((!err && queryRes) ? queryRes[0] : null))); + const columns = Object.keys(telemetry.fields).map(field => telemetry.fields[field]).concat(['last_extended']); + db.query('SELECT ' + columns.join(', ') + ' FROM sync WHERE akey=?', [akey], + (err, queryRes) => callback(err, ((!err && queryRes) ? telemetry.serialize(queryRes[0]) : null))); }; /** @@ -241,8 +235,8 @@ module.exports = { }); }, postExtended: (req, res) => { - // check required params - if (!req.body.akey || !req.body.token) { + // Validate the entire submission before any telemetry writes. + if (!req.body.akey || !req.body.token || !telemetry.valid(req.body)) { return res.status(400).json({ error: srv_errors.INVALID_PARAMETERS }); diff --git a/modules/sync/telemetry.js b/modules/sync/telemetry.js new file mode 100644 index 0000000..9d66df2 --- /dev/null +++ b/modules/sync/telemetry.js @@ -0,0 +1,58 @@ +// Only these known columns may participate in dynamic telemetry queries. +const fields = { + soh: 'soh', + charging: 'charging', + rapidChargePort: 'rapid_charge_port', + normalChargePort: 'normal_charge_port', + slowChargePort: 'slow_charge_port', + auxBatteryVoltage: 'aux_battery_voltage', + dcBatteryVoltage: 'dc_battery_voltage', + dcBatteryCurrent: 'dc_battery_current', + dcBatteryPower: 'dc_battery_power', + cumulativeEnergyCharged: 'cumulative_energy_charged', + cumulativeEnergyDischarged: 'cumulative_energy_discharged', + batteryMinTemperature: 'battery_min_temperature', + batteryMaxTemperature: 'battery_max_temperature', + batteryInletTemperature: 'battery_inlet_temperature', + externalTemperature: 'external_temperature', + odo: 'odo', + batteryCellVoltageMin: 'battery_cell_voltage_min', + batteryCellVoltageMax: 'battery_cell_voltage_max', + batteryCellVoltageDelta: 'battery_cell_voltage_delta', + batteryFanSpeed: 'battery_fan_speed' +}; + +const voltages = ['batteryCellVoltageMin', 'batteryCellVoltageMax', 'batteryCellVoltageDelta']; +const validatedFields = voltages.concat(['batteryFanSpeed', 'odo']); +const present = (body, field) => Object.prototype.hasOwnProperty.call(body, field); + +const valid = body => validatedFields.every(field => { + if (!present(body, field) || body[field] === null) return true; + const value = body[field]; + if (typeof value !== 'number' || !Number.isFinite(value) || value < 0) return false; + if (field === 'odo') return Number.isInteger(value) && value <= 2147483647; + if (field === 'batteryFanSpeed') return Number.isFinite(Math.fround(value)); + return value <= 999999.999999; +}); + +const supplied = body => Object.keys(fields).filter(field => present(body, field)); +const storedValue = (body, field) => { + const value = body[field]; + if (value === null) return null; + if (voltages.indexOf(field) !== -1) return Number(value.toFixed(6)); + // Normalize before SQL: a double just above FLOAT_MAX may still round to a + // finite float, but MySQL rejects the unrounded double as out of range. + if (field === 'batteryFanSpeed') return Math.fround(value); + return value; +}; + +const serialize = row => { + if (!row) return row; + validatedFields.forEach(field => { + const column = fields[field]; + if (row[column] != null) row[column] = Number(row[column]); + }); + return row; +}; + +module.exports = { fields, valid, supplied, storedValue, serialize }; diff --git a/modules/translation/lng/de.json b/modules/translation/lng/de.json index 73d9f10..4964762 100644 --- a/modules/translation/lng/de.json +++ b/modules/translation/lng/de.json @@ -1,4 +1,10 @@ { + "EXTENDED_TELEMETRY_HISTORY": "Erweiterte Telemetrie", + "BATTERY_CELL_VOLTAGE_MIN_V": "Minimale Zellspannung (V)", + "BATTERY_CELL_VOLTAGE_MAX_V": "Maximale Zellspannung (V)", + "BATTERY_CELL_VOLTAGE_DELTA_V": "Zellspannungsdifferenz (V)", + "BATTERY_FAN_LEVEL_RAW": "Batterielüfterstufe (Rohwert)", + "ODOMETER_KM": "Kilometerstand (km)", "SOC_HISTORY": "Ladezustandshistorie", "SOC_DISPLAY": "Ladezustand (Display)", "SOC_BMS": "Ladezustand (BMS)", diff --git a/modules/translation/lng/en.json b/modules/translation/lng/en.json index bb9ec32..02b7ea9 100644 --- a/modules/translation/lng/en.json +++ b/modules/translation/lng/en.json @@ -1,4 +1,10 @@ { + "EXTENDED_TELEMETRY_HISTORY": "Extended telemetry history", + "BATTERY_CELL_VOLTAGE_MIN_V": "Cell voltage minimum (V)", + "BATTERY_CELL_VOLTAGE_MAX_V": "Cell voltage maximum (V)", + "BATTERY_CELL_VOLTAGE_DELTA_V": "Cell voltage delta (V)", + "BATTERY_FAN_LEVEL_RAW": "Battery fan level (raw)", + "ODOMETER_KM": "Odometer (km)", "SOC_HISTORY": "State of charge history", "SOC_DISPLAY": "State of charge (Display)", "SOC_BMS": "State of charge (BMS)", diff --git a/tests/report/index.js b/tests/report/index.js new file mode 100644 index 0000000..b8c0ff9 --- /dev/null +++ b/tests/report/index.js @@ -0,0 +1,89 @@ +const assert = require('assert'); +const fs = require('fs'); +const util = require('util'); +const JSZip = require('jszip'); +const { loadModule, createResponse } = require('../helpers'); +const translation = require('../../modules/translation'); + +describe('extended telemetry XLSX exports', () => { + let loader; + beforeEach(() => { + loader = loadModule('./modules/report', { './modules/db': {}, './modules/token': {} }); + }); + afterEach(() => loader.restore()); + + it('writes numeric measurements, real zeros, unit headers, and blank unknown cells', async () => { + const file = await loader.module.logExportReport([ + { timestamp: 1600000000, battery_cell_voltage_min: '3.780000', battery_cell_voltage_max: 3.8, + battery_cell_voltage_delta: '0.020000', battery_fan_speed: 3, odo: 123456 }, + { timestamp: 1600000001, battery_cell_voltage_min: 0, battery_cell_voltage_max: 0, + battery_cell_voltage_delta: 0, battery_fan_speed: 0, odo: 0 }, + { timestamp: 1600000002, battery_cell_voltage_delta: 0.000001 }, + { timestamp: 1600000003, battery_cell_voltage_min: null, odo: null }, + { timestamp: 1600000004, soc_display: 80 } + ]); + try { + const zip = await JSZip.loadAsync(await util.promisify(fs.readFile)(file)); + const workbook = await zip.file('xl/workbook.xml').async('string'); + assert.ok(workbook.includes('Extended telemetry history')); + const strings = await zip.file('xl/sharedStrings.xml').async('string'); + ['Cell voltage minimum (V)', 'Cell voltage maximum (V)', 'Cell voltage delta (V)', + 'Battery fan level (raw)', 'Odometer (km)'].forEach(header => assert.ok(strings.includes(header))); + const sheet = await zip.file('xl/worksheets/sheet3.xml').async('string'); + const numeric = (cell, value) => assert.ok(new RegExp(']*r="' + cell + '"[^>]*t="n"[^>]*>' + value + '').test(sheet), cell + ' should contain numeric ' + value); + numeric('B2', '3.78'); numeric('C2', '3.8'); numeric('D2', '0.02'); + numeric('E2', '3'); numeric('F2', '123456'); + ['B3', 'C3', 'D3', 'E3', 'F3'].forEach(cell => numeric(cell, '0')); + numeric('D4', '0.000001'); + ['B4', 'C4', 'E4', 'F4', 'A5'].forEach(cell => assert.ok(!sheet.includes('r="' + cell + '"'))); + } finally { + await util.promisify(fs.unlink)(file); + } + }); + + it('provides English and German worksheet and measurement labels', () => { + ['en', 'de'].forEach(lng => { + ['EXTENDED_TELEMETRY_HISTORY', 'BATTERY_CELL_VOLTAGE_MIN_V', 'BATTERY_CELL_VOLTAGE_MAX_V', + 'BATTERY_CELL_VOLTAGE_DELTA_V', 'BATTERY_FAN_LEVEL_RAW', 'ODOMETER_KM'].forEach(key => { + assert.notStrictEqual(translation.translate(key, lng), key); + }); + assert.ok(translation.translate('BATTERY_CELL_VOLTAGE_DELTA_V', lng).includes('(V)')); + assert.ok(translation.translate('ODOMETER_KM', lng).includes('(km)')); + }); + }); + + it('includes localized telemetry in the general report path', () => { + loader.restore(); + const sheets = []; + function Workbook() { + this.addWorksheet = name => { + const sheet = { name, cells: {} }; + sheets.push(sheet); + return { cell: (row, column) => { + const set = type => value => { sheet.cells[row + ',' + column] = { type, value }; }; + return { date: set('date'), number: set('number'), string: set('string') }; + } }; + }; + this.write = () => {}; + } + loader = loadModule('./modules/report', { + excel4node: { Workbook }, + './modules/token': { validateToken: (akey, token, cb) => cb(null, true) }, + './modules/db': { query: (sql, params, cb) => { + assert.deepStrictEqual(params, ['ABC123']); + cb(null, sql.includes('SELECT lng') ? [{ lng: 'de' }] : [ + { timestamp: 1600000000, battery_cell_voltage_delta: '0.020000', battery_fan_speed: 0 } + ]); + } } + }); + const res = createResponse(); + loader.module.downloadReport({ query: { akey: 'ABC123', token: 'secret' } }, res); + assert.strictEqual(res.statusCode, 200); + assert.deepStrictEqual(res.body, { report: 'Excel.xlsx' }); + assert.strictEqual(sheets[2].name, 'Erweiterte Telemetrie'); + assert.strictEqual(sheets[2].cells['1,4'].value, 'Zellspannungsdifferenz (V)'); + assert.deepStrictEqual(sheets[2].cells['2,4'], { type: 'number', value: 0.02 }); + assert.deepStrictEqual(sheets[2].cells['2,5'], { type: 'number', value: 0 }); + assert.strictEqual(sheets[2].cells['2,2'], undefined); + }); +}); diff --git a/tests/sync/telemetry.js b/tests/sync/telemetry.js new file mode 100644 index 0000000..f605e1c --- /dev/null +++ b/tests/sync/telemetry.js @@ -0,0 +1,132 @@ +const assert = require('assert'); +const { loadModule, createResponse } = require('../helpers'); +const errors = require('../../srv_errors.json'); + +describe('extended telemetry validation and sparse writes', () => { + let loader, queries, events, recent, selected; + const measurements = { + batteryCellVoltageMin: 3.78, batteryCellVoltageMax: 3.8, + batteryCellVoltageDelta: 0.02, batteryFanSpeed: 3, odo: 123456 + }; + const names = { + batteryCellVoltageMin: 'battery_cell_voltage_min', batteryCellVoltageMax: 'battery_cell_voltage_max', + batteryCellVoltageDelta: 'battery_cell_voltage_delta', batteryFanSpeed: 'battery_fan_speed', odo: 'odo' + }; + const invoke = data => { + const res = createResponse(); + loader.module.postExtended({ body: Object.assign({ akey: 'ABC123', token: 'secret' }, data) }, res); + return res; + }; + const assignments = query => { + const columns = query.sql.match(/SET (.+) WHERE/)[1].split(', ').map(s => s.split('=')[0]); + return columns.reduce((values, column, i) => Object.assign(values, { [column]: query.params[i] }), {}); + }; + + beforeEach(() => { + queries = []; events = []; recent = false; selected = null; + loader = loadModule('./modules/sync', { + './modules/db': { query: (sql, params, cb) => { + queries.push({ sql, params }); + cb(null, sql.startsWith('SELECT') ? [selected || { last_extended: recent ? Math.floor(Date.now() / 1000) : 0 }] : { affectedRows: 1 }); + } }, + './modules/token': { validateToken: (akey, token, cb) => cb(null, true) }, + './modules/webhook': { emit: (...args) => events.push(args) }, + './modules/integrations/abrp': {} + }); + }); + afterEach(() => loader.restore()); + + it('maps every supplied measurement into current state and sparse history', () => { + assert.strictEqual(invoke(measurements).statusCode, 200); + const update = assignments(queries[1]); + Object.keys(names).forEach(field => { + assert.strictEqual(update[names[field]], measurements[field]); + assert.ok(queries[2].sql.includes(names[field])); + }); + assert.ok(!queries[1].sql.includes('soh=')); + assert.ok(!queries[2].sql.includes('soh')); + assert.strictEqual(queries[1].params.slice(-1)[0], 'ABC123'); + assert.strictEqual(queries[2].params.slice(-1)[0], 'ABC123'); + assert.strictEqual(events.length, 1); + }); + + [0, null].forEach(value => it('preserves all five ' + value + ' values', () => { + const data = Object.keys(names).reduce((body, field) => Object.assign(body, { [field]: value }), {}); + assert.strictEqual(invoke(data).statusCode, 200); + const update = assignments(queries[1]); + Object.keys(names).forEach(field => assert.strictEqual(update[names[field]], value)); + assert.strictEqual(queries[2].params.filter(v => v === value).length, 5); + })); + + it('preserves omitted legacy fields, ignores unknown fields, and rounds voltages', () => { + assert.strictEqual(invoke({ batteryCellVoltageDelta: 0.0000126, unexpected: 42 }).statusCode, 200); + assert.deepStrictEqual(Object.keys(assignments(queries[1])), ['battery_cell_voltage_delta', 'last_extended']); + assert.strictEqual(queries[1].params[0], 0.000013); + assert.strictEqual(queries[2].params[0], 0.000013); + assert.ok(!queries[1].sql.includes('unexpected')); + }); + + it('keeps old payloads valid and only updates submitted legacy fields', () => { + assert.strictEqual(invoke({ soh: 0, dcBatteryCurrent: -12 }).statusCode, 200); + assert.strictEqual(assignments(queries[1]).soh, 0); + assert.strictEqual(assignments(queries[1]).dc_battery_current, -12); + assert.ok(!queries[1].sql.includes('odo=')); + assert.ok(!queries[2].sql.includes('battery_cell')); + }); + + it('keeps credential-only payloads valid without clearing any measurements', () => { + assert.strictEqual(invoke({}).statusCode, 200); + assert.deepStrictEqual(Object.keys(assignments(queries[1])), ['last_extended']); + assert.ok(queries[2].sql.includes('(timestamp, akey)')); + }); + + it('updates current measurements but retains the five-second history throttle', () => { + recent = true; + assert.strictEqual(invoke(measurements).statusCode, 200); + assert.strictEqual(queries.length, 2); + assert.strictEqual(events.length, 1); + }); + + Object.keys(names).forEach(field => { + ['3.8', true, false, [], {}, undefined, NaN, Infinity, -Infinity, -0.1].forEach(value => { + it('rejects ' + field + '=' + String(value) + ' before writes or events', () => { + const res = invoke(Object.assign({}, measurements, { [field]: value })); + assert.strictEqual(res.statusCode, 400); + assert.deepStrictEqual(res.body.error, errors.INVALID_PARAMETERS); + assert.deepStrictEqual(queries, []); + assert.deepStrictEqual(events, []); + }); + }); + }); + + [ + { odo: 0.5 }, { odo: 2147483648 }, { batteryFanSpeed: 3.5e38 }, + { batteryCellVoltageMin: 1000000 }, { batteryCellVoltageMax: 1000000 }, + { batteryCellVoltageDelta: 1000000 } + ].forEach(data => it('rejects out-of-storage-range input ' + JSON.stringify(data), () => { + assert.strictEqual(invoke(data).statusCode, 400); + assert.deepStrictEqual(queries, []); + })); + + it('accepts storage boundaries', () => { + assert.strictEqual(invoke({ odo: 2147483647, batteryCellVoltageMax: 999999.999999, batteryFanSpeed: 3.4028234663852886e38 }).statusCode, 200); + }); + + it('normalizes fan values to their finite FLOAT representation before SQL', () => { + assert.strictEqual(invoke({ batteryFanSpeed: 3.4028235e38 }).statusCode, 200); + assert.strictEqual(assignments(queries[1]).battery_fan_speed, Math.fround(3.4028235e38)); + }); + + it('serializes selected decimals as numbers and preserves nulls and zeros', () => { + selected = { battery_cell_voltage_min: '3.780000', battery_cell_voltage_max: null, + battery_cell_voltage_delta: '0.000000', battery_fan_speed: 0, odo: 123456, last_extended: 123 }; + const res = createResponse(); + loader.module.getExtended({ query: { akey: 'ABC123', token: 'secret' } }, res); + assert.strictEqual(res.statusCode, 200); + assert.strictEqual(res.body.battery_cell_voltage_min, 3.78); + assert.strictEqual(res.body.battery_cell_voltage_delta, 0); + assert.strictEqual(res.body.battery_cell_voltage_max, null); + assert.ok(queries[0].sql.includes('battery_fan_speed')); + assert.deepStrictEqual(queries[0].params, ['ABC123']); + }); +}); diff --git a/tests/telemetry/index.js b/tests/telemetry/index.js new file mode 100644 index 0000000..5878894 --- /dev/null +++ b/tests/telemetry/index.js @@ -0,0 +1,222 @@ +const assert = require('assert'); +const fs = require('fs'); +const path = require('path'); +const util = require('util'); +const mysql = require('mysql'); +const express = require('express'); +const chai = require('chai'); +const chaiHttp = require('chai-http'); +const { loadModule } = require('../helpers'); +const config = require('../../srv_config.json'); + +chai.use(chaiHttp); + +describe('extended telemetry database and migration', function () { + this.timeout(20000); + const database = 'evnotify_telemetry_test_' + process.pid + '_' + Date.now(); + const freshDatabase = database + '_fresh'; + const root = path.resolve(__dirname, '../..'); + const columns = ['battery_cell_voltage_min', 'battery_cell_voltage_max', 'battery_cell_voltage_delta', 'battery_fan_speed']; + const archiveTables = ['statistics_9_2020', 'statistics_log_102']; + let connection, query, app, loaders = []; + let exportedRows; + let createdDatabase = false, createdFreshDatabase = false; + const credentials = { akey: 'AAAAAA', token: 'token-a' }; + const migration = async () => { + const parts = fs.readFileSync(path.join(root, 'modules/db/migrations/extended_vehicle_telemetry.sql'), 'utf8').split(/^DELIMITER .+$/m); + await query(parts[0]); + await query(parts[1].replace(/\/\/\s*$/, '')); + await query(parts[2]); + }; + const post = body => chai.request(app).post('/extended').send(Object.assign({}, credentials, body)); + const get = auth => chai.request(app).get('/extended').query(auth || credentials); + const allowHistory = () => query('UPDATE sync SET last_extended=0 WHERE akey=?', ['AAAAAA']); + + before(async () => { + connection = mysql.createConnection({ host: config.DB_HOST, user: config.DB_USER, + password: config.DB_PW, multipleStatements: true }); + query = util.promisify(connection.query).bind(connection); + await query('CREATE DATABASE ??', [database]); + createdDatabase = true; + await query('USE ??', [database]); + await query(fs.readFileSync(path.join(__dirname, 'legacy-schema.sql'), 'utf8')); + await query("INSERT INTO accounts (akey, pw_hash, token) VALUES ('AAAAAA', 'unused', 'token-a'), ('BBBBBB', 'unused', 'token-b')"); + await query("INSERT INTO sync (user, akey, soh, odo) VALUES ('AAAAAA', 'AAAAAA', 98, 56789), ('BBBBBB', 'BBBBBB', 90, 0)"); + await query("INSERT INTO statistics (akey, timestamp, odo) VALUES ('AAAAAA', 1600000000, 12345)"); + for (const table of archiveTables) { + await query('CREATE TABLE ?? LIKE statistics', [table]); + await query('INSERT INTO ?? SELECT * FROM statistics', [table]); + } + // A partially applied migration must also be recoverable. + await query('ALTER TABLE sync ADD COLUMN battery_cell_voltage_min DECIMAL(12,6) DEFAULT NULL'); + await migration(); + const db = { query: connection.query.bind(connection) }; + const token = loadModule('./modules/token', { './modules/db': db, './modules/account': {} }); + const sync = loadModule('./modules/sync', { './modules/db': db, './modules/token': token.module, + './modules/webhook': { emit: () => {} }, './modules/integrations/abrp': {} }); + const logs = loadModule('./modules/logs', { './modules/db': db, './modules/token': token.module, + './modules/notification/mail': { validateMail: () => true, simpleSend: () => {} }, + './modules/report': { logExportReport: rows => { exportedRows = rows; return Promise.resolve('unused.xlsx'); } } }); + loaders = [logs, sync, token]; + app = express(); + app.use(express.json()); + app.post('/extended', sync.module.postExtended); + app.get('/extended', sync.module.getExtended); + app.get('/logdetail', logs.module.getLog); + app.get('/logexport', logs.module.exportLog); + }); + + after(async () => { + loaders.forEach(loader => loader.restore()); + if (connection) { + try { + if (createdFreshDatabase) await query('DROP DATABASE ??', [freshDatabase]); + if (createdDatabase) await query('DROP DATABASE ??', [database]); + } finally { + await util.promisify(connection.end).bind(connection)(); + } + } + }); + + it('migrates both archive formats and preserves old data and unknown measurements', async () => { + for (const table of ['sync', 'statistics'].concat(archiveTables)) { + const rows = await query('SELECT * FROM ?? ORDER BY akey', [table]); + columns.forEach(column => assert.strictEqual(rows[0][column], null)); + assert.strictEqual(rows[0].odo, table === 'sync' ? 56789 : 12345); + const info = await query('SELECT COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? AND COLUMN_NAME IN (?)', [database, table, columns]); + assert.strictEqual(info.length, 4); + info.forEach(col => { + assert.strictEqual(col.IS_NULLABLE, 'YES'); + assert.strictEqual(col.COLUMN_TYPE, col.COLUMN_NAME === 'battery_fan_speed' ? 'float' : 'decimal(12,6)'); + }); + } + assert.strictEqual((await query("SELECT odo FROM sync WHERE akey='BBBBBB'"))[0].odo, 0); + const defaults = await query("SELECT COLUMN_DEFAULT FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME='sync' AND COLUMN_NAME='odo'", [database]); + assert.strictEqual(defaults[0].COLUMN_DEFAULT, null); + }); + + it('can rerun without changing data', async () => { + await query("UPDATE sync SET battery_cell_voltage_min=3.712345 WHERE akey='AAAAAA'"); + await migration(); + assert.strictEqual((await get()).body.battery_cell_voltage_min, 3.712345); + }); + + it('round-trips numeric values, six-place precision, and small deltas through HTTP and MySQL', async () => { + const response = await post({ batteryCellVoltageMin: 3.7812346, batteryCellVoltageMax: 3.8, + batteryCellVoltageDelta: 0.02, batteryFanSpeed: 3, odo: 123456 }); + assert.strictEqual(response.status, 200); + const body = (await get()).body; + assert.strictEqual(body.battery_cell_voltage_min, 3.781235); + assert.strictEqual(body.battery_cell_voltage_max, 3.8); + assert.strictEqual(body.battery_cell_voltage_delta, 0.02); + assert.strictEqual(body.battery_fan_speed, 3); + assert.strictEqual(body.odo, 123456); + await allowHistory(); + assert.strictEqual((await post({ batteryCellVoltageDelta: 0.000001 })).status, 200); + assert.strictEqual((await get()).body.battery_cell_voltage_delta, 0.000001); + const history = await query("SELECT * FROM statistics WHERE akey='AAAAAA' ORDER BY id DESC LIMIT 1"); + assert.strictEqual(history[0].battery_cell_voltage_delta, 0.000001); + assert.strictEqual(history[0].battery_cell_voltage_min, null); + assert.strictEqual(history[0].odo, null); + }); + + it('supports partial vehicle payloads, zero readings, explicit nulls, and legacy clients', async () => { + await allowHistory(); + assert.strictEqual((await post({ batteryCellVoltageMin: 0, batteryCellVoltageMax: 0, + batteryCellVoltageDelta: 0, batteryFanSpeed: 0, odo: 0 })).status, 200); + let body = (await get()).body; + columns.concat(['odo']).forEach(column => assert.strictEqual(body[column], 0)); + assert.strictEqual((await post({ soh: 97, dcBatteryCurrent: -10 })).status, 200); + body = (await get()).body; + columns.concat(['odo']).forEach(column => assert.strictEqual(body[column], 0)); + assert.strictEqual(body.soh, 97); + assert.strictEqual((await post({ batteryCellVoltageMin: null, batteryCellVoltageMax: null, + batteryCellVoltageDelta: null, batteryFanSpeed: null, odo: null })).status, 200); + body = (await get()).body; + columns.concat(['odo']).forEach(column => assert.strictEqual(body[column], null)); + }); + + it('stores range boundaries without MySQL overflow', async () => { + assert.strictEqual((await post({ batteryCellVoltageMax: 999999.999999, + batteryFanSpeed: 3.4028235e38, odo: 2147483647 })).status, 200); + const body = (await get()).body; + assert.strictEqual(body.battery_cell_voltage_max, 999999.999999); + assert.strictEqual(body.odo, 2147483647); + assert.ok(Number.isFinite(body.battery_fan_speed)); + }); + + it('rejects mixed invalid submissions before changing any telemetry', async () => { + const before = (await get()).body; + const count = (await query('SELECT COUNT(*) AS n FROM statistics'))[0].n; + for (const invalid of [{ odo: 1.5 }, { batteryFanSpeed: 3.5e38 }, { batteryCellVoltageMin: '3.7' }, + { batteryCellVoltageMax: 1000000 }, { batteryCellVoltageDelta: -1 }]) { + assert.strictEqual((await post(Object.assign({ soh: 11 }, invalid))).status, 400); + assert.deepStrictEqual((await get()).body, before); + } + assert.strictEqual((await query('SELECT COUNT(*) AS n FROM statistics'))[0].n, count); + }); + + it('rejects another account token and keeps current state and history scoped', async () => { + assert.strictEqual((await post({ token: 'token-b', odo: 99 })).status, 401); + assert.strictEqual((await get({ akey: 'AAAAAA', token: 'token-b' })).status, 401); + assert.strictEqual((await post({ akey: 'BBBBBB', token: 'token-b', batteryCellVoltageDelta: 0.05 })).status, 200); + const second = (await get({ akey: 'BBBBBB', token: 'token-b' })).body; + assert.strictEqual(second.battery_cell_voltage_delta, 0.05); + assert.strictEqual(second.odo, 0); + assert.strictEqual((await query("SELECT battery_cell_voltage_delta FROM statistics WHERE akey='BBBBBB'"))[0].battery_cell_voltage_delta, 0.05); + }); + + it('retains the five-second history throttle while updating current measurements', async () => { + await query("UPDATE sync SET last_extended=? WHERE akey='AAAAAA'", [Math.floor(Date.now() / 1000)]); + const count = (await query('SELECT COUNT(*) AS n FROM statistics'))[0].n; + assert.strictEqual((await post({ batteryCellVoltageDelta: 0.02 })).status, 200); + assert.strictEqual((await get()).body.battery_cell_voltage_delta, 0.02); + assert.strictEqual((await query('SELECT COUNT(*) AS n FROM statistics'))[0].n, count); + }); + + it('copies current statistics into existing and future archives and exposes them in logdetail', async () => { + await query('CREATE TABLE statistics_log_104 LIKE statistics'); + await allowHistory(); + await post({ batteryCellVoltageMin: 3.78, batteryCellVoltageMax: 3.8, batteryCellVoltageDelta: 0.02, batteryFanSpeed: 0, odo: 123456 }); + const source = (await query("SELECT * FROM statistics WHERE akey='AAAAAA' ORDER BY id DESC LIMIT 1"))[0]; + const archives = [ + { table: 'statistics', id: 101, delta: 0.02 }, { table: 'statistics_9_2020', id: 103, delta: 0.03 }, + { table: 'statistics_log_102', id: 102, delta: 0.04 }, { table: 'statistics_log_104', id: 104, delta: 0.05 } + ]; + for (const archive of archives) { + const table = archive.table; + const row = Object.assign({}, source, { battery_cell_voltage_delta: archive.delta }); + if (table !== 'statistics') { + // Match the dynamic named-column copy used by both archive jobs. + delete row.id; + row.timestamp = 1600000000; + await query('INSERT INTO ?? (??) VALUES (?)', [table, Object.keys(row), Object.keys(row).map(key => row[key])]); + } + const id = archive.id; + await query('INSERT INTO logs (id, akey, start, end, archived, standalone) VALUES (?, ?, ?, ?, ?, ?)', + [id, 'AAAAAA', row.timestamp, row.timestamp, table === 'statistics' ? 0 : 1, table.startsWith('statistics_log_') ? 1 : 0]); + const res = await chai.request(app).get('/logdetail').query(Object.assign({ id: id }, credentials)); + assert.strictEqual(res.status, 200); + assert.ok(res.body.stats.some(stat => stat.battery_cell_voltage_delta === archive.delta && stat.battery_fan_speed === 0)); + assert.strictEqual((await chai.request(app).get('/logdetail').query({ id: id, akey: 'BBBBBB', token: 'token-b' })).status, 422); + exportedRows = null; + const exportResponse = await chai.request(app).get('/logexport').query(Object.assign({ id: id, email: 'test@example.com' }, credentials)); + assert.strictEqual(exportResponse.status, 200); + assert.ok(exportedRows.some(stat => stat.battery_cell_voltage_delta === archive.delta && stat.battery_fan_speed === 0)); + } + }); + + it('creates fresh schemas with the same nullable telemetry and odometer default', async () => { + await query('CREATE DATABASE ??', [freshDatabase]); + createdFreshDatabase = true; + await query('USE ??', [freshDatabase]); + try { + await query(fs.readFileSync(path.join(root, 'modules/db/db_template.sql'), 'utf8')); + const info = await query("SELECT TABLE_NAME, COLUMN_NAME, COLUMN_TYPE, COLUMN_DEFAULT FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME IN ('sync', 'statistics') AND COLUMN_NAME IN (?)", [freshDatabase, columns.concat(['odo'])]); + assert.strictEqual(info.length, 10); + info.forEach(column => assert.strictEqual(column.COLUMN_DEFAULT, null)); + } finally { + await query('USE ??', [database]); + } + }); +}); diff --git a/tests/telemetry/legacy-schema.sql b/tests/telemetry/legacy-schema.sql new file mode 100644 index 0000000..a7395e0 --- /dev/null +++ b/tests/telemetry/legacy-schema.sql @@ -0,0 +1,91 @@ +-- Previous schema fixture for migration coverage. +-- accounts table structure +CREATE TABLE IF NOT EXISTS `accounts` ( + `akey` VARCHAR(6) NOT NULL PRIMARY KEY, + `pw_hash` VARCHAR(255) NOT NULL, + `lastactivity` INT(10) DEFAULT 0, + `token` VARCHAR(20) NOT NULL UNIQUE +); + +-- sync table structure +CREATE TABLE IF NOT EXISTS `sync` ( + `user` VARCHAR(6) NOT NULL, + `akey` VARCHAR(6) NOT NULL, + `soc_display` FLOAT DEFAULT 0, + `soc_bms` FLOAT DEFAULT 0, + `soh` FLOAT DEFAULT 0, + `charging` TINYINT(1) DEFAULT 0, + `rapid_charge_port` TINYINT(1) DEFAULT 0, + `normal_charge_port` TINYINT(1) DEFAULT 0, + `slow_charge_port` TINYINT(1) DEFAULT 0, + `aux_battery_voltage` FLOAT DEFAULT 0, + `dc_battery_voltage` FLOAT DEFAULT 0, + `dc_battery_current` FLOAT DEFAULT 0, + `dc_battery_power` FLOAT DEFAULT 0, + `cumulative_energy_charged` FLOAT DEFAULT 0, + `cumulative_energy_discharged` FLOAT DEFAULT 0, + `battery_min_temperature` FLOAT DEFAULT 0, + `battery_max_temperature` FLOAT DEFAULT 0, + `battery_inlet_temperature` FLOAT DEFAULT 0, + `external_temperature` FLOAT DEFAULT 0, + `odo` INT(6) DEFAULT 0, + `latitude` DECIMAL(10, 8) DEFAULT 0, + `longitude` DECIMAL(11, 8) DEFAULT 0, + `gps_speed` FLOAT DEFAULT 0, + `accuracy` INT(100) DEFAULT 0, + `location_timestamp` INT(20) DEFAULT 0, + `last_soc` INT(13) DEFAULT 0, + `last_extended` INT(13) DEFAULT 0, + `last_notification` INT(13) DEFAULT 0, + `last_location` INT(13) DEFAULT 0, + `last_qr` INT(13) DEFAULT 0, + PRIMARY KEY (`user`), + FOREIGN KEY (`akey`) REFERENCES `accounts`(`akey`) +); + +-- statistics table structure +CREATE TABLE IF NOT EXISTS `statistics` ( + `id` INT NOT NULL AUTO_INCREMENT, + `akey` VARCHAR(6) NOT NULL, + `soc_display` FLOAT DEFAULT NULL, + `soc_bms` FLOAT DEFAULT NULL, + `soh` FLOAT DEFAULT NULL, + `charging` TINYINT(1) DEFAULT NULL, + `rapid_charge_port` TINYINT(1) DEFAULT NULL, + `normal_charge_port` TINYINT(1) DEFAULT NULL, + `slow_charge_port` TINYINT(1) DEFAULT NULL, + `aux_battery_voltage` FLOAT DEFAULT NULL, + `dc_battery_voltage` FLOAT DEFAULT NULL, + `dc_battery_current` FLOAT DEFAULT NULL, + `dc_battery_power` FLOAT DEFAULT NULL, + `cumulative_energy_charged` FLOAT DEFAULT NULL, + `cumulative_energy_discharged` FLOAT DEFAULT NULL, + `battery_min_temperature` FLOAT DEFAULT NULL, + `battery_max_temperature` FLOAT DEFAULT NULL, + `battery_inlet_temperature` FLOAT DEFAULT NULL, + `external_temperature` FLOAT DEFAULT NULL, + `odo` INT(6) DEFAULT NULL, + `latitude` DECIMAL(10, 8) DEFAULT NULL, + `longitude` DECIMAL(11, 8) DEFAULT NULL, + `gps_speed` FLOAT DEFAULT NULL, + `accuracy` INT(100) DEFAULT NULL, + `location_timestamp` INT(20) DEFAULT NULL, + `timestamp` INT(13) DEFAULT NULL, + PRIMARY KEY (`id`), + FOREIGN KEY (`akey`) REFERENCES `accounts`(`akey`) +); + +-- logs table structure +CREATE TABLE IF NOT EXISTS `logs` ( + `id` INT NOT NULL AUTO_INCREMENT, + `akey` VARCHAR(6) NOT NULL, + `start` INT(10) DEFAULT 0, + `end` INT(10) DEFAULT 0, + `charge` TINYINT(1) DEFAULT 0, + `autogenerated` TINYINT(1) DEFAULT 1, + `archived` TINYINT(1) DEFAULT 0, + `standalone` TINYINT(1) DEFAULT 0, + `title` VARCHAR(255) DEFAULT NULL, + PRIMARY KEY (`id`), + FOREIGN KEY (`akey`) REFERENCES `accounts`(`akey`) +); From a6876111b009dfa7a072867bb325ae7a5a821c70 Mon Sep 17 00:00:00 2001 From: GPlay97 Date: Sun, 25 Oct 2026 22:29:28 +0100 Subject: [PATCH 2/4] Align JSHint syntax checks with Node 8 --- .jshintrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.jshintrc b/.jshintrc index ede9d2d..d864b4a 100644 --- a/.jshintrc +++ b/.jshintrc @@ -1,7 +1,7 @@ { "undef": true, "unused": true, - "esversion": 6, + "esversion": 8, "node": true, "mocha": true, "multistr": true, From 8654e761c7466471f4967d0a75b59776062b7d69 Mon Sep 17 00:00:00 2001 From: GPlay97 Date: Sun, 25 Oct 2026 22:41:41 +0100 Subject: [PATCH 3/4] Replace legacy Hound linting with pinned CI checks --- .github/scripts/lint-changes.js | 55 ++++++++++++++ .github/scripts/lint-changes.test.js | 108 +++++++++++++++++++++++++++ .github/workflows/ci.yml | 21 ++++++ .hound.yml | 2 +- README.md | 15 ++++ modules/logs/index.js | 1 + modules/report/index.js | 4 +- 7 files changed, 204 insertions(+), 2 deletions(-) create mode 100644 .github/scripts/lint-changes.js create mode 100644 .github/scripts/lint-changes.test.js diff --git a/.github/scripts/lint-changes.js b/.github/scripts/lint-changes.js new file mode 100644 index 0000000..26c5809 --- /dev/null +++ b/.github/scripts/lint-changes.js @@ -0,0 +1,55 @@ +const fs = require('fs'); +const path = require('path'); +const { execFileSync } = require('child_process'); + +const git = args => execFileSync('git', args, { encoding: 'utf8', maxBuffer: 10 * 1024 * 1024 }); + +const changedLines = diff => { + const ranges = []; + const hunk = /^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/gm; + let match; + while ((match = hunk.exec(diff)) !== null) { + ranges.push({ start: Number(match[1]), count: match[2] === undefined ? 1 : Number(match[2]) }); + } + return line => ranges.some(range => line >= range.start && line < range.start + range.count); +}; + +const lintChanges = (jshint, base, head) => { + const initialPush = !base || /^0+$/.test(base); + let listing = ['ls-tree', '-r', '--name-only', '-z', head]; + if (!initialPush) listing = ['diff', '--name-only', '--no-renames', '--diff-filter=ACMR', '-z', base, head, '--', '*.js']; + const files = git(listing).split('\0').filter(file => file.endsWith('.js')); + const config = JSON.parse(fs.readFileSync('.jshintrc', 'utf8')); + const issues = []; + let ignored = 0; + files.forEach(file => { + const isChanged = initialPush ? () => true : changedLines(git([ + 'diff', '--no-ext-diff', '--no-textconv', '--no-renames', '--unified=0', base, head, '--', file + ])); + jshint(git(['show', head + ':' + file]), config); + (jshint.errors || []).filter(Boolean).forEach(error => { + // Syntax/configuration errors must fail even when parsing stops + // outside the changed lines. Existing style warnings stay separate. + if (error.code.charAt(0) === 'E' || isChanged(error.line)) issues.push({ file, error }); + else ignored++; + }); + }); + issues.forEach(issue => console.error(issue.file + ':' + issue.error.line + ':' + issue.error.character + + ': ' + issue.error.reason + ' (' + issue.error.code + ')')); + console.log(files.length + ' JavaScript files checked; ' + ignored + + ' warnings outside changed lines ignored; ' + issues.length + ' issues.'); + return issues.length ? 1 : 0; +}; + +if (require.main === module) { + try { + const args = process.argv.slice(2); + if (args.length !== 3) throw new Error('Usage: node lint-changes.js '); + process.exitCode = lintChanges(require(path.resolve(args[0])).JSHINT, args[1], args[2]); + } catch (error) { + console.error(error.message); + process.exitCode = 1; + } +} + +module.exports = { changedLines, lintChanges }; diff --git a/.github/scripts/lint-changes.test.js b/.github/scripts/lint-changes.test.js new file mode 100644 index 0000000..cc2441b --- /dev/null +++ b/.github/scripts/lint-changes.test.js @@ -0,0 +1,108 @@ +// Standalone tooling tests run on the lint job's Node version, without backend dependencies. +const assert = require('assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { execFileSync, spawnSync } = require('child_process'); +const script = path.join(__dirname, 'lint-changes.js'); +const jshint = path.resolve(process.argv[2]); +let failed = 0; + +const checkCase = (name, run) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'evnotify-lint-')); + const git = args => execFileSync('git', args, { cwd: directory, encoding: 'utf8' }).trim(); + const write = (file, contents) => fs.writeFileSync(path.join(directory, file), contents); + const commit = () => { + git(['add', '.']); + git(['-c', 'user.name=Lint Test', '-c', 'user.email=lint@example.com', 'commit', '-qm', 'fixture']); + return git(['rev-parse', 'HEAD']); + }; + const lint = (base, head) => { + const result = spawnSync(process.execPath, [script, jshint, base, head], { cwd: directory, encoding: 'utf8' }); + if (result.error) throw result.error; + return { status: result.status, output: result.stdout + result.stderr }; + }; + try { + git(['init', '-q']); + write('.jshintrc', JSON.stringify({ esversion: 8, undef: true, unused: true, node: true })); + write('legacy.js', 'var unused = 1;\nprocess.stdout.write("ok");\n'); + const base = commit(); + run({ directory, git, write, commit, lint, base }); + console.log('PASS ' + name); + } catch (error) { + failed++; + console.error('FAIL ' + name + ': ' + error.message); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}; + +checkCase('ignores an existing warning on unchanged lines of a modified file', fixture => { + fixture.write('legacy.js', 'var unused = 1;\nprocess.stdout.write("ok");\nprocess.stdout.write("changed");\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 0, result.output); + assert.ok(result.output.includes('1 warnings outside changed lines ignored')); +}); + +checkCase('rejects a new warning on an added line', fixture => { + fixture.write('legacy.js', 'var unused = 1;\nprocess.stdout.write("ok");\nmissingValue();\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 1, result.output); + assert.ok(result.output.includes('legacy.js:3:')); + assert.ok(result.output.includes('W117')); +}); + +checkCase('rejects a new warning on a modified line', fixture => { + fixture.write('legacy.js', 'var unused = 1;\nprocess.stdout.write(missingValue);\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 1, result.output); + assert.ok(result.output.includes('legacy.js:2:')); +}); + +checkCase('does not treat the preceding line as changed for a deletion-only hunk', fixture => { + fixture.write('legacy.js', 'var unused = 1;\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 0, result.output); + assert.ok(result.output.includes('1 warnings outside changed lines ignored')); +}); + +checkCase('checks new JavaScript files completely, including paths with spaces', fixture => { + fixture.write('new file.js', 'missingValue();\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 1, result.output); + assert.ok(result.output.includes('new file.js:1:')); +}); + +checkCase('fails syntax errors even when the parser reports beyond changed lines', fixture => { + fixture.write('legacy.js', 'function broken() {\nprocess.stdout.write("ok");\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 1, result.output); + assert.ok(result.output.includes('(E')); +}); + +checkCase('ignores deleted JavaScript files', fixture => { + fixture.git(['rm', 'legacy.js']); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 0, result.output); + assert.ok(result.output.includes('0 JavaScript files checked')); +}); + +checkCase('succeeds when only non-JavaScript files change', fixture => { + fixture.write('README.md', 'Documentation only.\n'); + const result = fixture.lint(fixture.base, fixture.commit()); + assert.strictEqual(result.status, 0, result.output); + assert.ok(result.output.includes('0 JavaScript files checked')); +}); + +checkCase('checks all JavaScript on a first push without a base commit', fixture => { + const result = fixture.lint('0000000000000000000000000000000000000000', fixture.base); + assert.strictEqual(result.status, 1, result.output); + assert.ok(result.output.includes('legacy.js:1:')); +}); + +checkCase('fails when Git cannot resolve a comparison commit', fixture => { + const result = fixture.lint('not-a-commit', fixture.base); + assert.strictEqual(result.status, 1, result.output); +}); + +process.exitCode = failed ? 1 : 0; diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cd2b8b7..d45b835 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,6 +9,27 @@ permissions: contents: read jobs: + lint: + name: JavaScript lint + runs-on: ubuntu-22.04 + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + ref: ${{ github.event.pull_request.head.sha || github.sha }} + - uses: actions/setup-node@v4 + with: + node-version: 24 + - name: Install pinned lint tooling separately from the backend + run: npm install --prefix "$RUNNER_TEMP/evnotify-lint" --no-save --package-lock=false --ignore-scripts --no-audit --no-fund jshint@2.13.6 + - name: Test changed-line lint checks + run: node .github/scripts/lint-changes.test.js "$RUNNER_TEMP/evnotify-lint/node_modules/jshint" + - name: Lint changed JavaScript + env: + LINT_BASE: ${{ github.event.pull_request.base.sha || github.event.before }} + LINT_HEAD: ${{ github.event.pull_request.head.sha || github.sha }} + run: node .github/scripts/lint-changes.js "$RUNNER_TEMP/evnotify-lint/node_modules/jshint" "$LINT_BASE" "$LINT_HEAD" test: name: Node 8 / MySQL 5.7.34 runs-on: ubuntu-22.04 diff --git a/.hound.yml b/.hound.yml index 0508b09..c391971 100644 --- a/.hound.yml +++ b/.hound.yml @@ -1,3 +1,3 @@ jshint: - enabled: true + enabled: false config_file: .jshintrc diff --git a/README.md b/README.md index ac14884..f624b94 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,21 @@ Delivery happens in the background with a 10-second request timeout and no retri Install dependencies with `npm install`. GitHub Actions runs the full test suite in Docker using Node 8.17.0 and MySQL 5.7.34, with a disposable test database. +JavaScript linting runs separately in CI with Node 24 and pinned JSHint 2.13.6. This tooling does not change the backend's Node 8 runtime or dependencies. Hound's older JavaScript parser is disabled. + +The lint check reports warnings on added or modified JavaScript lines, checks new files completely, and fails syntax errors anywhere in a changed file. Existing warnings outside changed lines remain separate cleanup work. Pull requests compare against their base commit; pushes to `master` compare against the previous commit. The three existing report/export TODOs have individual ESLint suppressions; other CodeFactor checks remain enabled. + +To run the same lint check locally, install tooling outside the application directory and select the commits to compare: + +```bash +lint_tools=$(mktemp -d) +npm install --prefix "$lint_tools" --no-save --package-lock=false --ignore-scripts --no-audit --no-fund jshint@2.13.6 +node .github/scripts/lint-changes.test.js "$lint_tools/node_modules/jshint" +node .github/scripts/lint-changes.js "$lint_tools/node_modules/jshint" "$(git merge-base origin/master HEAD)" HEAD +``` + +Run these tooling commands with Node 24. The check reads committed revisions; commit local changes before checking them. + Run the isolated settings, webhook, sync, notification, and XLSX tests without a backend or database: ```bash diff --git a/modules/logs/index.js b/modules/logs/index.js index 99c3436..583c3ee 100644 --- a/modules/logs/index.js +++ b/modules/logs/index.js @@ -137,6 +137,7 @@ const exportLog = async (akey, id, email) => { const statistics = (await query(`SELECT * FROM ${table} WHERE akey=? AND timestamp >= ? AND timestamp <= ?`, [akey, log.start, log.end])); return report.logExportReport(statistics).then((file) => { + // eslint-disable-next-line no-warning-comments // TODO localize mail.simpleSend(email, 'Log-Export', 'Log-Export', [{ filename: 'Export.xlsx', diff --git a/modules/report/index.js b/modules/report/index.js index 9fb5332..e252c84 100644 --- a/modules/report/index.js +++ b/modules/report/index.js @@ -138,6 +138,7 @@ const createReport = (akey, token, callback) => { let wb = new excel.Workbook(), wss = worksheets.log.map(e => new Worksheet(wb, lng, e)); dataRes.forEach(e => wss.forEach(w => w.processRow(e))); + // eslint-disable-next-line no-warning-comments // TODO dynamic filename (tmp file which need to be unlinked and streamed to user..) wb.write('Excel.xlsx'); callback(null, 'Excel.xlsx'); @@ -153,6 +154,7 @@ const logExportReport = (data) => { return new Promise((resolve, reject) => { const randomFile = 'tmp_logexportreport_' + crypto.randomBytes(20).toString('hex') + '.xlsx'; try { + // eslint-disable-next-line no-warning-comments // TODO lng.. const wb = new excel.Workbook({ dateFormat: 'm/d/yy hh:mm:ss', @@ -201,4 +203,4 @@ module.exports = { }); }, logExportReport -}; \ No newline at end of file +}; From 39a8fac9db0fa2143aefcf2786fd01942fbff1df Mon Sep 17 00:00:00 2001 From: GPlay97 Date: Sun, 25 Oct 2026 22:50:58 +0100 Subject: [PATCH 4/4] Remove custom JavaScript lint workflow --- .github/scripts/lint-changes.js | 55 -------------- .github/scripts/lint-changes.test.js | 108 --------------------------- .github/workflows/ci.yml | 21 ------ README.md | 15 +--- 4 files changed, 1 insertion(+), 198 deletions(-) delete mode 100644 .github/scripts/lint-changes.js delete mode 100644 .github/scripts/lint-changes.test.js diff --git a/.github/scripts/lint-changes.js b/.github/scripts/lint-changes.js deleted file mode 100644 index 26c5809..0000000 --- a/.github/scripts/lint-changes.js +++ /dev/null @@ -1,55 +0,0 @@ -const fs = require('fs'); -const path = require('path'); -const { execFileSync } = require('child_process'); - -const git = args => execFileSync('git', args, { encoding: 'utf8', maxBuffer: 10 * 1024 * 1024 }); - -const changedLines = diff => { - const ranges = []; - const hunk = /^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/gm; - let match; - while ((match = hunk.exec(diff)) !== null) { - ranges.push({ start: Number(match[1]), count: match[2] === undefined ? 1 : Number(match[2]) }); - } - return line => ranges.some(range => line >= range.start && line < range.start + range.count); -}; - -const lintChanges = (jshint, base, head) => { - const initialPush = !base || /^0+$/.test(base); - let listing = ['ls-tree', '-r', '--name-only', '-z', head]; - if (!initialPush) listing = ['diff', '--name-only', '--no-renames', '--diff-filter=ACMR', '-z', base, head, '--', '*.js']; - const files = git(listing).split('\0').filter(file => file.endsWith('.js')); - const config = JSON.parse(fs.readFileSync('.jshintrc', 'utf8')); - const issues = []; - let ignored = 0; - files.forEach(file => { - const isChanged = initialPush ? () => true : changedLines(git([ - 'diff', '--no-ext-diff', '--no-textconv', '--no-renames', '--unified=0', base, head, '--', file - ])); - jshint(git(['show', head + ':' + file]), config); - (jshint.errors || []).filter(Boolean).forEach(error => { - // Syntax/configuration errors must fail even when parsing stops - // outside the changed lines. Existing style warnings stay separate. - if (error.code.charAt(0) === 'E' || isChanged(error.line)) issues.push({ file, error }); - else ignored++; - }); - }); - issues.forEach(issue => console.error(issue.file + ':' + issue.error.line + ':' + issue.error.character + - ': ' + issue.error.reason + ' (' + issue.error.code + ')')); - console.log(files.length + ' JavaScript files checked; ' + ignored + - ' warnings outside changed lines ignored; ' + issues.length + ' issues.'); - return issues.length ? 1 : 0; -}; - -if (require.main === module) { - try { - const args = process.argv.slice(2); - if (args.length !== 3) throw new Error('Usage: node lint-changes.js '); - process.exitCode = lintChanges(require(path.resolve(args[0])).JSHINT, args[1], args[2]); - } catch (error) { - console.error(error.message); - process.exitCode = 1; - } -} - -module.exports = { changedLines, lintChanges }; diff --git a/.github/scripts/lint-changes.test.js b/.github/scripts/lint-changes.test.js deleted file mode 100644 index cc2441b..0000000 --- a/.github/scripts/lint-changes.test.js +++ /dev/null @@ -1,108 +0,0 @@ -// Standalone tooling tests run on the lint job's Node version, without backend dependencies. -const assert = require('assert'); -const fs = require('fs'); -const os = require('os'); -const path = require('path'); -const { execFileSync, spawnSync } = require('child_process'); -const script = path.join(__dirname, 'lint-changes.js'); -const jshint = path.resolve(process.argv[2]); -let failed = 0; - -const checkCase = (name, run) => { - const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'evnotify-lint-')); - const git = args => execFileSync('git', args, { cwd: directory, encoding: 'utf8' }).trim(); - const write = (file, contents) => fs.writeFileSync(path.join(directory, file), contents); - const commit = () => { - git(['add', '.']); - git(['-c', 'user.name=Lint Test', '-c', 'user.email=lint@example.com', 'commit', '-qm', 'fixture']); - return git(['rev-parse', 'HEAD']); - }; - const lint = (base, head) => { - const result = spawnSync(process.execPath, [script, jshint, base, head], { cwd: directory, encoding: 'utf8' }); - if (result.error) throw result.error; - return { status: result.status, output: result.stdout + result.stderr }; - }; - try { - git(['init', '-q']); - write('.jshintrc', JSON.stringify({ esversion: 8, undef: true, unused: true, node: true })); - write('legacy.js', 'var unused = 1;\nprocess.stdout.write("ok");\n'); - const base = commit(); - run({ directory, git, write, commit, lint, base }); - console.log('PASS ' + name); - } catch (error) { - failed++; - console.error('FAIL ' + name + ': ' + error.message); - } finally { - fs.rmSync(directory, { recursive: true, force: true }); - } -}; - -checkCase('ignores an existing warning on unchanged lines of a modified file', fixture => { - fixture.write('legacy.js', 'var unused = 1;\nprocess.stdout.write("ok");\nprocess.stdout.write("changed");\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 0, result.output); - assert.ok(result.output.includes('1 warnings outside changed lines ignored')); -}); - -checkCase('rejects a new warning on an added line', fixture => { - fixture.write('legacy.js', 'var unused = 1;\nprocess.stdout.write("ok");\nmissingValue();\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 1, result.output); - assert.ok(result.output.includes('legacy.js:3:')); - assert.ok(result.output.includes('W117')); -}); - -checkCase('rejects a new warning on a modified line', fixture => { - fixture.write('legacy.js', 'var unused = 1;\nprocess.stdout.write(missingValue);\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 1, result.output); - assert.ok(result.output.includes('legacy.js:2:')); -}); - -checkCase('does not treat the preceding line as changed for a deletion-only hunk', fixture => { - fixture.write('legacy.js', 'var unused = 1;\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 0, result.output); - assert.ok(result.output.includes('1 warnings outside changed lines ignored')); -}); - -checkCase('checks new JavaScript files completely, including paths with spaces', fixture => { - fixture.write('new file.js', 'missingValue();\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 1, result.output); - assert.ok(result.output.includes('new file.js:1:')); -}); - -checkCase('fails syntax errors even when the parser reports beyond changed lines', fixture => { - fixture.write('legacy.js', 'function broken() {\nprocess.stdout.write("ok");\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 1, result.output); - assert.ok(result.output.includes('(E')); -}); - -checkCase('ignores deleted JavaScript files', fixture => { - fixture.git(['rm', 'legacy.js']); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 0, result.output); - assert.ok(result.output.includes('0 JavaScript files checked')); -}); - -checkCase('succeeds when only non-JavaScript files change', fixture => { - fixture.write('README.md', 'Documentation only.\n'); - const result = fixture.lint(fixture.base, fixture.commit()); - assert.strictEqual(result.status, 0, result.output); - assert.ok(result.output.includes('0 JavaScript files checked')); -}); - -checkCase('checks all JavaScript on a first push without a base commit', fixture => { - const result = fixture.lint('0000000000000000000000000000000000000000', fixture.base); - assert.strictEqual(result.status, 1, result.output); - assert.ok(result.output.includes('legacy.js:1:')); -}); - -checkCase('fails when Git cannot resolve a comparison commit', fixture => { - const result = fixture.lint('not-a-commit', fixture.base); - assert.strictEqual(result.status, 1, result.output); -}); - -process.exitCode = failed ? 1 : 0; diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d45b835..cd2b8b7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,27 +9,6 @@ permissions: contents: read jobs: - lint: - name: JavaScript lint - runs-on: ubuntu-22.04 - timeout-minutes: 5 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - ref: ${{ github.event.pull_request.head.sha || github.sha }} - - uses: actions/setup-node@v4 - with: - node-version: 24 - - name: Install pinned lint tooling separately from the backend - run: npm install --prefix "$RUNNER_TEMP/evnotify-lint" --no-save --package-lock=false --ignore-scripts --no-audit --no-fund jshint@2.13.6 - - name: Test changed-line lint checks - run: node .github/scripts/lint-changes.test.js "$RUNNER_TEMP/evnotify-lint/node_modules/jshint" - - name: Lint changed JavaScript - env: - LINT_BASE: ${{ github.event.pull_request.base.sha || github.event.before }} - LINT_HEAD: ${{ github.event.pull_request.head.sha || github.sha }} - run: node .github/scripts/lint-changes.js "$RUNNER_TEMP/evnotify-lint/node_modules/jshint" "$LINT_BASE" "$LINT_HEAD" test: name: Node 8 / MySQL 5.7.34 runs-on: ubuntu-22.04 diff --git a/README.md b/README.md index f624b94..1a0a2fb 100644 --- a/README.md +++ b/README.md @@ -101,20 +101,7 @@ Delivery happens in the background with a 10-second request timeout and no retri Install dependencies with `npm install`. GitHub Actions runs the full test suite in Docker using Node 8.17.0 and MySQL 5.7.34, with a disposable test database. -JavaScript linting runs separately in CI with Node 24 and pinned JSHint 2.13.6. This tooling does not change the backend's Node 8 runtime or dependencies. Hound's older JavaScript parser is disabled. - -The lint check reports warnings on added or modified JavaScript lines, checks new files completely, and fails syntax errors anywhere in a changed file. Existing warnings outside changed lines remain separate cleanup work. Pull requests compare against their base commit; pushes to `master` compare against the previous commit. The three existing report/export TODOs have individual ESLint suppressions; other CodeFactor checks remain enabled. - -To run the same lint check locally, install tooling outside the application directory and select the commits to compare: - -```bash -lint_tools=$(mktemp -d) -npm install --prefix "$lint_tools" --no-save --package-lock=false --ignore-scripts --no-audit --no-fund jshint@2.13.6 -node .github/scripts/lint-changes.test.js "$lint_tools/node_modules/jshint" -node .github/scripts/lint-changes.js "$lint_tools/node_modules/jshint" "$(git merge-base origin/master HEAD)" HEAD -``` - -Run these tooling commands with Node 24. The check reads committed revisions; commit local changes before checking them. +Hound's older JavaScript parser is disabled. The three existing report/export TODOs have individual ESLint suppressions; other CodeFactor checks remain enabled. Run the isolated settings, webhook, sync, notification, and XLSX tests without a backend or database: