State Estimation Measurements

Measurement sets travel as measurement tables or inside a case file. This page covers the files, the measurement generator, the case-file binding and the power flow chained onto an estimate. The estimator itself is on State Estimation; the format of the table (columns, keys, units, signs) on Measurement table.

Measurement files and the SE chain

The measurement table (first line # sparlectra-measurement-table v1, one row per measured value, keyed by element name or CGMES mRID, values in MW, MVAr, A, kV and degrees) is the measurement format of Sparlectra. The Web UI generator and "add noise" write it, and the sets shipped with the demo cases are tables. readMeasurementTable!(net; file) reads one set into net.measurements; writeMeasurementTable(file, rows) writes rows, for example from generateMeasurementTableFromPF. The measurement CSV v1 is no longer read (see Removed: measurement CSV v1).

Run artifacts. An estimation run writes the set it estimated as a measurement table, measurements.csv in the run directory, and the verdict per row in measurements_verdict.csv next to it. The name measurements.csv is kept: the Web UI links and the history know it, and it matches the <case>.measurements.csv sets. The table loads directly with readMeasurementTable! (for a CGMES case with index = cgmesMeasurementKeys(result)) or as the measurement file of a new run, and gives the same measurement set: the run estimates exactly the set this file reads back to.

• A table set keeps the rows that became a measurement, in file order, with their keys, value text, source and valid flag; rows that did not are listed in measurement_import.txt. A row without a sigma cell gets the sigma the run used and the source token sigma_level or sigma_default, so the file reads back the same under any configuration.
• Zero-injection rows the run added are bus balance rows with source se_run;zero_injection; a set carried by a case file is written with source case_file. A measurement the table cannot carry (a link flow) is named as a comment line and stays in the estimate.
• The comments of the input table are carried over (case binding, generator provenance, tap table, truth values).

The verdict file has one line per data row of the table, joined by its line number in measurements.csv (a key with its type is not unique: a bus sum covers the rows of all its prosumers, and each of them gets the verdict of the sum). Its CSV format follows output.csv_format.

Every other active row carries the residuals of the final diagnostics (after every elimination). The suspicious count in run.log, on the result page and in se_suspicious is taken from the same diagnostics; the pass before the elimination lists its own suspects in se_diagnostics.md. A run directory written before 0.50.0 holds the CSV v1 artifact (bad_data and status behind id); the Web UI shows such a run, and the topology hypothesis test asks for a new run instead of reading it.

Comment blocks. A table carries them as # lines before its header.

Bad data. The verdict file names the suspicious, eliminated, suppressed (replacement) and down-weighted (staged) rows. Until 0.50.0 se_bad_data.csv listed them; the verdict file replaces it. The tap steps of a run are in se_tap_estimates.csv. Eliminated rows are inactive in the reported run (its J is the J without them); se_diagnostics.md names the run, the case, the set and the reported numbers, then the pass that found them ("Diagnostics pass"). An elimination counts only when the control solve without the row converges; otherwise the row stays active, the attempt is named ("Reverted") and the sequence stops with not_converged. After a tap-estimation fallback the diagnostics run again on the frozen taps, without the rows the first pass removed; the rows of both passes are eliminated, each with the residual of the pass that removed it. The topology hypothesis test reads measurements.csv and leaves out the rows the verdict file marks as eliminated.

The set tab of the Web UI (download, re-upload, inline editor, tap comment table) is described in Web UI.

Measurement generator

The generator solves the case once (island-wise) and writes a new set <case>.measurements.<yyyymmdd-HHMMSS>.csv next to the case, never over an earlier one ("add noise" writes <case>.noisy.<stamp>.csv). A generated set takes precedence over the one a case file carries. Set selection and deletion: Web UI.

Measurement generator controls

Seeded noise

Adds Gaussian noise at the per-quantity sigmas to every generated value. The random generator is seeded, so regenerating with the same inputs reproduces the identical file.

Without noise the values are exact solutions of the power flow: the estimation reports J close to 0 and the band test flags :low. That is expected for a noise-free synthetic set, not an error.

Bad data (k times sigma)

Corrupts seed-randomly drawn telemetry rows by k times their sigma. 0 = off; 10 gives a clearly detectable bad measurement (a stuck transducer). Bad data rows says how many rows get the gross error; the seed decides which, protected ZI rows are never drawn, and the confirmation message names every corrupted row. The diagnostics should rank exactly these rows first, the sequential elimination remove them, the robust option suppress them without removal.

Tap deviation

Generates the measurements from a network state whose transformers run the given number of mechanical tap steps off the model position (0 = off; whole steps only); the model file stays untouched. Transformers (max) bounds how many transformers deviate; the seed picks them from the transformers whose deviation the tap estimation can absorb (in service, non-machine, with a declared changer), and the message names the affected branches. A machine transformer is used only when the case has nothing else, because the Web UI mass release skips it and the deviation would leave an unexplainable model error (J far above dof); the estimator run warns in that case.

The deviation lies on the ratio grid the estimator fixes on, so tap estimation recovers it exactly and the fixation run ends near J = 0; without tap estimation the band test reports :high with the suspects clustered at that transformer. Off-grid positions are not a generator setting; the estimator reports them through the off-grid note.

Measurement sigmas (U, I, P, Q)

Measurement accuracy per quantity in percent of the measured value (like a transducer accuracy class), written into the sigma column of every generated row and used for the noise, when enabled:

• sigma U (%): all voltage-magnitude rows. 0.5 corresponds to a class 0.5 device.
• currents (I) checkbox plus sigma I (%): branch current-magnitude rows at both branch ends, generated only when the checkbox is set. Currents are auxiliary in the estimator, see ImagMeas.
• sigma P (%): active-power injections and branch flows.
• sigma Q (%): reactive-power injections and branch flows.

Percent of reading keeps one setting meaningful across voltage levels. Near-zero readings get a per-type floor instead of a near-zero sigma (measurementSigmaFloors: 1e-4 pu, 0.05 MW/MVar, 0.1 A); the power floor also keeps zero-injection buses from stiffening the flat-start solve. A healthy noisy set lands near J = dof; sigmas too pessimistic against the noise push the band test to :low, too optimistic ones to :high.

Measurement generator v2

Truth state, flow placement, passive-node handling and a target number of critical measurements are configurable in the Web UI form.

Options

Notes

• from run rejects a missing artifact, a foreign case or an unknown run id up front, and warns (in the confirmation and in the file's # truth: line) when the source run is no solution of the model (state estimation outside the band, taps frozen at model positions, not converged): flows derived from such a state violate the node balances. The tap-deviation option is locked with from run.
• Critical-measurement thinning reads the criticality from diag(Omega) (see Observability) and removes per step the strongest partner (largest normalized covariance |Omega_ij| / sqrt(Omega_ii Omega_jj)) of the telemetry row closest to critical; above the dense-path size caps it removes the redundant row with the smallest wii. Zero-injection and passive balance rows are never removed; a removal that fails the rank test is undone.
• The zero-injection sigma floor is 1 kW (ZERO_INJECTION_SIGMA); a tighter row, squared in G = H' W H, pushes the normal equations past double precision.
• Delta file. An SE run on a set with # truth_value,... comments writes se_deltas.csv: per measurement the measured, truth and estimated values with the three deltas, the sigma, the normalized residual and the elimination flag, plus one row per released tap comparing the documented deviation with the fixed step.

Case files carry their own measurements

A Sparlectra Case Format case stores its measurement set inside the file. An estimation on such a case needs no separate CSV; the run still writes measurements.csv into its output directory for reproducibility. Naming a set explicitly takes precedence, and a set generated for the case replaces the one the file carries (the newer data). Every other format requires a measurement file.

A noise-free set returns $J = 0$ by construction, which says nothing about the estimate. A run on a case file produces the same se_deltas.csv and tap warning as a run from a measurement file, and a set bound to the case an SCF file was exported from stays usable, because the file records its source. A set in which every quantity appears twice (one generation appended onto existing rows) inflates $J$ without any single residual looking wrong, which reads as a topology error; the reader and the run report the repeated rows, and regenerating the set is the fix.

PF from the estimate

A power flow can start from the last estimate, either as start values only or as the study base with the estimated nodal balances. The estimated state persists in se_state.csv, which is how the Web UI hands an SE result to a later power-flow or N-1 run. Both modes leave the persistent model untouched (Write-back policy).

With consistent PV setpoints the snapshot power flow converges in 0 or 1 iterations and slack_pickup_mw/mvar stays below tolerance; active outer-loop controllers (taps, machine control, Q-limits) still iterate and legitimately move the operating point, which shows as a small residual pickup.

Removed: measurement CSV v1

The reader of the measurement CSV v1 (readMeasurementsCSV!) was removed in 0.50.0. The SE service refuses a v1 file with an error that names the converter, and so does "add noise" in the Web UI. The Web UI export and "Save case as" write the case without a v1 set and name the skipped file and the converter in their message. Convert a v1 file and its case with convertMeasurementsV1 or sparlectra convert-measurements (see Migrating from the v1 CSV); the converter still reads every v1 file.

writeMeasurementsCSV(net; file) stays and writes this layout, the input of the converter. The first line is the version comment, the second the header:

# sparlectra-measurements v1
type,bus,from_bus,to_bus,branch_nr,link_nr,direction,value,sigma,active,id

Exactly one location group per row (bus | from_bus+to_bus+branch_nr | link_nr). Delimiter and decimal separator follow output.csv_format of the writing run or session (excel_de: semicolon and decimal comma); the converter tells the format from the header line and reports errors with file and line (file:line: reason).