State Estimation Measurement Table

A measurement table is a CSV file with one row per measured value and one measurement set per timestamp. It is addressed by CGMES mRID for a CGMES case and by element name for every other case format. It is the measurement format of Sparlectra: the Web UI generator writes it and the shipped sets use it. The measurement CSV v1 is no longer read since 0.50.0; the two formats are told apart by their first line, and a v1 file is converted once (see Migrating from the v1 CSV).

The format is new and may change before 1.0.

File

• CSV, UTF-8 (a BOM at the start is ignored), comma as separator, decimal point, LF line ends (a CR before the LF is tolerated).

• The version line comes first, then optional comment lines, then the header:

  # sparlectra-measurement-table v1
  # free comments: origin, generator, seed
  timestamp,measurement_mrid,terminal_mrid,node_mrid,type,value,sigma,source,valid

• Lines starting with # before the header are comments; blank lines are skipped. After the header there are no comments.

• A field with a comma or a double quote is enclosed in double quotes, "" inside is one quote (RFC 4180). In practice this concerns source only. A field cannot hold a line break: each row is one line, and writeMeasurementTable refuses a cell or comment with a line break.

• Blanks around a cell are not part of it: the reader strips them and the writer does not write them.

• readMeasurementTable refuses a file without the header, without the version line or with another version, with an ArgumentError that says what was expected. Blanks after the version are allowed; other text after it is refused with its own message. isMeasurementTable decides by content (version line plus header), not by the file name. The measurement CSV v1 starts with # sparlectra-measurements v1 and is no measurement table.

• A file that starts with the table header but lacks the version line is still treated as a table by the SE service, so the run names the missing line; the Web UI lists it on upload as measurement_table_without_version_line.

Columns

Exactly one of the three key columns is set per row.

Addressing

The names of the key columns are historical. In both cases they hold keys; which kind of key depends on the case.

CGMES case (cgmesMeasurementKeys):

• terminal_mrid: mRID of a Terminal.
• node_mrid: mRID of a TopologicalNode, ConnectivityNode or BusbarSection.
• measurement_mrid: mRID of an Analog. Its measurementType gives the type (ThreePhaseActivePower P, ThreePhaseReactivePower Q, LineCurrent I, LineToLineVoltage U), its terminal the location (without a terminal its PowerSystemResource, when that is a node or a busbar). Other measurement types are reported as unknown_type. A U Analog on a terminal gives the voltage of the terminal's node (also on a switch terminal); at a node that is not energized it is not_modelled.
• A terminal of a branch out of service or open at one terminal is not_modelled at both ends: the estimator does not take flow measurements on a partially open branch yet. The same holds for name keys.
• mRIDs are compared in canonical form: a leading _ (rdf:ID in CGMES 2.4.15) and a urn:uuid: prefix are removed.
• A bus or branch end the delivery has no mRID for (the star point of a three-winding transformer, the legs that end there, auxiliary buses of the import) is keyed by its name as for any other case (below). The mRID comes first: a name key exists only where there is no mRID, and it never replaces an mRID key.

Any other case (MATPOWER, SCF, DTF, PowSyBl, a net built in code; nameMeasurementKeys):

• node_mrid: the bus name.
• terminal_mrid: <branch name>:from or <branch name>:to for a branch end; the prosumer name for a load or generator; the shunt name for a shunt.
• A branch name that several branches share (parallel lines) is qualified with the branch number, <branch name>#<k>:from, where k is the position in net.branchVec (the branch_nr of the CSV v1). A prosumer or shunt name that several of them share is qualified with its bus, <name>@<bus name>. When several at one bus share that as well (two generators of one voltage level carry the same generated name), each gets its order at the bus, <name>@<bus name>#<k> (prosumers first, then shunts, each in model order). A key that is still not unique resolves to not_modelled with the reason.
• Both #<k> forms are a way out, not a stable address: k follows the element order of the import and can differ after the next import or export. Element names should be unique; a table meant to be reused keys on unique names.
• Every prosumer and every in-service shunt at an energized bus is a member of the sum at its bus, whatever its key (see Group sums). An element without a name has no key; its bus sum is therefore always partial_injection.
• measurement_mrid does not apply; a row with it is reported as unknown_key.
• Names are compared as written, only surrounding blanks are ignored. The canonical mRID form does not apply: buses _B9 and B9 are two keys. Two bus names that differ only in surrounding blanks give one key; a U row on it is not_modelled (ambiguous), never one of the buses.

The names are the ones Sparlectra shows in its results (comp.cName, the bus names of net.busDict). For imported MATPOWER and SCF cases the branch and prosumer names are generated by the importer, for example B_ACL_110_1_2 or Ld_110.0.

Units

P and Q at a node are the bus balance in MW and MVAr (PinjMeas, QinjMeas). Angles refer to the time base of the measuring device; the estimator treats the offset to the slack as in the measurement CSV v1 (see State Estimation).

For U the import divides value and sigma by the nominal voltage Vn_kV of the bus: vm_pu = U_kV / Vn_kV, sigma_pu = sigma_kV / Vn_kV. No other quantity is converted: no power or current base is applied. A U cell with more than 17 significant digits is divided from its decimal text at 256 bits and rounded once. The converter (convertMeasurementsV1) writes such a cell where no Float64 kV value divides back to the pu value of the CSV v1, so the pu values come back bit for bit. An empty sigma takes the level sigma u_kv / Vn_kV when the level has one, else the VmMeas default directly in pu. I stays in ampere: ImagMeas is an ampere-valued measurement (see State Estimation), the import applies no current base.

Signs and mapping

• P and Q at a terminal are positive into the equipment at that terminal (the CGMES SvPowerFlow convention).
• Branch end: P, Q, I become PflowMeas, QflowMeas, ImagMeas at that end. The Sparlectra flow direction from the bus into the branch is the same direction.
• Injections (loads, machines, external grids, equivalent injections, SVCs, mapped energy sources; for name keys every prosumer) at one bus: P and Q of all modelled members are summed into one PinjMeas and one QinjMeas with the sign turned (bus injection = minus the power into the equipment), sigma = sqrt(sum sigma_i^2).
• Shunts at one bus: Q summed the same way into one ShuntQMeas, sign kept (power into the shunt, the q_shunt convention). The estimator predicts ShuntQMeas as the sum over all in-service shunts at the bus.
• I at a shunt: the bay current, ImagMeas at the bus, when the bus has one in-service shunt. With several shunts the rows are unsupported: the bay current is the magnitude of the summed currents, not the sum of the magnitudes.
• Ia at a branch end: IaMeas at that end.
• P at a shunt, I at an injection and Ia at an injection or shunt are unsupported.
• Node with U: VmMeas at the bus.
• Node with P or Q: the bus balance, generation minus load at the bus (injection positive), PinjMeas or QinjMeas with the value as written. It does not replace the prosumer rows of the same bus: where both exist, the balance and the prosumer sum enter the set as two redundant measurements.
• A bus balance row whose source carries the token zero_injection is a zero-injection constraint: it gets the id ZI_PINJ_bus_<k> or ZI_QINJ_bus_<k> and is protected from bad-data elimination like the rows state_estimation.zero_injection adds. Its sigma is the one of the row.
• Node with Va: VaMeas at the bus.
• A bus link (a switch or coupler joining two buses outside the Y-bus) is no branch of the model: rows on its terminals are not_modelled with the reason. Name keys: <link name>:from, <link name>:to.

Group sums

A bus sum needs a value for every modelled member of the group at that bus.

• All members have a row with valid=true: the sum is active. A valid=false row of a member that also has a valid row is not used and is reported as invalid_unused.
• Every member has a row, but at least one member only rows with valid=false: the sum is created inactive (the input says "do not use", as for a single branch row). The other members still use only their valid rows; an invalid row next to a valid one is invalid_unused here too.
• A member has no row at all: no sum is formed; its rows are counted as partial_injection.
• Several rows of one member (redundant meters on one terminal) are merged to their inverse-variance mean before the sum and counted as merged.

Redundancy and duplicates

Several keys at one location are redundant measurements and all enter the set: two Analogs on one terminal, or U at a busbar section and at its TopologicalNode give two VmMeas at one bus. duplicate means the same key with the same type twice in one set; the second row is reported.

Row outcomes

No row is dropped silently. Each row ends as an active measurement, an inactive measurement, or with a reason in the import report (measurementImportReport, one line per problem row with its line number):

Generators

Both generators return rows with the header fields; writeMeasurementTable writes them with the version line. The same arguments, seed included, give the same rows and therefore the same file.

generateMeasurementTableFromSV(result) works on a CGMES import result and writes mRID keys: U at every energized TopologicalNode with an SvVoltage, P and Q at both ends of every branch closed at both terminals, computed with the network model at the SV voltage state.

generateMeasurementTableFromPF(net) works on any net after runpf! and writes name keys: U at the buses and P and Q at both branch ends, computed with the network model at the solved voltages.

The sigma keywords are checked before anything is generated: sigma_pq_floor and sigma_u_rel must be positive, sigma_pq_rel and bad_data_sigmas finite and at least 0, otherwise an ArgumentError names the keyword. Every written sigma is then positive and passes the reader.

With noise = false an estimation on the table reproduces the state it was generated from: on sp_case14 the estimated voltages agree with the power flow to better than 1e-15 pu (all buses and branch ends, with or without the injection rows; the test asserts 1e-6 pu).

Pseudo measurements

A real delivery rarely measures every load and generator. generatePseudoMeasurementTable(net, rows; sigma) (name keys) and generatePseudoMeasurementTable(result, rows; sigma) (CGMES import, mRID keys) write rows for the injection members that rows does not observe: P and Q of each load, generator, external grid, equivalent injection or SVC, taken from the scheduled values of the case (for CGMES the SSH setpoints), as power into the equipment. A member counts as observed for a type when a well-formed row carries its terminal key, or an Analog on its terminal, with that type; a valid=false row counts as well. Members are filled one by one, so a partial bus sum becomes complete and keeps its real rows. A unit without a Q setpoint gets no Q row; shunts get none.

Both sigmas must be finite and positive (ArgumentError otherwise). Choose them large against the real meters, so the real rows win. The rows carry source = "pseudo;ssh" and valid = true; appended to the table they are read and summed like any other row:

rows = readMeasurementTable("delivery.table.csv")
pseudo = generatePseudoMeasurementTable(result, rows; sigma = 5.0)
writeMeasurementTable("filled.table.csv", vcat(rows, pseudo))

On the SmallGrid fixture the second set observes every injection and gets no pseudo rows; the first set has no injection rows and gets P and Q for all 125 members.

Incomplete measurement sets

A real delivery is never complete. Two kinds of rows close the gaps:

• Zero injection at passive buses (no generation, load or shunt) is exact network information, a constraint and not a measurement. With state_estimation.zero_injection: on the run adds it where the set has no injection row; see Use in the SE service and the Web UI.
• Pseudo measurements from the schedule (the SSH setpoints, see above) carry a large sigma. They restore observability where the gaps are and lose against every real meter.

Missing flows are not derived from measured neighbours. Such a value is a linear combination of measurements the estimator already has through the network model, so it adds no information. It counts those measurements a second time, and its error is correlated with theirs, while WLS treats every row as independent. Observability does not improve; only the residuals look better.

SmallGrid fixture

test/fixtures/measurements/smallgrid_synthetic.table.csv is a table for the ENTSO-E SmallGrid (NodeBreaker BaseCase in test/fixtures/cgmes_smallgrid), built by tools/gen_smallgrid_measurements.jl with generateMeasurementTableFromSV. It holds two sets. The numbers below come from importing the fixture and running runse_diagnostics with state_estimation.max_eliminations = 10.

The first set also reports three handmade rows, one per reason (unknown_key, not_modelled for a breaker terminal, unknown_type), and one healthy branch row with valid=false that becomes an inactive measurement. These numbers depend on the network model; a change of the CGMES mapping that moves them is a change of the model, not of the table.

Use in the SE service and the Web UI

The SE service reads a table while state_estimation.measurement_table is true (the default); with the switch off a table is refused and the key is named. The format of the case picks the index: mRIDs for a CGMES case, names for every other format. The run estimates the first set of the file and writes the import report of all sets to measurement_import.txt. With state_estimation.zero_injection: on (the default) every passive bus without an injection row in the set gets a zero-injection pair at ZERO_INJECTION_SIGMA, the rows the measurement generator writes for passive buses; measurement_import.txt and the run log give the count (zero-injection rows N). applyMeasurementSet! does the same for a table in a script and stores the count in set.counts[:zero_injection]; its keyword zero_injection overrides the configured switch. The switch is case scope: a case with its own <case>.config.yaml takes it from that file (written by the generator checkbox), not from the general configuration file. Before 0.50.0 a table run added no such rows, so a table that leaves passive buses unmeasured estimates differently now; off restores the former result. Redundant readings of one quantity (several keys at one location) are named in the run log as a note; unlike a doubled measurement CSV v1 set they are no duplicate finding of the run. See State-Estimation Configuration.

The Web UI generator and "add noise" write tables, keyed the way the service reads them. The state-estimation section offers a table next to the case (upload, selection, row counts per type, text and value editor, download, delete); the run goes through the same service. See Web UI Reference.

In a script, readMeasurementTable!(net; file) does the read, the import with nameMeasurementKeys(net) (or the index given, for example cgmesMeasurementKeys(result)) and applyMeasurementSet! in one call and returns the set.

The example examples/state_estimation/state_estimation_measurement_table.jl runs the generator and the estimation on sp_case14 and the bad-data elimination on the SmallGrid fixture.

Migrating from the v1 CSV

convertMeasurementsV1(v1_file, net; out) writes the table of a measurement CSV v1 file. The v1 file addresses buses and branches by name and number, the table by keys, so the converter takes the net of the case; for a CGMES case pass index = cgmesMeasurementKeys(result) so the table carries mRIDs as the SE service expects.

net = importSCF("sp_case14.scf.json")
res = convertMeasurementsV1("sp_case14.measurements.v1.csv", net; out = "sp_case14.measurements.csv")
print(res.report)

From the command line: sparlectra convert-measurements <case> <v1.csv> in the built application, or julia --project=app tools/convert_measurements.jl <case> <v1.csv> [<out.csv>] in a checkout; both key the table like the SE service.

• Every v1 row becomes one row with the same measurement. Vm becomes U in kV, written with up to 18 digits where needed so the import gives back the pu value bit for bit; P and Q stay in MW and MVAr, I in A, angles in degrees. Bus injections become node balances (P, Q at the node), zero-injection rows (ZI ids) keep their sigma and get the source token zero_injection, active becomes valid.
• The comments of the v1 file move in front of the header; the ids in the truth_value and critical-row lines are rewritten to the ids of the import, so tap estimation and se_deltas.csv work as before.
• Rows the table cannot carry (a link flow, a shunt bay at a bus with several shunts, a negative U or I, a second row with the same key and type) are not written; the result and the comment lines of the table name each with its line.
• A row at an isolated bus is written with that bus key; the import reports it as not_modelled, as the SE service left such a v1 row out.

The estimate on the converted table equals the one on the v1 file: the same J, state and eliminations on every set shipped with Sparlectra (shown in 0.50.0, when both readers existed). The v1 reader readMeasurementsCSV! was removed in 0.50.0; the SE service and the Web UI refuse a v1 file with an error that names the converter.