Skip to content

Model confidence pack

A consultant cannot hand over a model with “it looks about right”. They have to evidence it. Model data ▸ Model confidence pack assembles that evidence from numbers the app already computes, and prints it as a deliverable.

Six sections, each graded and each carrying the figure that produced the grade. Section grades describe evidence, not workmanship - the lowest is “Not demonstrated”, meaning the evidence does not yet support that particular claim, which is a different and far more actionable statement than “poor”. There is no grade above them; see below.

SectionWhat it answers
What the model issize, total length, whether it is one connected system
Measured against assumedwhich fields hold survey data and which hold a default, from data completeness
Outstanding validationerrors and warnings still open
Mass balancecontinuity error against the band in force
Comparison against observationevery gauge, judged twice
How much of the network that coversthe honest limiter, below

There is no overall grade, and that is deliberate

Section titled “There is no overall grade, and that is deliberate”

The pack reports evidence by dimension and stops there. It does not print a single word at the top telling you whether the model is good.

Whether a model is fit for a purpose depends on the purpose, and the purpose is not in the model. A calibration good enough for a strategic capacity study is not good enough for a flood-risk determination at one property. Only the reader knows which question is being asked, so only the reader can draw the conclusion.

Coverage, which is the fact that matters most

Section titled “Coverage, which is the fact that matters most”

NSE 0.95 at one flow monitor is a real result about one pipe. It is not a statement about the network. So the pack says, near the top and in plain words, what share of the network drains through a gauge that was actually compared:

2 gauges compared, covering 6% of the network by conduit length. The remaining 94% is not evidenced by this pack.

Coverage is measured by conduit length, not by element count. Counting elements flatters a monitor sitting at the foot of many short laterals and penalises one covering a few long trunk mains, and it is the trunk main the client cares about.

Two verdicts per gauge, and they can disagree

Section titled “Two verdicts per gauge, and they can disagree”

Each gauge is judged twice:

  • Statistical - Moriasi et al. (2007) from NSE and |PBIAS| over the whole overlapping record.
  • Criteria - the event bands in force: peak flow, event volume, peak depth.

They genuinely disagree, and the pack reports the disagreement rather than averaging it. A long record the model tracks well, with one storm peak it misses by 30%, scores very good statistically and unsatisfactory on criteria - because NSE is dominated by the quiet majority while the criteria judge the event the model is designed against.

That disagreement is exactly what a reviewer is looking for. Blending it into one middling grade would hide it.

The criteria are yours, and there are two presets

Section titled “The criteria are yours, and there are two presets”

Urban drainage · routine SCADA is the default: peak flow −25% to +30%, volume ±25%, peak depth −0.3 m to +0.5 m.

Flow survey · strict is the tighter set: peak flow −15% to +25%, volume ±20%, peak depth −0.1 m to +0.5 m, of the kind applied to dedicated flow-survey verification.

Both are editable, and every band is printed in the pack beside the result it judged. Editing any number switches the preset to Custom.

The depth band is absolute, not a percentage: half a metre is half a metre whether the chamber runs at 0.4 m or 4 m. It is also asymmetric, like the flow band - over-predicting depth is the conservative error, under-predicting it hides a flood.

  • A series stored as a preview rather than a full import is flagged, because statistics from a fraction of a record should not reach a deliverable unmarked.
  • A gauge whose record does not overlap the simulated period is reported as such, not scored as zero.
  • An observed series that does not resolve to an element is named.
  • A model that has never been run still gets its build and data sections, with everything downstream marked not evaluated.

Run simulation is in the panel footer, so you can re-run and watch the pack recompute without leaving it. Editing a criterion recomputes the grades immediately, with no re-run needed.

Requires a Pro plan.