Model confidence pack
A consultant cannot hand over a model with “it looks about right”. They have to evidence it. Model data ▸ Model confidence pack assembles that evidence from numbers the app already computes, and prints it as a deliverable.
What it reports
Section titled “What it reports”Six sections, each graded and each carrying the figure that produced the grade. Section grades describe evidence, not workmanship - the lowest is “Not demonstrated”, meaning the evidence does not yet support that particular claim, which is a different and far more actionable statement than “poor”. There is no grade above them; see below.
| Section | What it answers |
|---|---|
| What the model is | size, total length, whether it is one connected system |
| Measured against assumed | which fields hold survey data and which hold a default, from data completeness |
| Outstanding validation | errors and warnings still open |
| Mass balance | continuity error against the band in force |
| Comparison against observation | every gauge, judged twice |
| How much of the network that covers | the honest limiter, below |
There is no overall grade, and that is deliberate
Section titled “There is no overall grade, and that is deliberate”The pack reports evidence by dimension and stops there. It does not print a single word at the top telling you whether the model is good.
Whether a model is fit for a purpose depends on the purpose, and the purpose is not in the model. A calibration good enough for a strategic capacity study is not good enough for a flood-risk determination at one property. Only the reader knows which question is being asked, so only the reader can draw the conclusion.
Coverage, which is the fact that matters most
Section titled “Coverage, which is the fact that matters most”NSE 0.95 at one flow monitor is a real result about one pipe. It is not a statement about the network. So the pack says, near the top and in plain words, what share of the network drains through a gauge that was actually compared:
2 gauges compared, covering 6% of the network by conduit length. The remaining 94% is not evidenced by this pack.
Coverage is measured by conduit length, not by element count. Counting elements flatters a monitor sitting at the foot of many short laterals and penalises one covering a few long trunk mains, and it is the trunk main the client cares about.
Two verdicts per gauge, and they can disagree
Section titled “Two verdicts per gauge, and they can disagree”Each gauge is judged twice:
- Statistical - Moriasi et al. (2007) from NSE and |PBIAS| over the whole overlapping record.
- Criteria - the event bands in force: peak flow, event volume, peak depth.
They genuinely disagree, and the pack reports the disagreement rather than averaging it. A long record the model tracks well, with one storm peak it misses by 30%, scores very good statistically and unsatisfactory on criteria - because NSE is dominated by the quiet majority while the criteria judge the event the model is designed against.
That disagreement is exactly what a reviewer is looking for. Blending it into one middling grade would hide it.
The criteria are yours, and there are two presets
Section titled “The criteria are yours, and there are two presets”Urban drainage · routine SCADA is the default: peak flow −25% to +30%, volume ±25%, peak depth −0.3 m to +0.5 m.
Flow survey · strict is the tighter set: peak flow −15% to +25%, volume ±20%, peak depth −0.1 m to +0.5 m, of the kind applied to dedicated flow-survey verification.
Both are editable, and every band is printed in the pack beside the result it judged. Editing any number switches the preset to Custom.
The depth band is absolute, not a percentage: half a metre is half a metre whether the chamber runs at 0.4 m or 4 m. It is also asymmetric, like the flow band - over-predicting depth is the conservative error, under-predicting it hides a flood.
Things it will tell you rather than hide
Section titled “Things it will tell you rather than hide”- A series stored as a preview rather than a full import is flagged, because statistics from a fraction of a record should not reach a deliverable unmarked.
- A gauge whose record does not overlap the simulated period is reported as such, not scored as zero.
- An observed series that does not resolve to an element is named.
- A model that has never been run still gets its build and data sections, with everything downstream marked not evaluated.
Run simulation is in the panel footer, so you can re-run and watch the pack recompute without leaving it. Editing a criterion recomputes the grades immediately, with no re-run needed.
Requires a Pro plan.
Related
Section titled “Related”- Calibration · tuning the model the pack then evidences
- Auditing a model · the measured-against-assumed input