Quality Indicators Dictionary and Markup Language - QualityML

The scenario of rapidly growing geodata catalogues requires tools focused on facilitate users the choice of products. Having quality fields populated in metadata allow the users to rank and then select the best fit-for-purpose products. In this direction, QualityML is a dictionary that contains hierarchically structured concepts to precisely define and relate quality levels: from quality classes to quality measurements. This levels are used to encode quality semantics for geospatial data by mapping them to the corresponding metadata schemas. The benefits of having encoded quality semantics, in the case of data producers, are related with improvements in their product discovery and better transmission of their characteristics. In the case of data users, they would better compare quality and uncertainty measures to take the best selection of data as well as to perform dataset intercomparison. Also it allows other components (such as visualization, discovery, or comparison tools) to be quality-aware and interoperable.

On one hand, the QualityML is a profile of the ISO geospatial metadata standards (e.g. ISO 19157) providing a set of rules for precisely documenting quality measure parameters that is structured in 5 levels. On the other hand, QualityML includes semantics and vocabularies for the quality concepts. Whenever possible, it uses statistic expressions from the UncertML dictionary (http://www.uncertml.org) encoding. However it also extends UncertML to provide a list of alternative metrics that are commonly used to quantify quality beyond the uncertainty concept.

Finally, keep in mind that QualityML is not just suitable for encoding geospatial dataset level quality but also considers pixel and object level uncertainties. This is done by linking the metadata quality descriptions with layers representing not just the data but the uncertainty values associated with each geospatial element.

This page is structured in the following sections:

Both metrics and domain have a URI.

All relative URIs in this page refer to http://www.qualityml.org/1.0/

Generic quality element

A Quality element is a combination of a quality class, a quality indicator, a quality domain, a quality metric (which include a metrics name, metrics description, metrics parameters, its values and units of measure). The combination of a quality domain and a quality metrics are commonly known as quality measures. These concepts are mapped to the ISO 19139 and ISO 19115-3 in the following way:

Concept ISO 19139 mapping ISO 19115-3 mapping Example
Quality class and
quality indicator
Name of the DQ_Element Name of the DQ_Element ISO 19139: gmd:DQ_CompletenessComission
ISO 19115-3: mdq:DQ_CompletenessOmission
Quality measure name gmd:nameOfMeasure mdq:nameOfMeasure Excess
Quality measure
identification
gmd:measureIdentification/gmd:MD_Identifier/
gmd:code/gmx:Anchor@xlink:href
mdq:measureIdentification/mcc:MD_Identifier/mcc:code/gcx:Anchor@xlink:href
mdq:measureIdentification/mcc:MD_Identifier/mcc:codeSpace/gcx:Anchor@xlink:href
mdq:measureIdentification/mcc:MD_Identifier/mcc:version
http://qualityml.geoviqua.org/1.0/measure/Excess
http://www.qualityml.org (Only in ISO 19115-3)
1.0 (Only in ISO 19115-3)
Quality measure description gmd:measureDescription mdq:measureDescription Indication of elements within the dataset or sample that should not have been present.
The conformance or non-conformance can be expressed as a boolean, count or rate.
Quality domain gmd:value/gco:Record/* mdq:value/gco:Record/* qmld:NonConformance
Quality domain
parameters
gmd:value/gco:Record/qmld:NonConformance/*
qmld:range/qmld:min and/or qmld:range/qmld:max and/or qmld:rule
mdq:value/gco:Record/qmld:NonConformance/*
qmld:range/qmld:min and/or qmld:range/qmld:max and/or qmld:rule
qmld:rule: Indication of excess items
Usually parameters for the domain are not needed
Metrics description gmd:valueType/gco:RecordType mdq:valueRecordType/gco:RecordType Excess items
Metrics identifier gmd:valueType/gco:RecordType@xlink:href mdq:valueRecordType/gco:RecordType@xlink:href http://www.qualityml.org/metrics/items
Metrics parameters gmd:value/gco:Record/qml:Items/* mdq:value/gco:Record/qml:Items/* qml:rate and qml:max
"qml:indicator" or "qml:count" are also options
Metrics values gmd:value/gco:Record/qml:Items/qml:rate
gmd:value/gco:Record/qml:Items/qml:rate@max
mdq:value/gco:Record/qml:Items/qml:rate
mdq:value/gco:Record/qml:Items/qml:rate@max
66
100
Units of measure gmd:valueUnit@xlink:href mdq:valueUnit@xlink:href urn:ogc:def:uom:OGC:1.0:percent

The section about XML encodings provides some examples on how to encode the QualityML concepts in ISO 19139 or 19115-3 documents.

Quality classes

The ISO 19157:2013 Geographic information - Data quality defines 7 data quality elements (or classes) describing a certain aspect of the quality of geographic data in which quality elements can be classified.

Name Description
Completeness Presence and absence of features, their attributes and relationships
Logical consistency Degree of adherence to logical rules of data structure, attribution and relationships (data structure can be conceptual, logical or physical)
Positional accuracy Accuracy of the position of features within a spatial reference system
Thematic accuracy Accuracy of quantitative attributes and the correctness of non-quantitative attributes and of the classifications of features and their relationships
Temporal quality Accuracy of the temporal attributes and temporal relationships of features
Usability element Usability is based on user requirements. All quality elements may be used to evaluate usability
Metaquality Metaquality elements are a set of quantitative and qualitative statements about a quality evaluation and its result

Quality indicators

The ISO 19157:2013 Geographic information - Data quality defines 18 quality indicators (or sub-classes) in which quality elements can be classified.

Quality class Quality indicator Description
Completeness Commission Excess data present in a dataset
Completeness Omission Data absent from a dataset
Logical consistency Conceptual consistency Adherence to rules of the conceptual schema
Logical consistency Domain consistency Adherence of values to the value domains
Logical consistency Format consistency Degree to which data is stored in accordance with the physical structure of the dataset
Logical consistency Topological consistency Correctness of the explicitly encoded topological characteristics of a dataset
Positional accuracy Absolute or external accuracy Closeness of reported coordinate values to values accepted as or being true
Positional accuracy Relative or internal accuracy Closeness of the relative positions of features in a dataset to their respective relative positions accepted as or being true
Positional accuracy Gridded data positional accuracy Closeness of gridded data spatial position values to values accepted as or being true
Thematic accuracy Classification correctness Comparison of the classes assigned to features or their attributes to a universe of discourse (e.g. ground truth or reference data)
Thematic accuracy Non-quantitative attribute correctness Measure of whether a non-quantitative attribute is correct or incorrect
Thematic accuracy Quantitative attribute accuracy Closeness of the value of a quantitative attribute to a value accepted as or known to be true
Temporal quality Accuracy of a time measurement Closeness of reported time measurements to values accepted as or known to be true
Temporal quality Temporal consistency Correctness of the order of events
Temporal quality Temporal validity Validity of data with respect to time
Metaquality Confidence Trustworthiness of a data quality result
Metaquality Representativity Degree to which the sample used has produced a result which is representative of the data within the data quality scope
Metaquality Homogeneity expected or tested uniformity of the results obtained for a data quality evaluation

Quality measures matrices

We provide a matrix of the combinations of indicators, measures and metrics commonly used for each quality class. Both ISO 19157 and ISO 19138 have a list of metrics in the Annex D and share the same identifier (last field in the tables). Actually ISO 19157 supersedes ISO 19138 and contains all metrics in ISO 19138 but adds the new Id. 119 (Physical structure conflicts), Id. 128 (Bias of positions), Id. 159 (Chronological error) and Ids. 101 to 104 (Aggregation Measures).

Quality class: Completeness

Quality class Quality indicator gmd:DQ_Element Quality measure Domain Metrics Observations Origin
Completeness Commission gmd:DQ_CompletenessCommission

Excess

measure/Excess

domain/NonConformance

metrics/items   ISO 19157:
Id. 1 (boolean), Id. 2 (count), Id. 3 (rate)
Completeness Commission gmd:DQ_CompletenessCommission Duplicate

measure/Duplicate

domain/NonConformance metrics/items   GVQ: boolean, rate
ISO 19157: Id. 4 (count)
Completeness Commission gmd:DQ_CompletenessCommission Misclassification

measure/Misclassification

domain/predictedValues
domain/actualValues
metrics/CommissionError
metrics/FalsePositive
metrics/FalsePositiveRate

 

A3C
Completeness Omission gmd:DQ_CompletenessOmission MissingItems

measure/MissingItems

domain/NonConformance metrics/items   ISO 19157:
Id. 5 (boolean), Id. 6 (count), Id. 7 (rate)
Completeness Omission gmd:DQ_CompletenessOmission MissingClass

measure/MissingClass

  metrics/items   A3C
Completeness Omission gmd:DQ_CompletenessOmission Nodata areas
measure/NodataAreas
  metrics/items   A3C
Completeness Omission gmd:DQ_CompletenessOmission Flag areas
measure/FlagAreas
(used to flag elements that are detected as anomalous such as "cloud flag" or "snow flag")
  metrics/items   GVQ
Completeness Omission gmd:DQ_CompletenessOmission Misclassification

measure/Misclassification

domain/predictedValues
domain/actualValues
metrics/OmissionError
metrics/FalseNegative

 

A3C

Quality class: Logical consistency

Quality class Quality indicator gmd:DQ_Element Quality measure Domain Metrics Observations Origin
Logical consistency Conceptual consistency gmd:DQ_ConceptualConsistency Conceptual schema

measure/ConceptualSchema

domain/Conformance

domain/NonConformance

metrics/items   ISO 19157: Id. 8 + Id. 9 (boolean), Id. 10 + GVQ (count), Id. 12 + Id. 13 (rate)
Logical consistency Conceptual consistency gmd:DQ_ConceptualConsistency Overlaps of surfaces

measure/InvalidOverlapsSurfaces

domain/NonConformance metrics/items in relation to the overall number of overlap surfaces GVQ (boolean, rate), ISO 19157 Id. 11 (count)
Logical consistency Domain consistency gmd:DQ_DomainConsistency

Value domain

measure/ValueDomain

domain/Conformance

domain/NonConformance

metrics/items   ISO 19157 Id. 14 + Id. 15 (boolean), Id. 16 + GVQ (count), Id. 17 + Id. 18 (rate)
Logical consistency Format consistency gmd:DQ_FormatConsistency Physical structure conflicts

measure/PhysicalStructureConflicts

domain/NonConformance metrics/items   GVQ (boolean), ISO 19157 Id. 119 (boolean), Id. 19 (count), Id. 20 (rate)
Logical consistency Topological consistency gmd:DQ_TopologicalConsistency Faulty point-curve connections

measure/FaultyPoint-curveConnections

domain/NonConformance metrics/items in relation to the total number of point-curve connections GVQ (boolean), ISO 19157 Id. 21 (count), Id. 22 (rate)
Logical consistency Topological consistency gmd:DQ_TopologicalConsistency Missing connections due to undershoots

measure/MissingConnectionsDueUndershoots

domain/NonConformance metrics/items in relation to the total number of undershoots GVQ (boolean, rate), ISO 19157 Id. 23 (count)
Logical consistency Topological consistency gmd:DQ_TopologicalConsistency Missing connections due to overshoots

measure/MissingConnectionsDueOvershoots

domain/NonConformance metrics/items in relation to the total number of overshoots GVQ (boolean, rate), ISO 19157 Id. 24 (count)
Logical consistency Topological consistency gmd:DQ_TopologicalConsistency Slivers

measure/InvalidSlivers

domain/NonConformance metrics/items in relation to the total number of slivers GVQ (boolean, rate), ISO 19157 Id. 25 (count)
Logical consistency Topological consistency gmd:DQ_TopologicalConsistency Self intersects

measure/InvalidSelfIntersects

domain/NonConformance metrics/items in relation to the total number of intserctions GVQ (boolean, rate), ISO 19157 Id. 26 (count)
Logical consistency Topological consistency gmd:DQ_TopologicalConsistency Self overlaps

measure/InvalidSelfOverlaps

domain/NonConformance metrics/items in relation to the total number of overlaps GVQ (boolean, rate), ISO 19157 Id. 27 (count)

Quality class: Positional accuracy

Quality class Quality indicator gmd:DQ_Element Quality measure Domain Metrics Observations Origin
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Mean Absolute Error (MAE)

measure/MeanAbsoluteError

domain/DifferentialErrorsX, domain/DifferentialErrorsY
domain/DifferentialErrors2D
domain/DifferentialErrors3D
metrics/MeanAbsolute
metrics/MeanAbsolute2D
metrics/MeanAbsolute3D

this concept is called "uncertainty" in ISO 19157

ISO 19157 Id. 28
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Mean Absolute Error excluding outliers

measure/MeanAbsoluteErrorExcludingOutliers

domain/DifferentialErrorsXThreshold, domain/DifferentialErrorsYThreshold
domain/DifferentialErrors2DThreshold
domain/DifferentialErrors3DThreshold
(level="below")
metrics/MeanAbsolute
metrics/MeanAbsolute2D
metrics/MeanAbsolute3D

 : ISO 19157 Id. 29
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Bias of positions

measure/BiasOfPositions

domain/DifferentialErrorsX, domain/DifferentialErrorsY
domain/DifferentialErrors2D
domain/DifferentialErrors3D

metrics/MeanBias

  ISO 19157 Id. 128
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Uncertainties above a given threshold

measure/UncertaintiesAboveGivenThreshold

domain/DifferentialErrorsXThreshold, domain/DifferentialErrorsYThreshold
domain/DifferentialErrors2DThreshold
domain/DifferentialErrors3DThreshold
(level="above")

metrics/items

this concept is called "uncertainty" in ISO 19157

ISO 19157 Id. 30, Id. 31
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Mean Bias Error (MBE)

measure/MeanBiasError

domain/DifferentialErrorsX

domain/DifferentialErrorsY

metrics/NormalizedMeanBias

 
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy
Covariance Matrix

measure/CovarianceMatrix

domain/predictedValues
domain/actualValues
(when applied to 2D, 3D, etc, the vector-value random variables are formed by a list starting for the coordinate components of the first coordinate, the coordinate components of the second coordinate and so on)
http://www.uncertml.org/statistics/covariance-matrix   ISO 19157 Id. 32
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy
Confidence ellipse

measure/ConfidenceEllipse

domain/predictedValues
domain/actualValue
(applied to 2D)

metrics/ConfidenceEllipse

confidence: uncertainty ellipse (68.3% --> 1), confidence at 95%--> 5.99, confidence at 99%--> 9.21)

ISO 19157 Id. 50, Id. 51
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Linear Map Accuracy

measure/LinearMapAccuracy

domain/DifferentialErrorsX

domain/DifferentialErrorsY

metrics/Half-lengthConfidenceInterval Common confidence intervals: 50%, 68.3%, 90%, 95%, 99%, 99.8% ISO 19157 Id. 33, Id. 34, Id. 35, Id. 36, Id. 37, Id. 38
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Root mean square

measure/RootMeanSquare

domain/DifferentialErrorsX

domain/DifferentialErrorsY

metrics/RootMeanSquareError

metrics/NormalizedRootMeanSquareError

metrics/CoefficientOfVariationRootMeanSquareError

ISO 19157 Id. 39
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Root mean square

measure/RootMeanSquare

domain/DifferentialErrors2D

metrics/RootMeanSquareError2D

ISO 19157 Id. 47
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Root mean square

measure/RootMeanSquare

domain/DifferentialErrors3D

metrics/RootMeanSquareError3D

 
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Absolute linear error at 90% significance level. Alternative1

measure/AbsoluteLinearError90_SignificanceLevel_Alternative1

domain/DifferentialErrorsX

domain/DifferentialErrorsY

metrics/LMAS   ISO 19157 Id. 40
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Absolute linear error at 90% significance level. Alternative2

measure/AbsoluteLinearError90_SignificanceLevel_Alternative2

domain/DifferentialErrorsX

domain/DifferentialErrorsY

metrics/ALE  

ISO 19157 Id. 41

Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Circular Map Accuracy

measure/CircularMapAccuracy

domain/DifferentialErrors2D metrics/CircularError Common confidence intervals: 39.4%, 50%, 90%, 95%, 99.8% ISO 19157 Id. 42 (39.4%), Id. 43 (50%), Id. 44 (90%), Id. 45 (95%), Id. 46 (99.8%)
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Absolute circular error at 90% significance level. Alternative1

measure/AbsoluteCircularError90_SignificanceLevel_Alternative1

domain/DifferentialErrors2D metrics/ACE   ISO 19157 Id. 48
Positional accuracy Absolute or external accuracy gmd:DQ_AbsoluteExternalPositionalAccuracy
gmd:DQ_GriddedDataPositionalAccuracy

Absolute circular error at 90% significance level. Alternative2

measure/AbsoluteCircularError90_SignificanceLevel_Alternative2

domain/DifferentialErrors2D metrics/CMAS   ISO 19157 Id. 49
Positional accuracy Relative or internal accuracy gmd:DQ_RelativeInternalPositionalAccuracy

Relative Error

measure/RelativeError

domain/DifferentialErrorsX

domain/DifferentialErrorsY

metrics/RelativeError confidence 68.3% --> 1, 90%-->1.645  
Positional accuracy Relative or internal accuracy gmd:DQ_RelativeInternalPositionalAccuracy

Relative Error

measure/RelativeError

domain/DifferentialErrors2D metrics/RelativeError2D confidence 68.3% --> 1, 90%-->2.146 ISO 19157 Id. 53
Please note that ISO 19157 Id. 52 has not been included in this table yet since is defined "vertical".

Quality class: Temporal quality

Quality class Quality indicator gmd:DQ_Element Quality measure Domain Metrics Observations Origin
Temporal quality Accuracy of a time measurement gmd:DQ_AccuracyOfATimeMeasurement Time accuracy

measure/TimeAccuracy

domain/DifferentialErrors1D

metrics/Half-lengthConfidenceInterval Common confidence intervals: 50%, 68.3%, 90%, 95%, 99%, 99.8% ISO 19157 Id. 54, Id. 55, Id. 56, Id. 57, Id. 58, Id. 59
Temporal quality Temporal validity gmd:DQ_TemporalValidity

Value domain

measure/ValueDomain

domain/Conformance
domain/NonConformance
metrics/items   ISO 19157 Id. 14 + Id.15 (boolean), Id.16 + GVQ (count), Id.17 + Id.18 (rate)
Temporal quality Temporal consistency gmd:DQ_TemporalConsistency

Chronological Error
measure/ChronologicalError

TBD TBD   ISO 19157 Id. 159

Quality class: Thematic accuracy

Quality class Quality indicator gmd:DQ_Element Quality measure Domain Metrics Observations Origin
Thematic accuracy Classification correctness gmd:DQ_ThematicClassificationCorrectness Misclassification

measure/Misclassification

domain/predictedValues
domain/actualValues
metrics/items
http://www.uncertml.org/statistics/confusion-matrix
metricsAggregation\ConfusionMatrixDerivates
metrics/ConfusionMatrix
metrics/RelativeConfusionMatrix
metrics/KappaCoefficient
metrics/OmissionError
metrics/Reliability
metrics/AverageReliability
metrics/CommissionError
metrics/Accuracy
metrics/AverageAccuracy
metrics/OverallAccuracy

metricsAggregation\DiscreteClassification
metrics/DiscreteConfusionMatrix
metrics/Sensitivity
metrics/Specificity
metrics/TruePositive
metrics/TrueNegative
metrics/FalsePositive
metrics/FalseNegative
metrics/FalsePositiveRate
metrics/PositivePredictiveValue
metrics/FalseDiscoveryRate
metrics/MatthewsCorrelationCoefficient
metrics/AreaUnderROCCurve

 

GVQ (boolean), ISO 19157 Id. 60 (count), Id. 61 (rate), Id. 62 (confusion matrix, count), Id. 63 (confusion matrix, rate), Id. 64 (Kappa coefficient)
Thematic accuracy Non-quantitative attribute correctness gmd:DQ_NonQuantitativeAttributeAccuracy

Categorical attributes

measure/ValueDomain

domain/Conformance
domain/NonConformance

metrics/items

 

GVQ (boolean), ISO 19157 Id. 65 (count), Id. 67 + Id. 66 (rate)

Thematic accuracy Non-quantitative attribute correctness gmd:DQ_NonQuantitativeAttributeAccuracy

Categorical attribute proportions

measure/ValueDomain
List of attributes assigned to a geometric object (e.g. polygon)

domain/predictedValues

metrics/items

metrics/Purity

 

GVQ

Thematic accuracy Non-quantitative attribute correctness gmd:DQ_NonQuantitativeAttributeAccuracy

Categorical misclassification

measure/CategoricalCorrespondence

domain/predictedValues
domain/actualValues

metrics/CorrespondenceMatrix

metrics/RelativeCorrespondenceMatrix

metrics/NumberMajorCategories

metrics/Promiscuity

A Remote sensing hybrid classification method uses the Fidelity matrix to assigns a spectral class to the thematic class which the highest fidelity. A fidelity and representative thresholds may be defined; therefore items in spectral classes in which all fidelities are is below a threshold will not be assigned to any thematic class and therefore resulting in non-classified items (NODATA pixels) and thematic classes in which all representatibities are is below a threshold are discarded.

GVQ

Thematic accuracy Quantitative attribute correctness gmd:DQ_QuantitativeAttributeAccuracy

Root mean square

measure/RootMeanSquare

domain/DifferentialErrors1D

metrics/RootMeanSquareError

metrics/NormalizedRootMeanSquareError

metrics/CoefficientOfVariationRootMeanSquareError

   
Thematic accuracy Quantitative attribute correctness gmd:DQ_QuantitativeAttributeAccuracy

Mean Absolute Error (MAE)

measure/MeanAbsoluteError

domain/DifferentialErrors1D

metrics/MeanAbsolute

   
Thematic accuracy Quantitative attribute correctness gmd:DQ_QuantitativeAttributeAccuracy

Mean Bias Error (MBE)

measure/MeanBiasError

domain/DifferentialErrors1D

metrics/NormalizedMeanBias

   
Thematic accuracy Quantitative attribute correctness gmd:DQ_QuantitativeAttributeAccuracy Quantitative attribute correctness

measure/QuantitativeAttributeCorrectness

domain/DifferentialErrors1D

metrics/Half-lengthConfidenceInterval
http://www.uncertml.org/statistics/correlation (also known as Pearson Coefficient r)
metrics/CoefficientOfDetermination

For the Half-lengthConfidenceInterval the common confidence intervals are: 50%, 68.3%, 90%, 95%, 99%, 99.8% ISO 19157 Id. 68, Id. 69, Id. 70, Id. 71, Id. 72, Id. 73
Thematic accuracy Quantitative attribute correctness gmd:DQ_QuantitativeAttributeAccuracy SubSetting sampling stability

measure/SubSettingSamplingStability

http://www.uncertml.org/samples/random

http://www.uncertml.org/statistics/variance

Climatic Atlas Of the Iberian Peninsula
Thematic accuracy Quantitative attribute correctness gmd:DQ_QuantitativeAttributeAccuracy Extrapolated Areas
measure/ExtrapolatedAreas
  metrics/items   GVQ

Quality domain

This section contains the domains where metrics can be applied.

URI Name Parameters Origin

domain/predictedValues

Predicted or observed values

threshold

level (select of: "above", "below")

Formula a la D.28. Copiar a la fitxa

ISO 19157

domain/actualValues

Actual Values in the ground truth

threshold

level (select of: "above", "below")

ISO 19157
domain/DifferentialErrors1D 1D Differential error   ISO 19157
domain/DifferentialErrorsX 1D Differential error, X   ISO 19157
domain/DifferentialErrorsY 1D Differential error, Y   ISO 19157
domain/DifferentialErrorsXAboveThreshold 1D Differential Error Measure above a threshold, X

level (select of: "above", "below")

threshold

ISO 19157
domain/DifferentialErrorsYAboveThreshold 1D Differential Error Measure above a threshold, Y

level (select of: "above", "below")

threshold

ISO 19157
domain/DifferentialErrors2D 2D Differential error

 

ISO 19157
domain/DifferentialErrors2DAboveThreshold 2D Differential Error Measure above a threshold

level (select of: "above", "below")

threshold

ISO 19157
domain/DifferentialErrors3D 3D Differential error   ISO 19157
domain/DifferentialErrors3DAboveThreshold 3D Differential Error Measure above a threshold

threshold

level (select of: "above", "below")

ISO 19157
domain/DiagonalDifferencialError Diagonal Differential Error

 

 

domain/Conformance

Domain correctness range ISO 19157

domain/NonConformance

Domain error range ISO 19157

Quality Metrics

Whenever possible, uncertml statistics are used. When needed new metrics are defined here based on uncertml basic types. In some sense QualityML extends UncertML.

XML schemas and examples are available here (for convinience a exact copy of uncertml is included).

URI Metric Parameters Origin
metrics/items Items

value (choice of: indicator (boolean), count (int), rate (real, attribute: max (real)))

ISO 19157
metrics/Half-lengthConfidenceInterval Half-length of the confidence interval Extends http://www.uncertml.org/statistics/confidence-interval (upper and lower values) adding a half-lenght value ISO 19157
metrics/MeanAbsolute Mean Absolute Error (MAE)   ISO 19157
metrics/MeanAbsolute2D Mean Absolute Error (MAE)   ISO 19157
metrics/MeanAbsolute3D Mean Absolute Error (MAE)   ISO 19157
metrics/MeanBias Mean Bias   ISO 19157
metrics/NormalizedMeanBias Mean Bias Error (MBE)   ISO 19157
metrics/RootMeanSquareError Root mean square error   ISO 19157
metrics/NormalizedRootMeanSquareError Normalized root mean square error   ISO 19157
metrics/CoefficientOfVariationRootMeanSquareError coefficient of variation Root Mean Square Error   ISO 19157
metrics/LMAS Absolute linear error at 90% significance level. Alternative 1   ISO 19157
metrics/ALE Absolute linear error at 90% significance level. Alternative 2   ISO 19157
metrics/ACE Absolute circular error at 90% significance level. Alternative 1   ISO 19157
metrics/CMAS Absolute cirular error at 90% significance level. Alternative 2   ISO 19157
metrics/ConfidenceEllipse Confidence ellipse a, b, angle in Multiple values, confidence ISO 19157
metrics/ConfusionMatrix Confusion matrix   ISO 19157
http://www.uncertml.org/statistics/covariance-matrix Covariance Matrix   ISO 19157
metrics/CorrespondenceMatrix Correspondence Matrix   GVQ
metrics/RelativeCorrespondenceMatrix Relative Correspondence Matrix   GVQ
metrics/RelativeError 1D Relative error confidence ISO 19157
metrics/RelativeError2D 2D Relative error confidence ISO 19157
metrics/NormalizedConfusionMatrix Normalized confusion matrix Inspired in http://www.uncertml.org/statistics/confusion-matrix but using double type for count ISO 19157
metrics/KappaCoefficient Kappa coefficient   ISO 19157

metrics/OmmisionError

Omission Error actual categories GVQ

metrics/CommissionError

Commission Error predicted categories GVQ
metrics/Promiscuity Promiscuity   GVQ
metrics/MajorityCategories Majority Categories   GVQ
metrics/Purity Purity   GVQ
metrics/CoefficientOfDetermination Coefficient of determination   GVQ
metrics/DiscreteConfusionMatrix Discrete Confusion Matrix   GVQ
metrics/TruePositive True Positive   GVQ
metrics/TrueNegative True Negative   GVQ
metrics/FalsePositive False Positive   GVQ
metrics/FalseNegative False Negative   GVQ
metrics/Sensitivity Sensitivity   GVQ
metrics/Specificity Specificity   GVQ
metrics/OverallAccuracy Overall Accuracy   GVQ
metrics/FalsePositiveRate False Positive Rate   GVQ
metrics/PositivePredictiveValue Positive Predictive Value   GVQ
metrics/FalseDiscoveryRate False Discovery Rate   GVQ
metrics/MatthewsCorrelationCoefficient Matthews Correlation Coefficient   GVQ
metrics/AreaUnderROCCurve Area Under ROC Curve   GVQ
metrics/Reliability Reliability   GVQ
metrics/AverageReliability Average Reliability   GVQ
metrics/Accuracy Accuracy   GVQ
metrics/AverageAccuracy Average Accuracy   GVQ

Quality metrics sets

This table provides a way to aggregate some quality metrics in a single element that can group them. This is particularly useful when using several related metrics in the ISO 19139 encoding.
URI
Metrics aggregation Elements
Origin
metricsAggregation\DiscreteClassification
Discrete Classification
metrics\DiscreteConfusionMatrix
metrics\TruePositive
metrics\TrueNegative
metrics\FalsePositive
metrics\FalseNegative
metrics\Sensitivity
metrics\Specificity
metrics\OverallAccuracy
metrics\FalsePositiveRate
metrics\PositivePredictiveValue
metrics\FalseDiscoveryRate
metrics\MatthewsCorrelationCoefficient
metrics\AreaUnderROCCurve
GVQ
metricsAggregation\ConfusionMatrixDerivates
Confusion Matrix Derivates
metrics\ConfusionMatrix
metrics\OmissionError
metrics\CommissionError
metrics\Reliability
metrics\AverageReliability
metrics\Accuracy
metrics\AverageAccuracy
metrics\OverallAccuracy
metrics\KappaCoefficient
GVQ

 

Other reserved words

This table provides a list of keywords that are useful for the WMS-Q specification.
URI
Name
Meaning
values
Values
Parameter or component that more closely represent the actual values measured
qualityCollection
Quality collection
Variable that is decomposed in components or parameters
qualityComposition
Quality composition
Component that represents a composition of other components for visualization purposes

 

XML encoding

Here, some examples on how to encode quality metadata in ISO 19139 metadata documents using the qualityML framework and the qualityML URIs

Example using UncertML in an ISO metadata record

<gmd:MD_Metadata> <md:dataQualityInfo> <gmd:MD_DataQuality> <gmd:report> <gmd:DQ_QuantitativeAttributeAccuracy> <gmd:result> <gmd:DQ_QuantitativeResult> <gmd:valueType> <gco:RecordType xlink:href="http://www.uncertml.org/distributions/normal"> Value for vertical DEM accuracy </gco:RecordType> </gmd:valueType> <gmd:valueUnit xlink:href="urn:ogc:def:uom:OGC:1.0:metre"/> <gmd:value> <gco:Record> <un:NormalDistribution> <un:mean>1.2</un:mean> <un:variance>3.0625</un:variance> </un:NormalDistribution> </gco:Record> </gmd:value> </gmd:DQ_QuantitativeResult> </gmd:result> </gmd:DQ_QuantitativeAttributeAccuracy> </gmd:report> </gmd:MD_DataQuality> </md:dataQualityInfo> </gmd:MD_Metadata>

Example using QualityML in an ISO metadata record for dataset level quality

<gmd:MD_Metadata> <md:dataQualityInfo> <gmd:MD_DataQuality> <gmd:report> <gmd:DQ_CompletenessOmission> <gmd:nameOfMeasure> <gco:CharacterString> Missing </gco:CharacterString> </gmd:nameOfMeasure> <gmd:measureIdentification> <gmd:MD_Identifier> <gmd:code> <gco:CharacterString> http://www.qualityml.org/1.0/measure/Missing </gco:CharacterString> </gmd:code> </gmd:MD_Identifier> </gmd:measureIdentification> <gmd:result> <gmd:DQ_QuantitativeResult> <gmd:valueType> <gco:RecordType xlink:href="http://www.qualityml.org/1.0/metrics/items"> Rate of missing items </gco:RecordType> </gmd:valueType> <gmd:valueUnit/> <gmd:value> <gco:Record> <qml:Items> <qml:rate max="100">3</qml:rate> </qml:Items> </gco:Record> </gmd:value> </gmd:DQ_QuantitativeResult> </gmd:result> </gmd:DQ_CompletenessOmission> </gmd:report> </gmd:MD_DataQuality> </md:dataQualityInfo> </gmd:MD_Metadata>

Example using QualityML in an ISO metadata record pixel level quality

<gmd:MD_Metadata> <gmd:distributionInfo> <gmd:MD_Distribution> <gmd:transferOptions> <gmd:MD_DigitalTransferOptions id="TemperatureConfidenceInterval"> <gmd:onLine> <gmd:CI_OnlineResource> <!-- Any service that allow to retrieve the data values or a visualization of them --> <gmd:linkage> <gmd:URL>http://www.server.bob/wms.cgi?REQUEST=GetMap&amp;VERSION=1.3.0&amp;SERVICE=WMS&amp; LAYERS=TemperatureConfidenceInterval&amp;STYLES=</gmd:URL> </gmd:linkage> <gmd:protocol> <gco:CharacterString>serviceType:WebMapService:1.3.0:HTTP-GET</gco:CharacterString> </gmd:protocol> <gmd:name> <gco:CharacterString>TemperatureConfidenceInterval</gco:CharacterString> </gmd:name> </gmd:CI_OnlineResource> </gmd:onLine> </gmd:MD_DigitalTransferOptions> </gmd:transferOptions> </gmd:MD_Distribution> </gmd:distributionInfo> <gmd:dataQualityInfo> <gmd:DQ_DataQuality> <gmd:report> <gmd:DQ_QuantitativeAttributeAccuracy> <gmd:nameOfMeasure> <gco:CharacterString> Quantitative Attribute Correctness </gco:CharacterString> </gmd:nameOfMeasure> <gmd:measureIdentification> <gmd:MD_Identifier> <gmd:code> <gco:CharacterString> http://www.qualityml.org/1.0/measure/QuantitativeAttributeCorrectness </gco:CharacterString> </gmd:code> </gmd:MD_Identifier> </gmd:measureIdentification> <gmd:result> <gmd:DQ_QuantitativeResult> <gmd:valueType> <gco:RecordType xlink:href="http://www.qualityml.org/1.0/metrics/metrics/Half-lengthConfidenceInterval"> Half-length Confidence Interval </gco:RecordType> </gmd:valueType> <gmd:valueUnit/> <gmd:value> <gco:Record> <qml:HalfLengthConfidenceInterval level="0.683"> <un:values href="#TemperatureConfidenceInterval"/> </qml:HalfLengthConfidenceInterval> </gco:Record> </gmd:value> </gmd:DQ_QuantitativeResult> </gmd:result> </gmd:DQ_QuantitativeAttributeAccuracy> </gmd:report> </gmd:DQ_DataQuality> </gmd:dataQualityInfo> </gmd:MD_Metadata>

The research leading to these results has been carried out in the GeoViQua project that has received funding from the European Union Seventh Framework Programme (FP7/2010-2013) under grant agreement no. 265178