Air quality sensor performance: how does lack of standards influence?

Table of Contents

Reading time 11 min

Download the complete report of the
Co-location field study in Barcelona

Large cities in developed countries are usually dotted with a network of reference monitoring stations. However, it means a high cost of installation and maintenance, providing accurate data but only in few locations. On the other hand, this network is not possible in smaller cities or underdeveloped regions.

Hence, air quality monitoring using another type of equipment is necessary. One solution is to integrate air pollution sensors in a device, that takes the raw data and convert them in useful information for the end-user. Those air quality devices are lower in costs, easier to use and less bulky than traditional equipment. They provide the possibility for citizens and communities to monitor in real-time their local air quality that may affect their health.

However, nowadays, there is not enough information on how to test these types of devices to ensure their adequate performance. As it is explained by EPA in Peer Review and Supporting Literature Review of Air Sensors Technology Performance Targets document, “The lack of accepted performance specifications for air sensors is limiting the understanding of the quality of the data produced with this technology and is leading to confusion in the marketplace, as the buyers are uncertain of how well air sensors currently perform, how to operate them, and how well sensors need to perform to be suitable for a given purpose”.

In this context and over the past decade, there has been a worldwide effort to evaluate the usefulness and possible applications for air quality device technology. Performance evaluation projects have focused on determining the quality of the data produced by the sensor systems. The methodology used, comparing their response to reference instruments in the laboratory and in the field. Several independent and foundational evaluation efforts are occurring in Europe and U.S. This is the case of the U.S. Environmental Protection Agency (EPA), the South Coast Air Quality Management District, which started the Air Quality sensor performance evaluation center (AQ-SPEC) program, and the AIRLAB Microsensor Challenge organized by AIRPARIF.

There is not enough information on how to test these types of devices to ensure their adequate performance.

Moreover, two different air quality sensor performance standards are identified. In the case of Europe, the CEN Technical Committee 264, Working Group 42 is currently developing Technical Specification for the performance requirements and test methods for low-cost sensors under prescribed laboratory and field conditions. On the other hand, China’s Ministry of Environmental Protection Department of Hebei province has developed a generic performance standard for sensors.

Hereinafter, we present some real study cases results as well as three external evaluation processes carried out by different organizations:

Real Case Studies and Evaluation processes

In the study cases, R² value is included for each pollutant. It is a partial measure of how much air quality data agree with reference measurements according to a regression model. We decided to use the R² since it is the most common metric used in the evaluations. However, this metric has some limitations. The data need to be homogeneous, following a normal distribution, which is not always the case in real studies. It could improve when the range of the reference measurements increases or depending on the seasonality of sampling regarding the different studies.

Finally, R² is insensitive to bias between the air quality device and the reference data. Thus, we have included the Mean Absolute Error (MAE) in our studies, to show the accuracy of the results.

Co-location field study in Sabadell and Barcelona (Spain)

It is recommended to carry out evaluations used co-locations alongside reference instruments as a means to evaluate performance (Figure 1).

The device is placed in the field near a reference instrument for a period to provide a direct comparison of the device outputs to a calibrated reference instrument. Unfortunately, it is a challenge to observe the entire dynamic target gases, cross-sensitive pollutants, and environmental parameters.

Sabadell unit was co-located with a reference station from the Sabadell city, while Barcelona unit was co-located with the CSIC (Consejo Superior de Investigaciones Científicas) reference station.

Download the complete report of the
Co-location field study in Barcelona

DOWNLOAD

Co-location field study in Mexico City (Mexico)

World Athletics started in 2018 to create a real-time air quality network in stadiums to monitor air pollutant concentrations and help athletes choose the best times to train and compete, and help organizers to protect the health of athletes. In the long-term, and working with United Nations (UN), NGOs and governments, the project aims to raise awareness on how air quality impacts people’s quality of life.

Additionally, the World Athletics Health and Science Department is evaluating the correlation between air quality and athletic performance. As part of the World Athletics program, the athletics track at the Mexican Olympic Sports Center (CDOM) had an air quality monitor installed in January 2019.

Before Mexico City, the majority of the pilot project had been conducted in stadiums in Monaco, Addis Ababa, Sydney, and Yokohama.

Kunak Air A-10 obtained high R² values regarding the different gas pollutants and PMs in the three real case studies, R²> 0.85 for gases and R²> 0.75 for particles. The exception is NO₂ in Mexico City study, due to the large humidity transients (from 80% to 40%) at high temperatures (>20°C) (Figure 2).

This humidity transient issue is more significant in lower concentrations of NO₂, as it occurs during summer.

Nevertheless, the daily, weekly and monthly trends are perfectly tracked, with similar average concentrations.

EPA Wildland Fire Sensor Challenge

In 2017, EPA and five other federal agencies issued a Wildland Fire Sensors Challenge to improve smoke monitoring and provide data to protect public health.

Sensor developers and researchers were encouraged to develop new and innovative air sensor monitoring technologies to measure air pollutants from smoke during wildland fires. Smoke from fires is harmful to health that can irritate the eyes, nose, and throat, cause persistent coughing, wheezing and difficulty breathing and worsen heart and lung disease.

Comparative results of the field studies and evaluation processes:

The results at the laboratory reached high R² values, proving that in controlled laboratory conditions, Kunak Air10 performance is optimal. This is observed in Figure 3, in the case of the CO at different temperatures and humidity values.

AQ-SPEC. Air Quality sensor performance evaluation center

AQ-SPEC program aims at performing a thorough characterization of currently available “low-cost” sensors under ambient (field) and controlled (laboratory) conditions.

The main objectives are: evaluate the performance of commercially available “low-cost” air quality sensors in both field and laboratory settings, provide guidance and clarity for ever-evolving sensor technology and data interpretation and catalyze the successful evolution, development, and use of sensor technology.

Comparative results of the field studies and evaluation processes:

The results obtained in AQ-SPEC evaluation are good, with low Relative Intramodel Variabilities, even though the devices were testing at high temperatures, in which their performance is usually of poorer quality. In this specific case, the R² metric limitation is observed, in which the NO₂ concentrations are much lower during summer, leading to a poorer correlation, where most of the values are close to cero. Besides, the minimum value reached by the reference method is not cero, as the values obtained from Kunak Air, see Figure 4, leading again to a poorer R² result.

Apart from this, Kunak Air A-10 obtained the best PM₁₀ results despite the high humidity, when comparing with other devices tested at AQ-SPEC.

Finally, the Kunak Air devices included meteorological data, such as the temperature, humidity and pressure sensors, as well as an anemometer.

AIRLAB Microsensor Challenge Edition 2019

With the Challenge «AIRLAB microsensors», the objective is to pursue independently the evaluation and comparison of microsensors to illuminate users between the suitability of the product and the possible uses. This challenge highlights the qualities of these devices and allows, in a global way, an inventory of the devices, in order to promote innovation and even technological breaks in this field.

Comparative results of the field studies and evaluation processes:

Regarding the Jury’s opinion, “Kunak Air is a multi-pollutant solution for monitoring outdoor air quality. Its very professional design provides a large list of measured pollutants (NO₂, O₃, PM₁, PM_2.5, and PM₁₀), and weather parameters (temperature, humidity, pressure, and wind). The data quality is excellent for O₃, good for PM₁ and NO₂, average for PM₁₀ and very poor for PM_2.5. Its integrated solar panel allows it to run autonomously. Although not cheap, its price is relative competitive for the monitoring category.”

Even though some of the pollutants’ performances were not the best, as it occurred with PM_2.5, in general, the accuracy is good (Figure 5) . Regarding the PMs, it is observed how in other studies Kunak Air achieved higher accuracies. Besides, Kunak Air device was provided with NO and CO sensors, even though they could not be tested by the challenge, as well as an anemometer, being the only device in the whole challenge with this instrument, which it is very useful in order to have knowledge of the source of the pollutants. Finally, Kunak Air had the most competitive price, being provided by 4 gas sensors, an OPC, meteorological sensors (temperature, humidity, and pressure) an anemometer, as well as access to the Cloud.

Conclusion

Air quality devices are not a direct substitute for existing reference instruments or networks, but they are appropriate for other applications, i.e. research, supplementary existing monitoring data, source identification, and characterization, etc.

On the other hand, and as mention before, reference stations are not available in small cities, underdeveloped regions or in specific areas, such as close to schools, hospitals, urban areas close to industry, natural ecosystems at risk, etc.

Air quality devices are not currently a direct substitute for existing reference instruments or networks, but they are appropriate for other applications.

In this context, it is shown how Kunak Air devices have high accuracy regarding the different studies and under different conditions, being aware of the problematic of the temperature and humidity transients which have significant effects on these types of sensors, especially in the case of NO₂ sensor. Nevertheless, NO₂ pollution episodes occur during winter, in which higher concentrations of this pollutant appear in cities. In this case, NO₂ concentration values can be measured accurately, as it is observed in Sabadell and Barcelona field studies.

Nowadays, there exist several external evaluation methods to assure the performance of different air quality monitoring devices. Still, there is a lack of a normalization of the external evaluations and some aspects need to be improved, especially in field evaluations. During the article, we have shown that there are no common criteria in external evaluations. These criteria are hard to understand or even, some of them cannot be comparable within the different air quality solutions in an objective way.

There is a lack of a normalization of the external evaluations and some aspects need to be improved, especially in field evaluations.

For instance, AQ SPEC evaluations are carried out no matter the season, leading to not comparable results within different devices, as it occurs with the Kunak Air results, which were tested during summer under high temperatures and rapid transients of humidity, while other devices are tested during winter under higher pollutant concentrations and lower temperatures. The same occurs in Airlab, in which the criteria used for the assessment were the same for indoor, outdoor and citizen air quality devices, being difficult to be comparable within each other.

As it is exposed in Karagulian, et al. 2019, “In the absence of an internationally accepted standardized protocol for testing LCs (low-cost sensors), there is a lack of harmonization of the tests being carried out. Consequently, the conditions of tests and the metrics reported are generally diverse, making it difficult to compare the performance of LCs in different evaluation studies”.

However, in our opinion, this review does not expose a clear evaluation of LCs either, since they used R² values from different studies carried out in different seasons, eg. summer and winter, with different time periods (weeks vs. months), and different test. For instance, they compare the R² of field tests and laboratory tests. Besides, regarding the prices that are shown in the article, it is not possible to compare a device with open software, which only includes the cost of materials, not including either the cost of support, equipment upgrades or maintenance.

We have created a solution that enable the evaluation of urban measures, making possible to design efficient environmental health policies and give citizens access to quality information.

Hence, we consider that the evaluation methodology should be more precise in terms of periods, ambient conditions, and accuracy assessment. That is why we participate actively in the development of the future EU Technical Specification (WG 42) about “Performance evaluation of air quality sensors”. We, as manufacturers of air quality devices that will be standardized, with more than 5 years of experience, contribute with knowledge, experience in both laboratory and field, as well as being up to date in the latest innovations of the sector. We consider that it is important to promote the development of this standardization. Thus, any manufacturer of air quality devices could improve and innovate in the same terms, creating air quality solutions that can reach everywhere.

Kunak Air solution obtains accurate and respeatable results, regarding different study cases, evaluations and season.

At last, but not least, it is observed how our Kunak Air solution obtains accurate and respeatable results, regarding different study cases, evaluations and season.

Cities, industries and researchers need to make better decisions based on supplementary systems with larger & accurate data.

As we say,

“Large data is the goal,
but accurate data is the key”

Our solution

In this context, we have created air quality control and monitoring end-to-end solutions designed to monitor air quality, noise levels and other ambient parameters in real-time, capturing high-quality data 24 hours a day, 7 days a week.

We design and manufacture the Hardware, Communications, Software, Cloud Services and the integration with the client’s systems. A complete solution that generates high-value information to aid with the design of efficient policies, help evaluate actions and day-to-day management through any Smart City platform.

Specifically, Kunak AIR Pro Air Quality Monitoring System enable the easy monitoring of ambient pollution parameters and therefore, enable the evaluation of urban measures, making it possible to design efficient environmental health policies and give citizens access to quality information from your Website, Smart City platform or Kunak Cloud.

We have created a solution with easy and quick installation, maximum benefits with lower cost, high precision and remote calibration and with an Air Quality Cloud platform that to manage and configure the air quality devices and networks.

Download the complete report of the Co-location field study in Barcelona

Did you like this article?

If you wish to obtain more detailed information the Co-location field study developed in Barcelona, you can download the complete report filling the following form:

Name and Surname *

Email *

Company *

Phone *

Country *

Interests *

I. SMART ENVIROMENT (AQE)

II. UTILITIES & IIOT

III. INDUSTRY 4.0

IV. SMART PRODUCTS

I accept terms and conditions*

I allow Kunak to contact me for further information

Sources consulted

Lung, SC Candice, et al. 2018. “Low-Cost Sensors for the Measurement of Atmospheric Composition: Overview of Topic and Future Applications.” : 1–69.
Karagulian, Federico, et al. 2019. “Review of the Performance of Low-Cost Sensors for Air Quality Monitoring”. Atmosphere, vol. 10, no 9, p. 506.
Williams, Ron, et al. 2014. “Air Sensor Guidebook.” Epa/600/R-14/159 (1): 1–5.
Williams, Ron, et al. 2018. “Peer Review and Supporting Literature Review of Air Sensor Technology Performance Targets.” (September): 33.