Fiducial Reference Measurements for Ground-Based DOAS Air-Quality Observations

NDACC MAX-DOAS Service

1. Introduction

The NDACC (Network for the Detection of Atmospheric Composition Change) MAX-DOAS Service, launched in October 2020 in a demonstration mode, provides a NRT (24h latency) central processing facility for data from MAX-DOAS spectrometers. Including also specification of best practices for instrument operation, the aim of this service, developed under the ESA project FRM4DOAS is to produce homogenous and quality-controlled ground-based MAX-DOAS datasets at long-term monitoring sites or during field campaigns. The current official products are lower tropospheric NO2 vertical profiles and total O3 columns. The lower tropospheric HCHO and stratospheric NO2 vertical profile products are still under consolidation.

The purpose of this section is to present the key aspects of the NDACC MAX-DOAS Service.

2. NDACC MAX-DOAS Central Processing Facility Description

A general overview of the NDACC MAX-DOAS Service is presented in the flow-chart below.

user oriented frm4doas flowchart v14Detailed flow-chart of the NDACC MAX-DOAS Central Processing Facility (NRT: near-real-time; OL: off-line; RE: reprocessing)

The Central MAX-DOAS Facility uses the following retrieval algorithms:

  • QDOAS spectral fitting software (Fayt et al., 2011)
  • Parameterisation-based MAPA (Beirle et al., 2019) and Optimal-Estimation-based MMF (Friedrich et al., 2019) algorithms for lower tropospheric profiles and vertical columns of aerosols, NO2 and HCHO (see also Friess et al., 2019)
  • Standard AMF-based NDACC approach for the total O3 column retrieval (Hendrick et al., 2011)
  • Optimal-Estimation-based profiling tool for stratospheric NO2 vertical profiles (Hendrick et al., 2004)

and includes the following main processing steps:

  • Upon successful registration, level-1 files (spectrally calibrated radiance spectra) are uploaded by instrument PIs on a dedicated incoming FTP server. A standardised netCDF format (see https://frm4doas.aeronomie.be/index.php/frm4doas-guidelines) is required for those level-1 files, offering the possibility to include key data (e.g. reference spectrum and slit function) as well as ancillary data that can be used by the retrieval algorithms, e.g. pressure and temperature profiles and/or aerosol data.
  • Once being ingested, level-1 data files undergo subsequent processing and corresponding QA/QC for the data products listed above to produce final level-2 data files. Those are generated in (1) an internal netCDF file format that contains the complete and fully traceable set of retrieval variables and ancillary data information, and (2) in the standard GEOMS HDF4 file format (see https://avdc.gsfc.nasa.gov/index.php?site=596748165).
  • Finally, the GEOMS HDF4 files are automatically transferred to the NDACC Rapid Delivery repository with mirroring on the EVDC database, while the access of the full netCDF master output files is currently restricted to instrument PIs. Various diagnostic tools allow to detect anomalies in the processing chain and to generate reports (status on processed files, statistics, list of anomalies, etc.). Upon eventual detection of anomalies, e-mail alerts are sent to concerned instrument PIs.

It should be noted that in addition to the NRT processing, there will be also the possibility to perform off-line and/or reprocessing retrievals. The following three different processing chains will therefore work in parallel:

  • NRT: processing in near-real-time, with a target time lag of maximum one day after data acquisition and corresponding GEOMS data files stored in the NDACC/RD repository.
  • OFF-LINE: consolidated processing applied to same data as NRT processing, but with optimised algorithms and ancillary information and produced within a maximum delay of 3 months. Data files will be stored in the NDACC/RD repository in replacement of the corresponding NRT GEOMS data files.
  • REPROCESSING: processing of historical time-series of observations, performed on demand (from instrument PI) or after major upgrade of NRT/OFF-LINE algorithms. Upon PIs authorization and for instruments formally affiliated to NDACC, such reprocessed data sets are catalogued on the NDACC Consolidated repository on a yearly basis.

So far, only the NRT data stream is in operation for a limited number of stations beloging to the project partners. However, it has been designed for the efficient ingestion and processing of radiance spectra from a larger number of instruments and sites. In subsequent phases, the system will be extended to allow for processing of additional data products. Possible candidates are e.g. SO2, CHOCHO, BrO, HONO, H2O.

3. Added value for PIs joining the NDACC MAX-DOAS Central Processing System

Instrument PIs joining the NDACC MAX-DOAS Service will benefit from the following advantages:

  • Free-of-charge systematic level-1 (radiance spectra) to level-2 (vertical columns and profiles) NRT (24h latency) processing service
  • Continuous data quality monitoring with automated feedback to instrument PIs in case of anomalies
  • Increased data visibility as part of an international network (NDACC)
  • Possibility to contribute in international operational validation projects, e.g. in the frame of the EU Copernicus program
  • Processed level-2 data made available for scientific use by instrument PIs but also by the overall scientific community

In order to be eligible for the processing service, instrument PIs must provide evidences attesting of the quality of their measurements and commit to follow the FRM4DOAS guidelines and standards in terms of best practices, data acquisition protocol, and QA/QC for instrument calibration and operation. Those are described in living documents available at https://frm4doas.aeronomie.be/index.php/frm4doas-guidelines. Instrument PIs have also to share their resulting GEOMS files at least on the NDACC RD repository.

To protect the Intellectual Property Rights of the instrument PIs and avoid any misuse of the generated data sets, a strict data policy based on the Creative Common license system (see https://creativecommons.org/licenses/) is applied. The traceability of the datasets is ensured via systematic DOI assignment.

4. References

Beirle, S., Dörner, S., Donner, S., Remmers, J., Wang, Y., and Wagner, T.: The Mainz profile algorithm (MAPA), Atmos. Meas. Tech., 12, 1785–1806, https://doi.org/10.5194/amt-12-1785-2019, 2019

Fayt, C., De Smedt, I., Letocart, V., Merlaud, A., Pinardi, G., and Van Roozendael, M.: QDOAS Software user manual, http: //uv-vis.aeronomie.be/software/QDOAS/index.php, 2011

Friedrich, M. M., Rivera, C., Stremme, W., Ojeda, Z., Arellano, J., Bezanilla, A., García-Reynoso, J. A., and Grutter, M.: NO2 vertical profiles and column densities from MAX-DOAS measurements in Mexico City, Atmos. Meas. Tech., 12, 2545–2565, https://doi.org/10.5194/amt-12-2545-2019, 2019.

Frieß, U., Beirle, S., Alvarado Bonilla, L., Bösch, T., Friedrich, M. M., Hendrick, F., Piters, A., Richter, A., van Roozendael, M., Rozanov, V. V., Spinei, E., Tirpitz, J.-L., Vlemmix, T., Wagner, T., and Wang, Y.: Intercomparison of MAX-DOAS vertical profile retrieval algorithms: studies using synthetic data, Atmos. Meas. Tech., 12, 2155–2181, https://doi.org/10.5194/amt-12-2155-2019, 2019.

Hendrick, F., B. Barret, M. Van Roozendael, H. Boesch, A. Butz, M. De Mazière, F. Goutail, C. Hermans, J.-C. Lambert, K. Pfeilsticker, and J.-P. Pommereau, Retrieval of nitrogen dioxide stratospheric profiles from ground-based zenith-sky UV-visible observations: Validation of the technique through correlative comparisons, Atmos. Chem. Phys., 4, 2091-2106, 2004.

Hendrick, F., J.-P. Pommereau, F. Goutail, R. D. Evans, D. Ionov, A. Pazmino, E. Kyrö, G. Held, P. Eriksen, V. Dorokhov, M. Gil, and M. Van Roozendael, NDACC/SAOZ UV-visible total ozone measurements: Improved retrieval and comparison with correlative ground-based and satellite observations, Atm. Chem. Phys., 11, 5975-5995, 2011.