Issue with ERA5 Land hourly dataset?

by

Summary

The issue at hand involves the presence of infinite (Inf) values in the ERA5-Land hourly temperature dataset, particularly in coastal grid cells. This problem arises when computing population-weighted temperature for coastal cities, resulting in substantially lower exposure values compared to the observed regional mean temperature. The primary concern is whether this issue reflects an inherent characteristic of the dataset or is related to analytical choices, such as boundary masking during preprocessing.

Root Cause

The root cause of the issue can be attributed to several factors, including:

  • Infinite values in ERA5-Land temperature data: The presence of Inf values in the dataset, especially in coastal grid cells, which are not properly handled during analysis.
  • Boundary masking during preprocessing: The use of custom shapefiles to define boundaries may be contributing to the issue, particularly if the masking process is not accurately accounting for coastal grid cells.
  • Population-weighted exposure function: The way the function handles Inf values may be leading to discrepancies in the calculated population-weighted exposure.

Why This Happens in Real Systems

This issue can occur in real systems due to:

  • Data quality issues: The presence of Inf values in the dataset can be a result of data quality issues, such as errors during data collection or processing.
  • Analytical choices: The choice of boundary masking and population-weighted exposure functions can significantly impact the results, especially when dealing with coastal grid cells.
  • Dataset characteristics: The ERA5-Land dataset may have inherent characteristics, such as the masking of ocean and water bodies, that can affect the analysis.

Real-World Impact

The real-world impact of this issue includes:

  • Inaccurate population-weighted exposure values: The presence of Inf values and incorrect handling of boundary masking can lead to substantially lower population-weighted exposure values, which can have significant implications for climate-related studies and decision-making.
  • Biased results: The issue can result in biased results, particularly for coastal cities, which can affect the overall accuracy of the analysis.
  • Limited representativeness: The limited number of valid grid cells contributing to the calculation can reduce the representativeness of the results, making it challenging to draw meaningful conclusions.

Example or Code

import numpy as np

# Example of population-weighted exposure function
def population_weighted_exposure(temperature, population):
    # Handle Inf values
    temperature = np.where(np.isinf(temperature), np.nan, temperature)

    # Calculate population-weighted exposure
    exposure = np.nansum(temperature * population) / np.nansum(population)

    return exposure

How Senior Engineers Fix It

Senior engineers can fix this issue by:

  • Implementing robust data quality checks: Verifying the dataset for Inf values and other data quality issues before analysis.
  • Using alternative boundary masking approaches: Exploring alternative boundary masking methods, such as using buffer zones or smoothing techniques, to reduce the impact of coastal grid cells.
  • Developing custom population-weighted exposure functions: Creating functions that can handle Inf values and other dataset characteristics, such as using weighted averages or imputing missing values.

Why Juniors Miss It

Juniors may miss this issue due to:

  • Lack of experience with dataset characteristics: Limited familiarity with the ERA5-Land dataset and its inherent characteristics, such as the masking of ocean and water bodies.
  • Insufficient understanding of analytical choices: Inadequate knowledge of the impact of boundary masking and population-weighted exposure functions on the results.
  • Inadequate data quality checks: Failure to implement robust data quality checks, leading to the presence of Inf values and other data quality issues in the analysis.

Leave a Comment