Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity

Article Type: Research (Machine Learning Methodology)

This is a methodological research paper presenting a novel technical solution (PON) to a specific problem in federated reinforcement learning (FedRL). The analysis focuses on the design rationale, the comparative advantage over alternatives, and empirical validation.

The Core Problem: Heterogeneity Breaks Standard FedRL Assumptions

The central challenge identified is environmental heterogeneity. In FedRL, agents operate in different physical or simulated environments with unique state-transition dynamics. This heterogeneity causes two critical issues during the standard federated averaging process:

• Non-identical input distributions: The raw observation (state) data received by each agent follows different statistical patterns.
• Imbalanced parameter updates: When a global model is aggregated, updates from agents with mismatched observations can interfere, leading to poor convergence.

The PON Solution: Local Statistics for Global Consistency

The proposed Personalized Observation Normalization (PON) is a decentralized preprocessing layer. Each agent maintains its own running mean and variance to normalize its raw state inputs locally before feeding them into the policy network. The key design principle is to achieve consistent scaling of local features across the federation. By having each agent adapt its normalization to its own environment, the method aims to align the processed input distributions for the global model, even though the raw data remains different and private.

The Case Against Shared Normalization Parameters

A critical insight from the paper is the demonstration that sharing normalization statistics across agents is ineffective. The authors argue that because input distributions are diverse due to heterogeneity, a single set of global normalization parameters (e.g., a global mean and variance) would be a poor fit for any individual agent. This would either fail to properly normalize local data or, worse, distort it. Therefore, the personalization of these statistics is not merely a convenience but a necessary condition for the method to function correctly in heterogeneous settings.

Experimental Validation and Implications

Experiments are conducted on MuJoCo tasks engineered to be heterogeneous. The results show that agents using PON accelerate training and achieve superior final performance compared to baselines that either use no normalization, use local normalization without federated learning, or (crucially) use federated averaging with shared normalization parameters. This empirical evidence supports the core claim: for FedRL in non-IID (not independently and identically distributed) settings, allowing for personalized preprocessing components is a vital design choice. The work underscores that effective federated learning may require moving beyond uniform aggregation of all parameters, especially when facing fundamental data heterogeneity.