Cross-Modal Learning for Classification of Optical and SAR Imagery

Abstract:
Most automatic systems to analyze Earth observation data are designed for a particular combination of sensor and task. This limits their applicability in real world scenarios where for a certain point in time and space an acquisition of a certain sensor might not be available. We propose a cross-modal learning system that trained on multiple modalities can be applied to any of these modalities during inference time. We show that the proposed model not only maintains performance compared to the baseline approach of having independent modality specific models but also provides predictions with increased homogeneity regarding the modalities.