When foundation models meet astronomical data

This study evaluates the ability and limitations of existing visual foundation models, based on ViT, SwinV2, BEiT, and DINOv2, for astronomical applications, particularly galaxy image classification using the Galaxy10 DECaLS dataset. We employ a fine-tuning methodology to assess the viability of adapting pretrained models to astronomical tasks, aiming to get around the resource-intensive process of pretraining foundation models from scratch. This approach allows us to evaluate the effectiveness of transfer learning from computer vision to astronomy, and to assess if a full and costly pretraining of foundation models on astronomical data is required. The present study also evaluates the advantages of fine-tuning all model parameters, as opposed to solely the classification head fine-tuning, with a view to optimising model performance within a reduced training cost. Furthermore, the study evaluates the influence of a domain-specific pretraining and the benefits that it engenders. Our results show that full fine-tuning consistently outperforms head-only adaptation while requiring comparable training effort, indicating that simple off-the-shelf usage of foundation models may not fully exploit their representational capacity in the context of galaxy morphology classification. Furthermore, modern CNN-based architectures achieve performance comparable to, or exceeding, several vision foundation models when trained on a modest astronomical dataset, suggesting that domain-specific features remain critical for this task. These findings highlight both the potential and current limitations of visual foundation models for astronomical applications, particularly in low-data regimes. They suggest that effective deployment in astronomy may require either domain adaptation strategies or the integration of astrophysical priors. Overall, this work contributes to the emerging field of astronomical foundation models by clarifying their current applicability and outlining directions for future research in data-driven galaxy morphology analysis.

Preprint, under review. Available at SSRN 6911300