Open-Set Underwater Acoustic Recognition: A Multi-Protocol Benchmark with Mixture-of-Experts and OOD-Aware Evaluation

Underwater acoustic recognition faces challenges in robustness and generalization. This paper provides a unified framework evaluating joint motor--state, motor-only, and state-only classification on the Wolfset dataset. We compare handcrafted features with classical and deep classifiers, analyzing closed-set and open-set (OOD) performance. A specialized Mixture-of-Experts (MoE) model with dual LOFAR+DEMON representation successfully concentrates the best performances across multiple metrics, achieving ID Macro-F1 of 0.879 and OOD rejection rate of 0.625, matching or exceeding all baselines. A multi-criteria decision framework selects optimal models under accuracy-robustness trade-offs. Results show no single model dominates all scenarios: Mel/MFCC excel in joint tasks, LFCC+CNN2D yields strongest OOD robustness for motor-only, and MFCC+classical models best detect unknown states. Our findings highlight the importance of open-set evaluation for practical recognition.