Automobile gear transmission systems are vulnerable to wear, cracks, and broken teeth under long-term alternating loads, lubrication fluctuations, and assembly errors. Weak impact signals become difficult to detect due to noise interference and variable working conditions. This study proposes a fault detection method combining whale optimization algorithm, variational mode decomposition (VMD), maximum correlation kurtosis deconvolution (MCKD), and ant lion optimizer (ALO)-optimized stacked sparse denoising autoencoder (SSDA). For the representative tooth root crack condition at 1800 r/min and 50% rated load, the optimized model achieves an envelope kurtosis of 7.81, a main frequency energy ratio of 42.8%, and a decomposition signal-to-noise ratio of 14.63 dB, with clear fault impact peaks near sampling points 256, 768, 1280, and 1792. The detection model achieves accuracy, precision, recall, comprehensive evaluation index, and area under the curve values of 98.41%, 98.22%, 98.11%, 98.16%, and 99.40%, respectively. The inference time is only 6.38 ms, and stability reaches 97.36%. Under mixed variable operating conditions and -5 dB noise, recognition accuracy remains 97.81% and 93.48%, respectively. The proposed method effectively enhances weak fault feature extraction, cross-condition detection accuracy, and noise resistance for automobile transmission system monitoring.