Sleep apnea is a common symptomatic disease affecting nearly 1 billion people around the world. The gold standard approach for determining the severity of sleep apnea is full-night polysomnography conducted in the laboratory, which is very costly and cumbersome. In this work, we propose a novel scalogram-based convolutional neural network (SCNN) to detect obstructive sleep apnea (OSA) using single-lead electrocardiogram (ECG) signals. Firstly, we use continuous wavelet transform (CWT) to convert ECG signals into conventional scalograms. In parallel, we also apply empirical mode decomposition (EMD) to the signals to find correlated intrinsic mode functions (IMFs) and then apply CWT on the IMFs to obtain hybrid scalograms. Finally, we train a lightweight CNN model on these scalograms to extract deep features for OSA detection. Experiments on the benchmark Apnea-ECG dataset demonstrate that our proposed model results in an accuracy of 94.30%, sensitivity 94.30%, specificity 94.51%, and F1-score 95.85% in per-segment classification. Our model also achieves an accuracy of 81.86%, sensitivity 71.62%, specificity 86.05%, and F1-score 69.63% for UCDDB dataset. Furthermore, our model achieves an accuracy of 100.00% in per-recording classification for Apnea-ECG dataset. The experimental results outperform the existing OSA detection approaches using ECG signals.