Electrocardiogram (ECG) and phonocardiogram (PCG) are both noninvasive and convenient tools that can capture abnormal heart states caused by coronary artery disease (CAD). However, it is very challenging to detect CAD relying on ECG or PCG alone due to low diagnostic sensitivity. Recently, several studies have attempted to combine ECG and PCG signals for diagnosing heart abnormalities, but only conventional manual features have been used. Considering the strong feature extraction capabilities of deep learning, this paper develops a multi-input convolutional neural network (CNN) framework that integrates time, frequency, and time-frequency domain deep features of ECG and PCG for CAD detection. Simultaneously recorded ECG and PCG signals from 195 subjects are used. The proposed framework consists of 1-D and 2-D CNN models and uses signals, spectrum images, and time-frequency images of ECG and PCG as inputs. The framework combining multi-domain deep features of two-modal signals is very effective in classifying non-CAD and CAD subjects, achieving an accuracy, sensitivity, and specificity of 96.51%, 99.37%, and 90.08%, respectively. The comparison with existing studies demonstrates that our method is very competitive in CAD detection. The proposed approach is very promising in assisting the real-world CAD diagnosis, especially under general medical conditions.