Phase and polarization singularities are two important quantities for cylindrical vector vortex beams (CVVBs), and the efficient and accurate detection of both singularities simultaneously is of great importance. At present, various approaches such as meta-surfaces, holograms and other techniques have been implemented to detect optical singularities, but all of them suffer from either low light efficiency or poor discrimination. Besides, it remains a challenge to detect both singularities simultaneously. In this manuscript, we proposed a novel approach to simultaneously detect phase and polarization singularities of CVVBs by implementing deep learning into the optical system. CVVBs are firstly scattered to speckle patterns by a randomly changing scatterer. Here, the implement of the diffuser realistically reflects the air disturbance imposed on light field during transmission. Then the powerful feature finding capability of deep learning is used to reveal the mapping relationship between singularities carried by the incident beam onto the generated speckle pattern image. It is demonstrated that a well-trained network can recognize incident phase and polarization singularities simultaneously with an accuracy of up to 99%. Moreover, the scatter is positioned on a constantly rotating motor, which means one identical beam can be scattered into many decorrelated speckle pattern images. It has been demonstrated that the trained network can still fulfill the incident mode recognition task perfectly which confirms the generalization ability of the network. We finally implemented the method to an optical information transmission application. We believe the proposed method would open a new gate to manipulation and utilization of optical singularities.