Objection detection of vehicles and pedestrians in fog is of great significance for intelligent transportation and autonomous driving. Polarization image is beneficial to improve the object detection under adverse weather conditions. This study proposed a polarization image fusion method based on grouped convolutional attention network (GCAnet) to improve the object detection for cars and persons in foggy street scenes. Based on the international available Polar LITIS image dataset, a multi-channel grouped convolution matrix was first constructed to input different types of polarization images. Then, a grouped attention module was added to enhance the features in each type of polarization image, and finally each convolutional matrix was further connected to the detection network in series to perform objection detection. The experimental results prove that three types of polarization image fusion are obviously better than those of any two types of polarization image fusion and one single polarization image; and adding ECA attention module after multi-channel convolution can further enhance the accuracy of I04590 + Pauli + Stokes fused image to the highest value of 76.46%. The improvement of network lightweight shows that the Mobilenet-ECA has increased the speed by 26% with a slightly reduced accuracy. The proposed GCAnet method has significantly surpassed traditional objection detection networks of SSD300, SSD512, Faster R-CNN600, Yolov3, and Yolov4, which has increased the mAP@0.5 by 28.90%, 27.60%, 15.01%, 24.98%, and 16.45%, respectively; and has increased the mAP@0.5 by 9.36% and 6.20% compared to foggy image detection methods of AOD-Net SSD and DeRF-Yolov3-X, respectively. This work demonstrates the potential of GCAnet enabled polarization image fusion technology to be used as an effective foggy objection detection method in the field of intelligent transportation and autonomous driving.