Smartphone-based colorimetric sensors can be operated out of the laboratory, meeting urgent needs such as point-of-care testing and on-site diagnostics. The quality of the captured colorimetric image is directly related to the detection sensitivity, and the experimenter can optimize the image by customizing accessories and setting parameters. This paper introduces common colorimetric transduction mechanisms and reviews the advantages and disadvantages of various color spaces. The reports of paper-based (25 reports cited), plastic microfluidic chip-based (7 reports cited), microtiter plate-based (6 reports cited), and cuvette-based (7 reports cited) platforms combined with smartphones from 2017 to 2022 are reviewed, with the performance in signal enhancement and structural design being analyzed and discussed. Finally, the optimization algorithms of data processing and feature extraction are introduced, and the existing difficulties are discussed. This review aims to provide comprehensive solutions to the problems that appear in the colorimetric sensing assisted by smartphones.