This paper reviews the existing vision-based tactile sensor (VBTS) designs reported in the literature. Although some reviews on VBTSs already exist in the literature. We believe it is necessary to review existing VBTS designs to formulate a guideline for developing such systems considering recent developments in the manufacturing and imaging technologies. Therefore, the main emphasis of this paper is to investigate current manufacturing trends and component selection criteria for developing a complete VBTS system. Further, the motivation behind this review is to identify the shortcomings in the current VBTS development technology and to furnish viable solutions to overcome such challenges. First, three different modalities of VBTSs are discussed: i) Waveguide-type designs, ii) marker-tracking based designs, and ii) reflective membrane designs. Next, a detailed discussion on various design aspects, like manufacturing, selection, and arrangements of the various sensor components, of the VBTSs is included. Then, a discussion on the validation/testing of various VBTSs is presented. Finally, based on the review, several challenges related to the development of VBTS are presented and the future research directions to overcome such challenges are recommended. This will serve the research community in determining the future research directions in the area of VBTS development.