A convenient and effective method was developed to fabricate zirconium-based calcium sulfate whiskers (Zr-CSWs), an environmentally friendly adsorption material designed to clean excessive fluoride from wastewater. Zr-CSWs, which potentially provide high adsorption activity toward fluoride in wastewater, overcome an aggregation problem typical of Zr-based species that may block the pore structure of the supporting material and degenerate its treatment performance. The results indicate that Zr-CSWs show excellent equilibrium adsorption capacity compared to other Zr-based materials in practical water solutions with pH values of 6–7. The adsorption of fluoride on Zr-CSWs was further evaluated in accordance with the Langmuir equation and the pseudosecond-order model. Coexisting anions had a minor effect on the removal of fluoride, and a strong alkali solution regenerated the used adsorbent. Density functional theory (DFT) calculations were performed to determine the mechanism of fluoride removal. The calculations demonstrated that the removal of fluoride by Zr-CSWs was mainly achieved by ion exchange between fluoride ions and OH^- on the surface of the adsorbent, where a protonation process created low-coordination Zr atoms.