Chloroquine (CQ) and hydroxychloroquine (HCQ) show good efficacy in the treatment of SARS-CoV-2 in the early stage, while they are no longer recommended due to their side effects. As an important drug delivery carrier, serum albumin (SA) is closely related to the efficacy of drugs. Here, the affinity behaviour of chloroquine and hydroxychloroquine with two SA were investigated through the multispectral method of biochemistry and computer simulation. The results showed that the intrinsic emission of both SA was quenched by CQ and HCQ in a spontaneous exothermic entropy reduction static process, which relied mainly on hydrogen bonding and van der Waals forces. The lower binding constants suggested weak binding between the two drugs and SA, which might lead to differences in efficacy and possibly even to varying side effects. Binding site recognition demonstrated that CQ preferred to bind to the two sites of both SA, while HCQ tended to bind to site I of SA. The results of conformational studies demonstrated that CQ and HCQ could affect the structure of both SA by slightly increasing the α-helix content of SA. Finally, we combine the results from experimental start with molecular simulations to suggest drug modifications to guide the design of drugs. This work has important implications for guiding drug design improvements to select CQ derivatives with fewer side effects for the treatment of COVID-19.