The application of machine learning (ML) models to optimize antibody affinity to an antigen is gaining prominence. Unfortunately, the small and biased nature of the publicly available antibody-antigen interaction datasets makes it challenging to build an ML model that can accurately predict binding affinity changes due to mutations (ΔΔG). Recognizing these inherent limitations, we reformulated the problem to ask whether an ML model capable of classifying deleterious vs non-deleterious mutations can guide antibody affinity maturation in a practical setting. To test this hypothesis, we developed a Random Forest classifier (Antibody Random Forest Classifier or AbRFC) with expert-guided features and integrated it into a computational-experimental workflow. AbRFC effectively predicted non-deleterious mutations on an in-house validation dataset that is free of biases seen in the publicly available training datasets. Furthermore, experimental screening of a limited number of predictions from the model (<10^2 designs) identified affinity-enhancing mutations in two unrelated SARS-CoV-2 antibodies, resulting in constructs with up to 1000-fold increased binding to the SARS-COV-2 RBD. Our findings indicate that accurate prediction and screening of non-deleterious mutations using machine learning offers a powerful approach to improving antibody affinity.