Neoantigens play a key role in the recognition of tumour cells by T cells; however, only a small proportion of neoantigens truly elicit T-cell responses, and few clues exist as to which neoantigens are recognized by which T-cell receptors (TCRs). We built a transfer learning-based model named the pMHC–TCR binding prediction network (pMTnet) to predict TCR binding specificities of the neoantigens—and T cell antigens in general—presented by class I major histocompatibility complexes. pMTnet was comprehensively validated by a series of analyses and exhibited great advances over previous works. By applying pMTnet to human tumour genomics data, we discovered that neoantigens were generally more immunogenic than self-antigens, but human endogenous retrovirus E (a special type of self-antigen that is reactivated in kidney cancer) is more immunogenic than neoantigens. We further discovered that patients with more clonally expanded T cells that exhibit better affinity against truncal rather than subclonal neoantigens had more favourable prognosis and treatment response to immunotherapy in melanoma and lung cancer but not in kidney cancer. Predicting TCR–neoantigen/antigen pairing is one of the most daunting challenges in modern immunology; however, we achieved an accurate prediction of the pairing using only the TCR sequence (CDR3β), antigen sequence and class I major histocompatibility complex allele, and our work revealed unique insights into the interactions between TCRs and major histocompatibility complexes in human tumours, using pMTnet as a discovery tool.