The selenoprotein thioredoxin reductase (TXNRD) is a promising therapeutic target for cancer. To discover novel TXNRD inhibitors, a library of α, β-unsaturated carbonyl compounds were applied in structure-based virtual screening for the selection of hit compounds. Fifteen top-ranked compounds were further validated experimentally, exhibiting potent inhibition of TXNRD and remarkable cytotoxicity to cancer cells. The further binding mode analysis indicated that multiple noncovalent interactions between the inhibitors and the active pocket of TXNRD facilitated the formation of covalent bonds between the Sec498 on TXNRD and the α, β-unsaturated carbonyl groups on inhibitors. Results from both simulations and experiments demonstrated that Sec498 is the prime interaction site for the inhibition of TXNRD. Taking compound 7 as an example, the inhibition of TXNRD by compounds promoted oxidative stress-mediated apoptosis of cancer cells. Given these findings, novel TXNRD inhibitors may be discovered and introduced to the growing fields of small molecule drugs against TXNRD.