The transforming growth factor beta (TGFβ) signaling cascade is considered as one of the pivotal oncogenic pathways in most advanced cancers. Inhibition of the TGFβ signaling pathway by specific antagonists, neutralizing antibodies, or small molecules is considered as an effective strategy for the treatment of tumor growth and metastasis. Here we demonstrated the identification of a series of tetrahydro-β-carboline derivatives from virtual screening which potentially inhibit the TGFβ signaling pathway. Optimization of the initial hit compound 2-benzoyl-1,3,4,9-tetrahydro-β-carboline (8a) through substitution at different positions to define the structure-activity relationship resulted in the discovery of potent inhibitors of the TGFβ signaling pathway. Among them, compound 8d, one of the tested compounds, not only showed potent inhibition of lung cancer cell proliferation and migration in vitro but also strongly suppressed growth of lung cancer and breast cancer in vivo.