Two rutaecarpine (RUT) derivatives, substituted with methoxy groups, namely, 2-methoxyl rutaecarpine (RUT-OCH3, 3a), and 2,10-dimethoxy rutaecarpine (RUT-(OCH3)2, 3b), were synthesized and characterized using 1H nuclear magnetic resonance (NMR), 13C NMR and mass spectra. The in vitro antitumor activities of compounds RUT, 3a, and 3b against A549, H1299, and HepG2 cells were studied by 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyl tetrazolium bromide assay. The results showed that the activity of compounds 3a and 3b was stronger than that of compound RUT, and the activity of compound 3a was stronger than that of 3b, indicating that the activity of the compounds was improved after structural modification. The apparent oil-water partition coefficients of compound RUT, 3a, and 3b were explored using ultraviolet spectrometry. The results indicated that hydrophobicity affects the physicochemical properties of the molecules and influences antitumor activities. In addition, the Natural Electron Configuration, frontier molecular orbital (highest occupied molecular orbital, lowest unoccupied molecular orbital) bandgaps of compounds have been studied based on density functional theory (DFT) by means of DFT-B3LYP/6‐31G (d) in Gaussian 16. The calculation results showed that bandgap of 3a is highest, indicating that the stability of 3a is weakest, so 3a has higher activity than RUT and 3b, which agrees with the results of antitumor activities experiment. © The Author(s) 2021.