Evodiamine (EVO) and rutaecarpine (RUT) are the main active compounds of the traditional Chinese medicinal herb Evodia rutaecarpa. Here, we fully optimized the molecular geometries of EVO and RUT at the B3LYP/6-311++G (d, p) level of density functional theory. The natural population analysis (NPA) charges, frontier molecular orbitals, molecular electrostatic potentials, and the chemical reactivity descriptors for EVO and RUT were also investigated. Furthermore, molecular docking, molecular dynamics simulations, and the analysis of the binding free energies of EVO and RUT were carried out against the anticancer target topoisomerase 1 (TOP1) to clarify their anticancer mechanisms. The docking results indicated that they could inhibit TOP1 by intercalating into the cleaved DNA-binding site to form a TOP1–DNA–ligand ternary complex, suggesting that they may be potential TOP1 inhibitors. Molecular dynamics (MD) simulations evaluated the binding stability of the TOP1–DNA–ligand ternary complex. The calculation of binding free energy showed that the binding ability of EVO with TOP1 was stronger than that of RUT. These results elucidated the structure–activity relationship and the antitumor mechanism of EVO and RUT at the molecular level. It is suggested that EVO and RUT may be potential compounds for the development of new anticancer drugs. © 2022 by the authors.