Prostate cancer (PCa) is the second leading cause of death in men. Recently, some researches have showed that 5α-reductase inhibitors were beneficial in PCa treatment as well. In this study, a series of novel 3-oxo-4-oxa-5α-androst-17β-amide derivatives have been designed and synthesized in a more simple and convenient method. Most of the synthesized compounds displayed good 5α-reductase inhibitory activities and androgen receptor binding affinities. Their anti-proliferation activities in PC-3 and LNCaP cell lines were also evaluated and the results indicated that most of the synthesized compounds exhibited potent anti-proliferative activities. It is obvious that the androgen-dependent cell line LNCaP was much more sensitive than the androgen-independent cell line PC-3. Among all the synthesized compounds, 11d and 11k displayed the best inhibition activity with 4-fold more sensitive toward LNCaP than PC-3, which was consistent with their high affinities observed in AR binding assay. Molecular modeling studies suggested that 11k could bind to AR in a manner similar to the binding of dihydrotestosterone to AR. Compared to the finasteride, 11k showed a longer plasma half-life (4h) and a better bioavailability. Overall, based on biological activities data, compound 11d and 11k can be identified as potential dual 5α-reductase inhibitors and AR antagonists which might be of therapeutic importance for prostate cancer treatment.