In a pancreatic tumor microenvironment, activated pancreatic stellate cells (PSCs) produce extracellular matrix (ECM) to form a barrier to drug penetration. Moreover, the interaction between cancer cells and activated PSCs promotes the tumor growth. Vitamin D receptor (VDR), as a key regulator to promote the recovery of PSCs to the resting state, is an attractive therapeutic target for pancreatic cancer. Herein, we reported the design and synthesis of 57 nonsecosteroidal VDR modulators based on the skeleton of phenyl-pyrrolyl pentane. Among them, compounds C4, I5, and I8 exhibited excellent VDR affinity and effective inhibition of the activation of PSCs, as well as potent suppression of the interaction between cancer cells and PSCs in vitro. In vivo, compound I5 combined with gemcitabine achieved efficacious antitumor activity without causing hypercalcemia. In conclusion, the compounds designed in our study can remodel the tumor microenvironment and are expected to be candidates for the treatment of pancreatic cancer.