Although Hsp90 inhibitors can inhibit multiple tumorigenic pathways in cancer cells, their anticancer activity has been disappointingly modest. However, by forcing Hsp90 inhibitors into the mitochondria with mitochondrial delivery vehicles, they were converted into potent drugs targeting the mitochondrial Hsp90 paralog TRAP1. Here, to improve mitochondrial drug accumulation without using the mitochondrial delivery vehicle, we increased freely available drug concentrations in the cytoplasm by reducing the binding of the drugs to the abundant cytoplasmic Hsp90. After analyzing X-ray cocrystal structures, the purine ring of the Hsp90 inhibitor 2 (BIIB021) was modified to pyrazolopyrimidine scaffolds. One pyrazolopyrimidine, 12b (DN401), bound better to TRAP1 than to Hsp90, inactivated the mitochondrial TRAP1 in vivo, and it exhibited potent anticancer activity. Therefore, the rationale and feasible guidelines for developing 12b can potentially be exploited to design a potent TRAP1 inhibitor.