Histone deacetylase 6 (HDAC6) is associated with multiple neurological disorders as well as aggressive cancers, making its selective inhibition highly desirable for therapeutic purposes. The basic molecular design of an effective HDAC6 inhibitor consists of a zinc-binding group, a linker, and a capping group capable of making interactions at the mouth of the active site. To date, more than 50 high-resolution X-ray crystal structures of HDAC6-inhibitor complexes have been reported, many of which reveal intermolecular interactions that contribute to isozyme affinity and selectivity. Here, we review the key features of HDAC6 inhibitor design illuminated by these structural studies.