6H-Isoindolo[2,1-a]indoles (5, 7, 10, 13), 5,6-dihydroindolo[2, 1-a]isoquinolines (20, 21), and 6,7-dihydro-5H-benzo[c]azepino[2, 1-a]indoles (23, 25, 27, 30) have been prepared as melatonin analogues to investigate the nature of the binding site of the melatonin receptor. The affinity of analogues was determined in a radioligand binding assay using cloned human mt(1) and MT(2) receptor subtypes expressed in NIH 3T3 cells. Agonist and antagonist potency was measured using the pigment aggregation response of a clonal line of Xenopus laevis melanophores. The 2-methoxyisoindolo[2, 1-a]indoles (7a-d) showed much higher binding affinities than the parent isoindoles (5a-e), and whereas 7a-c were agonists in the functional assay, 7d and 5a-e were antagonists. The 2-ethoxyisoindolo[2,1-a]indoles (10a-d) showed reduced binding affinities compared to their methoxy analogues, while the 5-chloro derivative 13 showed a considerable reduction in binding affinity and potency compared to 7a. The 10-methoxy-5,6-dihydroindolo[2, 1-a]isoquinolines (21a-c) had higher binding affinities than the corresponding parent indoloisoquinolines (20a-c) in the human receptor subtypes, and the parent compounds were antagonists whereas the 10-methoxy derivatives were agonists in the functional assay. The N-cyclobutanecarbonyl derivatives of both the parent (20d) and 10-methoxyl (21d) series had similar binding affinities and were both antagonists with similar potencies. The 11-methoxy-6, 7-5H-benzo[c]azepino[2,1-a]indoles (25a-d) had higher binding affinities than the corresponding parent compounds (23a-d) at the MT(2) receptor but similar affinities at the mt(1) site; all of the compounds were antagonists in the functional assay. Changing 11-methoxy for 11-ethoxy decreased the binding affinity slightly, and this was more evident at the MT(2) receptor. All of the derivatives investigated had either the same or a greater affinity for the human MT(2) receptor compared to the mt(1) receptor (range 1:1-1:132). This suggests that the mt(1) and MT(2) receptor pockets differ in their ability to accommodate alkyl groups in the indole nitrogen region of the melatonin molecule. Two compounds (7c and 25c) were tested in functional assays on recombinant mt(1) and MT(2) melatonin receptors. Compound 7c is a potent agonist with some selectivity (44-fold) for the MT(2) receptor, while 25c is an MT(2)-preferring antagonist. Increasing the carbon chain length between N-1 of indole and the 2-phenyl group from n = 1 through n = 3 leads to a fairly regular decrease in the binding affinity, but, remarkably, when n = 3, it converts the methoxy compounds from melatonin agonists to antagonists. The Xenopus melatonin receptor thus cannot accommodate an N-n-alkyl chain attached to a 2-phenyl substituent with n > 2 in the required orientation to induce or stabilize the active receptor conformation.