To develop novel α-glucosidase inhibitors, a series of chromone-based phenylhydrazone and benzoylhydrazone derivatives were designed, synthesized, and evaluated their inhibitory effects on α-glucosidase. The target compounds were characterized using 1H NMR, 13C NMR, and high-resolution mass spectra. Some of the compounds showed a varying degree of α-glucosidase inhibitory activity with IC50 values ranging from 6.59 ± 0.09 to 158.55 ± 0.87 μM. Among them, compound 5c (IC50 = 6.59 ± 0.09 μM) was the most potent inhibitor by comparison with positive control acarbose (IC50 = 685.11 ± 7.46 μM). Enzyme kinetic, fluorescence analysis, circular dichroism spectra, and molecular docking techniques were employed to explain the underlying molecular mechanisms of 5c inhibition on α-glucosidase. In vivo sucrose-loading test showed that 5c could suppress the rise of blood glucose levels after loading sucrose in normal Kunming mice. The cytotoxicity assay indicated that 5c exhibited low cytotoxicity.