Some α,β-unsaturated ketones 5a-h were prepared from 3-acetyl-4-hydroxycoumarin and (hetero)aromatic aldehydes with yields of 58–62%. These ketones were converted into novel coumarin-benzothiazepine hybrid compounds 6a-h by their reaction with o-aminothiophenol in the presence of glacial acetic acid as catalyst with yields of 62–82%. All the synthesized compounds 6a-h were screened for their in vitro anticancer activity against human squamous cell carcinoma KB and hepatocellular carcinoma HepG2 cancer lines. Compound 6h was the most potential inhibitory activity in the series, with IC50 values of 4.0 and 6.25 μM against KB and HepG2 cell lines, respectively. The cytotoxicity assay on WI-38 cells showed that the most potential activity compounds 6b, 6d, 6f, and 6h had the low cytotoxicity, and of these compounds, compound 6h exhibited the lowest toxicity. ADMET properties showed that compound 6h possessed the drug-likeness behaviour. The molecular docking results for the most active compounds 6b, 6d, 6f, and 6h indicated the active interactions between each ligand and the residues in the binding pocket of enzyme 1Z5M. The molecular (MD) dynamics simulation that applied for compound 6h showed the interactions of this ligand with residues LEU88, ALA109, GLU166, LYS111, LEU212 in the active pocket of 1Z5M during 200-ns MD simulation. Furthermore, the electronic characteristics of the most active compounds 6b, 6d, 6f, and 6h were investigated by using density functional theory (DFT) method at the B3LYP/6-311++G(d,p) basis level. The frontier molecular orbital (FMO) energy and atomic net charges were examined.