The axion-like objects such as axion mini-clusters and axion clouds around spinning black holes induce parametric resonances of electromagnetic waves through the axion-photon interaction. In particular, it has been known that the resonances from the axion with the mass around mueV may explain the observed fast radio bursts (FRBs). Here we argue that similar bursts of high frequency gravitational waves, which we call the fast gravitational wave bursts (FGBs), are generated from axion clumps with the presence of gravitational Chern-Simons coupling. The typical frequency is half of the axion mass, which in general can range from mHz to GHz. We also discuss the secondary gravitational wave production associated with FRB, as well as the possible host objects of the axion clouds. Future detections of FGBs together with the observed FRBs are expected to provide more evidence for the axion.