Seismic signal spectral decomposition can be solved as an inverse problem, known as Inverse Spectral Decomposition (ISD). It generates a higher time-frequency resolution spectrum but also takes a higher cost. ISD can be solved using matrix or operator iterations, the former of which requires much higher cost. In this paper, we introduce In-Crowd algorithm to matrix based ISD. We use FISTA to solve sub-problem at each In-Crowd update. Proposed algorithm performs much faster even than operator based FISTA algorithm. Inverse denoising can be done in the same way by adjusting the inversion parameters. We also utilize an approach to further improve the speed of ISD or inversion denoising using the lateral continuity of seismic data. Synthetic examples are used to demonstrate the advantages of the proposed method.