The distribution and heterogeneity characteristics of microbial communities in cold seep water columns are significant factors governing the efficiency of methane filtering and carbon turnover. However, this process is poorly understood. The diversity of vertically stratified microbial communities and the factors controlling the community assemblage process in the water column above the Haima cold seep were investigated in this study. The prokaryotic community diversities varied distinctly with vertical changes in hydrochemistry. Cyanobacteria dominated the light-transmitting layers and Proteobacteria dominated the deeper layers. With respect to microbial community assemblages and co-occurrence networks, stochastic processes were particularly important in shaping prokaryotic communities. In the shallow (≥85 m) and mesopelagic water columns (600–800 m), microbial community characteristics were affected by deterministic processes, reduced network connectivity, and modularity. Microbial community diversities and assemblage processes along a vertical profile were influenced by the vertical variations in pH, temperature, DIC, and nutrients. Stochastic processes may have facilitated the formation of complex co-occurrence networks. Briefly, the distribution of local environmental heterogeneity along the vertical dimension could drive unique microbial community assemblage and species coexistence patterns. This study provides new perspectives on how microorganisms adapt to the environment and build communities, and how species coexist in shared habitats.