Rugged karst terrain relief that creates shadows in satellite imagery, combined with high karst landscape heterogeneity stand in the way of fractional cover retrieval on karst rocky desertification (KRD) monitoring. In this study, we explored the feasibility of applying multispectral high spatial resolution Advanced Land Observing Satellite (ALOS) imagery for the fractional cover extraction of rocky outcrops. Dimidiate pixel model (DPM) and spectral mixture analysis (SMA) approaches (including simple endmember spectral mixture analysis and multiple endmember spectral mixture analysis) were selected to explore their feasibility for KRD monitoring through accuracy improvement for fraction estimation. Results showed fractional cover retrievals at the sub-pixel scale is essential in highly heterogeneous karst landscapes. Indeed, mixed pixels accounted for 93.7% of the study area in southwest China. Multiple endmember spectral mixture analysis achieved high overall accuracy (80.5%) in monitoring the percentage of rocky outcrop land cover. Furthermore, the predicted exposed bedrock coverage via spectral mixture analysis were similar in sunlit and shadow areas for the same surface types. This reflected that SMA methods could effectively reduce topographic effects of satellite imagery to improve the accuracy of fractional cover extraction at sub-pixel level in heterogeneous and rugged landscapes.