Cells convert complex metabolic information into stress-adapted autophagy responses. Canonically, multilayered protein kinase networks converge on the conserved Atg1/ULK kinase complex (AKC) to induce non-selective and selective forms of autophagy in response to metabolic changes. Here we show that, upon phosphate starvation, the metabolite sensor Pho81 interacts with the adaptor subunit Atg11 at the AKC via an Atg11/FIP200 interaction motif to modulate pexophagy by virtue of its conserved phospho-metabolite sensing SPX domain. Notably, core AKC components Atg13 and Atg17 are dispensable for phosphate starvation-induced autophagy revealing significant compositional and functional plasticity of the AKC. Our data indicate that, instead of functioning as a selective autophagy receptor, Pho81 compensates for partially inactive Atg13 by promoting Atg11 phosphorylation by Atg1 critical for pexophagy during phosphate starvation. Our work shows Atg11/FIP200 adaptor subunits bind not only selective autophagy receptors but also modulator subunits that convey metabolic information directly to the AKC for autophagy regulation.