Mechanisms generating diverse cell types from multipotent progenitors are fundamental for normal development. Pigment cells are derived from multipotent neural crest cells and their diversity in teleosts provides an excellent model for studying mechanisms controlling fate specification of distinct cell types. Zebrafish have three types of pigment cells (melanocytes, iridophores and xanthophores) while medaka have four (three shared with zebrafish, plus leucophores), raising questions about how conserved mechanisms of fate specification of each pigment cell type are in these fish. We have previously shown that the Sry-related transcription factor Sox10 is crucial for fate specification of pigment cells in zebrafish, and that Sox5 promotes xanthophores and represses leucophores in a shared xanthophore/leucophore progenitor in medaka. Employing TILLING, TALEN and CRISPR/Cas9 technologies, we generated medaka and zebrafish sox5 and sox10 mutants and conducted comparative analyses of their compound mutant phenotypes. We show that specification of all pigment cells, except leucophores, is dependent on Sox10. Loss of Sox5 in Sox10-defective fish partially rescued the formation of all pigment cells in zebrafish, and melanocytes and iridophores in medaka, suggesting that Sox5 represses Sox10-dependent formation of these pigment cells, similar to their interaction in mammalian melanocyte specification. In contrast, in medaka, loss of Sox10 acts cooperatively with Sox5, enhancing both xanthophore reduction and leucophore increase in sox5 mutants. Misexpression of Sox5 in the xanthophore/leucophore progenitors increased xanthophores and reduced leucophores in medaka. Thus, the mode of Sox5 function in xanthophore specification differs between medaka (promoting) and zebrafish (repressing), which is also the case in adult fish. Our findings reveal surprising diversity in even the mode of the interactions between Sox5 and Sox10 governing specification of pigment cell types in medaka and zebrafish, and suggest that this is related to the evolution of a fourth pigment cell type.