Sexual dimorphisms in body size are widespread throughout the animal kingdom but their underlying mechanisms are not well characterized. Most models for how sex chromosome genes specify size dimorphism have emphasized the importance of gonadal hormones and cell-autonomous influences in mammals versus strictly cell-autonomous mechanisms in Drosophila melanogaster. Here, we use tissue-specific genetics to investigate how sexual size dimorphism (SSD) is established in Drosophila. We find that the larger body size characteristic of Drosophila females is established very early in larval development via an increase in the growth rate per unit of body mass. We demonstrate that the female sex determination gene, Sex-lethal (Sxl), functions in central nervous system (CNS) neurons as part of a relay that specifies the early sex-specific growth trajectories of larval but not imaginal tissues. Neuronal Sxl acts additively in 2 neuronal subpopulations, one of which corresponds to 7 median neurosecretory cells: the insulin-producing cells (IPCs). Surprisingly, however, male-female differences in the production of insulin-like peptides (Ilps) from the IPCs do not appear to be involved in establishing SSD in early larvae, although they may play a later role. These findings support a relay model in which Sxl in neurons and Sxl in local tissues act together to specify the female-specific growth of the larval body. They also reveal that, even though the sex determination pathways in Drosophila and mammals are different, they both modulate body growth via a combination of tissue-autonomous and nonautonomous inputs.