Visual information processing in animals with large image forming eyes is carried out in highly structured retinotopically ordered neuropils. Visual neuropils in Drosophila form the optic lobe, which consists of four serially arranged major subdivisions; the lamina, medulla, lobula and lobula plate; the latter three of these are further subdivided into multiple layers. The visual neuropils are formed by more than 100 different cell types, distributed and interconnected in an invariant highly regular pattern. This pattern relies on a protracted sequence of developmental steps, whereby different cell types are born at specific time points and nerve connections are formed in a tightly controlled sequence that has to be coordinated among the different visual neuropils. The developing fly visual system has become a highly regarded and widely studied paradigm to investigate the genetic mechanisms that control the formation of neural circuits. However, these studies are often made difficult by the complex and shifting patterns in which different types of neurons and their connections are distributed throughout development. In the present paper we have reconstructed the three-dimensional architecture of the Drosophila optic lobe from the early larva to the adult. Based on specific markers, we were able to distinguish the populations of progenitors of the four optic neuropils and map the neurons and their connections. Our paper presents sets of annotated confocal z-projections and animated 3D digital models of these structures for representative stages. The data reveal the temporally coordinated growth of the optic neuropils, and clarify how the position and orientation of the neuropils and interconnecting tracts (inner and outer optic chiasm) changes over time. Finally, we have analyzed the emergence of the discrete layers of the medulla and lobula complex using the same markers (DN-cadherin, Brp) employed to systematically explore the structure and development of the central brain neuropil. Our work will facilitate experimental studies of the molecular mechanisms regulating neuronal fate and connectivity in the fly visual system, which bears many fundamental similarities with the retina of vertebrates.