Jon Kaas (2006) discusses some of the research on the evolution of the neocortex throughout mammals. He notes some interesting facets of this process, such as the “late makes great” rule, where brain regions that form late developmentally are especially large in bigger brains. The inference that humans and other large-brained mammals must have an especially large neocortex follows trivially.
He describes early primates as small, nocturnal creatures that spent most of their time feeding in tree branches. This lends credence to the hypothesis that primate brain size increased initiallly due to an opportunity for animals with improved visual and sensorimotor abilities to thrive. Based on the fossil record of extinct and extant primates, the temporal cortex in particular was expanded, which was probably devoted to vision.
Finally, he speculates that the reason for the especially cognitive success of humans is due to excessive modulation of brain regions that chopped them up into more and more areas. These areas were then forced to become more automonous in order to cut down on the need for anatomically long processing.
Semendeferi et al showed that the volume of the frontal lobes had not changed throughout primate evolution. Perhaps this increased specialization hypothesis could explain how executive functions changed so dramatically without altering the volume of the region.
Kaas JH. 2006 Evolution of the neocortex. Current Biology 16:910-914. Link to PubMed.
Semendeferi K, Damasio H, Frank R, Van Hoesen GW. 1997 The evolution of the frontal lobes: a volumetric analysis based on three-dimensional reconstructions of magnetic resonance scans of human and ape brains. Journal of Human Evolution 32:375-88.