Neolithic sites show an increasingly settled way of life as exemplified by evidence of food storage. However, farming was hard work, and skeletal evidence shows signs of the heavy effort needed, which--combined with a diet adequate in calories but barely or less than adequate in minerals from the depleting effects of phytate (phytates in grains bind minerals and inhibit absorption)--led to a state of low general health. The considerable decrease in stature at this time (roughly 4-6 inches, or 12-16 cm, shorter than in pre-agricultural times) is believed to have resulted from restricted blood calcium and/or vitamin D, plus insufficient essential amino acid levels, the latter resulting from the large fall in meat consumption at this time (as determined by strontium/calcium ratios in human bone remains).
Disease effects were minor in the Upper [Late] Paleolithic except for trauma. In postglacially hot areas, porotic hyperostosis [indicative of anemia] increased in Mesolithic and reached high frequencies in Neolithic to Middle Bronze times. [Reminder note: The end of the last Ice Age and the consequent melting of glaciers which occurred at the cusp of the Paleolithic/Neolithic transition caused a rise in sea level, with a consequent increase in malaria in affected inland areas which became marshy as a result.] Apparently this resulted mainly from thalassemias, since children show it in long bones as well as their skulls. But porotic hyperostosis in adults had other causes too, probably from iron deficiency from hookworm, amebiasis, or phytate, effect of any of the malarias. The thalassemias necessarily imply falciparum malaria. This disease may be one direct cause of short stature.
The other pressure limiting stature and probably also fertility in early and developing farming times was deficiency of protein and of iron and zinc from ingestion of too much phytic acid [e.g., from grains] in the diet. In addition, new diseases including epidemics emerged as population increased, indicated by an increase of enamel arrest lines in Middle Bronze Age samples....
We can conclude that farmers were less healthy than hunters, at least until Classical to Roman times. [Due to the difficulty in disentangling all relevant factors, as Angel explains a bit earlier] [w]e cannot state exactly how much less healthy they were, however, or exactly how or why.