I will review our work in extracting clinically relevant characterizations of anatomy and pathology from medical images in two domains. First, joint modeling of image, genetic and clinical data is used to gain insight into the patterns of disease in large heterogeneous clinical populations. Examples include studies of white matter disease in stroke patients from brain MRI, of genetically defined patterns of emphysema in COPD patients as observed in chest CT, and others. The second family of applications aims to provide accurate delineations of pathology and make predictions in medical scans of individual patients. Examples include functional imaging of the placenta and cardiac image analysis for surgical planning.