Droughts affect more people than any other natural disaster. Drought severity is not merely a function of precipitation; it emerges from a web of interrelations between human and natural systems. The impacts of drought are equally complex, shifting across temporal scales, economic sectors, and regions. Even in regions with similar hydroclimatic characteristics, there is tremendous variation in the effects of drought. This study combines satellite imagery, geospatial data, and qualitative data to identify the multi-scalar factors that drive variations in agricultural responses to drought. We analyzed eleven years of remotely sensed imagery to identify agricultural areas in which cultivation occurred during an extreme drought in Sri Lanka. We visited a subset of these communities and conducted interviews with officials and farmers to identify the factors that influenced agricultural adaptation. Results suggest that though structural factors such as infrastructural capacity and physical environment significantly affect agricultural adaptation, dynamic factors such as local control of water supply, perceived risk, community cohesion, and farmer experience explain significant variation in the adaptive capacity of agricultural systems.