Forecasts of natural hazards save lives. But the accuracy of forecasts must be evaluated before use. Here we evaluate heavy rainfall advisories over Kenya. We assess their ability to anticipate heavy rainfall and show how well they warned of recent floods which had significant impacts. We find that although they effectively warn of heavy rainfall and flooding, issues such as a lack of spatial detail limit their utility for systematic approaches to preparedness.