Comment on nhess-2021-216

with great interest I have read your manuscript on simulating the effects of climate change and policy intervention on the uptake and success of drought adaptation measures in smallholder farming in Kenya. Assessing the effects of knowledge constraints, communication patterns, risk considerations and self-efficacy that determine the actual adaptation decisions of farmers in combination with more basic economic considerations is a timely and valuable contribution. I appreciate the considerable thought and the considerable effort in empirical data collection that you have put into building this model.

As far as I understand, this article is not about introducing the basic model idea and implementation (which has been done in Wens et al. 2020), but also not yet about a pure policy assessment in which the comparison of the consequences of different policy options is thought reliable enough to directly inform policy decisions. Rather, it is a methodological step in between, intended to illustrate how this previously introduced, but not yet wellestablished model can be used for policy assessment and discuss its potential contributions and limitations. I concur that this is the appropriate framing for the article given the state of the analysis --this could be formulated somewhat more clearly in the introduction and used to focus the conclusions -and based my assessment on this understanding. I have subdivided my comments into three parts: I) Main general comments on the analysis and its description and discussion in the article II) Comments on specific lines and paragraphs of the article III) Simple orthographic mistakes I GENERAL COMMENTS 1) One of my main concerns with the manuscript is an insufficient description of essential elements of the model that are key to understand and judge the theoretical and empirical foundation of the simulation results. This concerns both the description in the main text as well as the ODD protocol in the appendix. I referred to Wens et al. 2020, which helped to clarify some questions, but key information remained vague there, too. (In any case, since the model is not well-established and in widespread use, so that you can assume familiarity of the reader with the model, the key assumptions, on which the conclusions rely, should be explained in the article itself and not require reading a second article.) You model the adoption of adaptation measures as a stochastic decision, in which the probability to adopt is influenced by risk perception, self-efficacy and farmer-perceived benefits and cost of the technology. Policy options affect these determining factors in different ways. Your simulation analysis then calculates the resulting consequences of this assumed influences of policy options to quantitatively compare their overall effect for adoption speed, poverty and food security. In my understanding, the Protection Motivation Theory that you use as a theoretical starting point does provide you with indications on which explaining factors to include in the model, but it does not provide any further arguments for a specific functional form to quantify and map the influence of policy options on explanatory factors and then map the explanatory factors to a probability. This mapping is the essential and central element underlying the model and must be explained in sufficient detail in the main body of the manuscript. Your current manuscript shows a graph and describes it at the level of "this influences that", but to understand the theoretical and empirical foundation, the reader needs to know: a) the functional form of the mappings (linear, proportional, or more complex with thresholds, nonlinearities), b) the quantities assumed for the weights and parameters in that mapping (the basic model and the effect of policy options): what affects what by how much? c) Most importantly, the justification for those choices: with what reasons did you choose the forms, the values for the parameters and weights (or the boundaries you set on them in uncertainty analysis)? How did you come to them: are they ad hoc assumptions? Is there an underlying well-founded theoretical or structural value for them? Have they been calibrated (chosen such that they yield a desired model outcome) and if yes under which auxiliary assumptions, considering which error distribution, which alternative model formulations, was their identifiability actually asseed. In other words how strong is the evidence for these key parameters?
Information on these central elements remain too vague in your current manuscript.
In addition, the ODD that you present is lacking much detail. The sequencing of processes and the effects of processes on state variables remains unclear. After reading it I still don't have a clear idea of what happens after what in your model and which state variables are affected and updated when by what. (A few more details on this in part II of my assessment.) The submodels section of the ODD is actually not only for external submodels (like AcquaCrop in your case), but should describe the details of the individual processes in the model. While the Processes and Scheduling section in the overview part only names the processes and describes their sequence, the submodels section should contain descriptions of equations, algorithms and parameters of each of these processes (see section 3.7 in Grimm et al. 2010 2) Since your manuscript is intended to exemplify the use of your model as a decision support tool for ex ante policy assessment, I miss a more in-depth discussion about the validity of generalizing your model from the current situation to the future scenarios. This can, as you rightly remark, not come in the form of an empirical verification, but must be a form of structural validation. While I assume that the domain for which AquaCrop has been parameterized covers the climatic conditions that you analyze for the future, this is not self-evident for the parameters and weights used in the adaptation decision model. Given their origin (theoretical or calibrated), can all of their values be considered stable over time and across the structural breaks introduced by climate change? This should at least be shortly discussed in the context of the more detailed description I suggested above.
3) I am uncertain about some basic economic relationships in your model: a) What about the market dynamics in your model. Does your model contain market dynamics at all? Do prices vary at all --whether exogenously forced or endogenously determined? You speak of the influence of regional supply on household food security, but this seems to be purely quantity-based, without any effect on prices. Supply from outside the region does not seem to be considered. Prices that react to supply shortages would improve income opportunities for crop sellers and complicate matters for crop buyers, in good years that mechanism would work vice versa. Strictly, then also poverty thresholds that are based on valuations of minimal consumption baskets (if they are?) would have to be updated. If not, this should be explained and discussed as a potential limitation. b) I am unsure how the actual cost and benefits of the adaptation measures enter your model. I can see that they enter as part of the calculation of the probability to adopt and if I understand correctly from your previous article they can be prohibitive, i.e. lead to probabilities of zero if cost are not covered. Does this only refer to cash demands? Or is there also a consideration of labor availability. How are opportunity costs for labor used on farm instead of off-farm and vice versa? You write that most adaptation measures provide advantages in most years. Does "advantages" here only refer to positive effects on yield, or to an overall positive economic balance after subtracting additional costs from increased revenue? Would such near-certain benefits trigger uptake of measures in your model irrespective of risk appraisal for droughts, or would this every-year benefits be ignored and only the risk mitigation aspect is considered? c) As far as I understand , in your model, agents always have "receiving food aid" as a backup: Do agents consider this in their risk appraisal? Would decisions change if this food aid was not available (reality and model)? 4) I am pleased to see that you do an uncertainty analysis to check for the robustness of results. Given the uncertainty in many parameters that you admit, this is essential and communicating the robustness of your findings against this uncertainty strengthen your results. However, your current uncertainty analysis is only vaguely described and remains underexplored.
I can see three different sources of uncertainty that warrant analysis: a) The epistemic uncertainty in the weights and parameters in your model b) The initialization of the starting population c) The sequence of random numbers drawn (aleatory uncertainty) Please better describe what exactly was varied following which kind of sampling scheme? (I suggest a separate subsection on the design of the uncertainty analysis in the methodology section, currently some incomplete information is scattered over several parts). With respect to the random sequence: How did you control for randomness to ensure that observed difference between scenarios are due to structural differences between the scenarios and not through differences in random draws between runs? (Did you assess a sufficient number of agent decisions so that you can assume convergence/averaging out of random effects or did you employ a common random numbers scheme (see Troost & Berger 2016 for a discussion)?) In more general terms, while you show uncertainty distribution in a few cases, you mostly communicate single numbers often without indicating whether these constitute averages over repetitions or not. Please always report ranges and also assess whether the differences between scenarios or ranking of the scenarios remains stable over the repetitions in the uncertainty analysis, or whether their ranking/direction is reversed in some cases. Where uncertainty ranges don't overlap, the robustness of the ranking is clear, but where they do overlap this is not self-evident (see discussion in Berger & Troost 2014, which you already cite, on focusing on the distribution of the pointwise scenario differences rather than the differences between the scenario averages over the distributions). 5) I am somewhat concerned about the sampling you chose to initialize the agent population. You say that you sample based on average and standard deviation, but: Are mean and sd sufficient descriptors of the empirical statistical distribution? Are household assets, family and farm sizes really normally distributed? And are they uncorrelated (looks like you sample individual assets independently, but this is not clear from your statement)? In my experience distributions of farm characteristics are often skewed and correlated. Empirically accurate statistical properties of the starting population are potentially very important in your case, because according to your model description the speed of diffusion will depend on the composition of your model population (e.g. few or many innovating households).
II SPECIFIC COMMENTS l. 42f.: "Uncertainties in adaptive behavior are often addressed by using different adaptation scenarios, but this approach fails to capture the two-way interaction between risk dynamics and adaptive behaviour dynamics (Elshafei, 2016)." This statement is very condensed, not entirely clear what you mean here. Do you contrast fixed exogenous adaptation behavior with endogenously modeled adaptation? Or do you contrast a few scenarios vs a full global uncertainty analysis? l. 44: I have problems with the use of the term "economically rational" as a contrast to boundedly rational. In my view, it is a misconception that bounded rationality does not constitute economically rational behavior. I believe this comes from a conflation of the "rational expectations hypothesis", often equated with the assumption of perfect foresight, and the more basic assumption of rational behavior. The former refers to how expectations are formed (and how expectations relate to true outcomes), the latter how decisions are taken after an appraisal of the situation including forming expectations. It is true that especially aggregated economic models typically have to rely on both assumptions at the same time to analytically derive aggregate equations.* Nevertheless, these two aspects should be kept apart to avoid confusion -especially, since the term 'rational' bears a connotation of "doing the objectively right/the sensible" and the opposite of rational would be "irrational". Boundedly rational behavior can well be economically rational given the state of knowledge, expectations and cognitive resources of the agent (Day 2008). I suggest to use something like "perfectly optimizing under perfect foresight" or "perfect global optimization" instead. (*It is also true that for a time some orthodox economists may have considered only perfectly informed optimization as proper economic rationality, but there has always been a more fundamental, wider conception of rationality in economics, with the narrower understanding being a subset employed for specific conditions (see e.g. the discussion in Caldwell 1991). l. 58: Your wording implies "intention = motivation". Can that be equated? l. 70: "such as" is a bit vague. What does that mean? Has it been applied exactly to these communities as representatives of this context or to a stylized community that represents communities such as these (then why only mention them and not all for which the stylized community is representative). l. 82: Not clear what you want to show with the statement about what farmers "produce ... to ensure adequate supplies": the production potential (this is what they can produce per farm. How large are farms?) or the consumption demand (this is what they need and hence produce. How large are households? How does this combine with bought maize mentioned below?) or the actual production (they could maybe produce more, but they actually produce this. Why?) l. 104: what does "calibrated" refer to here? ADOPT as a whole or the AquacropOS part of ADOPT. Clarify. Also much more details on calibration are needed (for what set of conditions, how transferable/stationary are these conditions compared with the situation analyzed in this article, see general comments 1 and 2). l. 106:" which was derived as most suitable in an earlier study (Wens et al, 2020) and has proved to best describe the observed behavior of farm households (Wens et al 2021)." -> You should be more cautious in your statements here. In Wens et al, (2020) your specific operationalization of PMT in an heterogeneous ABM formulation was tested only against constant behavior and expected utility theory (assuming perfect foresight and homogeneous, simplistic preferences) in an nonheterogeneous agent setting, and you concluded "The inclusion of PMT behavior is thus better able to capture some of the variability in adoption decisions, but is nevertheless still not a complete explanation of the observed adaptive behavior of households in semi-arid Kenya". From Wens et al. 2021 I don't read a clear preference for PMT over ToPB, CONSUMAT from conclusions and the discussion. Besides methodological concerns that I would raise towards the deduction of strong conclusions on "best" models based on the methodological setups of these cited studies (confounding of effect of PMT and effects of heterogeneity vs homogeneous agents, lack of assessment of identifiability of contrasted models, lack of explicit error model/likelihood function for empirical tests in the 2020 article, lack of rigorous ex ante definition of which kind of observation would be able to clearly discriminate between contrasted theories) your wording evokes a notion of certainty and comprehensiveness in empirical support for PMT that is not consistent with the earlier manuscript themselves. I would be fine with "better than EUT with perfect foresight", "provided strong arguments for the use of PMT", but not "proved to best describe". l. 109: "12 different initialisations to allow for uncertainty in the relative importance of the behavioural factors." <--this requires much more detail about the uncertainty analysis (see discussion in the General comment 4) l. 130f.: How is farm income calculated? Especially, at what prices do you value consumption of self-produced food? (Judging from this line in Wens et al. 2020: "any additional harvest is sold (farm income, increasing the households' financial assets)" you seem to actually refer to sales revenue, not precisely income in accounting or economic terms).
Is the poverty line fixed in nominal monetary terms or is it based on a consumption basket and recalculated when food prices change? Do food prices react to supply shortages? Is supply from world markets considered? I could not find any description of an actual market module or food price variation in the model description.
l. 138: Why are these "proxies" for drought risk? Aren't these the relevant indicators themselves? Is drought risk an (unmeasurable) independent quantity or isn't it rather a statistical property of such indicators (e.g. the drought risk for income etc.). Or did you mean "proxies" for farmer well-being? l. 151f / l. 160ff: Too vague: What does added mean? Was a 365th part added to each day of the year? Or was it unevenly distributed over the days of the year? You are already defensive about the scenarios, but since you focus on droughts as extreme events the former would be a very coarse, potentially biased allocation of changes. Not sure how relevant it is as it depends on the sensitivity of the Aquacrop model to such imprecision. You should explicitly discuss the potential implications/robustness of such a simple addition on Aquacrop results. Fig. 5: The graph should be improved. While one can clearly distinguish the strategic and proactive scenarios from the other scenarios, the remaining scenarios are so close together that one can see that the climate scenario seems to make a difference, but the the details and directions cannot really be discerned. It would also be much more understandable if the shorthands for the interventions used in the graph had been mentioned in the earlier scenario description as used here. Please not in the caption, how the values have been aggregated over the uncertainty runs.
You show the number of adopted measures, but since it is higher than the number of households, many households seem to adopt several measures. Just the number of measures may however not be a clear indicator of the strength or effectiveness of adoption. Three low-cost, low barrier, low protection measures may actually be indicative of less motivation and effectiveness in adaptation than the adoption of one high cost, high barrier, high effect measure. You should reflect and discuss this somehow or justify why the number of measures is a good indicator in your case. l. 212: "This means that adaptation intention is indeed limited by a low risk perception, high (initial) adaptation costs, a limited knowledge of the adaptation efficacy or a low self-efficacy." -> That is already an interpretation and should probably better go to the discussion section, rather than the result presentation. Also it remains very coarse. It would be interesting to get some more information on which of these factors that you enumerate is most important. l.223: Instead of the clumsy "(non-)governmental", why not leave this out (just "intervention") or use "policy intervention"? Since both non-governmental and governmental are meant, basically "all" outside interventions are meant and there is no need to constantly (non-)distinguish them. Alternatively, the established distinction between "autonomous" (by farmers themselves) and "planned" (by government and nongovernmental actors) adaptation could be very helpful here (see Easterling et al. 2007 l. 235: "surprisingly" -> maybe not really surprising given the arguments in this sentence. l. 266: "Droughts, climate change and adaptation levels influence the availability of maize on this market."-> Is there a market mechanism in the model? Is there endogenous market supply and closure? l. 356: "Because results of the future scenario runs cannot be falsified or verified, this study claims not to provide a prediction of the future for south-eastern Kenya". Results for the future can never be verified before the future becomes the present (and often not even then, due to limited control of outside influence in open systems). That's the nature of prediction. In this sense, the reason you give here misses the point. I fully agree that your simulations do not provide forecasts, i.e. point predictions of the future. But the reason is that the system you are predicting is, on the one hand, unstable and not fully understood, and on the other hand, too open to outside influence that cannot be controlled in a few exogenous scenarios (which is underlined by the uncertainty in those scenarios you already have). So the system is simply not fully predictable. The question of verification and validity concerns the model and simulation setup rather then the scenario runs for the future. l. 359: "Future research can use ADOPT to study the differentiated effect of these interventions on different types of households, in order to tailor strategies and target the right beneficiaries of government interventions." This is the correct target for your simulations and should be the focus of discussing the limitations of your study. Mention why you can't provide a point prediction (see above, you already provide essential arguments), but focus the main part of your discussion on limitations towards this actual goal of your modeling (which is not the forecast): What are the limitations of your model with respect to contrasting the effect of different policy options under different circumstances? What are the limitations in highlighting and differentiating vulnerability and heterogeneity between farmers? l. 376: "We show that all investigated interventions have a positive effect on the uptake of adaptation measures,..." Was that in doubt? Or better: would your model --by design -allow for one of these interventions to slow down uptake? (Please discuss this in the discussion section and, if it does not, don't emphasize this as a major outcome, but directly come to the quantification of the effects. ) l. 377ff: "Extension services increase the adoption of low-cost, unknown drought adaptation measures while credit schemes are useful for costeffective but expensive drought adaptation measures. Ex-ante cash transfers allow the least endowed households to adopt lowcost popular drought adaptation measures. Early warning systems are more effective in climate scenarios with less frequent drought if used as a tool to create awareness and risk perception." These statements are very general and most people would subscribe to them before having seen your model or its outcomes. Could you conclude more precisely? What exactly could be seen in your model results that is not self-evident from the assumptions that you put in the model? l. 386: estimates → predicts or implies l. 388: "this study proves" → Prove is too strong: It would require proving that all the assumptions of your model are fully correct and a proof of a complete incorporation of all relevant systematic effects in your model, which you admit is not possible. The results "suggest" or "shows that under the following assumptions..." Appendix: ODD I.i "The ADOPT model...management policies" This is a description of what the model is supposedly capable of, but not what the actual purpose is. To really judge whether it fulfils the intended purpose, this purpose must be formulated much more precisely: To me it seems that the purpose is -to simulate the (welfare, income, ..?) of smallholder farming households as a function of climate effects on agricultural production mitigated adaptation decisions.
-to simulate the speed of uptake of adaptation measures by these households as a function of policy interventions, household characteristics, ...
--> in this way, to allow identifying vulnerability by detecting where adaptation does not suffice and contrasting the effects of different policy options ODD I.ii: -"stock of assets": could you name them (at least with examples)? -More precisely: I would say that "weather" is an exogenous factor, not climate change. Climate change is reflected through different input scenarios for this external factor weather. (Or do you model weather endogenously?) -What about prices? Are they exogenous (then they should be mentioned here) or are they endogenous? (Then is there an auctioneer agent or market agent or and entities/states that represent the market that should be mentioned in entities?) ODD I.iii: This is a bit too short and coarse. Please also try to formulate this in terms of the state variables and entities that you mentioned in I.ii and as actual process steps with timing/sequencing (this does that then, then this happens) -not as general relations with