Review for “Probabilistic and Machine Learning Methods for Uncertainty Quantification in Power Outage Prediction due to Extreme Events”

This is a comprehensive work that compares different machine learning methods. I find the methods and presentations are solid and the authors did nice job in summarizing the substantial works they have finished. However, I do find some critical information is missing. Namely, they need a comparison of performances for all ML models with the separate testing data because it will show whether their models have overfitting issues and how their performances for new data never encountered. The author also needs more illustrations of how their model input data is obtained and possible associated uncertainties. Based on those, I suggest a major revision for this version. Please see detailed suggestions below.

Line 34, “including hurricane”, is it wind field or else? Please clarify.

Uncertainty in the poweroutage.us data. 

Line 105 to 111, what are the possible uncertainties in interpolate all covariates into the city scale? Which interpolation method is used? Please be specific.

Line 113: number of outages, is it the same as customers without power?

Line 125: Uncertainty in wind speed estimates since the sizes of cities vary.

Line 147: Which rescaling technique is used for this one?

Line 175: Please fix the citation.

Table 2: how to interpret the difference between R2DEV and R2ψ?

Why is random forest used only for the fraction of customers without power? Is the number of power outages not fit the RF algorithm?

Figure 5: what are R2 and other error statistics in the holdout test? It will be helpful to report them in the same figure.

Do you have any prediction vs observation plot for the RF model like Figure 5?

Section 8.2, there is a heavy discussion on how winds control the power outage from the models. However, how precipitation is related to power outages is not shown, as it is the second most important variable in the RF model. You have demonstrated some nonlinear relationships between wind speed and power outage fraction. Therefore, it is worthwhile to show precipitation’s relationship to outrage fraction or show precipitation and wind jointly with outage prediction in a separate pdp plot. That may explain some nonlinear relationships in Figure 7b.

Section 9, the author mentioned beta regression may have better performance. But no comparison is made with the previous method. I suggest shortening the arguments after line 454 because there is no evidence in the paper supporting them.

Line 469 to 470, unlike linear models, RF does not have the assumption of non-collinearity.

This paper investigated the limitations of existing power outage models, including bounded prediction, out-of-distribution prediction, and physics-aware uncertainties The authors found some of the existing state-of-the-art models may generate unrealistic predictions, and cannot generalize well to extreme events that are not sufficiently represented in the training datasets. The authors discuss some potential ways to address the shortcomings of these models. I have some major comments that authors need to address before publication:

1. The problems mentioned by the authors, including limited generalization ability, unbounded predictions, and unreasonable uncertainty variations, are common problem for general machine learning models. Many machine learning community researchers proposed different methods to address these problems. How unique and critical are they for power outage predictions?

2. Now there is a variety of more complex power outage prediction models [1], are there any specific reasons for the authors to choose to evaluate traditional machine learning models? These traditional models are known to be less representative.

3. It is unclear to me why beta regression should perform well in general cases. I think it also has its own problems such as strict distribution assumption and does not address the representativeness issues which eventually cause the poor generalization problem. Could you provide any justifications and performance comparison regarding why Beta regression should be used?

[1]Xie, Jian, Inalvis Alvarez-Fernandez, and Wei Sun. "A review of machine learning applications in power system resilience." In 2020 IEEE Power & Energy Society General Meeting (PESGM), pp. 1-5. IEEE, 2020.