Thank you for your appreciation and for reading our work.

Ans1:

Generating aftershocks (or earthquakes) based on the probability map of a model does not involve any changes in the mainshock parameters. However, it is about simulating events based on the probability distribution.

In simulating catalogs with different earthquake numbers but a fixed number of cells containing earthquakes, there can be two ways to achieve this programmatically. We can keep simulating catalogs until we get the catalog with the desired number of earthquakes and active cells. This can be computationally expensive. Secondly, we can simulate one earthquake at a time and keep simulating until we get the desired number of active cells. In this work, we use the latter approach.

Please find all the codes to reproduce results here:

https://github.com/khawajasim/aftershock_forecast_testing.

The functions to simulate catalogs can be found in the module “utils”, along with the required documentation.



Ans2:

We decided to have just one ROC for R-model because it has a positive forecast everywhere and the ROCs remain similar. Since our goal is to observe the behavior of ROC when an earthquake occurs in a negative region of ΔCFS so introducing ROCs for the R-model would only clutter the figures with more lines without adding meaningful information.

We use an inverse cumulative distribution for one model (which happens to be R-model in this case), and a cumulative distribution for the other model, because we think that in this way we can easily distinguish visually whether two curves are clearly separated or not. We can also use cumulative distribution for the R-model and inverse cumulative for ΔCFS.

Ans3:

It refers to the difference between the performance of the ΔCFS and the R model. ΔCFS(1%) and R-model(99%) mean that we allow 1% flexibility or overlap according to our criteria. Alternatively, this could be set to 2.5% (i.e. ΔCFS(2.5%) and R-model(97.5%)), or 5% or 10%, etc. However, we have used 1% because we want to allow less overlap between the performance of the ΔCFS and the R model.



Ans4:

We selected earthquakes that occurred within the one year after the mainshock with a horizontal distance of less than 100 km from the mainshock fault, because the seismicity in this spatial and temporal vicinity to the mainshock is usually dominated by aftershocks, and previous studies have used similar selection rules (Sharma et al. 2020, Asayesh et al. 2022, cited in the manuscript). However, some of the selected earthquakes may be background events unrelated to the mainshocks. We refrain from labeling the test data with a sophisticated approach such as the ETAS model, because a simpler approach is that the forecast models already account for possible stationary background activity. Here we have ignored background activity, which may have a small effect on our result, but won’t change the main result because the background activity is only a small fraction of the total activity. However, we have now clarified in the revised manuscript that we have ignored possible background activity in our test of the two real aftershock sequences.

Note that these are just two case studies for aftershock forecast evaluation (for Chi-Chi and Landers), that we have discussed here. There are several options for other case studies or data selections. However, the final conclusion of this study does not depend on this evaluation.

Sharma, S., Hainzl, S., Zöeller, G., and Holschneider, M.: Is Coulomb stress the best choice for aftershock forecasting?, Journal of Geophysical Research: Solid Earth, 125, e2020JB019 553, 2020.

Asayesh, B. M., Zafarani, H., Hainzl, S., and Sharma, S.: Effects of large aftershocks on spatial aftershock forecasts during the 2017–2019 western Iran sequence, Geophysical Journal International, 232, 147–161, 2022

Hello Behnam, Thank you for providing some comments from readers’ perspective.

Ans1:

The best possible values of MCC and F1 are 1. So the best performance of the forecast in terms of MCC and F1 is the one closest to 1. So the performance of (MCC, F1) nearest to (1, 1) provides the best performance.

Ans2:

For MCC-F1, the procedure of drawing the curve between MCC and F1 is the same as proposed by Cao et al. (2020). We differ slightly from the idea of Cao et al. (2020) for computing a single performance metric using the MCC-F1 curve. Cao et al. (2020) propose to compute the average distance of the MCC-F1 curve from the point of ideal performance, i.e. (1, 1). However, we slightly differ from this idea. MCC already incorporates all the entries of a confusion matrix, i.e. True Positive (TP), False Positive (FP), True Negative (TN), False Negative (TN). It never increases or decreases monotonically with varying decision thresholds, unlike in the ROC curve. Thus with average distance, we can never achieve the perfect performance of one for an ideal forecast. On the other hand, the highest performance can reveal the maximum potential of the forecast, so we choose the testing metric based on the minimum distance from the point of ideal performance (1,1).

Thank you for taking the time to review the manuscript and appreciating the effort.

Line 29 (Response): Corrected.

Lines 34-35 (Response): We have changed the wording and tone of the sentence.

… “It is desirable to use those forecast models for societal decision making that have proven their applicability through testing”…

Line 41(Response): Done.

Line 86 (Response): Noted.

Line 128 (Response):  Modified the sentence to clarify the meaning.

... "The reliability of the testing models for any type of dataset is primarily associated" ...

Line 166 (Response):  We have now provided the reference in the text.

Line 177 (Response): ‘x’ refers to ∆CFS in this case. We have modified and replaced x with “Model”.

Line 182 (Response): Thank you for the suggestion. We have added the reference.

Lines 195-196 (Response): These lines do not discuss the forecast testing based on a real earthquake catalog. Instead, it refers to the synthetic experiment. In this experiment, we use the simple ∆CFS model as a seismicity generator, and the same is being used for the forecast evaluation. Thus in this context, the ∆CFS is the perfect forecast model because it is also the catalog generator, so it should be able to perform better than any competing uninformative model.

We think that we failed to convey this in the first version of the manuscript, so we have modified the text slightly to make it clearer. Please see line 201 of the revised annotated manuscript.

Line 326 (Response): In this manuscript, we have not discussed the issue of computational resources in the context of aftershock forecast testing, so it would be inappropriate to add this to the conclusion.

Figs. 1a and 1b. (Response): The forecasts shown in the Figures 1a and 1b show the forecast rates (ƛ), and their calculation is provided in the manuscript. We have further elaborated the text and added this to the legend and caption of the figure.

Figs. 2b and 3b. (Response): Thank you for pointing this out. We have corrected them.

Table 1. (Response): We have corrected it.

Ans1:

Although the manuscript proposing the MCC-F1 curve has not yet been published in a journal, but we find it useful. However, MCC and F1 are already published and discussed performance measures in the literature. We find the MCC-F1 curve useful because like ROC, it provides performance across different decision thresholds.

Ans2:

To address this question, we would like to refer to a recent study (Khawaja et al. 2023) which showed that aggregating a forecast on a multi-resolution grid does not change the consistency of a forecast model, as long as a model is actually consistent with the actual observation.

Khawaja, Asim M., et al. "Statistical power of spatial earthquake forecast tests." Geophysical Journal International 233.3 (2023): 2053-2066.

Ans3:

Yes, we considered the performance measures that work using varying thresholds, for example, the Precision-Recall curve. The Precision involves TP and FP, while Recall involves TP and FN. Similarly, both together take into consideration TP, FP, and FN, leaving out the effect of TN. Therefore, we preferred a performance metric that takes into account all four entries of a confusion matrix.

However, the novelty of this study is to provide a guideline on the minimum amount of data required to evaluate earthquake forecast models in terms of binary classification.