# General Comments
I thank the authors for carefully replying to and successfully addressing most of the comments. After reading the rebuttal and the new version of the manuscript, several points are clearer. Although a few imprecisions and, in some cases, less than optimal performance in subtasks of the pipeline remain, they are also hard to prevent completely for large-scale data with current computational tools. The pipeline and resulting dataset ought to be a valuable contribution to the field as long as the readers and users are well aware of its limitations and do not treat it as a definite ground truth of the underlying hazard manifestations.
Below are a few adjustments that I still consider necessary for publication. Then, I list various minor details throughout the text that would profit from clarification or rephrasing at several points.
# Recommended Adjustments
- The introduction should finish with a prominent sentence pointing the reader specifically to the limitations section and recommending that users become familiar with potential influence on their results before using the dataset.
- The authors successfully resolved the ambiguous use of the term LLM, but there is still a pending issue with NLP terminology. "Language models" is mainly used to refer to models that specifically *model language*; technically, models that take a context of words/tokens as input and output a probability distribution over *words/tokens* (either the word that comes next or a missing word in the middle). As far as I understand, none of the models in this paper are used for that task. The listed models do rely on text encoders but their output layer is performing classification, token labelling, translation etc. More suitable terms to refer to them, when applicable, are transformer-based models, text encoders, transformer-based classifiers, translation models or, more generally, just NLP models. Some (but not all) of such occurrences are pointed in the comments below.
- In the spatial and temporal clustering (Section 2.4), paragraphs are ignored based on arbitrary thresholds. On the one hand, it helps remove noise, but on the other hand it also potentially loses information on events that got less media attention, exactly the ones that may not be in existing inventories.
- Evaluation: the distribution of the labels in the annotated data samples used for the evaluation should be reported. Otherwise, it is not possible to correctly interpret the metrics' values. This could be part of Table B3, which should be part of the main text. The new classifier of "non-events" is an essential addition to the pipeline. However, it has not been evaluated in Section 2.5. Only an F1-score is reported without further information in Section 2.2. Please include its evaluation in Section 2.5.
- The unit of "media information" is still the article, even though the pipeline relies on paragraphs. A long article may have multiple paragraphs about the same event. Using the number of paragraphs per event cluster in the analysis can distort results in case many paragraphs are all coming from the same text. I suggest showing a scatter plot with the number of represented unique articles vs. the number of paragraphs per event cluster. If it not too uncommon for many paragraphs to belong to same article in a given event cluster, the number of unique articles should be used instead for the analyses and argumentations.
- Since each of the tasks in the pipeline are prone to errors, a final validation with an error analysis of the dataset would be very valuable. For instance, a sample of 100 events could be manually checked for their relevance (is it really an ongoing event of the hazard category?), coherence (are all the paragraphs indeed talking about the same event), geolocation (does the location boundaries align with the paragraphs?), impact (does the final value really represent the numbers reported in the paragraphs?) and time (during the event really happen at the estimated period?).
# Minor Suggestions and Pending Clarifications
## General
- As far as I understand now after the rebuttal, only paragraphs that mention both a valid hazard AND a location can end up in the final sample. The text, e.g. at the beginning of the methodology at L118, could make it explicit. It would have avoided some confusion in my first reading.
- Due to the selected keywords, the flood sample overlaps considerable with the flash flood sample, because the term "flood" also matches the same occurrences of the terms flash flood and sudden flood. Is that intended?
- Table 2: The total counts should be reported in the same table to enable interpretation. It is also unclear what exactly the last 3 rows represent. For instance, Table B2 shows 185 HI floods in EM-DAT, whereas HazMiner has 8,298. How do these numbers relate to the reported 45? The header says "relative variation", but the computation is not clear.
## By Section
### Introduction
- L72-74: This sentence is not well-formed. The "while" should introduce a contrast; and either the "a" is spurious or a noun is missing after "reliable".
- L79: A text corpora is already composed of its texts in some format. The "extraction" (news crawling) step is needed here because this work starts from a list of URLs instead, which is not always the case in text mining.
- L82: The task is typically called "Question Answering", not "Question and Answer".
- 88: Here, the term NLP seems more suitable than computational linguistics.
- L88-90: The claim about the underuse of encoder-based models would require a citation or some evidence.
### Methodology
- L119: The rebuttal explains why this step is named "text extraction". Still, it causes confusion due to the common usage of "information extraction". Besides, step 1 also includes splitting texts into paragraphs and translating them, which are not "extraction". Step 1 could thus be more generally called "text preparation" or "preprocessing".
- L120: just models, not language models (see above)
- L122: Suggestion for clarity: "is designed to potentially document any type of natural hazards from text sources"
- Figure 1: in the output of step 3 and input of step 4, paragraphs are not only geocoded but also annotated with impact and time? Maybe use "annotated" instead, otherwise it seems that they are *only* geocoded at this point.
- L134: Please say explicitly where this collection of URLs is coming from. It is not clear it comes from GDELT. In other news databases, there might not even be URLs to crawl.
- L139: translation models instead of language models
- L141: NLP models instead of encoder language models
- L152: Is this performed with a multi-label classifier? Please specify. The use of the scores do not sound like it was a multiclass model. If it is indeed multi-label, what happens when a paragraph talks about both a flood and a landslide and both are above the thresholds: is it assigned only to the one with the highest score? Why not to both categories?
- L175: The previous paragraphs about the other two classifiers did not report any evaluation metrics. It is better to either report on all of them or move this information to the evaluation section.
- L175: Please be specific on which version of the F1 score was used. In this case, the binary F1 score version is more suitable. Is this the binary F1-score (i.e. considering relevant paragraphs as the positive class) or an average of both classes? Reporting precision and recall of the positive class would be even more informative.
- L182: "for these articles" -> how are they identified among all instances?
- L201: What happens to paragraphs that do not mention any location? Are they excluded or information from other paragraphs are assigned to it? This information is relevant here but comes only much later on.
- L236: Media attention should rather be counted in the number of *articles*. Here, it may be the case that many paragraphs about the same event come from a single long article (see comment above). If that is the case, this should rather be computed using the number of unique article identifiers of a cluster of paragraphs.
- L237: Eisensee and Stromberg (2007) are cited as a source of such claim but they do not seem to separate floods from flash floods in their analysis.
- L239: "two separate events": the concept of event here depends on the temporal segmentation, which at this point has not been performed yet. Do you mean two separate regions?
- L256: Why was a clustering algorithm applied for the temporal dimension? If periods are set with epsilon=2, simple segmentation would be enough and simpler.
- L276: What about the new "non-event" classifier?
- L278: NLP models instead of language models.
- L318: The names of the two translation models were mentioned in Section 2. They could be here instead, as they are specific to this assessment and not to the pipeline.
### Results
- L370: Section 3.2 should preferably come before the actual resulting dataset starts to be discussed. First the pipeline is evaluated, then its outputs (events) are described and discussed.
- L374: What was the distribution of the positive and negative label among these 750 instances? It is important to know how imbalanced the problem is when interpreting the metrics. One can infer it later from Figure 3, but an explicit reporting would be much appreciated.
- L377: What was the distribution of the three labels among these 2,000 instances? One can infer it later from Figure 3, but an explicit reporting would be much appreciated.
- L384: "correctly detects" -> not always, as it does not achieve recall = 1.
- L386: What exactly was the precision? Table B6 belongs to the main text.
- Table 4: Why are the min and max so extreme? If an EM-DAT event and a news event are months apart, they probably do not refer to the same fact. The alignment procedure should handle that.
- L443: Maybe the underestimations are due to using the median? Maybe a larger quantile or the mode would work better?
- L474: Here is another case where reporting the number of unique articles within the cluster of paragraphs is more reliable than the number of paragraphs, as more than one paragraph might have been extracted from the same text.
### Discussion
- L598: "its multilingual approach significantly improves..." The paper has not provided evidence to generalise this, it only showed a few specific examples.
- L663: "languages lack equivalents" -> it is still often possible to express the same concepts, despite the lack of a single term.
- L705: NLP models
### Appendix
- L805: Please also report the distribution of articles by language in the initial sample and of paragraphs by language in the final dataset.
- Table B1: Please add hyperlinks to the models' pages.
- Table B5: A paragraph that does not mention e.g. any injured people is different from a paragraph that states clearly that no one was injured. Were these cases treated separately in the classification and in the subsequent impact estimations? |
General Comments
This paper addresses a pressing research gap: the creation of reliable and validated (global) text-based datasets for natural hazard impact and adaptation research. The authors propose and implement an automated pipeline with 14 steps to cluster news paragraphs into temporally and geographically thematic clusters for three types of hazards: floods, flash floods and landslides. The work demonstrates a remarkable effort in coordinating all processing steps for large scale multilingual text data and analyzing the resulting structures. Importantly, known limitations and biases are transparently and extensively discussed in Section 4. In that regard, it is a valuable contribution with good potential to pave the way for relevant further research in various fields. Still, a few drawbacks must be pointed out and require improvement or at least a more detailed discussion of consequences and possible solutions.
Specific Comments
Clarifications / corrections needed with some suggestions
Abstract
- l1: Since the number of pages is not critically limited, the abbreviation GH could be avoided throughout the whole text. The full version "geo-hydrological hazards" does not take that much extra space and really facilitates reading.
- l15: What is data-scared?
- l18: The 58 languages should be listed in the appendix.
- l20: It does not really advance the text mining field itself as no new techniques or findings related to the text mining methods themselves are presented.
Introduction
- Almost 2.5 pages is quite long for an introduction. It mentions many related works that deserve a dedicated related work section. I suggest splitting this part into two sections, one with a compact overview of the paper and the other for the list of related work, ideally extended to account for the various tasks in the pipeline.
- l63: Text mining methods are not perfect and cannot always extract the right information. It is important to phrase it in a way that makes it clear that these techniques do not always work.
- l65: And disadvantages.
- l74: LLMs do not "understand" texts. I recommend avoiding verbs related to human cognition to refer to how LLMs process language.
- l75-78: LLMs are mentioned before GPT and BERT, which makes it sound as if these two appeared after the popularization of LLMs. I suggest reversing the order of these sentences. First, refer to the advent of Transformer-based architectures like GPT and BERT, then to the popularization of LLMs as a tool for many NLP tasks.
- l82: The "While only a few..." needs to be connected to another sentence.
- l94: It is ambiguous whether "the dataset" refers to GDELT or your dataset.
- l96: The "Followed by..." needs to be connected to another sentence.
Methodology
- It is a matter of personal taste, but since Figure 1 already shows an overview of the pipeline, it would be better if the description of each subtask was already accompanied by its own evaluation. It is hard to read and move on without knowing how well each model performs at each step. Then Section 3 could focus only on describing the resulting dataset.
- l109: What does it mean that HazMiner was "specifically" configured to extract GH events, isn't it its main purpose?
- Figure 2: "hazardous text" sounds like the text itself is hazardous.
- Figure 2: An arrow connecting the output of a step to the input of the other would be useful to avoid the reliance on color, which may not be visible for color-blind readers.
- 2.1 should be named "text selection" or "document selection", as it involves steps that are not about extraction.
- l115: Subtasks rather than suprocesses.
- l116: Scrapping news is usually not allowed by media outlet websites. Was it performed in accordance with the terms of use?
- l119: The simple heuristic to use backspace to identify paragraphs can be problematic, as there is no guarantee that all websites follow this rule and this information may be lost if the raw texts were already altered by news aggregators. This step should thus also be evaluated.
- l121: It is not really more efficient because it makes the number of instances to process much larger and removes important context needed for disambiguation.
- l122: It's not true that current LLMs are optimized for shorter inputs (unless it's not actually LLMs but text encoders, see comment above). Not fewer parameters but fewer computations.
- l150: I don't know how these specific embeddings were optimized. But generally, embeddings are trained to capture contextual relations, it is expected that the representations of all these concepts will be close in the embedding space.
- l161: Publication date is not very reliable as the timing of the hazard because many important events resurface in the news much later, and some are delayed.
- l170: It's unlikely that a rule-based approach really solves this problem. There are many news about studies that refer to events as a general category, or about events that occur in movies or books, which are not always in future tense.
- l178: This assigns coordinates to all locations mentioned in the text, but how is the event location detected among them? Normally, there is a third step.
- l180: Hallucinations is a common (although not very suitable) term for wrong content generated by text completers / chatbots. It seems that, here, it refers to normal translation errors instead.
- l196: Weighted average may likely end up in a location that is not even in the text, maybe even in a different country.
- l220: The sentence "As hazard events..." is not finished.
- l262: It is unclear what "until the number of false positives remained constant" means here.
- l290: The fact that the timing algorithm does not rely on neural network-based models does not mean it works well. It's a rule-based approach that will not always work.
Results
- For classification, traditional metrics like binary precision, recall and F1 scores should be reported in a Table. Only recall (true positive rate) was reported but, in such tasks, high precision is as (or even more) important, as "inventing" events that never happened can be a more serious mistake than missing real events depending on the use case.
- l345: Precision is quite low, though (only around 0.6 based on the confusion matrix). So the performance is mostly coming from identifying the vast majority of unrelated instances.
- l364: How exactly are the errors computed? Are the bounding boxes considered or only the centroid?
- l380: Very often, news report on preliminary number that keep being updated as the event unfolds. How was this handled? If multiple paragraphs from different articles are reporting about the same event, how are their estimates harmonized?
- l393: The mean duration is not of the GH itself but of its news coverage?
- l468: "one death but fewer than 10 fatalities" sounds like a death is not a fatality. Maybe rephrase.
Discussion
- l509: with the goal
- l520: The sentence "Mostly because..." should be connected to another sentence.
- l524: The term "only" is not correct here as it can include events with less deaths if they affect enough people.
- l545: The sentence "As studies..." should be connected to another sentence.
- l574: country's
- l643: It's not the language that is non-alphabetical, it's the writing system (see https://en.wikipedia.org/wiki/List_of_writing_systems).
- l671: The sentence "Such as news..." should be connected to another sentence.
- l674: into the (or rather "as input to the"?)
- l716: Switching models is likely not that easy, and each may use different labels, output format, programming libraries or require a different input structure.
- l725: Subconsciously is not a suitable term since model/study are the agents in this sentence.