An Intelligent Recording Method for Field Geological Survey Data in Hydraulic Engineering Based on Speech Recognition

Field data collection and recording in geological surveys are crucial tasks in hydraulic engineering construction, characterized by their necessity and preeminence [1]. The raw data from hydraulic engineering geological surveys are critical for describing the on-site geological environment and serve as a primary source for geological big data [2], featuring large data volumes and diverse types. The complex work environment of hydraulic engineering geological surveys makes data collection by handwriting or typing in the field inconvenient, hindering the accurate and efficient gathering and recording of geological survey data. Therefore, designing and developing a novel method for field geological data recording that simplifies the data collection process is essential for reducing the difficulty of data acquisition and improving the accuracy and efficiency of hydraulic engineering geological data collection.

Since the 1970s, methods for geological data collection have undergone significant reform and innovation. Traditional methods involved recording data in field notebooks and later storing the collected data in databases for management, which often resulted in issues such as non-standardized recording formats and low collection efficiency [3]. To address these problems, the Queensland Geological Survey combined handheld devices with supporting software to develop a mobile GIS-based field geological data collection technology [4]. This approach offers advantages such as low power consumption and ease of operation, integrating data input, storage, management, and output. However, it also has drawbacks, including insufficient utilization of built-in sensors and cumbersome data collection processes [5].

In recent years, as mobile devices have become increasingly intelligent and hardware performance has rapidly improved, the variety of hardware sensors has also expanded. Utilizing these latest software environments and hardware devices to make field data collection more efficient and accurate is a growing trend in modern information technology. Mobile GIS-based field geological data collection technology has thus become a more efficient collection method [6]. Furthermore, the significant advancements in artificial intelligence (AI) have led to its widespread application across various fields. The use of AI-based mobile devices for data collection is simple and convenient, enhancing the efficiency of fieldwork for geologists [7], [8], and holds substantial significance for revolutionizing traditional field data collection methods.

The complex fieldwork environment in hydraulic engineering geological surveys makes geological investigation and data collection particularly challenging. For intelligent geological data collection, it is necessary to quickly and conveniently record observed data into devices by applying various advanced technologies to achieve efficient and convenient data recording. However, using mobile devices for field data collection and recording often requires one hand to observe samples while the other operates the mobile device. This sometimes requires additional tools such as magnifying glasses, geological hammers, and compasses [9], making single-handed operation challenging, prone to errors, and inefficient. Therefore, a data collection method that avoids occupying both hands is needed to make the geological data collection process more convenient, reducing the complexity and operational difficulty of field geological work.

With the rapid development of deep learning methods, human-computer interaction has gradually evolved from mouse-keyboard interaction and touchscreen interaction to natural language interaction [10], [11]. Using voice control for field geological data collection, where data is collected by voice control of mobile devices without manual operation, allows data to be converted to text through speech recognition, which is faster than manual text entry on mobile devices, significantly improving collection efficiency. Speech recognition technology has been widely applied in areas such as terminal control [12], software interaction [13], and meeting transcription [14]. Currently, open platforms like iFlytek and Baidu Voice provide technologies such as speech recognition, lexical analysis, and speech synthesis, which can support general field geological data observation and recording [15]. However, there are limitations in recognizing specialized geological terminology.

The current implementation methods for speech recognition mainly include offline and online speech recognition, each suited to different application scenarios [16]. Offline speech recognition is characterized by high accuracy and fast recognition speed, but its primary drawback is the limited scope of recognizable speech, as it can only recognize content within its built-in language models and acoustic model libraries [17]. Online speech recognition, on the other hand, has a broader recognition range, supporting multiple languages such as Chinese and English [18]. It is suitable for general tasks such as daily communication, software interaction, and intelligent device control, but its recognition accuracy for specialized geological terminology is relatively lower than offline methods, as it lacks a specialized geological lexicon. Using a general lexicon for field geological data collection often results in high rates of misrecognition. Considering the challenges of weak signals and low bandwidth in fieldwork environments, it is necessary to develop an offline speech recognition method tailored for hydraulic engineering geological survey scenarios. This would enable the verbal description of observed geological information in the field, which is then intelligently converted into text records, improving the efficiency of field recording and addressing the inefficiencies in field geological data collection.

In response to these challenges, this paper proposes an intelligent recording method for field geological survey data in hydraulic engineering based on speech recognition interaction. This method simplifies the data collection process in geological surveys, improving the accuracy and efficiency of data collection in hydraulic engineering geological surveys. Initially, geological survey text data were collected and used to create a geological survey speech dataset through manual recording and speech synthesis. Then, a speech recognition acoustic model tailored for geological surveys was trained based on the Conformer acoustic model architecture [19], and a geological survey-specific language model was trained using the N-gram algorithm [20]. Finally, the recognition performance of the proposed method was evaluated using CER and single-character decoding time as metrics, and its effectiveness was validated through comparative testing. Experimental results demonstrate that the proposed offline recognition method outperforms classical speech recognition models and mainstream commercial software, efficiently and accurately enabling the collection and recording of hydraulic engineering geological survey data, effectively addressing the current inefficiencies and error-prone nature of field geological data collection.

Hydraulic engineering geological terminology is characterized by its strong domain specificity, high level of specialization, low frequency of common usage, and a high occurrence of uncommon characters, making it difficult to be accurately recognized by mainstream speech recognition models and programs currently available on the market. Therefore, geological-related corpora are used to construct a speech recognition dataset for hydraulic engineering geological surveys, which is then used to train the speech recognition model. The geological specialized speech data includes audio and its corresponding text, with the text data sourced from various geological professional materials, mainly including hydraulic engineering geological monographs, textbooks on hydraulic engineering geology, fieldwork specifications for geological surveys in water conservancy and hydropower engineering, geological borehole exploration entry materials, and engineering geological survey reports.

To convert the text data into dataset labels for input into the acoustic model, the following four preprocessing tasks are required: (1) Content Extraction: Convert the geological specialized materials into editable text and manually remove information sections with low relevance to geological surveys. (2) Text Cleaning: This includes removing meaningless data such as spaces and punctuation marks, replacing Arabic numerals, letters, and other non-Chinese characters with Chinese characters, and removing meaningless data such as spaces and punctuation marks [21]. (3) Sentence Segmentation: Break down long sentences into multiple short sentences of 2 to 25 characters based on key punctuation marks such as commas, periods, and semicolons, forming 16,352 sentences of text data. (4) Word Segmentation: For each short sentence, segment the sentence into word information based on sentence components and vocabulary attributes, using spaces as delimiters, while retaining specialized geological terms. The overall preprocessing workflow is shown in Figure 1.

To obtain audio files matching the text data, audio data is generated through two methods: human voice recording and speech synthesis. Human voice recording is carried out by 15 recording personnel who read the text data content in Mandarin in a quiet environment, recording each line using a mobile phone microphone. Speech synthesis is done by randomly using multiple Text to Speech (TTS) models [22] to generate audio sentence by sentence, simulating multiple different speakers to enhance the diversity of the audio data, thereby ensuring the generalization ability of the speech recognition model. The TTS models used for speech synthesis are shown in Table 1.

Referring to the format of the Aishell dataset [23], the text sequences and audio sequences are mapped to one-to-one relationships, and placed line by line in the record file, constructing the speech recognition dataset for hydraulic engineering geological surveys. This dataset contains 16,532 audio files with a total duration of 13.53 hours. In terms of professional domain recognition, this dataset can ensure recognition accuracy while effectively maintaining the model's generalization performance due to its large volume, thereby ensuring its anti-interference stability during actual use. Considering the large scale of the dataset, it is randomly divided into training, validation, and test sets according to a ratio of 18:1:1 [24] to simulate field geological survey conditions as closely as possible, ensuring the reference value of the model evaluation. The process of constructing the dataset is shown in Figure 2.

The conversion of acoustic signals is the fundamental task of speech recognition. According to the waveform of the speech signal, acoustic signals are converted into effective acoustic features to describe and capture different speech signals. The effectiveness of feature extraction directly impacts the accuracy of subsequent speech recognition.

To fully extract the features of the human voice in the speech signal while reducing the impact of background noise on linguistic information, this study uses Mel-scale Frequency Cepstral Coefficients (MFCCs) as the features of the input acoustic signal, which are widely used in speech recognition [25]. MFCCs are cepstral parameters extracted in the Mel-scale frequency domain. The Mel scale describes the nonlinear characteristics of human ear frequency perception, and its relationship with frequency is as follows:

Based on the audio files, MFCC features are obtained through the following eight steps:

(1) Pre-emphasis. The high-frequency part of the speech is easily lost after being emitted by the human vocal organs. Pre-emphasis is used to compensate for the amplitude of the high-frequency part of the speech signal. The calculation formula is as follows:

where, $x(n)$ is the $n$-th sample point of the input speech signal, and $\alpha$ is the pre-emphasis coefficient.

(2) Framing. Speech signals exhibit short-term stationarity, so a segment of speech over a short period is taken as a frame. To ensure continuity of the speech signal after framing, a portion of the previous frame is retained in the next frame, called the frame shift. Typically, the frame shift is set to 10ms, and the frame length is set to 25ms.

(3) Windowing. Windowing is associated with framing and smooths the frame through a window function. A Hamming window is used to retain the frequency characteristics of the speech signal effectively, and its function is as follows:

where, $a$ is 0.46, and $N$ is the window length.

(4) Fourier Transform. The Fourier Transform converts the signal from the time domain to the frequency domain for spectrum analysis. The Short-Time Fourier Transform (STFT) is suitable for short-term stationary signals, and its expression is:

where, $x$ represents the speech signal, $t$ is the sequence number of the frame, $w$ is the window function, and $b$ represents the transformation coefficient at the $k$-th frequency point.

(5) Mel Filter Bank. The Mel filter bank is used to extract frequency bands in the low-frequency region, and its function is represented as shown in Figure 3. The response of the $m$-th filter at the $k$-th frequency point can be calculated by the following formula:

where, $f(m)$ is the center frequency of the filter.

(6) Logarithmic Power Operation. The amplitude spectrum obtained from the Fourier Transform is squared to obtain the short-term power spectrum. By multiplying and accumulating through the filter, the logarithm is taken to obtain the logarithmic power spectrum, which relatively amplifies the low-frequency signal. The resulting feature is the F-bank feature of the audio, and its expression is as follows:

where, $M$ is the total number of filters.

(7) Discrete Cosine Transform (DCT). The logarithmic energy obtained above is input into the DCT to concentrate the signal's energy, and the MFCCs are obtained according to the following formula:

where, $L$ represents the order of the MFCCs, usually set to 12$\sim$16.

(8) Dynamic Differential. The DCT only obtains static MFCC features. Dynamic MFCC features can be obtained by taking the differential of the static features. The calculation method for the first-order differential coefficient is as shown in Eq. (8), and the second-order differential coefficient is obtained by repeatedly substituting the first-order differential coefficient into the equation. The combination of static, first-order, and second-order coefficients results in the complete MFCC features.

where, $t$ is the sequence number of the frame, and $N$ represents the frame sequence difference of the first-order derivative, which can be 1 or 2.

The overall extraction process of MFCCs features is shown in Figure 4.

The acoustic model is one of the most important modules in a speech recognition system, directly affecting the system's performance [26]. The acoustic model takes the vector sequence obtained from feature extraction of the speech signal as input and establishes a mapping relationship between speech features and phonemes to obtain the probability of the speech waveform corresponding to the model's output speech signal.

The Conformer model combines the strengths of the Transformer [27] and Convolutional Neural Network (CNN), leveraging CNN's advantage in capturing local features while retaining Transformer's capability of acquiring long-range dependencies. This enhances the network's ability to model both global and local dependencies simultaneously, demonstrating outstanding performance in recognition accuracy, inference speed, and model parameter size.

The core component of the Conformer model is the Conformer block, which mainly consists of four modules: the feedforward network, multi-head attention mechanism module, convolution module, and the second feedforward network. Its structure is similar to a macaron structure, where the two feedforward networks each contribute half of the Conformer's output. The speech features xi are processed by the Conformer model as follows:

where, $FFN$ represents the feedforward network, $MHSA$ stands for the multi-head self-attention module, and $LN$ represents layer normalization. Residual connections are used between each module.

In the encoding stage, the audio is first processed by a convolutional subsampling layer, and then multiple Conformer blocks are used for further processing, as shown in Figure 5. In the figure, (a) represents the Conformer model structure, (b) shows the upsampling and downsampling operations for aligning the input feature space, and (c) illustrates the fully connected layer structure of the feedforward network.

The multi-head attention mechanism [28] module replaces the positional encoding in the Transformer with the relative sinusoidal positional encoding from Transformer-XL, allowing the attention module to generalize better across different input lengths and making the generated encoder more stable to changes in audio sequence length.

The self-attention mechanism is a variant of the attention mechanism, reducing the dependency on external information and being more adept at capturing the internal correlations within data or features. In the self-attention mechanism, the $Q$ (query) vector, $K$ (key) vector, and $V$ (value) vector all originate from the same input sequence $X$ obtained from character encoding. This means that after the model reads the input information, it determines the most important information based on the input itself. The calculation process is shown in Figure 6.

In this study, the correlation between $Q$ and $K$ is first calculated using scaled dot-product to capture the local relationships within the input sequence. To handle the scale differences between $Q$ and $K$, their scores are scaled to stabilize the variance of the attention weights. The results are then normalized using the softmax function to highlight the weights of important elements, resulting in the attention weight coefficients. Finally, the $V$ values are weighted and summed according to the attention weight coefficients to obtain the Self-Attention Value, with the calculation formula as follows:

In the formula, $Q, K$, and $V$ are obtained from the original input sequence $X$ by multiplying it with the linear transformation matrices $W_Q, W_K$, and $W_V$ respectively; $\operatorname{dim}_k$ represents the dimension of $K$ and $V$.

When encoding the information at the current position, the self-attention mechanism tends to focus excessively on its own position, making it less effective than CNN in capturing useful information. To address this issue, the multi-head attention mechanism is used. It applies $h$ (generally $h$=8) different sets of independently learned linear transformation matrices to the input sequence $X$ to transform $Q$, $K$, and $V$. These $h$ sets of transformed $Q$, $K$, and $V$ then undergo self-attention mechanism operations in parallel, resulting in multiple Self-Attention Values. Finally, the $h$ Self-Attention Values are concatenated and transformed through another learnable linear transformation matrix $W_0$ to produce the final Multi-Head Self-Attention output. The overall calculation process is shown in Figure 7.

The convolution module consists of five parts: a pointwise convolution layer, a Gated Linear Unit (GLU) activation function, a depth-wise convolution layer, a Swish activation function, and a second pointwise convolution layer. The structure is shown in Figure 8.

Depth-wise convolution and pointwise convolution are used together as an efficient combination to replace traditional CNNs, effectively reducing the number of network parameters and improving computational efficiency. In depth-wise convolution, one convolution kernel is responsible for one channel, meaning that each channel is convolved by only one convolution kernel, focusing solely on the dependencies within the sequence in each channel, without considering dependencies between different channels. On the other hand, pointwise convolution is very similar to regular convolution operations, with a convolution kernel size of 1×1×$M$, $M$ equals to the number of channels in the previous layer. Its convolution operation weights and combines the feature maps from the previous step along the depth direction to generate a new set of feature maps with the same number as the convolution kernels, thus focusing on dependencies between different channels while ignoring intra-channel dependencies.

The GLU activation function is an activation function used in neural networks that incorporates a gating mechanism, helping the network better capture long-term dependencies in sequence data. The GLU activation function is defined as follows:

where, $X$ is the input vector, $\otimes$ represents element-wise multiplication, $g\left(x_i\right)$ is the intermediate vector obtained through the convolution layer, and the Sigmoid function is defined as follows:

The Swish activation function has characteristics such as a lower bound, smoothness, and non-monotonicity, effectively addressing the problems of gradient vanishing and neuron death encountered in the ReLU activation function. Its calculation formula is:

where, $\beta$ is a trainable parameter.

A FFN is a fully connected feedforward neural network consisting of two fully connected layers and a nonlinear activation function. The feedforward module in the Conformer model uses the Swish activation function. The first fully connected layer is used for dimensionality expansion, and the second fully connected layer is used for dimensionality reduction. This design aims to perform nonlinear transformations and mappings of the embedding vectors, allowing the model to learn more abstract features. Additionally, layer normalization, the Dropout mechanism, and residual summation are introduced to accelerate network training, improve the model's generalization ability, and alleviate the gradient vanishing problem [29]. The feedforward module's calculation process is shown in Figure 9.

During training, the Connectionist Temporal Classification (CTC) loss and attention loss are used to supervise the training of the CNN and Transformer branches, respectively [30], to obtain features that exhibit both CNN and Transformer characteristics. During inference, the outputs of these two classifiers are summed to serve as the prediction results. The cross-entropy function is expressed as follows:

where, $Ti$ represents the actual probability distribution, and $Pi$ represents the predicted probability distribution, both of which are normalized using the softmax function.

The CTC loss function measures the gap between the predicted result and the actual result when no annotated text is provided. For an input sequence $x$ of length $T$ and an output sequence $y$ of length $U$, the cross-entropy loss function is:

where, $P(y \mid x)$ represents the probability of the output sequence $y$ given the input sequence $x$.

The attention loss function measures the model's focus on different positions in the input sequence. For an input sequence $x$ of length $T$, an output sequence $y$ of length $U$, and corresponding attention weights $a$, the loss function is expressed as:

where, $a_{u, t}$ represents the degree to which the model focuses on the $t$-th position in the sequence when predicting $y_u$; $y_u^t$ represents the predicted probability of the $u$-th character. The logarithmic likelihood of these two parts is combined in a weighted sum.

The loss function for the joint training process of decoding usually includes both cross-entropy loss and attention loss, calculated as follows:

where, $\lambda$ is used to set the weights of the two types of loss, generally set to 0.3.

The purpose of the language model is to further decode the output of the acoustic model, converting the speech signal into the corresponding text sequence [31], and it is used for model CTC beam search decoding prediction. The CTC beam search method involves having B-Size candidate sequences, and at each time step, generating a new set of the best B-Size candidate sequences. The final result is the sequence with the highest probability among the B-Size candidate sequences. The principle of CTC beam search is shown in Figure 10.

A language model is a knowledge representation of a sequence of words. By using a statistical-based language model and processing a large specific text corpus, the probability distribution of a given word sequence can be obtained, further determining the likelihood of a text sequence and finally obtaining the accurate prediction result of the text sequence with the highest probability.

Currently, the N-gram algorithm is the most commonly used language model in speech recognition systems. Its basic idea is to introduce the Markov assumption, considering that the probability of each word occurring is only related to several preceding words, and it ignores longer-distance context. Based on this assumption, the joint probability of words in a sentence is calculated to determine the likelihood of the sentence. N is usually an integer between 1 and 5, and when $N$ is 3, the corresponding Tri-gram algorithm is as follows:

where, $S$ represents the sentence, and $w_i$ represent the word sequence forming the sentence.

In scoring the candidate results of decoding, in addition to the score of the acoustic model, there are also additional language model scores and length penalty scores. Let $W$ be the decoding result and $X$ be the input speech. The final score is calculated as follows:

where, $am$ represents the acoustic model score, $lm$ represents the language model score, $\alpha$ and $\beta$ are the set hyperparameters.

Field geological survey text data in hydraulic engineering involves many specialized terms and rare characters. The current N-gram language models are mostly trained based on common Chinese text [32], such as the large-scale Giga Chinese model (zh_giga.no_cna_cmn.prune01244.klm, 2.75 GB) and the lightweight People’s Daily 2014 corpus model (people_2014_corpus_char.klm, 0.14 GB). These language models, due to their corpora not fully encompassing the content of the hydraulic engineering geological domain, result in higher CER in recognition and may lead to situations where specific characters cannot be displayed in the recognition results, as shown in Figure 11.

To solve the problem of missing characters and further optimize the speech recognition results, geological specialized text data was converted to form a geological specialized Chinese text corpus. Additionally, by combining the ToRCH2009 Modern Chinese Balanced, ToRCH2014 Modern Chinese Balanced, ToRCH2019 Modern Chinese Balanced, and The BFSU DiSCUSS four open-source general Chinese corpora, the KenLM toolkit was used to train the model and compress it into binary format, forming a specialized language model for hydraulic engineering field geological surveys. To ensure that the model accurately and comprehensively learns the probability distribution of text sequences, the N value of the N-gram model was set to 5 [33].

Model fine-tuning is an important technique for addressing the challenge of limited computational power under large-scale data conditions [34]. It allows a general model to quickly adapt to the specific needs of a domain using a small amount of labeled data, thereby improving model performance, increasing recognition speed, and enhancing the accuracy of recognizing targets in specific fields. Additionally, training on top of an existing large-scale dataset helps the model fully learn acoustic features, enhancing its generalization capability and anti-interference stability [35].

The task of specialized speech recognition for hydraulic engineering field surveys involves numerous geological terms and related expressions that are rarely encountered in everyday language. The current mainstream speech recognition models are trained on datasets mostly derived from publicly available common language materials, where geological survey-specific vocabulary and speech account for a small proportion, resulting in weak recognition capabilities for geological domain-specific speech. Therefore, it is necessary to fine-tune the existing mainstream speech recognition models for the geological domain to improve their performance in this field.

When fine-tuning the geological-specific speech recognition model, the WenetSpeech [36] large-scale Chinese speech dataset is first used to train the model as a base model for transfer learning. Then, fine-tuning is performed on the specialized dataset for geological field surveys, updating all parameters of the entire network during the fine-tuning process. Cepstral Mean and Variance Normalization (CMVN) features [37] are used to normalize the speech signal and remove noise, while CTC loss and attention loss are used for supervision.

To ensure the recognition performance of the model, this study uses decoding speed and CER as metrics to evaluate the performance of the proposed speech recognition model. The time taken to decode a single character represents the time required by the model to decode the audio corresponding to a single character, and it is used to assess the model's recognition efficiency. The smaller the unit decoding time, the higher the model's recognition efficiency. CER is used to evaluate the degree of difference between the predicted text and the original reference text, reflecting the accuracy of text recognition. The lower the CER, the better the model's recognition performance. The calculation method is as follows:

where, $S$ represents the number of substitutions in the predicted result relative to the original text, $D$ represents the number of deletions in the predicted result, $I$ represents the number of insertions in the predicted result, $N$ represents the total number of characters in the original text, and $C$ represents the number of correctly recognized characters in the predicted result.

The audio editing software WavePad was used to trim all audio files, removing silent sections. FFmpeg was used to modify the sampling rate of all audio to 16 kHz, lock the channel to mono, and convert the files to WAV format. The parameter information is shown in Table 2.

After processing the audio, a file-by-file reading and matching method was used to generate a data list file, which was used to index the audio files and their corresponding text information. Each line of data includes the relative path of the audio file and the annotated content corresponding to that audio file. The format is shown in Figure 12.

Afterward, all characters contained in the dataset are counted to generate a vocabulary file. Finally, the number of frames, mean, and standard deviation are calculated for CMVN operations, with the default being to use all speech data to calculate the mean and standard deviation.

Finally, feature extraction is performed on all audio files to obtain their MFCC features, and JSON-formatted data feature list files are generated for both the training set and the test set for model training. In practice, the features of the same phoneme may differ due to the influence of different microphones, recording environments, and audio channels. Through CMVN operations, standard features with a mean of 0 and a variance of 1 can be obtained, thereby improving the anti-interference stability of the acoustic features. The MFCCs spectrogram is shown in Figure 13.

To further enrich the diversity of the data and thereby enhance the model's generalization and anti-interference capabilities, small random perturbations were added to the original audio during model training to generate new audio for data augmentation. The augmentation methods include noise perturbation, speed perturbation, volume perturbation, and SpecAugment [38]. The overall data preprocessing process is shown in Figure 14.

All model experiments were conducted on the same computer equipment, with the operating system being Windows 10 Professional, the programming language Python 3.11, the CPU being Intel® Core™ i5-13490F, and the GPU being NVIDIA® GeForce RTX™ 3070.

In this study, the configuration parameters for the training and validation stages of the Conformer acoustic model were determined based on relevant literature in the field [39]. The specific parameter information is shown in Table 3. The number of epochs represents the total number of training cycles for the model; the batch size is the number of audio samples the model processes simultaneously in one cycle; gradient accumulation is used to achieve the effect of expanding the batch size; the initial learning rate and its decay determine the convergence speed of the objective function; and MFCC size represents the dimensions of the MFCCs.

The Conformer model was trained for 200 epochs, with validation after each epoch. The model converged at the 87th epoch and achieved the best validation performance at the 194th epoch, with a CER of 4.38%, as shown in Figure 15. After combining with the geological specialized language model, the CER was reduced to 2.36%, and the single-character decoding time was 15.5 ms.

To further evaluate the performance of the proposed model, the constructed hydraulic engineering field geological survey speech dataset was used as an example, with other advanced deep learning speech recognition models selected for comparison under the same configuration parameters as the hydraulic engineering geological survey speech recognition model. CER and decoding time were used as evaluation metrics. The decoding times and CERs of each model are listed in Table 4. The comparison results show that the proposed language model, due to its specialization in the geological domain, effectively reduced the CER compared to the Giga Chinese model. It can be seen that this method shows no significant difference in single-character decoding time compared to other methods, meeting the requirements for real-time speech recognition; the CER metric is superior to other methods, indicating that it is more suitable for the speech recognition tasks involved in hydraulic engineering field geological survey data collection. Additionally, fine-tuning and loading the specialized language model significantly improved the recognition performance of the Conformer model, further demonstrating the advantages of the proposed method.

To directly demonstrate the recognition performance of the proposed model, actual field data recording speech was used as a sample, comparing the predicted results of this model with the output of mainstream commercial software. This speech sample contains a large number of geological terms, with a duration of 23 seconds and a text length of 95 characters. The software and their recognition results, along with the CER, are shown in Table 5.

It can be seen that the CER of the proposed model is 3.16%, which is 2.10% lower than that of the Baidu Input Method in online mode. Compared to the offline versions of Baidu, iOS, and Sogou Input Methods, the CER is reduced by 16.84%, 25.26%, and 37.89%, respectively, showing better performance than the existing mainstream commercial software. For mainstream commercial software, their online recognition is more geared toward everyday conversational dialogue. Although their dataset size is larger than that used in the proposed method, the total number of geological terms is low, and their proportion is small during training, resulting in poor model performance in the related field, leading to more recognition errors, especially when geological terms are misrecognized as common conversational words. Notably, the proposed model shows a significant advantage in offline recognition, effectively recognizing geological terms, meeting the needs for recording speech data into text during field geological data collection.

This study proposed a method for the collection and recording of hydraulic engineering geological survey data in the field, based on speech recognition interaction, to meet the demands for digital and intelligent data collection in hydraulic engineering geological surveys. The proposed method effectively improves the efficiency and accuracy of field data collection for hydraulic engineering geological surveys. The main conclusions are as follows:

(1) Various geological professional materials were compiled to form a specialized speech dataset for hydraulic engineering geological field surveys. This dataset includes 16,352 sentences of geological survey professional text data, totaling 218,498 characters, and 13.53 hours of speech data generated through both human voice recording and speech synthesis.

(2) Based on the self-made dataset and fine-tuning on the WenetSpeech large-scale dataset model, a Conformer acoustic model was trained using CTC-Attention joint loss supervision. Compared to the WenetSpeech pre-trained model, the CER was reduced to 3.9%.

(3) A speech recognition model for hydraulic engineering geological field surveys was developed. Using tokenized text data combined with open-source Chinese corpora, an N-gram algorithm was used to train and generate a specialized language model for hydraulic engineering geological field surveys, which was used for CTC beam search decoding. The proposed method achieved a single-character decoding time of 15.5 ms and a CER of only 2.6%, a reduction of 1.3%, outperforming other speech recognition methods such as Deepspeech2, Transformer, Efficient Conformer, and Squeezeformer.

(4) By combining the Conformer acoustic model with the N-gram language model, an intelligent speech recognition and efficient recording method tailored for hydraulic engineering geological field surveys was proposed. Compared with mainstream commercial software, this method achieved better results in terms of CER and showed significant advantages in offline recognition, making it suitable for mobile data environments in field geological surveys.

The proposed method performed well on the self-made dataset and in practical speech tests for geological surveys, but there is still room for improvement in recognition accuracy. Future work will include further expanding the dataset, such as collecting geological survey data for bridges, underground projects, and increasing the number of recording personnel. Additionally, the model architecture will be improved using the EfficientNet network to further optimize performance. Moreover, the acoustic and language models developed in this study are still relatively large after export, so future research will explore model lightweighting using knowledge distillation techniques.