Enhanced Abnormal Event Detection in Surveillance Videos Through Optimized Regression Algorithms

In recent years, the proliferation of surveillance systems has been pivotal in enhancing security and safety across various sectors, ranging from urban safety to private enterprises. As technology advances, the integration of machine learning and artificial intelligence in video surveillance has become a focal point of research, aiming to develop smarter, more efficient methods of monitoring and anomaly detection [1], [2]. Despite these technological strides, existing surveillance systems exhibit significant limitations. Traditional approaches such as SVMs and DTs, while foundational, often fail to effectively handle the dynamic and complex nature of real-world environments. These methods struggle with high rates of false positives and are inefficient in differentiating between normal variations and genuine anomalies, particularly in diverse and uncontrolled settings. Furthermore, most current systems do not adequately address issues like occlusion, varying lighting conditions, and large-scale data management, leading to gaps in surveillance effectiveness.

The research gaps in this field are pronounced, with a critical need for systems that can adapt to varied environments and update their parameters in real-time without human intervention. Additionally, there is a lack of frameworks that can seamlessly integrate with existing infrastructure to enhance both accuracy and efficiency without substantial overhead [3], [4].

The potential applications of an optimized abnormal event recognition system are extensive. Improved surveillance technology could be transformative for public safety, traffic monitoring, and management in urban areas, as well as for securing sensitive locations such as airports, banks, and governmental buildings. In the private sector, such systems can be employed to monitor commercial spaces, aiding in loss prevention and enhancing customer safety. This study introduces the ARER method, a novel approach using advanced regression algorithms that significantly improve the detection and analysis of abnormal events in surveillance videos. By addressing the aforementioned drawbacks and gaps, ARER represents a substantial step forward in the application of intelligent video analysis for real-time, accurate, and efficient surveillance [5]. The ARER model advances the state-of-the-art by improving detection accuracy by 0.35% and reducing false positives by 0.40% compared to the SVM. It achieves this through advanced 3D Convolutional Neural Networks (CNNs) for better feature extraction and Long Short-Term Memory (LSTM) networks for temporal analysis. Additionally, ARER uses dynamic thresholding to adapt to changing environments in real-time, reducing detection latency by 0.30% over DTs. The model also improves system throughput by 0.25%, making it more effective for real-time surveillance applications. Its fusion of classification and regression enhances anomaly detection accuracy and efficiency.

Figure 1 shows a sophisticated machine learning workflow designed for the classification of video segments into normal and anomalous events in surveillance applications. At the outset, video clips are pre-processed and segmented into discrete time slices (S1 through S32), which are then fed into the feature extraction phase. This phase utilizes a range of neural network architectures, including a Volumetric Convolutional Network (VCN), Spatio-Temporal Convolutional Network (STCN-3D), multiple instances of 3D Deep Convolutional Networks (3D-DCN) coupled with a Multi-Dimensional Neural Network (MDNN), and a Video CNN (V-CNN). These networks are responsible for distilling spatial and temporal characteristics from the video sequences to identify distinguishing features of abnormal activities [6], [7], [8], [9].

Upon extracting these features, the clips are categorized into positive or negative bags—reflective of a multiple instance learning framework—where positive bags contain at least one abnormal event and negative bags contain none. Within the classification stage, the features are scrutinized through a multi-layered neural network, assigning each one an instance score that corresponds to the probability of it depicting an anomaly. Subsequently, a ranking method evaluates these instance scores, presumably to prioritize the events by their likelihood of being anomalous. This ranking could serve a practical purpose, like alerting operators to the most critical events first [10], [11].

Finally, the architecture employs a loss function, which is integral to the model's training process. The loss function assesses the disparity between the model's predictions and the true classifications, guiding the optimization process to hone the model's ability to accurately discern between normal and anomalous events. The aim is to minimize the loss, thereby enhancing the model's predictive fidelity and overall surveillance efficacy [12], [13].

Recent advancements in surveillance technology have introduced several novel approaches to abnormal event detection, each with unique methodologies and challenges. Traditional methods like SVMs and DTs struggle with high false positive rates and adaptability. To address these issues, modern approaches use deep learning techniques such as CNNs for spatial feature extraction and the LSTM for temporal analysis. Generative Adversarial Networks (GANs) have also been used to generate synthetic data, helping mitigate data scarcity, but they often face training stability issues. Unsupervised learning methods aim to reduce reliance on labeled data, yet they still encounter difficulties with false positives and generalization across different environments.

SVM-based approaches are widely used for abnormal event detection due to their ability to handle high-dimensional data and perform binary classification. These methods are effective in detecting anomalies by creating a decision boundary between normal and abnormal events. However, they often struggle with scalability when applied to large datasets and can result in high false positive rates, particularly in dynamic environments. Despite their simplicity, SVM-based models are less adaptable to real-time processing and complex, evolving surveillance scenarios. Zaheer et al. [14] presented an algorithm based on the SVM, which, despite its improved accuracy over prior models, struggles with scalability in larger datasets.

CNN-based approaches are effective for abnormal event detection due to their ability to automatically extract spatial features from video frames. By capturing patterns and anomalies within the spatial dimensions of the data, CNNs improve the accuracy of detection. However, CNNs are typically limited in handling temporal dynamics, making them less effective for video sequences that require an understanding of events over time. While CNNs enhance performance in static image analysis, they often need to be combined with other methods like the LSTM for more comprehensive event detection in surveillance. Yuan and Xu [15] utilized a CNN to enhance spatial feature extraction, but the technique lacks temporal context for dynamic scenes.

LSTM-based methods are designed to capture the temporal dependencies in video sequences, making them highly effective for abnormal event detection in dynamic environments. The LSTM is capable of learning patterns over time, which helps differentiate between normal variations and true anomalies in video streams. However, despite their effectiveness in handling time-series data, LSTM models can be computationally intensive and may lead to higher processing costs, limiting their application in real-time surveillance systems. Bazhenov et al. [16] introduced an LSTM-based model to capture temporal features; however, the complexity of the model significantly increased the computational cost. Liu et al. [17] incorporated a hybrid model combining CNNs and the LSTM to address this gap, yet they encountered difficulties with overfitting due to the limited diversity in training data.

GAN-based methods have emerged as a promising approach for abnormal event detection, particularly in generating synthetic data to address the scarcity of labeled anomalous events. GANs consist of two networks, a generator and a discriminator, that work together to produce realistic data, enhancing model robustness. These methods help improve detection by learning from both real and synthetic data. However, GANs often face challenges with unstable training and can be sensitive to variations in input, making them less reliable for consistent performance in real-world surveillance scenarios.

Further exploration into deep learning was seen in the study by Sarkar et al. [18], where GANs were used for anomaly detection, which showed promise but was hampered by unstable training procedures. Yu et al. [19] advanced the field through the introduction of a novel deep reinforcement learning technique, though it remained untested in real-world scenarios. In contrast, Kwak et al. [20] applied a real-time anomaly detection system based on DTs, which was fast but less accurate compared to deep learning methods. Wang and Ji [21] took a different approach, employing unsupervised learning to identify anomalies without labeled data. While this improved the model’s adaptability, it also increased the number of false positives.

Two recent surveys by Coşar et al. [22] and Togare and Andurkar [23] synthesize these individual efforts, noting that while there has been significant progress in the field, issues such as high false-positive rates, the need for vast amounts of labeled data, and the ability to generalize across different surveillance environments persist as predominant research gaps. Both surveys call for further research into unsupervised and semi-supervised learning techniques, as well as the development of more robust and scalable models that can be deployed in a variety of settings. The field of machine learning applied to abnormal event detection in surveillance footage is rapidly evolving, with recent literature presenting a breadth of techniques and identifying various limitations and gaps. In the work of Önal et al. [24], an SVM-based approach was enhanced with a semi-supervised learning framework to reduce the need for labeled data. While this method showed a 20% improvement in accuracy over standard SVMs, its performance declined in scenarios with significant background noise. Kamthe and Patil [25] improved upon traditional CNN architectures by integrating attention mechanisms to better focus on relevant spatial features. Despite achieving a lower false-positive rate in controlled environments, this model's application to crowded or chaotic scenes was not extensively tested, suggesting a potential gap in real-world applicability.

Huang et al. [26] emphasized the importance of temporal features and proposed an LSTM network that excelled in recognizing patterns over time. However, the substantial increase in computational resources required for this model raised concerns about its viability for widespread adoption in real-time surveillance systems. In a different vein, Alijanpour and Raie [27] explored the effectiveness of transfer learning by applying pre-trained CNN models to anomaly detection tasks. Their findings revealed a significant reduction in training time without sacrificing accuracy. However, the transferability of these models to different types of surveillance footage remained a challenge, indicating a research gap in model generalization.

Nawaratne et al. [28] were pioneers in applying GANs to generate synthetic training data for anomaly detection, addressing the scarcity of labeled abnormal event data. Although their approach reduced the dependency on real anomalous data, the synthetic data did not entirely capture the complexity of genuine abnormalities, highlighting a need for improved generative models. Ranjitha et al. [29] introduced an edge computing framework to distribute the computational load of deep reinforcement learning models. This reduced latency for real-time anomaly detection; however, it introduced new challenges in data security and privacy, marking an area for further investigation. Huszár et al. [30] took a practical approach, developing a lightweight model based on DTs that could be easily deployed in existing surveillance systems. While this model was efficient, it was not as adept at detecting subtle anomalies as deep learning models, suggesting a trade-off between efficiency and sensitivity.

Lopatka et al. [31] tackled the problem of unsupervised anomaly detection by employing autoencoders to learn a representation of normal behavior and flag deviations. Their model proved useful in environments with well-defined normal patterns but struggled in areas with high variability in typical behavior. The surveys by Chen et al. [32] and Li et al. [33] provide comprehensive overviews of these developments, converging on the consensus that while machine learning models have become more sophisticated, they still face challenges in data dependency, generalization across diverse environments, computational efficiency, and balancing accuracy with real-time processing capabilities. These reviews highlight the ongoing need for innovation in the field, particularly in developing models that can learn from limited data and adapt to new environments without extensive retraining.

The methodology section of the proposed ARER framework details the systematic approach taken to enhance abnormal event detection in surveillance footage through a blend of regression algorithms and deep learning. The methodology starts with the collection and preprocessing of a diverse set of surveillance videos, involving frame segmentation, normalization, and data augmentation to fortify the dataset [34]. Feature extraction was carried out using a specially designed 3D-CNN that captures both spatial and temporal information. This network is structured with several convolutional and pooling layers, allowing it to discern complex spatio-temporal features while preventing overfitting [35].

In a departure from traditional classification, the anomaly detection was treated as a regression problem where the output is a continuous anomaly score [36]. An ensemble of regression trees was employed to enhance predictive accuracy and stability. Temporal sequencing was integrated through LSTM networks, enabling the model to differentiate between normal variability and true anomalies over time [37]. The output is an anomaly score assessed against a dynamically adjusted threshold that accommodates the changing definition of 'normal' within the surveillance context. The training of the ARER model uses both supervised and unsupervised learning, with a loss function designed to minimize the difference between predicted scores and actual labels [38].

For evaluation, a cross-validation technique was used to ensure the model's generalizability, with metrics such as mean squared error (MSE) for regression accuracy and precision, recall, and F1-score for classification performance. Finally, the model was implemented and tested in a simulated surveillance environment to assess its real-time detection capabilities, aiming for precise and efficient identification of abnormal events [39], [40], [41].

Figure 2 shows the proposed architecture of the ARER system, which is aimed at optimizing abnormal event recognition in surveillance videos. The image provides a visual representation of the workflow and the integration of different machine learning techniques to enhance the accuracy and efficiency of detecting abnormalities in surveillance footage [42]. The workflow begins with a set of raw video data, depicted as a series of fragmented images that undergo a process of initial fusion. This initial fusion is likely a preprocessing step that involves techniques such as frame normalization, noise reduction, and data augmentation to prepare the video segments for further analysis [43], [44], [45], [46], [47]. This step may also include the synchronization of video frames to align temporal sequences, which is critical for accurate motion analysis. Once the data is preprocessed, it is divided into two streams. The first stream employs a Naive Bayes Classifier (NBC), a probabilistic algorithm that calculates the likelihood of an event being normal or abnormal based on historical data. This classifier assesses the features extracted from the video data and computes a normality score ($\textit{Normality Score_1}$), which quantifies the probability of the observed event conforming to the pattern of normal events. The NBC is particularly effective in handling complex datasets where the relationships between attributes are not linear. The second stream utilizes a regression algorithm to process the same preprocessed video data. Regression algorithms are designed to predict numerical values, and in this context, it is used to determine a continuous normality score ($\textit{Normality Score_2}$) that reflects the extent of abnormality in the surveillance footage [48]. The regression algorithm may incorporate innovative methods such as Support Vector Regression (SVR) or Gradient Boosting Regression (GBR), which can handle nonlinear relationships and are robust to outliers, thereby enhancing the predictive power of the model. Both normality scores from the classification and prediction streams are then fed into a set of proposed regression algorithms. This stage represents the innovation core of the ARER system, integrating the two streams to leverage their strengths and mitigate their individual limitations. The proposed regression algorithms may include advanced machine learning techniques such as ensemble methods, which combine multiple models to improve prediction accuracy, or deep learning approaches like Recurrent Neural Networks (RNNs) or CNNs that can capture complex patterns in spatial and temporal data [49], [50].

The fusion of these advanced techniques ensures a comprehensive analysis of the surveillance footage, leading to a more accurate assessment of normality or abnormality. The decision-making module at the end of the workflow evaluates the combined insights from both normality scores, potentially applying further context-aware analysis or anomaly scoring thresholds that adapt to the changing environment. The architecture embodies the cutting-edge of machine learning in surveillance technology, pushing the boundaries with a sophisticated integration of statistical, traditional machine learning, and deep learning methods to create a system that not only detects but also understands the nuances of abnormal events in a variety of settings [51], [52], [53].

Figure 3 shows that the classification stream model represented in the image operates as an integral component of the ARER system. It begins with raw images, which serve as input data. These images are analyzed to extract two critical pieces of information: the class, which identifies the type or category of the objects within the image, and the pose, which describes their orientation or position [54]. These characteristics are essential in recognizing patterns of normality or abnormality within the visual data. The extracted features are then fed into a NBC, a probabilistic machine learning model that uses statistical methods to predict the likelihood that the given data points belong to certain predefined categories, based on prior knowledge. The NBC outputs a $\textit{Normality Score_1}$, which quantifies the extent to which the analyzed features conform to the model of what is deemed 'normal' within the context of the surveillance environment. This score serves as an early indicator of potential anomalies in the video stream, flagging events that deviate from established patterns [55], [56], [57].

Figure 4 shows that the prediction stream model outlined in the image is another facet of the ARER system, operating concurrently with the classification stream [58], [59]. It also starts with raw images and processes them through a regression algorithm. Regression algorithms in the context of this system are designed to predict a continuous variable, which in this case is the degree of abnormality in the visual data. A key part of this stream is the use of optical flow, a technique used to estimate the motion of objects between different frames of the video. This analysis helps in understanding the dynamics and temporal aspects of the scene. The result of this processing is the $\textit{Normality Score_2}$, which, like $\textit{Normality Score_1}$, is a measure of how much the events in the images depart from the baseline of normal events established during the training phase of the ARER system [60].

The ARER model was trained and evaluated using two well-known publicly available datasets: the UCSD Anomaly Detection Dataset and the Avenue Dataset. The former comprises 98 video clips capturing pedestrian movement on walkways, where abnormal behaviors such as deviations from normal walking paths were marked as anomalies. The video clips have a resolution of 238$\times$158 pixels and are recorded at 10 frames per second (fps). Each video clip was carefully annotated with frame-level labels to mark anomalous activities. The dataset includes approximately one hour of footage, and the annotations were manually created based on predefined behaviors to ensure the accurate identification of abnormal events.

The Avenue Dataset includes a total of 68 video clips, divided into 21 training videos and 47 testing videos. The dataset contains videos with varying conditions such as different camera angles and lighting scenarios, which provide a more challenging and realistic surveillance environment. The resolution of the videos is 640$\times$360 pixels, and they are recorded at 25 fps. Like the UCSD Dataset, the Avenue Dataset has frame-level annotations that mark abnormal events like unusual pedestrian movements or unexpected object behaviors, which are labeled manually based on predefined behaviors. This ensures precise event detection during the model's training and evaluation phases.

By utilizing these datasets, the ARER model was rigorously evaluated across different real-world surveillance conditions. The datasets’ diversity in terms of environmental factors, camera perspectives, and anomalous activities contributes to the robustness and adaptability of the ARER model in detecting abnormal events.

The ARER model was trained and evaluated using a diverse set of surveillance video clips gathered from publicly available datasets, including the UCSD Anomaly Detection Dataset and the Avenue Dataset. These datasets contain video sequences captured in real-world surveillance environments with varying conditions, such as lighting, camera angles, and background activities, as follows:

1. UCSD Anomaly Detection Dataset

$\bullet$ It contains 98 video clips of pedestrian walkways, with annotations marking abnormal events such as individuals deviating from normal paths.

$\bullet$ Total duration: approximately 1 hour, with frame-level annotations for each anomaly.

$\bullet$ Resolution: 238$\times$158, recorded at 10 fps.

2. Avenue Dataset

$\bullet$ It comprises 21 training videos and 47 testing videos.

$\bullet$ The dataset is annotated with frame-level labels identifying abnormal events such as unusual pedestrian movement or object behaviors.

$\bullet$ Resolution: 640$\times$360, recorded at 25 fps.

Annotations were performed manually by labeling events considered anomalous, based on predefined behaviors and movement patterns. Each dataset was split into training and testing sets, and additional augmentation techniques (such as rotation and cropping) were applied to improve model generalization.

The proposed ARER system uses machine learning and deep learning to process and analyze surveillance video data for the detection of abnormal events. While specific mathematical equations are not detailed, a set of mathematical frameworks that could be applied in the ARER system can be inferred and constructed, considering the innovative methods and processes described.

The initial stage involves preprocessing video clips into discrete time slices and extracting features using neural network architectures such as VCN, STCN-3D, 3D-DCN, MDNN, and V-CNN. The feature extraction equation using CNNs is given as Eq. (1):

where, $F(x)$ represents the extracted features, $\sigma$ is a nonlinear activation function like ReLU, $W$ represents the weights of the convolutional filters, $*$ denotes the convolution operation, $x$ is the input frame, and $b$ is the bias.

For the classification stream utilizing a NBC, the normality score is likely based on the posterior probability. The NBC probability can be calculated using the Eq. (2):

where, $P(A \mid B)$ is the probability of hypothesis $A$ given the data $B, P(B \mid A)$ is the probability of data $B$ given that the hypothesis $A$ holds, $P(A)$ and $P(B)$ are the probabilities of observing $A$ and $B$ independently of each other.

The prediction stream involving regression algorithms may be based on regression trees or SVR methods. The SVR is given as Eq. (3):

where, $f(x)$ is the predicted output (normality score), $\omega$ is the weight vector, $\phi(x)$ is the feature vector obtained from the input data $x$, and $b$ is the bias term.

The fusion of the classification and prediction streams might involve ensemble techniques, where the final normality score could be derived from a weighted average of both streams. Ensemble weighted average is computed using the Eq. (4):

where, $N_S$ is the final fused normality score, $N_{S 1}$ and $N_{S 2}$ are the normality scores from the classification and prediction streams, respectively, and $\alpha$ is the weight factor determining the contribution of each stream. The decision-making process might then involve a threshold-based system that classifies events as normal or abnormal.

The decision threshold in a machine learning context, particularly for anomaly detection in surveillance systems like the ARER, is a critical value that distinguishes between normal and abnormal events. It's the point at which the system decides whether an event is anomalous based on the computed normality scores. This threshold is not just a static value but is often dynamically computed to adapt to varying conditions and to maintain the accuracy of the system over time. It can be calculated using the Eq. (5):

where, $T$ is the threshold for deciding between normal and abnormal events.

Table 1 shows the comparison of simulation parameters between conventional methods and the proposed ARER method. The table lists parameters such as detection accuracy, false positive rate, detection latency, system throughput, training time, memory usage, and adaptability to environmental changes. Each parameter was measured for both conventional and proposed methods, showcasing the improvements brought by ARER in terms of efficiency, accuracy, and adaptability.

To ensure reproducibility, the following hyperparameter settings were used during the training process for the 3D-CNN feature extractor and the ensemble of regression trees. For the 3D-CNN feature extractor, the model was configured with five convolutional layers, each using a 3$\times$3$\times$3 filter size, and the ReLU activation function was applied. Max pooling was performed with a 2$\times$2$\times$2 filter, and to prevent overfitting, a dropout rate of 0.5 was used. The model was trained with a batch size of 32, using the Adam optimizer with a learning rate of 0.001 and cross-entropy loss as the loss function. A total of 50 epochs were run, with L2 regularization (coefficient of 0.0001) applied. Additionally, data augmentation techniques such as rotation, scaling, and cropping were used to improve the model’s generalization. For the ensemble of regression trees, the hyperparameters were set as follows: the ensemble consisted of 100 trees with a maximum depth of 10. The minimum number of samples required to split a node was 2, and the minimum number of samples for a leaf node was 1. The criterion used for regression was MSE, and bootstrap sampling was enabled to allow random sampling with replacement. The maximum number of features considered was set to the square root of the total number of features. For boosting methods, a learning rate of 0.1 was used, and subsampling was set to 0.8 to reduce overfitting.

Figure 5 illustrates the comparative detection accuracy of the proposed ARER system against traditional surveillance systems employing algorithms such as SVMs, DTs, and K-Nearest Neighbors (KNN). This graph was plotted as detection accuracy (%) on the Y-axis against varying environmental conditions or operational scenarios on the X-axis. It clearly depicts that ARER consistently achieves higher accuracy levels across all tested conditions, emphasizing its enhanced ability to correctly identify abnormal activities without being misled by noise or normal variations in the data. This superiority is attributed to ARER's advanced machine learning frameworks that integrate deep learning for more nuanced understanding of video content.

Figure 6 presents a detailed analysis of detection latency, showcasing the efficiency of the ARER system in comparison with conventional methods. Detection latency, measured in milliseconds (ms) on the Y-axis, was plotted against different levels of scene complexity on the X-axis. The graph demonstrates that ARER substantially reduces the latency in detecting events, thereby facilitating quicker response actions. This performance is crucial in high-stake environments where timely alert generation can prevent potential threats or assist in immediate decision-making processes. ARER's optimized algorithmic structure allows for rapid processing of input data, minimizing delays and enhancing overall system responsiveness.

Figure 7 shows the evaluated system throughput, expressed in fps, highlighting the operational efficiency of the ARER system relative to traditional detection methods. The throughput was graphed against varying data volumes on the X-axis, illustrating how each system sustains its performance as the workload increases. This figure reveals that ARER maintains a higher throughput under all tested data volumes, underscoring its robustness and scalability. This is particularly beneficial in environments with high data inflow, where maintaining a high throughput ensures that no critical information is missed and all video data is processed in real-time.

The ARER surveillance technology, like any advanced surveillance system, raises potential ethical considerations. One concern is privacy, as increased monitoring can lead to over-surveillance, particularly in public spaces, which may infringe on individual privacy rights. Additionally, there is the potential for misuse if the system is deployed in a way that targets certain groups unfairly or without adequate oversight. To mitigate these concerns, it is important to ensure the technology is used transparently, with clear guidelines on data collection, usage, and storage. There should also be measures in place to prevent biases in the detection algorithms and ensure that the system is used for legitimate security purposes without compromising civil liberties.

The ARER system demonstrates significant enhancements over traditional surveillance technologies such as SVMs and DTs. Specifically, ARER improved detection accuracy by 0.35% and reduced false positives by 0.40% compared to SVMs. Additionally, it enhanced system throughput by 0.25% and lowered detection latency by 0.30% relative to DTs. These improvements highlight ARER's superior capability in handling the complexities of real-world surveillance environments. ARER combines advanced regression algorithms and deep learning techniques to efficiently process and analyze surveillance data, ensuring high accuracy and quick response times essential for security operations. The system's robust performance in diverse settings underscores its potential to significantly advance current surveillance practices, offering a more reliable and efficient monitoring solution. Further developments aim to integrate ARER more seamlessly with existing systems, enhancing its adaptability and reducing operational overhead, thereby solidifying its role as a leading solution in the field of surveillance technology.

Future enhancements for the ARER system will focus on integrating Internet of Things (IoT) technologies for faster processing, enhancing adaptability to diverse conditions, and advancing machine learning capabilities to improve accuracy and efficiency in surveillance applications.