Assessing Automatic Dependent Surveillance-Broadcast Signal Quality for Airplane Departure Using Random Forest Algorithm

The three main causes of aviation accidents during takeoff and landing—technical issues, weather, and human error—generally account for 63% of all accidents involving aircraft [1]. Accident analysis during the take-off or landing phase now depends solely on data from the black box, whereas, as information technology advances, data from the Automatic Dependent Surveillance-Broadcast (ADS-B) should be a concern in order to see alternative points of view in the accident analysis process [2]. Yet, the ADS-B signal quality is the most important component in convincing the analytical results utilizing ADS-B data. ADS-B allows aircraft to broadcast their identification and current position, determined by the global navigation satellite system, to ground stations or other aircraft above 1090 MHz [3]. The ADS-B ground station equipment on the ground receives the signal from ADS-B aircraft and sends it to the ATC (Air Traffic Controller). The ATC display then uses the data to track the aircraft [4]. ADS-B is particularly significant, especially for airports with a high operating level, since it is utilized to assist operational activities with the goal of boosting the safety, capacity, and efficiency of national airspace system operations, lowering radar installation, and providing off-radar coverage. One of the Object studies in this research is Sultan Mahmud Badaruddin II Airport ($02^{\circ} 54^{\prime} 01^{\prime \prime} \mathrm{S}$ $104^{\circ} 42^{\prime} 00^{\prime \prime} \mathrm{E}$), which sees an average of 16 departures each day from both local and foreign destinations. This airport's busy schedule ought to be complemented by a strong ADS-B signal.

The purpose of this study was to evaluate Sultan Mahmud Badaruddin II Airport's ADS-B signal quality for measuring its level of safety. To delve deeper, a mechanism for classifying the quality of ADS-B data at Sultan Mahmud Badaruddin II Airport is required. This research has its own level of difficulty because there is no reference for analyzing ADS-B signals at Indonesian airports, so this is a novelty for this research, which will ultimately provide an overview of data quality around the airport area. The Random Forest classification method was picked as one of the options. Based on prior research, the Random Forest method produced an accuracy of 86.14% and an f1-score of 86.93% when used to classify maritime transportation, specifically ships [5].

We organize this study as follows: The first section is the introduction, which contains the study's aims; the second section is the methodology, which covers the technique adopted to reach the objectives, as well as the methods of data collecting, data processing, and random forest algorithm implementation. The third section provides the results of the methodology's execution, as well as talks about the process of analyzing the results. The conclusions are revealed in the last section.

The Research Methodology in this research describes the procedures or techniques used to identify the problem or objective, select the data, and process and analyze the information. The general flow of methodology is illustrated in Figure 1.

The main data source in this study is an application named AERO-TRACK [6], which is used to record flight departure data at Sultan Mahmud Badaruddin II Airport, after which the data is downloaded in database format and translated into CSV format with the following data information as seen in Table 1.

Sultan Mahmud Badaruddin II International Airport (IATA: PLM, ICAO: WIPP) is a commercial airport in Talang Betutu, South Sumatra, owned by the government of Indonesia. The airport has one runway with two landing or takeoff directions, 11 and 29, with a length of 9,834 feet (3000 meters) with an asphalt surface. According to statistics, the total number of passengers who used this airport in 2018 was 5,126,298.

Preprocessing is where the data from Table 1 is processed. Reduced data size, association discovery, data normalization, irregularity removal, and feature extraction are the main goals of preprocessing. The strategies used in this process include data cleaning, integration, transformation, and reduction [7]. The data cleaning phase is the initial step in preprocessing procedures used to identify anomalies, clean out noise in the data, detect missing values, and fix incorrect data [8]. Figure 2 shows a variable in a dataset that is missing a value. A missing value can reveal bias and lower the quality of a flight analysis' results [9]. Resolving missing values can be done in several ways, including (a) eliminating objects that contain missing values, (b) manually resolving missing values, (c) employing global or object-to-object consistency, and (d) finding the most likely answer to the problem [10].

We choose the data that will be used in the visualization process and discard the remainder at the preprocessing stage. Preprocessing is carried out using Jupyter Notebook, the Python programming language, and the panda’s library. The following are the phases of preprocessing:

1. Date and icao24 filtering. This step is important so that the data may be sorted according to a flight with a specific date schedule.

2. Removing any duplicate information discovered in time variables. Duplicate data in rows has an impact on how well the data is visualized.

3. Don't keep altitude rows that aren't datasets. The following criteria apply to altitude deletion:

a. Deleting altitude 0 and using the latest line altitude recorded for each trip as its final altitude.

b. If the flying altitude does not start at 0, then the altitude is still taken.

c. The altitude line is eliminated if the final altitude is zero.

4. New columns next to the time column: difference, average, and tier. These columns are utilized in the processing and evaluation of data quality.

5. Using the formula second time minus first time, followed by other iterations, to determine the difference. The data needed to process data quality includes the difference.

6. Determining the average by dividing the total number of differences by the sum of all differences.

7. The criteria should be used to classify the signal quality as viewed from the average as seen in Table 2.

Description:

Tier 1: Data standards with high-performance traffic separation services are included in this class.

Tier 2: Data standards with traffic situational awareness services and procedural separation can be found in this class.

Tier 3: This group of data standards includes those with traffic advisory functions (flight information services).

Tier 4: Unacceptable data standards are used in this class.

8. Using the date, icao24, aircraft code, registration, airline icao, average, and category columns as the dataset for the classification method as well as additional research material.

One classification and regression-based method using a decision tree aggregation procedure is the Random Forest. This technique is employed because it results in fewer errors and has a decent classification accuracy [12]. When the majority of the data is absent, Random Forest provides an efficient approach for predicting the missing data, maintains accuracy, and has a mechanism for balancing class mistakes in the data set [13]. With Random Forest, the stages of compilation and estimation are:

a. The bootstrap stage, where random samples with recoveries of size n are drawn from the cluster of training data.

b. The random sub-setting stage, where a tree is constructed from the data but m < d explanatory variables are randomly selected; the best splits are produced in this stage.

c. Repeating steps, a – b as much as k times in order to produce k random trees.

d. Doing a joint estimation using k-trees (e.g., using the majority vote for classification cases or the average for regression cases).

Using Jupyter Notebook, the Python programming language, and the sklearn package, this classification technique is implemented. The dataset is initially split into three categories: 20% of it is unseen data or data whose labels have been purposefully removed; The training data is drawn from the remaining dataset (80%), which is divided into 70%, which is used to teach the computer to recognize patterns; and the remaining 30% is test data, which is used to evaluate the effectiveness of the machine's training. This method of splitting the data is called the "holdout" method [14], [15], [16], [17], [18], [19]. There are 1,444 rows of training data after being separated, as can be seen. The confusion matrix is used to test the data and assess the algorithm's effectiveness. Table 3 shows the multi-class confusion matrix's shape.

Accuracy is a formula to determine the comparison of true (negative and positive) predictions with the overall data, as seen in Eq. (1), while precision is a formula to determine the ratio of data that is predicted positive to the overall data, as seen in Eq. (2). Next is referred to as "recall." Recall is a formula for knowing the true positive predicted value compared to the true positive value of the overall data, as seen in Eq. (3) and the last is the F1 score, which is the sum of recall and precision averaged, with a formula that can be written as in Eq. (4) [20].

The Random Forest classification algorithm will be used to train and evaluate the data after preprocessing in an effort to categorize it. Figure 3 displays the performance of the training data, showing that Random Forest can accurately categorize all categories because there is enough data scattered throughout each class for the model to do so.

The 620 rows of test data were used to test the trained model. Figure 4 displays the test data's performance; there are 0–2 misclassifications in each category. This is because each class only has a tiny amount of test data, which leads to errors in the model.

This study examines performance based on the f1-score caused by imbalanced datasets. The test data performance, which results in an f1-score value of 100% in Tier 1, 94% in Tier 2, 67% in Tier 3, and 92% in Tier 4, giving an overall f1-score value of 88%, is shown in Figure 5. The fact that there is a sizable disparity in the amount of data for each category has an impact on the f1-score's value.

Tier 1 has the best performance because there is enough training data and test data for the model to accurately classify, whereas Tier 3 has the worst performance because there is not enough training data and test data for the model to perform at its best. Because there are 2,381 flights in Tier 1, 108 in Tier 2, 24 in Tier 3, and 67 in Tier 4, the data categorization findings, shown in Figure 6, show that the Sultan Mahmud Badaruddin II Airport's ADS-B signal quality is in the good category.

As most aircraft at Sultan Mahmud Badaruddin II Airport are categorized as Tier 1, it can be inferred from the trend in Figure 6 that signal quality will be high between February 1 and August 1 of 2022.

The findings of the ADS-B signal quality classification in this study revealed that the Random Forest algorithm produced an accuracy of 99% with this imbalanced data, which was significantly better than the accuracy of multi-class classification with imbalanced data in earlier studies. From the results obtained, it is necessary to test again with other algorithms, which in this study used the SVM and Naïve Bayes algorithms with the same procedure as the procedure in Random Forest. The accuracy obtained using the SVM algorithm is 93%, while the Naïve Bayes algorithm produces an accuracy of 97%. However, random forest also has drawbacks such as a high probability of overfitting, making it very difficult to interpret the prediction results, taking a long training time, and using a lot of memory. All of these are influenced by the number of trees in the forest. However, these shortcomings are not so impactful if we still use a small number of datasets and classes [21], [22], [23]. As proven in this study.

As a result of Sultan Mahmud Badaruddin II Airport's dominance in the Tier 1 category, the data analysis results revealed that the ADS-B signal quality on airplane departure data is generally good. It follows from this that most active pilots adhere to the ADS-B use regulations. This study performs well even with data that is skewed and accurately classifies existing groups of classes, especially Tier 1. The quantity of training and test data in that category supports this. With the test results using the generative classification type, namely the Naïve Bayes algorithm, and the discriminative type, namely SVM and Random Forest, further testing is recommended to attempt utilizing a distribution-free classification type.