Evading control flow graph based GNN malware detectors via active opcode insertion method with maliciousness preserving

The ongoing conflict between malware and detection systems has captivated interest since the inception of computers. To manage the increasing variety of new malware types, researchers have proposed numerous machine learning-based detection methods. These methods employ various data analysis techniques for malware detection, such as raw byte sequences^1,2, opcode N-Grams^3,4, binary visual grayscale graphs⁵, control flow graphs (CFGs)^6,7, and others. However, owing to the nature of machine learning models, which are often seen as black boxes with low interpretability and complex decision-making processes, their reliability is more challenging to verify, potentially concealing security risks.

Recently, the confrontation between malware and machine learning detection models has attracted increasing interest from research teams. Several related projects have been proposed, including gradient-based fast gradient sign method attacks^8,9,10,11, randomization-based attacks^10,12,13, generative adversarial network-based attacks^14,15,16, and reinforcement learning (RL)-based attacks^{17,18,19,20,21,22,23,24}, among others. These methods have proven effective in deceiving machine learning detection models, leading to erroneous predictions with high confidence.

GNN-based detection methods have been shown to be effective for malware detection and outperform convolutional neural network (CNN) and recurrent neural network (RNN)-based methods, demonstrating stronger robustness against attacks. GNN-based detection methods primarily leverage the CFG of malware for detection, and an increasing number of commercial malware detectors are incorporating CFGs to enhance their detection capabilities. Therefore, the key to executing a successful counter-attack lies in how to modify the CFG. However, traditional counter-attack methods for CNN and RNN models are unable to modify the CFG in a way that affects the detection results of GNN models. Furthermore, existing adversarial attack methods targeting GNN detection models do not preserve functionality. To make the adversarial attack more realistic, our approach must preserve functionality, which presents the following three challenges

• Direct modification of CFG is not feasible. Because the CFG is a visual representation of an executable file’s control flow, it can only be derived through reverse analysis of the binary executable. The execution logic of the binary file restricts direct modification of the CFG, making it impossible to inverse map to the binary file.

• Function-preserving limitations. The execution logic and relative offset addresses in the binary file make it highly sensitive to every byte. Inserting, deleting, or modifying even a single byte in the wrong location may render the program non-functional or alter its original function.

• Difficulty in identifying effective modification positions. Owing to the discrete and topologically dependent nature of CFG nodes and edges, suitable modification locations cannot be directly determined using gradient-based attacks. Moreover, unreasonable modifications may increase, rather than decrease, the probability of being predicted as malware.

In light of the aforementioned challenges, we propose a novel adversarial approach for malware detection based on RL, termed MalAOI, as illustrated in Figure 1. The goal of this approach is to modify the original malware samples in such a way that they can evade detection by the GNN model while preserving their functionality. CFGs from dataset A will be used to train a GNN detection model as an adversarial target, which detection model will then be employed to detect dataset B. From dataset B, we will obtain classified benign software samples and extract benign opcode sequences from the benign software samples. MalAOI employs an active opcode insertion modification method to insert these benign opcode sequences into the malware samples. RL is then used to determine the most effective modification locations and benign opcode sequences to reduce the malware’s maliciousness based on observed changes before and after the modifications.

MalAOI Overall Process.

Full size image

In this paper, the SOREL-20M dataset²⁵is used to train the GNN model, while the BODMAS dataset²⁶ is utilized to train the MalAOI model. Experimental results demonstrate that MalAOI can modify original malware samples to automatically generate adversarial malware that evades detection by the GNN model. It achieves an average attack success rate of 93.73% across multiple GNN malware detection models. Notably, for the graph attention network (GAT) model with 2-Grams representation method and 300 training rounds, the attack success rate reached 99.20%. Additionally, while maintaining a high attack success rate, the average file size increased by only 12.87%. In conclusion, our contributions can be summarized as follows:

• Novel base modification operations. MalAOI introduces a novel base modification method, termed active opcode insertion. This approach modifies the executable bytes of the malware according to the rules, thereby altering the CFG. The inserted opcode sequences form a new basic block, which can be executed during the malware’s runtime.

• Function preservation. This is the instance of a function-preserving adversarial attack against GNN malware detection models, generating adversarial malware that maintains the original sample’s functionality.

• High attack success rate. MalAOI has demonstrated a high success rate against various GNN malware detection models, achieving an average success rate of 93.73%.

• Transferability. Malware generated by MalAOI against a single GNN detection model also achieves high evasion rates against other GNN detection models with different structures.

The remainder of the paper is structured as follows. Section II provides an overview of the background for our work and a brief review of existing research related to adversarial attacks on malware detection models. In Section III, we introduce the proposed novel base modification method and the workflow of MalAOI. Section IV details the experimental settings and implementation specifics, followed by the presentation and analysis of results in Section V. Finally, Section VI summarizes the paper.

In order to better understand the work, the following content is what readers may need to know in advance. The algorithm symbols list refers to Table 1.

Background

In this section, we provide the background techniques utilized in our work, encompassing CFGs, node representation methods, GNNs, and RL.

1) Control Flow Graph

CFG is a graphical representation of program decompilation, representing all possible paths and associated opcodes during program runtime, typically realized through directed graphs. CFG nodes denote basic blocks in a program, comprising sequences of opcodes executed sequentially. Meanwhile, CFG edges signify the flow of control between these basic blocks. In this study, we use an GNN model to classify benign and malicious software based on CFG.

2) Node Representation Methods

The extraction of features from CFG nodes depends on the chosen representation method. The following two representation methods are used in this study for the GNN model:

• Malconv: Proposed by Raff et al.¹, this method disregards the compiled form of the basic block. Instead, it encodes the complete raw byte sequence of the basic block, with each byte transformed into an embedding vector.

• NGram:This approach views the opcode sequence in a basic block as natural language text^3,4, capturing local sequence dependencies by segmenting the text into consecutive phrases of length n. In this paper, we consider n values of 1, 2, and 3.

3) Graph Neural Networks

GNN is a deep learning model tailored for processing graph-structured data, aiming to learn representations of nodes, edges, and the overall graph by exchanging information between graph nodes. Previous research^6,7has demonstrated the effectiveness of GNN models in classifying malware CFGs. In this work, the studied adversarial target model uses three prominent GNN models: Graph Convolutional Network (GCN)²⁷, Dynamic Graph Convolutional Neural Network (DGCNN)²⁸, and GAT²⁹.

4) Reinforcement Learning

RL is an emerging field in machine learning, where an agent learns to achieve a particular goal by taking actions in an environment, receiving feedback in the form of states and rewards, and adjusting its strategy to maximize future rewards. RL excels in addressing problems characterized by long-term feedback, making it well-suited for adversarial malware generation, which inherently involves long-term feedback.

Related work

To date, a considerable body of research³⁰ has explored adversarial attacks targeting malware detection models. These attacks are often categorized by the attacker’s knowledge of the model (black-box and white-box) and the space they manipulate (feature space and problem space). To illustrate the nature of this research, we present relevant adversarial attack works categorized into two main groups: 1) black-box attacks targeting the problem space of malware detection models and 2) attacks aimed at the control flow graph-based malware detection model.

1) Black-box Attacks Against the Problem Space of Malware Detection Model

In this classification, the attacker modifies the malware without considering its specific characteristics, often with limited knowledge of the target. Attackers typically use RL-based methods to realize these attacks.

Anderson et al.¹⁷ pioneered an RL-based black-box adversarial attack and released their RL environment, gym-malware, for open-source use. They assembled a collection of ten function-preserving PE file base modification operations and integrated them into this environment. These operations constitute the available action space for the RL agent, and RL is employed to determine sequences of modification operations that enable modified malware to evade static feature-based machine learning malware detection models. This approach marked the first application of RL to adversarial malware generation, demonstrating that the generated adversarial malware could effectively evade detection by models based on raw byte sequences. Subsequently, this work garnered significant attention from the research community, leading to further exploration using gym-malware.

Labaca-Castro et al.²¹ introduced AIMED-RL, a novel framework for adversarial attacks based on RL. AIMED-RL redesigned the reward function and explored various weight strategies to maximize output, also introducing and demonstrating the significance of a penalty technique.

These RL-based adversarial attacks allow malware to bypass CNN- or RNN-based malware detection models while maintaining functionality. However, these methods heavily rely on basic modification operations, such as altering the portable executable header or adding benign byte sequences. These added byte sequences are not executed during malware runtime and, therefore, cannot modify the CFG to evade GNN model detection.

2) Attacks Targeting the Control Flow Graph (CFG) Malware Detection Model

GNN models^6,7 heavily rely on CFGs for malware detection. However, there is a lack of adversarial techniques for these models, primarily because CFGs cannot be directly mapped back to binary files.

Abusnaina et al.³¹ were the first to study modification methods for CFGs, entitled GNA. This method connects each selected graph with all samples from the opposite class using shared entry and exit nodes, generating functionally preserved adversarial samples. However, this method is only applicable to IoT software on ARM architecture and has not been extended to more complex PE files on the X86-32 architecture. Additionally, the adversarial target is a CNN malware detection model, and it has not been extended to GNN models, which can better capture relationships between nodes.

In a pioneering work, Zhang et al.²⁴ proposed the first adversarial approach for graph-based malware detection models, entitled semantics-preserving RL. The method uses RL to train an agent, iteratively selecting appropriate semantic null operations (Nops) and corresponding basic blocks, and performing semantic nops insertion operations, until an adversarial malware escape detection model is generated. While exhibiting a near-perfect success rate in adversarial outcomes, the method focuses solely on preserving semantics and falls short of generating functionally preserved adversarial malware.

Problem formulation

Given a malware sample M, which has been analysed in order to obtain its CFG (G = (V, E)) and its node feature set ({F}), it can be accurately classified as malware by the GNN malware detection model (1). The goal is to construct a function (f:M rightarrow M_a) that modifies M byte-wise to generate (M_a). A collection of CFG (G_a = (V_a, E_a)) and node features set ({F_a}) of (M_a) that enables the GNN malware detection model to misclassify as benign software (0).

In this framework, the function f is implemented through an active opcode insertion base modification method and RL-based MalAOI. The objective of MalAOI is to create adversarial malware capable of evading detection by the GNN model while preserving the functionality of the original malware sample and introducing minimal additional load.

In Equation 1, where L represents the objective function, (theta) denotes the parameter optimized by RL, (lambda _1) and (lambda _2) are the coefficients weighing the two objectives, (g(M’)) is the function computing the malice probability of decrease by the generated sample (M’), and (d(M, M’)) is the function computing the additional load introduced by the generated sample (M’).

Active opcode insertion

Before proceeding with a detailed discussion of MalAOI, we introduce a novel methodology for PE file modification known as active opcode insertion based on the concept of CFG composition. In this approach, we define opcodes executable during PE file runtime as active opcodes. Our focus is on PE software in the X86-32 architecture, making this methodology applicable solely to PE software implementations in this architecture. This method is used to modify the original malware binary sample by inserting a sequence of benign opcodes that retain functionality in the binary file, thus modifying the file’s CFG. Figure 2 shows the framework of this method.

Active Opcode Insertion Framework.

Full size image

1) Overall architecture

In the proposed framework, given input benign software B and original malware M to be modified, the initial step involves using a reverse tool to decompile both M and B. Subsequently, eligible opcodes in M are filtered, and the appropriate opcode is selected as the insertion position (opcode_i). The address (addr_i) of the next opcode after (opcode_i) is then recorded. Following this, a suitable basic block is chosen from B, and its sequence of opcodes is denoted as (sequence_b). This sequence is preprocessed, and adding the original opcode of (opcode_i) and the jump opcode to (addr_i) to obtain the modified (sequence’_b). Finally, the bytes of (sequence’_b) are appended to the end of M, and the address (addr_b) is calculated after insertion.The (opcode_i) is then modified to a jump opcode to (addr_b), completing the active opcode insertion modification operation. Furthermore, the connection between the insertion position and (sequence’_b) is established via the jmp opcode, (sequence’_b) can jump back to the next opcode of the insertion position after its execution.

2) Selection of Insertion Positions

To ensure compliance with the execution logic of the PE executable, the insertion position (opcode_i) must encompass one or more complete opcodes, with a primary criterion being a byte length of no less than 10. This stipulation arises from the need to transform (opcode_i) into an unconditional jump opcode that directs to the post-insertion address (addr_b). Such an unconditional jump opcode typically comprises the byte E9 with a 32-bit relative offset, resulting in a length of 10 bytes. However, in the X86-32 architectures instruction set, most opcodes have byte lengths less than 10. To maximize available insertion positions, it may be necessary to extend beyond a single opcode, employing consecutive multiple opcodes with byte lengths of no less than 10. Additionally, (opcode_i) must not include modifications to the extended stack pointer (ESP) register because such changes could yield unpredictable stack positions, potentially causing the modified malware to fail to execute or exhibit altered functionality.

3) Opcode sequence Preprocessing

In order for the modified malware to retain its functionality and operate correctly, it is imperative to ensure that the registers’ contents when transitioning from sequence (sequence’_b) back to opcode (opcode_i) are identical to those before the modification and that no modification of the memory storing the data occurs. Consequently, the inserted (sequence_b) must undergo a series of intricate preprocessing steps.

First, the benign opcode (sequence_b) must be cleansed of opcodes unsuitable for insertion operations. Because (sequence_b) is executed outside the register environment of the original benign software B, it is impossible to determine the register contents, leading to various uncontrollable scenarios. For instance, jmp family and call family opcodes may jump to unknown addresses, ret and loop opcodes may cause dead loops, div opcode may trigger division by zero errors, in and out opcodes may communicate with unknown hardware input/output (I/O) ports, and cli and sti opcodes may cause unexpected interruptions. Additionally, opcodes incompatible with insertion operations include those employing indirect memory addressing, such as opcodes using indirect addressing of registers or first operand is [CONST]. Addresses referenced by these opcodes may exceed permitted memory access, resulting in out-of-bounds errors or unintended memory modifications affecting the modified malware’s functionality.

Following this, a stack balancing operation is conducted on (sequence_b) to maintain the esp register in its original position after the execution of (sequence_b). This ensures the stack’s data remains unaltered, enabling the restoration of the register environment in a typical manner. The pop opcode removes the current element at the top of the stack, reducing the address in the esp register. Conversely, the push opcode inserts an element at the top of the stack, increasing the address in esp. As the sequence of (sequence_b) opcodes is traversed, the occurrences of pop and push are counted in real time. Any excess push opcodes are deleted to prevent a stack overflow.

Moreover, it is also necessary to reconstruct the stack in (sequence_b), clean up any opcodes that involve ebp, esp register, and then append push ebp; mov ebp, esp opcodes at the beginning of (sequence_b) and leave opcodes at the end of (sequence_b). The push ebp opcode is used to push the current value of ebp register onto the stack, preserving the base pointer of the previous function stack frame. The mov ebp, esp opcode is used to move the value of ebp register to esp register, which is the address of the current stack top. The leave opcode as a simplified version of mov esp, ebp; pop ebp, which is used to exit the stack frame of the current function and return the stack frame to the state it was in before the function was called. These operations enable (sequence_b) to safely manage local variables and restore the original state of the stack after execution.

Finally, to restore all registers to their original state after (sequence_b) execution, the pushf; pusha opcodes is inserted before (sequence_b) to push the flag register and general-purpose registers onto the stack. Subsequently, the popa; popf opcodes is added after (sequence_b) to restore the saved values to the registers. This concludes the preprocessing of (sequence_b) according to the aforementioned steps.

4) Insert

Upon completion of the preprocessing operations for (sequence_b), the necessary opcodes must be integrated into it. Because the insertion position (opcode_i) will be transformed into a jump opcode, it is crucial to include the original opcode of (opcode_i) in (sequence_b) to preserve the intended functionality of (opcode_i). To guarantee that (sequence_b) can return to the insertion position following execution, it is necessary to include a jmp opcode that directs to (addr_i). Furthermore, the way (opcode_i) is add to (sequence_b) must be adjusted according to its own opcode and position in the base block. For instance, if (opcode_i) is a jmp family opcode, the execution address of (opcode_i) changes. Therefore, it is necessary to calculate and modify the relative offset of (opcode_i) to ensure it jumps to the originally intended base block address. There is no need to add a jmp opcode to return to (addr_i). If (opcode_i) belongs to the cjmp or call family, it is necessary to modify both the relative offset of (opcode_i) and to add a jmp opcode that jumps back to (addr_i). If (opcode_i) is from the trap family, the original opcode of (opcode_i) should be added directly without including a jmp opcode to return to (addr_i).

Once the aforementioned preprocessing and opcode addition are completed, we obtain (sequence’_b). Upon appending (sequence’_b) to the end of M, a new section within M is created to accommodate the byte information of (sequence’_b). Simultaneously, the address (addr_b) of (sequence’_b) in M after insertion is calculated, and (opcode_i) is modified to a jmp opcode, directing to (addr_b). In cases where the length of (opcode_i) exceeds 10 bytes, excess bytes must be replaced with a nop opcode to prevent them from being recognized as separate opcodes. With this, the insertion procedure concludes successfully.

The described procedure outlines the modification operation for a single insertion position. Should there be a need to select multiple insertion positions simultaneously, the abovementioned steps must be repeated. The bytes of multiple (sequence’_b) are combined and added to M as the contents of a new section. Subsequently, the insertion address (addr_b) of each (sequence’_b) is calculated, and each (opcode_i) is modified accordingly.

Reinforcement learning environment

To enable MalAOI to automatically select suitable insertion positions in original malware samples and corresponding benign code sequences to generate adversarial malware evading GNN model detection, we developed an RL environment outlined in Figure 3.

Reinforcement Learning Environment Depicted.

Full size image

The RL environment commences by randomly selecting an original malware sample, denoted as M, from the malware dataset. This sample is then analyzed to obtain its CFG represented as (G = (V, E)), and its node’s feature set represented as ({F}). These components, (G = (V, E)) and ({F}), collectively form the state or the original observation space of the environment. Employing the GNN malware detection model as an adversarial target, (G = (V, E)) and ({F}) are assessed, generating a prediction score p forM. This score indicates the likelihood that the input file may be malware. During the decompilation process of the original malware sample, opcodes meeting the criteria of the active opcode insertion method are identified, forming a list of insertion locations for subsequent operations.

The (opcode_i)’s own opcode family, its position in the basic block, and other characteristics dictate how modifications to the malware impact the CFG. With these considerations in mind, we categorized insertion positions into 13 distinct types. These encompassed single and multiple opcodes, positions for the first, middle, and end of the basic block. Additionally, we accounted for various opcode families, including non-jmp, jmp, cjmp, call, and trap families. Figure 4 shows the typical CFG changes before and after modifications.

Consequently, a multidimensional discrete action space was devised for this environment. This space entails two discrete actions: one for selecting the insertion position based on its type and another for choosing a sequence of benign opcodes. Once the RL agent determines the optimal action, the environment selects the most suitable insertion location and benign opcode sequence based on this decision. Subsequently, an active opcode insertion base modification operation is performed on the malware, resulting in the generation of a modified malware (M’). The modified malware undergoes analysis to obtain the modified CFG (G’ = (V’, E’)) and its node feature set ({F’}). These are then inputted into the GNN malware detection model to obtain the modified malware prediction score (p’). By comparing changes in p and (p’), it is feasible to determine whether the modification effectively reduces the predicted score of the malware.

Changes occurring at control flow graphs (CFGs).

Full size image

The reward function within this RL environment is determined by the values of p and (p’). A positive reward signifies that the modification has decreased the model’s predicted likelihood of maliciousness. Conversely, a negative reward suggests that the modification has increased the model’s predicted likelihood of maliciousness. If the (p’) value after executing an action falls below a predefined threshold, it indicates the modified malware successful evasion of detection by the GNN detection model.

Meanwhile, the modified CFG (G’ = (V’, E’)) and ({F’}) serve as the new state, and (p’) acts as p for the subsequent action, which the agent learns to make the next decision action. This iterative process continues until there are no remaining insertion locations for the malware or the probability of the file being malware falls below the threshold. At this juncture, a new original malware sample is selected, the environment is reinitialized, and the sequence of operations recommences. In Algorithm 1, we present the execution of this RL environment.

RL Environment Step

Full size image

In Figure 5, the original malware sample and the generated adversarial malware are depicted in grayscale, with notable differences highlighted in red. Upon inspection, one can discern scattered red markings in the file, denoting the insertion point of the modification, while the extended red segment at the end signifies the inserted sequence of benign opcodes. Figure 6 shows the CFG of the malware before and after the modification, visualized through the interactive disassembler professional (IDA Pro) reverse tool. It is observed that the CFG of the malware has been altered after the modification.

Adversarial Malware Greyscale Image Differences.

Full size image

Adversarial Malware CFG Differences.

Full size image

Dataset preparation

The malware dataset used in the experiments originates from publicly available and widely recognized datasets: (1) SOREL-20M:The SOREL-20M dataset²⁵ consists of nearly 20 million files, each with pre-extracted features, metadata, high-quality labels obtained from multiple sources, and details regarding vendor detections of the malware samples at the time of collection. (2) BODMAS:The BODMAS dataset²⁶ is an open resource tailored for learning-based temporal analysis of PE malware. It encompasses 57,293 malware samples and 77,142 benign samples collected between August 2019 and September 2020. The dataset includes meticulously curated family information, comprising 581 families.

The two malware datasets cover a wide range of prevalent malware families, such as Ransomware, Crypto miner, Adware, and Downloader. Table 2 outlines the individual malware families and the corresponding number of malware instances in each dataset. However, due to copyright restrictions, binary files of benign software are not available in either dataset. Instead, the benign dataset used in our experiments comprises 12,500 samples collected from Windows system files and widely used software, including popular applications, development tools, and security software. Details of the benign dataset are shown in Table 3.

The decompilation process involved using the open-source reverse engineering framework, Radare2³², to analyze and extract the CFG of each sample, as outlined in Algorithm 2. Only scenarios under static analysis were considered during the extraction of CFGs. Basic blocks requiring indirect jumps to reach their destination could not have their edges established through static analysis alone and were therefore omitted from processing. As a result, the CFGs obtained from this extraction may be non-connected graphs. Additionally, invalid samples that failed to decompile and those consisting of more than 10,000 nodes were excluded from further analysis. Following this, features were extracted from the basic blocks using various representations. Finally, the extracted CFGs are converted into graph-structured data using the Deep Graph Library. This graph-structured data, along with its corresponding node feature list, can be used jointly for training and prediction of GNN detection models.

Metrics

1) Adversarial attack performance metric: To verify the efficacy of MalAOI in this study, we use the attack success rate (ASR) as an evaluation metric. Equation 2 is used to compute the ASR, which indicates the proportion of adversarial malware that can be successfully generated from all malware samples. The variable (N_{adv}) denotes the number of successfully generated adversarial samples, while (N_{orig}) denotes the total number of original samples.

2) Function-preserving metric: To verify the functionality retention of malware following the MalAOI modification in this study, we use the function-preserving rate (FPR) as an evaluation metric. Equation 3 is used to compute the FPR, which indicates the proportion of adversarial malware that operates normally and retains its functionality in the sampled test cases. The variable (N_{func-pres}) denotes to the number of function-preserving adversarial samples.

3) Extra load metrics: This study introduces a new metric, which comprises three components: average growing file size ratio (AGFR), average growing basic blocks ratio (AGBR), and average growing opcodes ratio (AGOR), outlined in Equations 4, 5, and 6, respectively.

Extracting CFGs

Full size image

Experimental groups and baseline

This section outlines the experimental setup and comparison baseline used in this study to validate the effectiveness of MalAOI against state-of-the-art methods.

1) Adversarial Attacker Performance

To evaluate the versatility of MalAOI across different GNN models employing varied structural designs and representation methods, we developed three GNN models with different structures, including GCN, GAT, and DGCNN. Four representations of the original byte sequences, namely MalConv and the opcodes N-Grams (n = 1,2,3 ), were employed to extract node features, resulting in 12 models. If MalAOI demonstrates high efficacy against all these malware detection models, this suggests its potential to effectively target multiple GNN detection models. In this experiment, MalAOI and the baseline were limited to a maximum of 1,000 modification operations and a 20% increase in file size.

In addition, we designed four baselines for comparison with MalAOI. One baseline is adapted from gym-malware¹⁷, while the other three are modified based on the baseline proposed by Zhang et al²⁴. The details of these four baseline attacks are as follows:

• Gym-malware With GNN: An adaptation was implemented on gym-malware¹⁷ by transforming its detection model into a GNN detection model within its environment, while maintaining the original action space.

• Function-preserving Random Insertion 1(FRI1): In this approach, random positions in the original malware sample are selected for insertion, where a fixed benign opcode sequence is inserted. The malware undergoes modification using the active opcode insertion base modification method.

• Function-preserving Random Insertion 2(FRI2): Insertion positions and benign opcode sequences are randomly selected for the original malware sample. The malware is then modified using the active opcode insertion base modification method.

• Function-preserving Accumulated Insertion(FAI): This method relies on hill-climbing techniques. For each modification, an insertion location and benign opcode sequence are randomly chosen. The predicted scores before and after the modification are compared. If the predicted score decreases, the modification is retained; otherwise, it is backtracked to the previous modification.

2) Adversarial Malware Quality

To assess the quality of the adversarial malware generated by the MalAOI method, we have structured four sets of sub-experiments:

a) Function-preserving: The function-preserving is an important prerequisite for adversarial attacks targeting malware detection models, ensuring the generated adversarial malware behaves as intended without changing its functionality. Owing to the difficulty of controlling register variables for testing each control flow in the CFG, a comparison between the original and modified malware was conducted in Windows SandBox. This entailed running the malware before and after modification separately. The HuoRong Sysdiag tool was also employed to analyze and compare the malware’s behavior. Various malware samples with different malicious functionality characteristics were randomly selected from the generated adversarial malware, including families such as Ransomware, Crypto miner, Adware, Downloader, and Trojan. Functional characteristics, especially evident in Ransomware, Crypto miner, adware, and Downloader variants, directly influence behaviors such as file encryption, CPU load, the appearance of pop-up advertisements, and downloading of additional applications. To analyze Trojan behavior, known for its camouflage and infiltration capabilities, a suite of tools is employed to identify actions such as network IO requests, addition of boot items, modification of system registry, and changes to key system files.

b) Extra Payload: MalAOI only operates insertion operations on original malware samples to ensure functional completeness. The modified adversarial malware exhibits increased byte size compared to the originals. Additionally, the CPU executes inserted benign code sequences, resulting in the modified adversarial malware executing more opcodes. Consequently, the modified malware inevitably introduces additional CPU load, leading to longer execution times compared to the original samples. To assess the impact of MalAOI-generated malware on system resources, an analysis of the additional load imposed by the malware was conducted. This analysis includes aspects such as an average increase in the size and percentage of malware, an average increase in the number and percentage of basic blocks, and an average increase in the number and percentage of opcodes.

c) Transferable Adversarial: The term “transferable adversarial” denotes the phenomenon wherein adversarial samples created for one model may also be misclassified by another model. We devised a transferable adversarial test experiment to confirm that the generated adversarial malware exhibits equal adversarial potency across other GNN models. In this test, the adversarial malwares generated by the MalAOI for one GNN detection model is input into other GNN models to test whether it can escape detection.

d) Commercial Malware Detector: In addition to machine learning-based malware detectors, we designed an experiment to test whether the generated adversarial malware can evade detection by traditional commercial malware detectors. For this purpose, we selected VirusTotal, a security platform that aggregates over 70 commercial malware detectors, including well-known solutions such as AVG, Kaspersky, and Microsoft, for malware detection. Since traditional detectors provide only a binary classification of samples as either benign or malicious, in this experiment, we define the “malicious probability” as the ratio of detectors that classify the sample as malicious to the total number of detectors. In this experiment, we generated 1,000 adversarial malware samples from prior experiments, and for each adversarial sample, we selected the corresponding original sample. Both the original and adversarial samples were uploaded to VirusTotal for analysis, and the results were compared to assess the “malicious probability” of the adversarial samples relative to the original ones.

3) Adversarial Training Test

Adversarial training is an important method for enhancing neural networks’ robustness. This is achieved by constructing adversarial samples and then integrating them with the original samples to form a new dataset. By using this new dataset for training a GNN malware detection model, this model may incorporate a level of defense capability. Evaluating this model’s vulnerability to the MalAOI approach can provide insights into its effectiveness in identifying potential defensive measures.

Implementation details

1) Dataset Detail

During the training of the GNN detection model, 8000 samples are randomly selected from each of the SOREL-20M malware dataset and the benign sample dataset to form the training dataset, 2000 samples from each to form the test dataset, and 2500 samples from each to form the validation dataset. During the validation process of the detection model using the validation dataset, benign opcodes can be extracted from the samples classified as benign by the detection model, and the extracted benign opcodes will constitute a library of benign opcode sequences for use in the attack model. During the training of the MalAOI attack model, 5000 samples are randomly selected from the BODMAS malware dataset as the training dataset and 2000 as the validation dataset, and the inserted opcode sequences are selected from the aforementioned benign opcode sequence library.

2) Experiments Environment Detail

All experiments were conducted on Ubuntu 22.04 using a system equipped with a Ryzen 9 5950X processor and an NVIDIA RTX 3090 graphics processing unit hardware and Torch 1.13.1 and DGL 2.0.0 with a Python 3.8 environment.

Threat model

1) Defender’s Capability

The defender is a GNN-based malware detection model that takes a control flow graph, derived from disassembling a PE file, as input and outputs the probability that the file is classified as malware. Table 4 shows that the structure of the GNN-based malware detection model, the node feature dimension is 32. Table 5 shows the classification performance of the GNN-based malware detection model.

2) Attacker’s Goals For an original sample predicted as malware by the GNN detection model, the attacker must carefully modify it to alter its control flow graph. The objective is to produce a modified sample that is ultimately misclassified as benign by the GNN model. During the modification process, the attacker is limited to making changes at the binary level, while ensuring that the functionality of the malware remains intact.

3) Attacker’s Knowledge In this black-box scenario, the attacker has no knowledge of the detection model’s parameters or structure. The only available information is the prediction probability provided by the model’s output. By comparing the detection probability to a predefined threshold, the attacker can assess whether the modified malware has successfully evaded detection by the model.

Average Attack Success Rate Histogram.

Full size image

Adversarial attacker performance

Table 6 presents the efficacy of the MalAOI, gym-malware with GNN, FRI, and FAI methods in evading detection by various GNN models. Figure 7 shows the comparative histogram. Experimental results indicate that MalAOI demonstrates effective performances, achieving an average attack success rate of 93.73% against a range of GNN detection models. GNN models subjected to more training rounds show increased susceptibility to MalAOI attacks, with an average attack success rate of 96.08% for models with 300 training rounds compared to 91.38% for those with 100 rounds. In the GAT model, particularly with the 2-Grams characterization method, MalAOI exhibits exceptional performance, with an adversarial success rate of 99.20%. In gym-malware with GNN, most base modification operation fails to modify the original malware sample CFG, only a section append modification method is successful, inserting a new section recognized by the decompiler tool as an unconnected diagram to the original CFG. The efficacy of this method is limited, with a success rate of only 3.22% for attacks. In FRI1 and FRI2, a purely random selection of insertion locations has no discernible impact on the attack success rate, which can average 40.68% in FRI1 and 42.10% in FRI2, regardless of whether benign opcode sequences are chosen as fixed or random. In FAI, a hill-climbing method based approach achieves a slightly higher attack success rate of 55.70% by continuously refining modified strategies to lower prediction scores.

The experiments demonstrate that MalAOI is effective against GNN detection models with diverse structures and characterization methods. The efficacy of MalAOI’s attack is significantly superior to other methods when the number of modifications is constrained to a maximum of 1,000, and the file size is allowed to expand by up to 20%.

A randomly selected GNN malware detection model was used to further compare attack success rates, with the maximum number of modifications set to 50, 100, 250, 500, 750, and 1,000, respectively. The results, shown in Figure 8, indicate that MalAOI maintains a high level of attack success even under these varying conditions.

Attack Success Rate at different limits.

Full size image

Adversarial malware quality

1) Function-preserving

Table 7shows the percentage of functionality retained in adversarial malware generated by each malware family modified by MalAOI. The experimental results demonstrate that most adversarial malware produced by MalAOI retains consistent functionality with the original samples. On average, an 78.43% FPR is achieved in MalAOI-modified adversarial malware, which is considered acceptable compared to the FPR reported in for gym-malware actions³³. The Downloader malware family exhibits the highest FPR of 91.89%.

In cases where adversarial malware is non-functional, our testing suggests the following potential explanations for this phenomenon: (1) Malware authors include code to detect debuggers or transform the original import table to prevent security personnel from analyzing the malware. (2) Malware authors incorporate self-checking code to ensure the integrity of the malware file and prevent others from modifying or cracking it. (3) In the activate opcode insertion base modification method, the PE file modification used the Lief module. However, while saving the modified PE file, the Lief module encountered a bug, resulting in the loss of some bytes from the binary program.

2) Extra Payload

The execution of software can result in scenarios such as sleep wait times or interruptions from external device requests, which may not be directly evaluated owing to the inability to traverse all control flow branches through direct execution. Therefore, this indicator needs to be approximated, which can be achieved by comparing the proportion of additional opcodes and basic blocks. The results in Table 8 indicate that the generated malware exhibited an average increase in file size of 72.23 KB, with 688.2 additional basic blocks and 9126.86 additional opcodes. This corresponds to a 12.87% increase in size, a 47.64% increase in basic blocks, and a 115.03% increase in opcodes compared to the original malware sample.

3) Transferable Adversarial

Furthermore, the adversarial mobility of MalAOI-generated malware was evaluated by inputting it into a graph model distinct from its original adversarial target for detection. The effectiveness of this approach was assessed by observing whether the adversarial malware could evade detection by other graph models, as illustrated in Figure 9. Experimental results indicate that the generated adversarial malware can exhibit high transferability to other GNN models, with an average escape rate of 44.83% across these models.

Transferable Adversarial Validation.

Full size image

4) Commercial Malware Detector

The detection results of the original and generated adversarial samples using VirusTotal are presented in Table 9. From the table, it can be observed that the average “malicious probability” of the original malware is 80.15%, while the average “malicious probability” of the generated adversarial samples is 59.77%, representing a reduction of 20.38%. The experimental results demonstrate that the proposed method exhibits a certain degree of counteracting capability against commercial malware detectors, with some detectors showing sensitivity to control flow graphs. A detailed comparison of the VirusTotal results reveals that the adversarial samples are more likely to evade detection by commercial malware detectors such as Panda, McAfee and ViRobot.

Adversarial training test

An adversarial training test was conducted to investigate potential defensive strategies. In this experiment, an additional 500 raw malware samples and 500 corresponding MalAOI-generated adversarial malware samples were incorporated into the malware dataset used to train the GNN detection model, forming the adversarial training dataset. The GNN detection model was then trained with defenses using four node characterization methods and the DGCNN structural model over 100 epochs. Subsequently, the abovementioned retrained model is used as an adversarial target and subjected to a second attack using MalAOI. The classification efficiency of the trained GNN detection model and the success rate of this attack are detailed in Table 10. This type of adversarial training can yield a degree of defense against MalAOI.

In this study, we introduce MalAOI, a black-box adversarial attack method based on RL tailored for the problem space of malware detection models for GNN. This is a new adversarial approach in the malware detection model confrontation field. MalAOI is designed to enable the evasion of GNN models based on CFG detection by modifying the malware CFG while preserving essential functionality. Experimental results show MalAOI’s robust performance across large datasets, including BODMAS and SOREL-20M. Moreover, its efficacy, minimal computational burden, and resilience against migration have been corroborated through extensive experimentation. This study underscores MalAOI’s potential to tackle challenges associated with function-preserving and CFG modifications.

The efficacy of MalAOI allows for the reveal of novel techniques potential employed by malicious actors to evade existing security protocols. Consequently, it encourages researchers to enhance their detection models and conduct adversarial training to mitigate identified threats. Furthermore, MalAOI aids in predicting malware development trends, uncovering attack paths and vulnerabilities overlooked by existing detection methods, implementing preemptive protection measures, and advancing security technology to meet evolving threats.

However, while MalAOI demonstrates promising results across comprehensive datasets and corresponding GNN models, real-world adversarial targets may pose greater complexities. Future research avenues may involve enhancing MalAOI by extending support for retaining malware features for X86-64 architectures or executable and linkable formats (ELFs) and achieving high transferability for other non-GNN models. Moving forward, we aim to explore MalAOI’s adaptability in intricate scenarios and conduct further investigations into implementing defenses for faster and more accurate responses to malware attacks.