Secure Autonomous and Cyber-Physical Systems

Instructor: Prof. Sibin Mohan, Oregon State University

CS/ECE 599 | Winter 2022 Term | MW 2:00 - 3:50 PM PT | BAT 150

MP I-B: V2X Misbehavior Detectors in VEINS

Administrivia

Objective

• Enable custom applications on V2X networks and implement a misbehavior detector with it.
• Observe a wide array of misbehavior patterns (10 more attacks) and check the performance of the implemented detector against these attacks.
• Observe and relate calculations of detectors to specific attacks.
• Understand how to modify misbehavior detectors for attacks, specific components of detectors in networking context, and how to implement them.

Prelude

MP1 instructions

In this MP, we will use and modify some of the basic misbehavior detection (MBD) applications in the vehicular networks. To gain familiarity, we will use pre-implemented MBD applications and examine how they work.

1. Every simulation is governed by the omnetpp.ini files residing in the scenario directories. We will again modify this file as first thing to set up attacks and some additional loggings. Go to Desktop/v2x_simulator/F2MD/F2MD/veins-f2md/f2md-networks/IRTSystemXScenario/omnetpp.ini
   in your favorite text editor and make the following changes:
   1. Change the simulation length sim-time-limit to 1200s for a 20 minute simulation:
   2. appl.MixLocalAttacks = true to use a mix of all attack types in the list.
   3. appl.LOCAL_ATTACKER_PROB = 0.3 to set 30% of all vehicles as attackers.
   4. Set appl.writeVeremi = true to have all vehicles record their received message logs seperately to f2md-results file.
   5. Set appl.writeBSMsV1 = true to record MBD BSM logs.
   6. Set appl.writeListBSMsV1 = true to record aggregated MBD BSM logs.
   7. Set appl.writeReportsV1 = true to record MBD reports by vehicles.
   8. Set appl.writeListReportsV1 = true to record aggregated MBD BSM logs.
   9. Set appl.EnableV1 = true to enable V1 app socket of our misbehavior detection app to listen to simulation messages.
   10. Set appl.SaveStatsV1 = true to record MBD app logs.
   11. Set appl.appTypeV1 = 0 which is the simple thresholding MBD application. It essentially checks the message plausibilities and marks message as attacks plausibility score falls below certain threshold.
   12. Set appl.checksVersionV1 = 0 which is the simple legacy plausibility calculations. Essentially, these checks calculate certain plausibility measures for each message and adding 0 or 1, depending on if the message seems plausible or malicious.
2. Run the IRTSX simulation with default thresholding MBD parameters. After it ends, you will see the following MBD records in addition to Veremi BSM logs:
   
   1. MDBSMs_V1: This is the seperate and timed records of all BSMs over the simulation, received by the Misbehaivor App. For instance, as the naming convention, MDBSM_V1_8-7028_10218_1090 file refers to BSM records sent at t= 8.7028 seconds sent by vehicle with pseudonym ID 1090 and received by vehicle with pseudonym ID 10218. Pseudonym is a temporary certificate that vehicle uses as ID, and mapping from real IDs to pseudonyms are avaiable in all logs.
   2. MDBSMsList: This is the records of all BSMs over the simulation, received by the Misbehaivor App, per vheicle pseudonym. It is essentially an aggregated version of records at (a). For instance, as the naming convention, MDBSM_V1_1090 file refers to BSM records received by vehicle with pseudonym ID 1090 during simulation.
   3. MDReports: This is the collective misbehavior reports generated by our detection app. For instance, as the naming convention, MDReport_V1_28-2104_10154_10331 file refers to misbehavior report sent at t= 28.2104 seconds sent by vehicle with pseudonym ID 10154, reporting the vehicle with pseudonym ID 10331 for misbehavior. Pseudonym is a temporary certificate that vehicle uses as ID, and mapping from real IDs to pseudonyms are avaiable in all logs.
3. Loop over all the records on the Veremi file names (easy way) to extract attacker and genuine vehicle ID's (A0 are genuine and others are attackers). These records will serve as the ground truth labels for us to evaluate the performance of our MBD app:
   
   
   The attack types are given in omnetpp.ini file, as below:
   


4. [M] Loop over all the MBD app records in the MDBSMList file, and extract received BSM messages for each vehicle. Each file (e.g. MDBSM_V1_10452) has the following record structure, with keys as BSMPrint:
   1. Metadata: Credentials for the set of received messages. Here, the receiver has ID:45, pseudonym:10452, is an attacker itself (hard way to extract labels), and this BSMPrint includes messages received at t=26.21 seconds.
   
   
   2. BSMCheck: Plausibility of self messages, we will not use this information for now. But essentially, some plausibility checks have low scores for vehicles that are attackers itself, which allow for detection.
   3. BSMs: List of received BSMs sent by other vehicles. Each BSM has following fields:
      1. Pseudonym& Real ID: Credentials of message sender. The real ID here matches the extracted labels from Part 3.
      2. MbType: "LocalAttacker" if the sender is an attacker.
      3. AttackType: "$ATTACKNAME$", type of the attack if the sender is an attacker.
      
   4. For each file and each BSM under BSMs list of MDReport_V1_.., explained in Section 2-c, check if the sender is marked as an attacker or genuine for each message. Compare these reports for each BSM with the ground truth labels from Part 3 (vehicle-based labels), and Part 4-c (BSM-based labels). Create a confusion matrix with the number of True positive, True Negative, False Positive and False Negative reports. For more information on confusion matrix, refer to scikit-learn:confusion_matrix.


5. Detect the attack based performance of the MBD for attacks A1-A10. Create a table that tells the TP, FP, FN rates for each attack, so we know for which attacks our MBD performs better (or worse):
   

   Attack Type	True Positive	False Positive	False Negative
   A1 (Constant Position)	#TP-A1 (X%)	#FP-A1 (X%)	#FN-A1 (X%)
   A2 (Constant Position Offset)	#TP-A2 (X%)	#FP-A2 (X%)	#FN-A2 (X%)
   ...	...	...	...
   A10 (Disruptive)	#TP-A10 (X%)	#FP-A10 (X%)	#FN-A10 (X%)

   
   The attack types are given in omnetpp.ini file, as below:
   


6. Check the source code for the following and describe briefly:
   1. Refer to the calculations at legacy checks (at directory below) that our thresholding app uses to check the if the message is an attack: Briefly describe PositionConsistancyCheck, SpeedConsistancyCheck, and PositionSpeedConsistancyCheck functions and list&comment on at least 4 attacks (from A1 through A10) that these checks can be useful.
   2. Briefly describe the thresholding app itself (at directory below), which uses the plausibility results (0 or 1) from legacy checks, compares it to a threshold and decides if message is attack or not. What happens if we decrease the Threshold parameter that our thresholdApp uses, i.e. do we catch more attackers, do we get more false positives etc.?

Submission Instructions

Save and submit your report as a PDF file of 2 pages or less. Your report should include the following:
1. A 2x2 confusion matrix of True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN): Confusion matrix of MBD performance from Part 4, including number of messages and percentages for each matrix entry. [5 pt]
2. The attack based MBD performance table, as given in Part 5. [5 pt]
3. Explanations of Legacy checks and thresholding app, as described in Part 6. [5 pt].

References

1. https://veins.car2x.org/documentation/
2. https://github.com/josephkamel/F2MD
3. Here is another well-put VEINS tutorial for reference:
   http://kjit.bme.hu/images/Tovabbi_allomanyok/Trafficlab/Research/SUMOTRACI/SUMO-VEINS_Tutorial.pdf
4. A custom simulation tutorial in VEINS:
   http://cse.iitkgp.ac.in/~soumya/micro/t2-3.pdf