Secure Autonomous Systems

Instructor: Prof. Sibin Mohan, The George Washington University

CSCI 6907/3907 | Fall 2022 | TR 12:45PM - 02:00PM ET | SMTH 115


MP I-B: V2X Misbehavior Detectors in VEINS

Administrivia

Announcement Date Sept. 22, 2022
Submission Date Oct. 07, 2022
Submission Time 11:59 PM ET
Total Points 15

Objective

In this MP, we will dive deeper on how to use and implement misbehavior detectors in VEINS simulator for V2X networks. We will check the implementation details and performance of a basic misbehavior detector and discuss how the misbehavior checks relate to specific attacks. Objectives of this MP are as follows:

Prelude

We will be using the Local Detection module in the chart in the VEINS and F2MD Simulators (from MP I-A) to implement our first misbehavior detector. We will generate protocol reports using the Data Output module and observe/evaluate these reports.

MP1 instructions

In this MP, we will use and modify some of the basic misbehavior detection (MBD) applications in the vehicular networks. To gain familiarity, we will use pre-implemented MBD applications and examine how they work.

  1. Every simulation is governed by the omnetpp.ini files residing in the scenario directories. We will again modify this file as first thing to set up attacks and some additional loggings. Go to Desktop/v2x_simulator/F2MD/F2MD/veins-f2md/f2md-networks/IRTSystemXScenario/omnetpp.ini
    in your favorite text editor and make the following changes:
    1. Change the simulation length sim-time-limit to 1200s for a 20 minute simulation:
    2. appl.MixLocalAttacks = true to use a mix of all attack types in the list.
    3. appl.LOCAL_ATTACKER_PROB = 0.3 to set 30% of all vehicles as attackers.
    4. Set appl.writeVeremi = true to have all vehicles record their received message logs seperately to f2md-results file.
    5. Set appl.writeBSMsV1 = true to record MBD BSM logs.
    6. Set appl.writeListBSMsV1 = true to record aggregated MBD BSM logs.
    7. Set appl.writeReportsV1 = true to record MBD reports by vehicles.
    8. Set appl.writeListReportsV1 = true to record aggregated MBD BSM logs.
    9. Set appl.EnableV1 = true to enable V1 app socket of our misbehavior detection app to listen to simulation messages.
    10. Set appl.SaveStatsV1 = true to record MBD app logs.
    11. Set appl.appTypeV1 = 0 which is the simple thresholding MBD application. It essentially checks the message plausibilities and marks message as attacks plausibility score falls below certain threshold.
    12. Set appl.checksVersionV1 = 0 which is the simple legacy plausibility calculations. Essentially, these checks calculate certain plausibility measures for each message and adding 0 or 1, depending on if the message seems plausible or malicious.
  2. Run the IRTSX simulation with default thresholding MBD parameters. After it ends, you will see the following MBD records in addition to Veremi BSM logs:
    mp1
    1. MDBSMs_V1: This is the seperate and timed records of all BSMs over the simulation, received by the Misbehaivor App. For instance, as the naming convention, MDBSM_V1_8-7028_10218_1090 file refers to BSM records sent at t= 8.7028 seconds sent by vehicle with pseudonym ID 1090 and received by vehicle with pseudonym ID 10218. Pseudonym is a temporary certificate that vehicle uses as ID, and mapping from real IDs to pseudonyms are avaiable in all logs.
    2. MDBSMsList: This is the records of all BSMs over the simulation, received by the Misbehaivor App, per vheicle pseudonym. It is essentially an aggregated version of records at (a). For instance, as the naming convention, MDBSM_V1_1090 file refers to BSM records received by vehicle with pseudonym ID 1090 during simulation.
    3. MDReports: This is the collective misbehavior reports generated by our detection app. For instance, as the naming convention, MDReport_V1_28-2104_10154_10331 file refers to misbehavior report sent at t= 28.2104 seconds sent by vehicle with pseudonym ID 10154, reporting the vehicle with pseudonym ID 10331 for misbehavior. Pseudonym is a temporary certificate that vehicle uses as ID, and mapping from real IDs to pseudonyms are avaiable in all logs.
  3. Loop over all the records on the Veremi file names (easy way) to extract attacker and genuine vehicle ID's (A0 are genuine and others are attackers). These records will serve as the ground truth labels for us to evaluate the performance of our MBD app:
    mp1

    The attack types are given in omnetpp.ini file, as below:
    mp1

  4. Loop over all the MBD app records in the MDBSMList directory, and extract received BSM messages for each vehicle. Each file (e.g. MDBSM_V1_10452) has the following record structure, with keys as BSMPrint:
    1. Metadata: Credentials for the set of received messages. Here, the receiver has ID:45, pseudonym:10452, is an attacker itself (hard way to extract labels), and this BSMPrint includes messages received at t=26.21 seconds. mp1

    2. BSMCheck: Plausibility of self messages, we will not use this information for now. But essentially, some plausibility checks have low scores for vehicles that are attackers itself, which allow for detection.
    3. BSMs: List of received BSMs sent by other vehicles. Each BSM has following fields:
      1. Pseudonym& Real ID: Credentials of message sender. The real ID here matches the extracted labels from Part 3.
      2. MbType: "LocalAttacker" if the sender is an attacker.
      3. AttackType: "$ATTACKNAME$", type of the attack if the sender is an attacker.
      4. mp1
    4. For each file and each BSM under BSMs list of MDReport_V1_.., explained in Section 2-c, check if the sender is marked as an attacker or genuine for each message. Compare these reports for each BSM with the ground truth labels from Part 3 (vehicle-based labels), and Part 4-c (BSM-based labels). Create a confusion matrix with the number of True positive, True Negative, False Positive and False Negative reports. For more information on confusion matrix, refer to scikit-learn:confusion_matrix.

  5. Detect the attack based performance of the MBD for attacks A5-A15. Create a table that tells the TP, FP, FN rates for each attack, so we know for which attacks our MBD performs better (or worse):
    Attack Type True Positive False Positive False Negative
    A5 (Constant Speed) #TP-A5 (X%) #FP-A5 (X%) #FN-A5 (X%)
    A6 (Constant Speed Offset) #TP-A6 (X%) #FP-A6 (X%) #FN-A6 (X%)
    ... ... ... ...
    A15 (DosDisruptive) #TP-A15 (X%) #FP-A15 (X%) #FN-A15 (X%)

    The attack types are given in omnetpp.ini file, as below:
    mp1

  6. Check the source code for the following and describe briefly:
    1. Refer to the calculations at legacy checks (at directory below) that our thresholding app uses to check the if the message is an attack: mp1 Briefly describe PositionConsistancyCheck, SpeedConsistancyCheck, and PositionSpeedConsistancyCheck functions and list&comment on at least 3 attacks (from A5 through A15) that these checks can be useful.
    2. Briefly describe the thresholding app itself (at directory below), which uses the plausibility results (0 or 1) from legacy checks, compares it to a threshold and decides if message is attack or not. What happens if we decrease the Threshold parameter that our thresholdApp uses, i.e. do we catch more attackers, do we get more false positives etc.? mp1
  7. Submission Instructions

    Save and submit your report as a PDF file of 2 pages or less. Your report should include the following:
    1. A 2x2 confusion matrix of True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN): Confusion matrix of MBD performance from Part 4, including number of messages and percentages for each matrix entry. [5 pt]
    2. The attack based MBD performance table, as given in Part 5. [5 pt]
    3. Explanations of Legacy checks and thresholding app, as described in Part 6. [5 pt].

    References

    1. https://veins.car2x.org/documentation/
    2. https://github.com/josephkamel/F2MD
    3. Here is another well-put VEINS tutorial for reference:
      http://kjit.bme.hu/images/Tovabbi_allomanyok/Trafficlab/Research/SUMOTRACI/SUMO-VEINS_Tutorial.pdf
    4. A custom simulation tutorial in VEINS:
      http://cse.iitkgp.ac.in/~soumya/micro/t2-3.pdf
    5. For the attack descriptions (A5-A15), refer to the original F2MD paper:
      https://hal.archives-ouvertes.fr/hal-02527873/document