\chapter{Evaluation}\label{ch:5-eval}
In this sectioned we evaluate \iottb, paying particular attention to the requirements defined in \cref{sec:requirements}.

\begin{table}[h!]
\centering
\begin{tabular}{|c|l|c|}
\hline
\textbf{Requirement ID} & \textbf{Description} & \textbf{Status} \\ \hline
\ref{req:auto_install_tools} & Installation of Tools & Not Met \\ \hline
\ref{req:auto_config_start} & Configuration and Start of Data Collection & $\downarrow$ \\ \hline
\ref{req:auto_config_start}a) & Automate Wi-Fi Setup & Not Met \\ \hline
\ref{req:auto_config_start}b) & Automate Data Capture & Met \\ \hline
\ref{req:auto_data_processing} & Data Processing & Partially Met \\ \hline
\ref{req:auto_reproducibility} & Reproducibility & Partially Met \\ \hline
\ref{req:auto_execution_control} & Execution Control & Not Met \\ \hline
\ref{req:auto_error_logging} & Error Handling and Logging & Partially Met \\ \hline
\ref{req:auto_documentation} & Documentation & $\downarrow$ \\ \hline
\ref{req:auto_documentation}a) & User Manual & Met \\ \hline
\ref{req:auto_documentation}b) & Developer Docs & Not Met \\ \hline
\ref{req:fair_data_meta_inventory} & Data and Metadata Inventory & Met \\ \hline
\ref{req:fair_data_formats} & Data Formats and Schemas & Met \\ \hline
\ref{req:fair_file_naming} & File Naming and Directory Hierarchy & Met \\ \hline
\ref{req:fair_preservation} & Data Preservation Practices & Partially Met \\ \hline
\ref{req:fair_accessibility} & Accessibility Controls & Not Met \\ \hline
\ref{req:fair_interoperability} & Interoperability Standards & Fully Met \\ \hline
\ref{req:fair_reusability} & Reusability Documentation & Met \\ \hline
\end{tabular}
\caption{Summary of Requirements Evaluation}
\label{tab:requirements-evaluation}
\end{table}

\cref{tab:requirements-evaluation} gives an overview of the requirements introduced in \cref{sec:requirements} and our assessment of their status. 
It is important to note that the status “Met” does not imply that the requirement is implemented to the highest possible standard. 
Furthermore, this set of requirements itself can (and should) be made more specific and expanded in both detail and scope as the project evolves.

Additionally, \ref{tab:requirements-evaluation} does not provide granularity regarding the status of individual components, which might meet the requirements to varying degrees. 
For example, while the requirement for data collection automation may be fully met in terms of basic functionality, advanced features such as handling edge cases or optimizing performance might still need improvement. 
Similarly, the requirement for data storage might be met in terms of basic file organization but could benefit from enhanced data preservation practices.

Thus, the statuses presented in \cref{tab:requirements-evaluation} should be viewed as a general assessment rather ground truth. 
Future work should aim to refine these requirements and their implementation to ensure that \iottbsc continues to evolve and improve.

To provide a more comprehensive understanding, the following sections offer a detailed evaluation of each requirement. This detailed analysis will discuss how each requirement was addressed, the degree to which it was met, and any specific aspects that may still need improvement. By examining each requirement individually, we can better understand the strengths and limitations of the current implementation and identify areas for future enhancement.

\section{\ref{req:auto_install_tools}: Installation of Tools}
\textbf{Status: Not Met} \\
\iottbsc does not install any software or tools by itself. Dependency management for Python packages is handled by installers like PIP, since the Python package declares its dependencies.
Tcpdump is the only external dependency, and \iottbsc checks if Tcpdump is available on the capture device. If it is not, the user is asked to install it.
Our position is that generally it is a good idea to not force installation of software and allow users the freedom to choose. The added benefit to the user of a built-in installer seems low. Adding some installer to \iottbsc does not promise great enough improvement in ease-of-use vis-à-vis the higher maintenance cost introduced to maintain such a module.
For future work, this requirement could be removed.

\section{\ref{req:auto_config_start}: Configuration and Start of Data Collection}
\textbf{Status: Partially Met} \\
The testbed automates the configuration and initiation of data collection processes, including wireless hotspot management and network capture. This automation reduces setup time and minimizes errors.
The testbed automates some aspects of configuring and initializing the data collection process. This project focused on package capture and adjacent tasks. \ref{req:auto_config_start}b can be considered \textit{complete} in that packet capture is fully supported thorough Tcpdump and important metadata is saved. Depending on the setup (see \cref{fig:cap-setup1} and \cref{fig:cap-setup2}) a Wi-Fi hotspot needs to be set up before packet capture is initiated. \iottbsc does not currently implement automated setup and takedown of a hotspot on any platform, so \ref{req:auto_config_start} a is not currently met. There are scripts for Linux systems bundled with the Python package which can be used with the \texttt{--pre} and \texttt{--post} options mentioned in \cref{sec:integrating-user-scripts}. But to consider this task fully automated and supported, this should be built in to \iottbsc itself.
Furthermore, there are other data collection tools like \textit{mitmproxy}\citep{mitmproxy} or more complicated setup tasks like setting up a routing table to allow for more capture scenarios, which are tedious tasks and lend themselves to automation. Future work should include extending the set of available automation recipes continuously. 
New task groups/recipe domains should be added as sub-requirements of \ref{req:auto_config_start}.
We propose the following new sub-requirement
\begin{itemize}
    \item \ref{req:auto_config_start}c: Testbed should implement automatic setup of NAT routing for situations where \ap is connection to the capture device and a bridged setup is not supported.
    \item \ref{req:auto_config_start}d: Testbed should dynamically determine which type of hotspot setup is possible and choose the appropriate automation recipe.
\end{itemize}
Extending \ref{req:auto_config_start} means stating which data collection and adjacent recipes are wanted.

\section{\ref{req:auto_data_processing}: Data Processing}
\textbf{Status: Partially Met} \\
While the testbed includes some basic data processing capabilities, there is room for improvement. 
Currently, the only one recipe exists for processing raw data. \iottbsc can extract a CSV file from a PCAP file. The possibilities for automation recipes which support data processing are many.
Having the data in a more standardized format allows for the creation of more sophisticated feature extraction recipes with application for machine learning. 
Before they are available, users can still use the \texttt{--post} option with their feature extraction scripts.

\section{\ref{req:auto_reproducibility}: Reproducibility}
\textbf{Status: Met} \\
Supported automation can be run with repeatedly, and used options are documented in the capture metadata. This allows others to repeat the process with the same options. 
So in this respect, this requirement is met. But, the current state can be significantly improved by automating the process of repeating a capture task with the same configuration as previous captures.
To support this, we propose the following new subrequirement which aids the automated reproduction of past capture workflows
\begin{itemize}
    \item \ref{req:auto_reproducibility}a: The testbed should be able to read command options from a file
    \item \cref{req:auto_reproducibility}b: The testbed should be able to perform a capture based on metadata files of completed captures.
\end{itemize}
Taking these requirements promises to seriously increase reproducibility.

\section{\ref{req:auto_execution_control}: Execution Control}
\textbf{Status: Not Met} \\
The testbed currently provides no controlled method to interfere with a running recipe. In most cases, \iottb will gracefully end if the user sends the process a SIGINT, but there are no explicit protections against data corruption in this case. Furthermore, during execution, \iottb writes to log files and prints basic information to the users' terminal. Extending this with a type of monitoring mechanism would be good steps toward complying with this requirement in the future.

\section{R1.6: Error Handling and Logging}
\textbf{Status: Met} \\
Robust error handling and logging are implemented, ensuring that issues can be diagnosed and resolved effectively. Detailed logs help maintain the integrity of experiments. It is also possible for the user to control how much output is given in the terminal. Here are four examples of the same command, with just increasing degrees of verbosity specified by the user:

\subsection{Logging Example}
\textbf{Command: } \verb|iottb sniff roomba  --unsafe -c 10 <verbosity>|
Verbosity can be unspecified, \verb|-v|, \verb|-vv| or \verb|-vvv|

\begin{figure}
    \centering
\begin{minted}[breaklines]{bash}
$ iottb sniff roomba  --unsafe -c 10
Testbed [I]
 Using canonical device name roomba
 Found device at path /home/seb/showcase/roomba
 Using filter None
 Files will be placed in /home/seb/showcase/roomba/sniffs/2024-07-01/cap0000-0214
 Capture has id 62de82ad-3aa2-460e-acd0-546e46377987
 Capture setup complete!
 Capture complete. Saved to roomba_62de82ad-3aa2-460e-acd0-546e46377987.pcap
 tcpdump took 2.16 seconds.
 Ensuring correct ownership of created files.
 Saving metadata.
 END  SNIFF SUBCOMMAND    
\end{minted}
    \caption{No verbosity.}
    \label{fig:example-no-verb}
\end{figure}

On the first verbosity level, only logger warnings are now printed to the standard output. During normal execution we do not expect significantly more output. This is also true for the second verbosity level.
\begin{figure}
    \centering
    \begin{minted}{bash}
$ iottb -v|-vv sniff roomba  --unsafe -c 10
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
WARNING - iottb_config - DatabaseLocations are DatabaseLocationMap in the class iottb.models.iottb_config
\end{minted}
    \caption{Only \textit{additional} output for \-v or \-vv.}
    \label{fig:example-one-verb}
\end{figure}

This changes once we reach the third verbosity level, because now additonally the logger level is set to "INFO".
Clearly, \cref{fig:example-lvl-three} contains far more output than \cref{fig:example-one-verb}.
It is possible to get even more output printed to standard output by also passing the \verb|--debug| flag.
This produces significantly more output as can be seen in \cref{fig:example-debug-output}.
\begin{figure}
    \centering
    \begin{minted}{bash}
$ iottb -vvv sniff roomba  --unsafe -c 10
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
INFO - main - cli - 48 - Starting execution.
INFO - iottb_config - __init__ - 24 - Initializing Config object
WARNING - iottb_config - warn - 21 - DatabaseLocations are DatabaseLocationMap in the class iottb.models.iottb_config
INFO - iottb_config - load_config - 57 - Loading configuration file
INFO - iottb_config - load_config - 62 - Config file exists, opening.
INFO - sniff - validate_sniff - 37 - Validating sniff...
INFO - sniff - sniff - 91 - sniff command invoked
INFO - string_processing - make_canonical_name - 20 - Normalizing name roomba
Testbed [I]
 Using canonical device name roomba
 Found device at path /home/seb/showcase/roomba
INFO - sniff - sniff - 152 - Generic filter None
 Using filter None
 Files will be placed in /home/seb/showcase/roomba/sniffs/2024-07-01/cap0003-0309
 Capture has id f1e92062-4a82-4429-996c-97bd7fa57bec
INFO - sniff - sniff - 186 - pcap file name is roomba_f1e92062-4a82-4429-996c-97bd7fa57bec.pcap
INFO - sniff - sniff - 187 - stdout log file is stdout_f1e92062-4a82-4429-996c-97bd7fa57bec.log
INFO - sniff - sniff - 188 - stderr log file is stderr_f1e92062-4a82-4429-996c-97bd7fa57bec.log
INFO - sniff - sniff - 246 - tcpdump command: sudo tcpdump -# -n -vvv -c 10 -w /home/seb/showcase/roomba/sniffs/2024-07-01/cap0003-0309/roomba_f1e92062-4a82-4429-996c-97bd7fa57bec.pcap
 Capture setup complete!
 Capture complete. Saved to roomba_f1e92062-4a82-4429-996c-97bd7fa57bec.pcap
 tcpdump took 2.12 seconds.
 Ensuring correct ownership of created files.
 Saving metadata.
 END  SNIFF SUBCOMMAND        
    \end{minted}
    \caption{Caption}
    \label{fig:example-lvl-three}
\end{figure}

\section{\ref{req:auto_documentation}: Documentation}
\textbf{Status: Partially Met} \\
For users, there is a 'Command Line Reference' (see \cref{appendix:cmdref}) which details all important aspects of operating the \iottb \cli. Furthermore, helpful messages are displayed regarding the correct syntax of the commands if an input is malformed. So user documentation does exist and, while certainly can be improved upon, is already helpful. 
Unfortunately, documentation for developers is currently poor. The codebase is not systematically documented and there is currently no developer's manual. 
Thoroughly documenting the existing codebase should be considered the most pressing issue and tackled first to improve developer documentation.

\section{\ref{req:fair_data_meta_inventory}: Data and Metadata Inventory}
\textbf{Status: Fully Met} \\
The testbed organizes data and metadata in a standardized and principled way. The database is complete with respects to the currently primary and secondary artifact which stem from operating  \iottb itself.
While complete now, extending \iottb carries the risk of breaking this requirement if not careful attention is given.
Since the database is a central part of the system as a whole, extensions must ensure that they comply with this requirement before they get built in. 

\section{\ref{req:fair_data_formats}: Data Formats and Schemas}
\textbf{Status:  Met} \\
The testbed standardizes directory and file naming. All metadata is in plain test and in the JSON format. This makes them very accessible to both humans and machines. Currently, the only binary format which \iottbsc creates are PCAP files. Luckily, the PCAP format is widely known and not proprietary, and  widely available tools (e.g., Wireshark\citep{wiresharkorg}) exist to inspect them. Furthermore, the data in the PCAP files can be extracted in to the plaintext CSV format, this further improves interoperability. Consistence is currently implicitly handles, that is, there are no strict schemas \footnote{Strict schemas for metadata file briefly were introduced, but then abandoned due to the lack of knowledge surrounding the PYdantic library \citep{pydantic}.} \iottb should generally not corrupt data during operation. But plaintext files are manually editable and can inadvertently be corrupted or made invalid (e.g. accidentally deleting a few digits from a UUID). 
It is important to keep this in mind when extending \iottbsc and the types of files residing in the database become more heterogeneous.


\subsection{\ref{req:fair_file_naming}: File Naming and Directory Hierarchy}
\textbf{Status: Met} \\
\iottb currently names all files which it creates according to a well-defined schema. In all cases, the file name is easily legible (e.g., metadata files like \cref{lst:cap-meta}) or the context of where the file resides provides easy orientation to a human reviewer. For instance, raw data files, which currently only are PCAP files, are all named with a \uuid. This is not helpful to the human, but the metadata file, which resides in the same directory, provides all the needed information to be able to understand what is contained within it. Furthermore, these files reside in a directory hierarchy which identifies what devices the traffic belongs to, the date the capture file was created. Finally, capture files reside in a directory which identifies where in the sequence of capture of a given day it was created. 
Automation recipes expanding the range of data types collected can just follow this convention. This ensures interoperability and findability between various capture methods.

\cref{ch4} \ref{sec:add-dev} already showed examples of the naming convention when adding devices.


\section{\cref{req:fair_preservation}: Data Preservation Practices}
\textbf{Status: Partially Met} \\
Specific data preservation practices are not taken. \iottb already follows the Library of Congress recommendations on data formats (see \citet{recommendedformatrsLOC}). Most data is stored in plain text, and the binary formats used are widely known within the field and there is no access barrier. 
To enhance the testbeds' compliance with this requirement, automation recipes which back-up the data to secure locations periodically can be developed. The need for built-in preservation should be balanced with the goal of not introducing dependencies not related to the core aim of automated collection and FAIR storage. One way is just to have a repository of scripts which are not built in to the \iottb executable, but which users can  use and adapt to their  needs\footnote{For instance rsync scripts with predefined filters appropriate for the database.}.

\section{\cref{req:fair_accessibility}: Accessibility Controls}
\textbf{Status: x} \\
While the \iottb executable is ware what data it can and cannot access or change, there are currently no  wider access controls implemented.