SofiaWL-LearnerPlusPlus 1.1.5

This package contains the necessary methods to replicate the information from the study "The Cost vs. the Benefit of Adding an Extra Code Reviewer to Mitigate Developer Turnover through Reviewer Recommenders" which is published in ICSE'26.

To run the simulations, follow these steps:

1. You should first download and restore the database backup into your local MS SQL Server from Figshare. There is a separate database for each studied project, and you should restore all of them. To restore a database from a .bacpac file, start by opening SQL Server Management Studio (SSMS) and connecting to your local SQL Server instance. In the Object Explorer, right-click on the Databases node and choose Import Data-tier Application. From there, click Browse to locate the .bacpac file on your local disk. Once you’ve selected the file, follow the wizard by clicking Next, reviewing the steps, and then clicking Finish to complete the import process. This will restore your database and make it available in your SQL Server instance. Note that the databases are approximately 2 GB in size.

2. Install the SofiaWL-LearnerPlusPlus package and download the simulation commands and configuration files from the official GitHub repository.

dotnet tool install --global SofiaWL-LearnerPlusPlus

3. Open and modify each configuration file to set up the connection with the database. You have to provide the server address along with the credentials to your local SQL server instance. The following snippet shows a sample of how the connection string should be set.

"ConnectionStrings": {
  "RelationalGit": "Server=ip_db_server;User Id=user_name;Password=pass_word;Database=Roslyn_PlusPlus"
},

4. Open the simulations.ps1 file using an editor and update all the paths to the configuration files. For instance, each of the following variables contains the absolute path of the corresponding configuration file for the first research question.

$corefx_conf_RQ1 = "\absolute\path\to\Replace_Risky\corefx_conf.json"
$coreclr_conf_RQ1 = "\absolute\path\to\Replace_Risky\coreclr_conf.json"
$roslyn_conf_RQ1 = "\absolute\path\to\Replace_Risky\roslyn_conf.json"
$rust_conf_RQ1 = "\absolute\path\to\Replace_Risky\rust_conf.json"
$kubernetes_conf_RQ1 = "\absolute\path\to\Replace_Risky\kubernetes_conf.json"

5. Open PowerShell and run the simulations.ps1 script to simulate the performance of all the studied recommendation strategies across all projects.

./simulations.ps1

6. To get your simulation IDs, connect to your local SQL Server instance and run the following query for each project. This query is a sample for the Rolsyn project.

-- Get the Id of the simulation 
SELECT  Id,
	KnowledgeShareStrategyType, 
	StartDateTime,
	EndDateTime
	PullRequestReviewerSelectionStrategy,
	SimulationType 
FROM [Roslyn_PlusPlus].[dbo].[LossSimulations]
WHERE EndDateTime > StartDateTime
ORDER BY StartDateTime DESC

7. After restoring the simulation IDs, comment out the commands at the end of the simulations.ps1 file and set the values for --analyze-result-path, --recommender-simulation, and --reality-simulation. Then, run these commands to calculate the outcome measures and store the values in --analyze-result-path.

dotnet-rgit --cmd analyze-simulations --analyze-result-path "\absolute\path\to\CoreFX_Results_Directory" --recommender-simulation <CoreFX-Simulation-Ids-SeparatedWithSpace> --reality-simulation <Reality-Id> --conf-path $corefx_conf_RQ1
dotnet-rgit --cmd analyze-simulations --analyze-result-path "\absolute\path\to\CoreCLR_Results_Directory" --recommender-simulation <CoreCLR-Simulation-Ids-SeparatedWithSpace> --reality-simulation <Reality-Id> --conf-path $coreclr_conf_RQ1
dotnet-rgit --cmd analyze-simulations --analyze-result-path "\absolute\path\to\Roslyn_Results_Directory" --recommender-simulation <Roslyn-Simulation-Ids-SeparatedWithSpace> --reality-simulation <Reality-Id> --conf-path $roslyn_conf_RQ1
dotnet-rgit --cmd analyze-simulations --analyze-result-path "\absolute\path\to\Rust_Results_Directory" --recommender-simulation <Rust-Simulation-Ids-SeparatedWithSpace> --reality-simulation <Reality-Id> --conf-path $rust_conf_RQ1
dotnet-rgit --cmd analyze-simulations --analyze-result-path "\absolute\path\to\Kubernetes_Results_Directory" --recommender-simulation <Kubernetes-Simulation-Ids-SeparatedWithSpace> --reality-simulation <Reality-Id> --conf-path $kubernetes_conf_RQ1

For more details, check out the official repository on GitHub.