Spectre and Meltdown fixes influence on iOS apps build time

At the beginning of 2018, a public report about Spectre and Meltdown vulnerabilities has been revealed to the general public. Since that day, a number of hardware and software vendors released various patches to eliminate a possibility to exploit mentioned vulnerabilities.
Due to hardware nature of both vulnerabilities, software workarounds were required to mitigate the risk. However, despite its convenience for end users, such solution came at a price. As soon as first fixed were deployed, some reports about decreased system performance started to show up around the Internet, with some of them claiming up to 5% slowdown for typical use cases [1]. Apple addressed Spectre and Meltdown in 10.13.2 and 10.13.3 releases of macOS High Sierra [2], and it remains to be seen how those patches will affect our productivity as developers who use Xcode to build software. A number of benchmarks were published at [3] describing Geekbench 4 score and XNU kernel build time difference between all High Sierra releases except 10.13.3 and newer. The performance loss of 2% to 9% has been reported for XNU kernel compilation.

In this post, I would like to focus specifically on the evaluation of potential slowdown during iOS apps compilation.

Spectre and Meltdown fixes aftermath

To find out whether there is any significant difference between build time on macOS versions with and without fixed vulnerabilities, I used 3 apps written in Swift: Firefox (master, 4cd9bcd), Kickstarter (master, 445373b) as well as my own free app Electronics Engineer Helper. Each project was compiled 16 times on each macOS version using a simple bash script

#! /bin/bash
PROJECT_NAME=$1
SCHEME_NAME=$2
CONFIGURATION=$3
for ((n=0;n<16;n++));
do
    rm -rf ~/Library/Developer/Xcode/DerivedData/*
    /usr/bin/time -p xcodebuild -project $PROJECT_NAME -scheme $SCHEME_NAME -configuration $CONFIGURATION -sdk iphonesimulator build > /dev/null
    sleep 10
done

…which was used with following input arguments

./profile.sh Kickstarter.xcodeproj Kickstarter-iOS Release
./profile.sh Client.xcodeproj Fennec Firefox
./profile.sh “Electronics Engineer Helper.xcodeproj” “Electronics Engineer Helper” Release

It’s important to manually clean DerivedData folder before each run, as simply passing clean command to xcodebuild will not remove the ModuleCache, and so the duration of a first and all following builds will be slightly different.

All apps were compiled using Xcode 9.2 (9C40b) on a Hackintosh featuring Core i7 6700K CPU (4 cores, 4 GHz), 32 GB of RAM (DDR4, 2400 MHz) and Kingston KC1000 SSD (NVMe, 240 GB, PCIe 3, 4 Lanes). DerivedData was located on a RAM disk created using the iRamDisk app.

With all data collected, a mean build time as well as margin of error at 95% confidence level was calculated. The calculation of margin of error will help us to better understand whether any difference between two OS releases is just a random fluctuation or not.

With that said, let’s see the results.

                                        Build time, seconds
          Application            ---------------------------------
                                     10.13.1           10.13.3      
 -----------------------------   ---------------   --------------- 
  Kickstarter                    302.776 ± 0.299   308.541 ± 1.344  
  Firefox                        146.935 ± 1.297   151.014 ± 0.949  
  Electronics Engineer Helper    165.211 ± 0.525   171.021 ± 0.537

Conclusion

The slowdown due to Spectre and Meltdown fixes is definitely present, but the good news is that it is barely noticeable. For mentioned projects, the build time increased on average from 1.9% to 3.5%, which is similar to the results of other benchmarks posted on the web.