Using WITH_META_MODE and ccache for FreeBSD build boosts
If you're looking to speed up your FreeBSD kernel and world builds, try using the
WITH_META_MODE compile option and/or installing and enabling
ccache. Turning these on gave me a 30x speed boost for the buildworld and buildkernel process. That's not a typo. Compiling a complete kernel and world on my test machine used to take over 6 hours, even with
/usr/obj/ having been left intact from prior builds. My most recent build finished in just 13 minutes, or around 1/30th of the old time!
The WITH_META_MODE compile option takes advantage of the
filemon kernel module, creating and inspecting metadata about the objects compiled during a system build. New metadata files get stored in
/usr/obj alongside the object files they refer to. On subsequent builds, when the compiler goes to build an object, if nothing about the build command or its related files (headers, libraries, etc.) has changed, the cached object can be used instead of compiling a new file.
On a FreeBSD 11.2 system, there are more than 52,000
.o files in
/usr/obj after a system build, not to mention the final binaries. That's a lot of compiling that can be skipped, especially with patch releases where not much has changed.
WITH_META_MODE, edit your
/etc/src-env.conf (or create this file if it doesn't exist) to contain this line:
Then, prior to performing your buildworld and buildkernel routine, you must run:
Loading the filemon module becomes a crucial part of the build process. If you forget, the compiler won't be able to perform the necessary comparisons against existing metadata files. Having this module loaded all the time can slow things down, so you should only load it before building the system. Just add
kldload filemon to the top of your build script if you use one, or make sure you remember to run it manually before kicking off a buildworld.
ccache is a compiler cache that stores copies of compiled object code (
.o files) on disk. When
make attempts to compile a file that's identical to one in the cache, the cached copy is used instead, bypassing the compile process. This benefit is system-wide and will apply to almost every
make job you run, including ports and other software.
cd /usr/ports/devel/ccache && make install or run
pkg add ccache. Then add a couple of lines to your
You can put the cache directory wherever you have room for it; after a kernel and world build, plus various ports builds over the past month, mine takes up 4.4GB. If you have a maintenance job that aggressively cleans out
/tmp, you should choose another location.
Here are my compile times for 11.2-RELEASE-p7, it took about 400 minutes. When this build was initiated,
/usr/obj contained artifacts from building 11.2-RELEASE-p5. Prior to running the build, I configured
ccache, but neither of them had generated their caches yet.
>>> World build started on Thu Jan 3 09:34:10 CST 2019 >>> World build completed on Thu Jan 3 15:45:43 CST 2019 >>> Kernel build for HOST started on Thu Jan 3 16:34:27 CST 2019 >>> Kernel build for HOST completed on Thu Jan 3 17:04:49 CST 2019
And here are my compile times for 11.2-RELEASE-p9 on the same machine, the process only took about 13 minutes. When this build was initiated,
/usr/obj contained artifacts from building 11.2-RELEASE-p7, and also contained all of the
.meta files that
WITH_META_MODE had generated last time.
>>> World build started on Tue Feb 5 16:44:24 CST 2019 >>> World build completed on Tue Feb 5 16:55:51 CST 2019 >>> Kernel build for HOST started on Tue Feb 5 17:04:11 CST 2019 >>> Kernel build for HOST completed on Tue Feb 5 17:06:02 CST 2019
Because I enabled both
ccache at the same time, I can't say for sure which one is most responsible for the performance improvement. Gauging by anecdotes from the FreeBSD forums, I suspect
WITH_META_MODE makes the most difference.