(new Soapbox())->shout(array_map('strtoupper', $opinions)); //Shaun's blog


Me, elsewhere

GitHub
parseword
Miscellaneous public code

Twitter
@parseword
I don't tweet much

XMPP chat
xmpp@shaunc.com
(Pidgin, Miranda, Swift, etc.)


Perfect is the enemy of good enough.

Using WITH_META_MODE and ccache for FreeBSD build boosts

Posted February 06, 2019 by shaun

If you're looking to speed up your FreeBSD kernel and world builds, try using the WITH_META_MODE compile option and/or installing and enabling ccache. Turning these on gave me a 30x speed boost for the buildworld and buildkernel process. That's not a typo. Compiling a complete kernel and world on my test machine used to take over 6 hours, even with /usr/obj/ having been left intact from prior builds. My most recent build finished in just 13 minutes, or around 1/30th of the old time!

WITH_META_MODE

The WITH_META_MODE compile option takes advantage of the filemon kernel module, creating and inspecting metadata about the objects compiled during a system build. New metadata files get stored in /usr/obj alongside the object files they refer to. On subsequent builds, when the compiler goes to build an object, if nothing about the build command or its related files (headers, libraries, etc.) has changed, the cached object can be used instead of compiling a new file.

On a FreeBSD 11.2 system, there are more than 52,000 .o files in /usr/obj after a system build, not to mention the final binaries. That's a lot of compiling that can be skipped, especially with patch releases where not much has changed.

To enable WITH_META_MODE, edit your /etc/src-env.conf (or create this file if it doesn't exist) to contain this line:

WITH_META_MODE=yes

Then, prior to performing your buildworld and buildkernel routine, you must run:

kldload filemon

Loading the filemon module becomes a crucial part of the build process. If you forget, the compiler won't be able to perform the necessary comparisons against existing metadata files. Having this module loaded all the time can slow things down, so you should only load it before building the system. Just add kldload filemon to the top of your build script if you use one, or make sure you remember to run it manually before kicking off a buildworld.

ccache

ccache is a compiler cache that stores copies of compiled object code (.o files) on disk. When make attempts to compile a file that's identical to one in the cache, the cached copy is used instead, bypassing the compile process. This benefit is system-wide and will apply to almost every make job you run, including ports and other software.

To install ccache, either cd /usr/ports/devel/ccache && make install or run pkg add ccache. Then add a couple of lines to your /etc/make.conf file:

WITH_CCACHE_BUILD=yes
CCACHE_DIR=/tmp/ccache

You can put the cache directory wherever you have room for it; after a kernel and world build, plus various ports builds over the past month, mine takes up 4.4GB. If you have a maintenance job that aggressively cleans out /tmp, you should choose another location.

Results

Here are my compile times for 11.2-RELEASE-p7, it took about 400 minutes. When this build was initiated, /usr/obj contained artifacts from building 11.2-RELEASE-p5. Prior to running the build, I configured WITH_META_MODE and ccache, but neither of them had generated their caches yet.

>>> World build started on Thu Jan 3 09:34:10 CST 2019 
>>> World build completed on Thu Jan 3 15:45:43 CST 2019 
>>> Kernel build for HOST started on Thu Jan 3 16:34:27 CST 2019 
>>> Kernel build for HOST completed on Thu Jan 3 17:04:49 CST 2019

And here are my compile times for 11.2-RELEASE-p9 on the same machine, the process only took about 13 minutes. When this build was initiated, /usr/obj contained artifacts from building 11.2-RELEASE-p7, and also contained all of the .meta files that WITH_META_MODE had generated last time.

>>> World build started on Tue Feb 5 16:44:24 CST 2019 
>>> World build completed on Tue Feb 5 16:55:51 CST 2019 
>>> Kernel build for HOST started on Tue Feb 5 17:04:11 CST 2019 
>>> Kernel build for HOST completed on Tue Feb 5 17:06:02 CST 2019

Because I enabled both WITH_META_MODE and ccache at the same time, I can't say for sure which one is most responsible for the performance improvement. Gauging by anecdotes from the FreeBSD forums, I suspect WITH_META_MODE makes the most difference.



Recent articles

📰 Using WITH_META_MODE and ccache for FreeBSD build boosts

📰 Resolving subversion error E000013: Unable to create pristine install stream

📰 Enhancements to SmokePing's AnotherDNS probe

📰 Generating vanity DNSSEC key tags

📰 DDoS involving forged packets from 23.225.141.70

📰 Website integrity monitoring through version control

📰 SpamAssassin 3.4.2 fixes security problems, adds HashBL and phishing plugins

📰 Bug or turf war? ICQ via Pidgin now fails with "startOSCARSession: Request Timeout"

📰 🎂

📰 SFSQuery, a PHP class to query the StopForumSpam API and DNSBL

📰 Resolving portmaster error "pkg-static: automake-1.16.1 conflicts with automake-wrapper-20131203"

📰 Resolving LibreNMS error "RuntimeException: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths"

📰 1.1.1.1: Fast, but not so accurate (yet)

📰 autodiscover.xml as an Indicator of Attack

📰 Blocking Facebook's Tracking and Surveillance: A Comprehensive Approach

▲ Back to top | Permalink to this page