(new Soapbox())->shout(array_map('strtoupper', $opinions)); //Shaun's blog

Me, elsewhere

Miscellaneous public code

A PHP API client for Reddit

I don't tweet much

XMPP chat
(Pidgin, Miranda, Swift, etc.)

Perfect is the enemy of good enough.

Using WITH_META_MODE and ccache for FreeBSD build boosts

Posted February 06, 2019 by shaun

If you're looking to speed up your FreeBSD kernel and world builds, try using the WITH_META_MODE compile option and/or installing and enabling ccache. Turning these on gave me a 30x speed boost for the buildworld and buildkernel process. That's not a typo. Compiling a complete kernel and world on my test machine used to take over 6 hours, even with /usr/obj/ having been left intact from prior builds. My most recent build finished in just 13 minutes, or around 1/30th of the old time!


The WITH_META_MODE compile option takes advantage of the filemon kernel module, creating and inspecting metadata about the objects compiled during a system build. New metadata files get stored in /usr/obj alongside the object files they refer to. On subsequent builds, when the compiler goes to build an object, if nothing about the build command or its related files (headers, libraries, etc.) has changed, the cached object can be used instead of compiling a new file.

On a FreeBSD 11.2 system, there are more than 52,000 .o files in /usr/obj after a system build, not to mention the final binaries. That's a lot of compiling that can be skipped, especially with patch releases where not much has changed.

To enable WITH_META_MODE, edit your /etc/src-env.conf (or create this file if it doesn't exist) to contain this line:


Then, prior to performing your buildworld and buildkernel routine, you must run:

kldload filemon

Loading the filemon module becomes a crucial part of the build process. If you forget, the compiler won't be able to perform the necessary comparisons against existing metadata files. Having this module loaded all the time can slow things down, so you should only load it before building the system. Just add kldload filemon to the top of your build script if you use one, or make sure you remember to run it manually before kicking off a buildworld.


ccache is a compiler cache that stores copies of compiled object code (.o files) on disk. When make attempts to compile a file that's identical to one in the cache, the cached copy is used instead, bypassing the compile process. This benefit is system-wide and will apply to almost every make job you run, including ports and other software.

To install ccache, either cd /usr/ports/devel/ccache && make install or run pkg add ccache. Then add a couple of lines to your /etc/make.conf file:


You can put the cache directory wherever you have room for it; after a kernel and world build, plus various ports builds over the past month, mine takes up 4.4GB. If you have a maintenance job that aggressively cleans out /tmp, you should choose another location.


Here are my compile times for 11.2-RELEASE-p7, it took about 400 minutes. When this build was initiated, /usr/obj contained artifacts from building 11.2-RELEASE-p5. Prior to running the build, I configured WITH_META_MODE and ccache, but neither of them had generated their caches yet.

>>> World build started on Thu Jan 3 09:34:10 CST 2019 
>>> World build completed on Thu Jan 3 15:45:43 CST 2019 
>>> Kernel build for HOST started on Thu Jan 3 16:34:27 CST 2019 
>>> Kernel build for HOST completed on Thu Jan 3 17:04:49 CST 2019

And here are my compile times for 11.2-RELEASE-p9 on the same machine, the process only took about 13 minutes. When this build was initiated, /usr/obj contained artifacts from building 11.2-RELEASE-p7, and also contained all of the .meta files that WITH_META_MODE had generated last time.

>>> World build started on Tue Feb 5 16:44:24 CST 2019 
>>> World build completed on Tue Feb 5 16:55:51 CST 2019 
>>> Kernel build for HOST started on Tue Feb 5 17:04:11 CST 2019 
>>> Kernel build for HOST completed on Tue Feb 5 17:06:02 CST 2019

Because I enabled both WITH_META_MODE and ccache at the same time, I can't say for sure which one is most responsible for the performance improvement. Gauging by anecdotes from the FreeBSD forums, I suspect WITH_META_MODE makes the most difference.

Recent articles

📰 Caveat with Vantec SATA/IDE to USB 2.0 Adapter and Macrium software

📰 Jay Niffley, Man of Mystery

📰 Compiling Doxygen on FreeBSD without LaTeX and Ghostscript

📰 Introducing Snuze, a PHP client for the Reddit API

📰 jisusaiche: Java's installer telemetry

📰 BIND client log error "query_find: query_getdb failed"

📰 Resolving "The lang/perl5.24 port has been deleted: Has expired" portmaster error

📰 Armagaddon2 interim fix for Firefox 56 and other old versions

📰 Strange DNS queries: qname "miep", qtype ANY

📰 Undeliverable as addressed: A massive broken spam campaign?

📰 Using WITH_META_MODE and ccache for FreeBSD build boosts

📰 Resolving subversion error E000013: Unable to create pristine install stream

📰 Enhancements to SmokePing's AnotherDNS probe

📰 Generating vanity DNSSEC key tags

📰 DDoS involving forged packets from

▲ Back to top | Permalink to this page