#Include All .Cpp Files into a Single Compilation Unit

#include all .cpp files into a single compilation unit?

It's referred to by some (and google-able) as a "Unity Build". It links insanely fast and compiles reasonably quickly as well. It's great for builds you don't need to iterate on, like a release build from a central server, but it isn't necessarily for incremental building.

And it's a PITA to maintain.

EDIT: here's the first google link for more info: http://buffered.io/posts/the-magic-of-unity-builds/

The thing that makes it fast is that the compiler only needs to read in everything once, compile out, then link, rather than doing that for every .cpp file.

Bruce Dawson has a much better write up about this on his blog: http://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/

Separate compilation units vs Single Compilation unit for faster compilation, linking, and optimised code?

It is compiler-specific, and depends upon the optimizations you are asking from your compiler.

Most recent free software C++11 (or C++14) compilers are able to do link-time optimization : both recent GCC & Clang/LLVM are accepting the -flto flag (for link time optimiation...). To use it you should compile and link your code with it, and some additional (same) optimization flags. A typical use thru the make builder could be:

make 'CXX=g++ -flto -O2' 

or, in separate commands:

g++ -flto -O2 -Wall -I/usr/local/include -c src1.cc
g++ -flto -O2 -Wall -I/usr/local/include -c src2.cc
g++ -flto -O2 -Wall src1.o src2.o -L/usr/local/lib -lsome -o binprog

^{Don't forget -flto -O2 at link time !}

Then the code is compiled nearly the same as if you put all src1.cc & src2.cc in the same compilation unit. In particular, the compiler is able to (and sometimes will) inline a call from a function in src1.cc to a function in src2.cc

What happens under the hoods with -flto (with GCC, but in principle it is similar in Clang) is that the compiler is putting some intermediate representation (in some Gimple/SSA form) of your source code in each object file. At "link-time" (actually done also by the compiler, not only the linker)
this intermediate representation is reloaded and processed and recompiled for the entire program. So the compilation time nearly doubles.

So -flto is slowing the compilation (approximately by a factor of 2) and might sometimes give a few percents of performance improvement (execution time of the produced binary). Hence I almost never use it.

I'm trying to understand what makes a compiler compile faster.

This is compiler specific, and depends a lot with the optimizations you are asking from it. Using a recent GCC5 or GCC6, with g++ -O2 (and IIRC also with clang++ -O2) by practical and empirical measure the compilation time is proportional not only to the total size of the compilation unit (e.g. the number of tokens or size/volume of AST produced after preprocessing & include & macro expansions, and even template expansion) but also to the square of the size of the biggest function. A possible explanation is related to the time complexity of register allocation and instruction scheduling. Notice that the standard headers of the C++11 or C++14 containers are expanded to something quite big (e.g #include <vector> gives about ten thousand lines). BTW, compiling with g++ -O0 is faster than with g++ -O1 faster that g++ -O2. And asking for debug information (e.g. g++ -g2) slows down the compiler. So g++ -O1 -g2 is a slower compilation that g++ -O0 (which would produce a slower executable).

Precompiled headers might help reducing the compilation time (but not always!). You would have a single common header, and you'll better have not too small compilation units: total compilation time is slightly faster with 20 *.cc files of about two thousand lines each than with 200 *.cc files of two hundred lines each (notably because header files expand to many tokens). I generally recommend having at least a thousand lines per *.cc file if possible, so having just one small file of a hundred lines per class implementation is often a bad idea (in terms of overall compilation time). For a tiny project of e.g. 4KLOC having a single source file is quite sensible.

Notice also that C++ template expansion happens very "syntactically" (there are no modules yet in C++; Ocaml modules & functors are much better in that aspect). In other words vector<map<string,long>> is "expanded" (and is as compile-time consuming...) almost as if <vector> and <map & <string> standard headers have been inserted at the first occurrence of vector<map<string,long>> ... Template expansion is somehow an internal rewriting of ASTs. So a vector<map<string,set<long>>> requires -on its first occurrence- a lot of compiler work and nearly the same amount of work would have to be done for the "similar" vector<map<string,set<double>>>

Of course, several compilation units could be compiled in parallel, e.g. with make -j

To understand where a given GCC compilation is passing compiler time, pass -ftime-report to g++, see this. To be scared by the complexity of internal GCC representations, try -fdump-tree-all once.

To speedup your overall compilation time (with a focus on Linux systems with GCC; but you could adapt my answer to your system):

• have a parallel build (e.g. make -j would run several g++ processes in parallel, perhaps one per translation unit e.g. per *.cc file). Learn to write good enough Makefile-s.

• consider having one common header file and pre-compile your header (but that might slow down the compilation time, you need to benchmark); if you keep several header files (and there are many good reasons to do so), avoid having too many tiny header files and prefer having fewer, but bigger, ones. A single common precompiled header file of nearly ten thousand lines is not unusual (and you might #include several other files in it).

• consider having larger source files, e.g. having 20 source files of 2000 lines each might compile faster than 200 source files of 200 lines each (because with many small source files, preprocessing & template expansion is more repeated) and I do sometimes have source files of nearly ten thousand lines. however you'll often do an incremental build (and then that could be false, YMMV and you need to benchmark)

• disable optimizations, or lower the optimization level, so compile with g++ -O0 or g++ -O1 instead of g++ -O2 and avoid -flto. In many (but not all) cases, g++ -O3 -with or without -flto ...- is not worth the effort (it compiles slower, but the resulting machine code is not significantly faster than g++ -O2). But YMMV. Some numerical computations profit a lot from -O3. You could consider using function-specific pragmas or attributes to optimize some functions more than others in the same *.cc source file.

• disable debugging info, or lower it, so compile with g++ -O1 instead of g++ -01 -g2; but higher debugging info (e.g. g++ -g3) is very useful to the gdb debugger so YMMV.

• you could disable warnings, but that is not worth the trouble. On the contrary, always enable all of them, so at least -Wall to g++ and probably also -Wextra and be sure that your code compiles without warnings.

• avoid using too much nested templates, like e.g. std::set<std::vector<std::map<std::string,long>>> ; in some cases having opaque pointers and using the PIMPL idiom could help. You might then only include some extra headers (e.g. for containers) in some *.cc and not all of them (but this is incompatible with precompiled headers, so YMMV).

Some compilers or versions are slightly faster than others. So you could prefer clang++ to g++. I do recommend using several compilers (with warnings enabled). Be scared of undefined behavior in your code.

Notice that C++ is unlike Java: you can and often should have several classes or functions per file. Again YMMV.

PS. See (the slides and documentations and follow the many links on) starynkevitch.net/Basile/gcc-melt for more about GCC internals. I have abandoned GCC MELT in 2018 but the slides are still useful.

Why not concatenate C source files before compilation?

Some software are built that way.

A typical example is SQLite. It is sometimes compiled as an amalgamation (done at build time from many source files).

But that approach has pros and cons.

Obviously, the compile time will increase by quite a lot. So it is practical only if you compile that stuff rarely.

Perhaps, the compiler might optimize a bit more. But with link time optimizations (e.g. if using a recent GCC, compile and link with gcc -flto -O2) you can get the same effect (of course, at the expense of increased build time).

I don't have to write a header file for each function

That is a wrong approach (of having one header file per function). For a single-person project (of less than a hundred thousand lines of code, a.k.a. KLOC = kilo line of code), it is quite reasonable -at least for small projects- to have a single common header file (which you could pre-compile if using GCC), which will contain declarations of all public functions and types, and perhaps definitions of static inline functions (those small enough and called frequently enough to profit from inlining). For example, the sash shell is organized that way (and so is the lout formatter, with 52 KLOC).

You might also have a few header files, and perhaps have some single "grouping" header which #include-s all of them (and which you could pre-compile). See for example jansson (which actually has a single public header file) and GTK (which has lots of internal headers, but most applications using it have just one #include <gtk/gtk.h> which in turn include all the internal headers). On the opposite side, POSIX has a big lot of header files, and it documents which ones should be included and in which order.

Some people prefer to have a lot of header files (and some even favor putting a single function declaration in its own header). I don't (for personal projects, or small projects on which only two or three persons would commit code), but it is a matter of taste. BTW, when a project grows a lot, it happens quite often that the set of header files (and of translation units) changes significantly. Look also into REDIS (it has 139 .h header files and 214 .c files i.e. translation units totalizing 126 KLOC).

Having one or several translation units is also a matter of taste (and of convenience and habits and conventions). My preference is to have source files (that is translation units) which are not too small, typically several thousand lines each, and often have (for a small project of less than 60 KLOC) a common single header file. Don't forget to use some build automation tool like GNU make (often with a parallel build through make -j; then you'll have several compilation processes running concurrently). The advantage of having such a source file organization is that compilation is reasonably quick. BTW, in some cases a metaprogramming approach is worthwhile: some of your (internal header, or translation units) C "source" files could be generated by something else (e.g. some script in AWK, some specialized C program like bison or your own thing).

Remember that C was designed in the 1970s, for computers much smaller and slower than your favorite laptop today (typically, memory was at that time a megabyte at most, or even a few hundred kilobytes, and the computer was at least a thousand times slower than your mobile phone today).

I strongly suggest to study the source code and build some existing free software projects (e.g. those on GitHub or SourceForge or your favorite Linux distribution). You'll learn that they are different approaches. Remember that in C conventions and habits matter a lot in practice, so there are different ways to organize your project in .c and .h files. Read about the C preprocessor.

It also means I don't have to include the standard libraries in each file I create

You include header files, not libraries (but you should link libraries). But you could include them in each .c files (and many projects are doing that), or you could include them in one single header and pre-compile that header, or you could have a dozen of headers and include them after system headers in each compilation unit. YMMV. Notice that preprocessing time is quick on today's computers (at least, when you ask the compiler to optimize, since optimizations takes more time than parsing & preprocessing).

Notice that what goes into some #include-d file is conventional (and is not defined by the C specification). Some programs have some of their code in some such file (which should then not be called a "header", just some "included file"; and which then should not have a .h suffix, but something else like .inc). Look for example into XPM files. At the other extreme, you might in principle not have any of your own header files (you still need header files from the implementation, like <stdio.h> or <dlfcn.h> from your POSIX system) and copy and paste duplicated code in your .c files -e.g. have the line int foo(void); in every .c file, but that is very bad practice and is frowned upon. However, some programs are generating C files sharing some common content.

BTW, C or C++14 do not have modules (like OCaml has). In other words, in C a module is mostly a convention.

^{(notice that having many thousands of very small .h and .c files of only a few dozen lines each may slow down your build time dramatically; having hundreds of files of a few hundred lines each is more reasonable, in term of build time.)}

If you begin to work on a single-person project in C, I would suggest to first have one header file (and pre-compile it) and several .c translation units. In practice, you'll change .c files much more often than .h ones. Once you have more than 10 KLOC you might refactor that into several header files. Such a refactoring is tricky to design, but easy to do (just a lot of copy&pasting chunk of codes). Other people would have different suggestions and hints (and that is ok!). But don't forget to enable all warnings and debug information when compiling (so compile with gcc -Wall -g, perhaps setting CFLAGS= -Wall -g in your Makefile). Use the gdb debugger (and valgrind...). Ask for optimizations (-O2) when you benchmark an already-debugged program. Also use a version control system like Git.

On the contrary, if you are designing a larger project on which several persons would work, it could be better to have several files -even several header files- (intuitively, each file has a single person mainly responsible for it, with others making minor contributions to that file).

In a comment, you add:

I'm talking about writing my code in lots of different files but using a Makefile to concatenate them

I don't see why that would be useful (except in very weird cases). It is much better (and very usual and common practice) to compile each translation unit (e.g. each .c file) into its object file (a .o ELF file on Linux) and link them later. This is easy with make (in practice, when you'll change only one .c file e.g. to fix a bug, only that file gets compiled and the incremental build is really quick), and you can ask it to compile object files in parallel using make -j (and then your build goes really fast on your multi-core processor).

---

Related Topics