Compile regular llvm+clang

Following the instruction in LLVM official website[1], we can compile regular llvm+clang. Steps 4, 5, 6 can be skipped.

Note that llvm making process really eats tons of memory, so make -j with a small number. Also RELEASE build is fast and requires less memory. One example cmake configuration can be

cmake -G "Unix Makefiles" ../llvm/ \
 -DLLVM_TARGETS_TO_BUILD="ARM;X86;AArch64" \
 -DCMAKE_BUILD_TYPE=RELEASE \
 -DCMAKE_INSTALL_PREFIX=../release

Generate LLVMgold.so

Clang CFI requires LLVM Link Time Optimization (LTO), which requires the gold ld and LLVMgold.so. You can use ld –version to check if it is gold linker or not, also use ld -plugin to check the plugin support.

1.ld.gold is usually install in /usr/bin/, we only need to make a symbolic link to let ld point to ld.gold Or as we also need to give binutils/include to llvm, so build ld.gold locally is a better choice, can follow instruction here[2]. This step will generate gold linker ld-new under build/gold/.

git clone --depth 1 git://sourceware.org/git/binutils-gdb.git binutils
mkdir binutils/build
cd binutils/build
../configure --enable-gold --enable-plugins --disable-werror
make all-gold -j4

2.Make a symbolic link to let ld point to binutils/build/gold/ld-new

sudo ln -sf binutils/build/gold/ld-new /usr/bin/ld

3.Generate LLVMgold.so, come back to llvm build fold, run

cmake -G "Unix Makefiles" -DLLVM_BINUTILS_INCDIR=PATH_TO/binutils/include ../llvm
make -j4 ENABLE_OPTIMIZED=1

Then LLVMgold.so is generated, will be loaded automatically by gold loader.

How is the CFI enforced

Clang provides CFI checks on certain levels, such as virtual table and indirect call checks. We can follow the instruction in Let’s talk about CFI: clang edition[3]. Use the indirect call source[4] as the example, we can compile it use

clang -g -fvisibility=hidden -flto -fsanitize=cfi-icall cfi_icall.c -o cfp.out -v

Clang CFI checks the dynamic type of a function pointer matching the callee function type. Function type is defined by the arguments and argument type. In other words, Clang CFI will classify all functions (address taken functions) into different sets according to their arguments and allow a function pointer to jump to any function in the correct set. Using the above example, three functions have the same arguments number and type, therefore in the same set.

static int int_arg(int arg) { ... }
static int bad_int_arg(int arg) { ... }
static int not_entry_point(int arg) { ... }

Clang will generate

00000000004008a0 int_arg:
  4008a0: e9 eb fe ff ff        jmpq   400790 int_arg.cfi
  4008a5: cc                    int3
  4008a6: cc                    int3
  4008a7: cc                    int3
 
00000000004008a8 bad_int_arg:
  4008a8: e9 03 ff ff ff        jmpq   4007b0 bad_int_arg.cfi
  4008ad: cc                    int3
  4008ae: cc                    int3
  4008af: cc                    int3
 
00000000004008b0 not_entry_point:
  4008b0: e9 6b ff ff ff        jmpq   400820 not_entry_point.cfi
  4008b5: cc                    int3
  4008b6: cc                    int3
  4008b7: cc                    int3 
  
Func Addr  Bit Index       98 7654 3210
4008a0 -> 0100 0000 0000 1000 1010 0000  -> 00 for int_arg
4008a8 -> 0100 0000 0000 1000 1010 1000  -> 01 for bad_int_arg
4008b0 -> 0100 0000 0000 1000 1011 0000  -> 10 for not_entry_point
  

Clang CFI allows int_arg_fn to jump to any functions in the set. From the bit index, it is easy to see that the 3rd and 4th bits combine will give functions an index. and the inserted instructions between —— in main will check if the jump-to address is in the set or not. If not, just check to 0x400785 to crash current process. This is how -fsanitize=cfi-icall is enforced.

0000000000400680 main:
  400680: 55                    push   %rbp
  400681: 48 89 e5              mov    %rsp,%rbp
  400684: 53                    push   %rbx
  400685: 50                    push   %rax
  400686: 48 89 f3              mov    %rsi,%rbx
  400689: 83 ff 02              cmp    $0x2,%edi
  40068c: 75 3e                 jne    4006cc main x4c=""
  40068e: bf 00 0a 40 00        mov    $0x400a00,%edi
  400693: e8 b8 fe ff ff        callq  400550 puts plt=""
  400698: 48 8b 43 08           mov    0x8(%rbx),%rax
  40069c: 48 0f be 38           movsbq (%rax),%rdi
  4006a0: 48 8b 0c fd c0 2e 40  mov    0x402ec0(,%rdi,8),%rcx
  4006a7: 00 
  ----------------------------------------------------
  4006a8: b8 a0 08 40 00        mov    $0x4008a0,%eax
  4006ad: 48 89 ca              mov    %rcx,%rdx
  4006b0: 48 29 c2              sub    %rax,%rdx  //callee addr - base 0x4008a0
  4006b3: 48 c1 c2 3d           rol    $0x3d,%rdx //left shift 61 bit, 
  						  //int_arg addr becomes 00
						  //bad_int_arg addr becomes 01
						  //not_entry_point addr becomes 10
  4006b7: 48 83 fa 03           cmp    $0x3,%rdx  //compare with 3
  4006bb: 0f 83 c4 00 00 00     jae    400785     //crash if >= 3, out of set
  ----------------------------------------------------
  4006c1: 48 83 c7 d0           add    $0xffffffffffffffd0,%rdi
  4006c5: ff d1                 callq  *%rcx
  .
  400785: 0f 0b                 ud2

References

  1. Getting Started: Building and Running Clang
  2. The LLVM gold plugin
  3. Let’s talk about CFI: clang edition
  4. trailofbits/clang-cfi-showcase
  5. Clang 5 documentation - Control Flow Integrity