Overlapping Instructions in x86

Instruction overlapping is when you have an executing instruction that starts in the middle of another, thus overlapping it.

Let's say you have assembly code that looks like the following:

section .text
global _start

_start:
call overlapping_function
call a
mov ebx, 123
; mov eax, 1 ; this should already be 1 from overlapping_function
int 0x80

overlapping_function:
mov eax, -1
inc eax
cmp eax, 0
je 0x3
a:
rol bl, 0
ret

All this program is designed to do is exit with 123. Since x86 assembly code can be infamously hard to read, so let's break down what exactly is going down here.

We start from the _start function and immediately call overlapping_function. Then we have some junk instructions that are meaningless until we get to the je 0x3 instruction. We take this branch to land us 3 bytes into the instruction rol bl, 0, turning into the instruction inc eax. Then the instructions realign to ret before a is called to do a meaningless rol bl, 0.

0x0000000000000000:  0F 84 FF FF FF FF    je  5
0x0000000000000006:  C0 C3 00             rol bl, 0
0x0000000000000009:  C3                   ret 

Here, we can see that what was previously je 0x3 got assembled to je 5. This is because the CPU calculates the jump from the instruction pointer (EIP/RIP). Let's pretend that it is at 0x00. When we jump by five, then we are left with the last byte of the je 5 instruction: ff.

Then you can imagine what happens next.

0x0000000000000005:  FF C0    inc eax
0x0000000000000007:  C3       ret

The remaining 00 C3 never gets executed because the ret instruction sets the instruction pointer back to the _start function.

The rest of the program execution is simple. Function a is called to do essentially nothing (rol bl, 0), and then because eax is already 1, the program exits when we call int 0x80 with exit code 123 stored in ebx.

Real World Application

A fair question to ask is when does this happen in real code. Overlapping instruction is uncommon because there is no guarantee that the control flow will converge in a sensible way unless carefully constructed. In our case, we artificially introduced the overlapping instruction by creating junk instructions that don't affect the functioning of the program. In a real program, however, overlapping would occur in the form of prefixes.

WIP because I know of its existence but haven't observed it myself.