HackTheBox: SimpleEncryptor challenge Writeup

Challenge Overview

• Category: Reverse Engineering
• Difficulty: Easy
• Target Binary: encrypt (ELF 64-bit LSB pie executable, x86-64, not stripped)
• Output File: flag.enc (Data file)

1. Initial Analysis & Triage

We began our analysis by checking the file properties and basic metadata of the provided files using the standard Linux file utility.

file encrypt
# Output: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, ..., not stripped

file flag.enc
# Output: data

Because the binary was not stripped, running strings encrypt exposed crucial system library calls that pointed directly to a custom time-seeded PRNG (Pseudo-Random Number Generator) encryption routine:

• srand / rand: Indicates pseudo-random sequence generation.
• time: Indicates the generator is likely seeded using the current system Unix timestamp (time(NULL)).
• fopen / fread / fwrite: Standard file I/O operations used to ingest the flag and spit out the payload.

2. Static Analysis (Ghidra Decompilation)

Opening the encrypt binary inside Ghidra and navigating to the decompiled main function revealed the explicit core cryptographic logic:

undefined8 main(void) {
  int iVar1;
  time_t tVar2;
  long in_FS_OFFSET;
  uint local_40;
  uint local_3c;
  long local_38;
  FILE *local_30;
  size_t local_28;
  void *local_20;
  FILE *local_18;
  long local_10;
  
  local_10 = *(long *)(in_FS_OFFSET + 0x28);
  local_30 = fopen("flag","rb");
  fseek(local_30,0,2);
  local_28 = ftell(local_30);
  fseek(local_30,0,0);
  local_20 = malloc(local_28);
  fread(local_20,local_28,1,local_30);
  fclose(local_30);
  
  tVar2 = time((time_t *)0x0);
  local_40 = (uint)tVar2;
  srand(local_40);
  
  for (local_38 = 0; local_38 < (long)local_28; local_38 = local_38 + 1) {
    iVar1 = rand();
    *(byte *)((long)local_20 + local_38) = *(byte *)((long)local_20 + local_38) ^ (byte)iVar1;
    
    local_3c = rand();
    local_3c = local_3c & 7;
    *(byte *)((long)local_20 + local_38) =
         *(byte *)((long)local_20 + local_38) << (sbyte)local_3c |
         *(byte *)((long)local_20 + local_38) >> 8 - (sbyte)local_3c;
  }
  
  local_18 = fopen("flag.enc","wb");
  fwrite(&local_40,1,4,local_18);
  fwrite(local_20,1,local_28,local_18);
  fclose(local_18);
  
  if (local_10 != *(long *)(in_FS_OFFSET + 0x28)) {
                    /* WARNING: Subroutine does not return */
    __stack_chk_fail();
  }
  return 0;
}

Key Takeaways from the Full Code:

1. The Seed is Exposed: The binary calls fwrite(&local_40,1,4,local_18); to save the 4-byte time_t timestamp seed directly into the first 4 bytes of flag.enc.
2. Deterministic Encryption: Because srand() is initialized with a known seed extracted directly from the payload header, we can fully replicate the exact stream of pseudo-random numbers produced by rand().
3. Two-Stage Mutation Loop: For every byte in the flag buffer:

• Stage 1: The byte is XORed (^) with the lower 8 bits of the first rand() call.
• Stage 2: The byte is circularly shifted left << local_3c and ORed with a right shift >> (8 - local_3c). This layout (x << n) | (x >> (8 - n)) is the classic C implementation for an 8-bit bitwise circular Left Rotation (ROL).

3. Vulnerability & Exploitation Strategy

To reverse the encryption loop, we process the encrypted bytes strictly in reverse order:

1. Extract the first 4 bytes of flag.enc to recover the srand() seed.
2. Initialize srand(seed) utilizing glibc's native random engine via Python's ctypes bindings.
3. For each encrypted payload byte:

• Re-generate the two corresponding rand() tokens used during that specific loop iteration.
• Undo Stage 2 (ROL): Apply a Rotate Right (ROR) bitwise operation to invert the original left bitwise rotation.
• Undo Stage 1 (XOR): Apply an XOR operation with the identical token (since \(A \oplus B \oplus B = A\)).

4. Automation Solver Script

The following Python script reads the compiled flag.enc payload file, extracts the seed, mirrors the native glibc random state constraints, and decodes the plaintext flag.

#!/usr/bin/python3
import ctypes
import os

def solve():
    if not os.path.exists("flag.enc"):
        print("[-] Error: flag.enc missing from current working directory.")
        return

    # Load standard Linux C library to match exact glibc rand() behavior
    libc = ctypes.CDLL("libc.so.6")

    with open("flag.enc", "rb") as f:
        data = f.read()

    # Parse 4-byte seed and payload
    seed = int.from_bytes(data[:4], byteorder='little')
    encrypted_payload = data[4:]
    
    print(f"[+] Recovered Seed: {seed} (Hex: {hex(seed)})")
    libc.srand(seed)

    def ror8(val, shift):
        """Perform bitwise circular right-rotation on an 8-bit byte"""
        return ((val & 0xFF) >> shift) | ((val << (8 - shift)) & 0xFF)

    flag = []
    
    for enc_byte in encrypted_payload:
        # Replicate internal loop PRNG taps
        r1 = libc.rand()
        r2 = libc.rand()
        
        xor_key = r1 & 0xFF
        shift_amount = r2 & 7
        
        # Invert transformations in reverse execution sequence
        unrotated = ror8(enc_byte, shift_amount)
        decrypted_char = unrotated ^ xor_key
        
        flag.append(chr(decrypted_char))

    print(f"[+] Decrypted Flag: {''.join(flag)}")

if __name__ == "__main__":
    solve()

5. Flag Recovery Execution

Run the script directly inside your challenge workspace:

python3 solve.py

Output:

[+] Recovered Seed: 1655780698 (Hex: 0x62b1355a)
[+] Decrypted Flag: HTB{REDACTED}