NOVA CTF {2024} — Mystical Cryptic State Reversal
REVERSE ENGINEERING CHALLENGE
Given Description:
The cipher used remains a mystery, requiring your keen reverse engineering skills. Only those who delve deep, find decryption cunningly, and think beyond the conventional can unravel the secrets within. Approach with curiosity, analyze meticulously, and piece together the puzzle.Are you up for the challenge? Dive in, crack the code, and claim your victory!
Analysing the File:
PE32+ is the Portable Executable 64-bit format. PE files are the standard file format for executables, object code, and Dynamic Link Libraries (DLLs) in Windows operating systems.
Strings:
It seems everything is fake flags.
To gather some context, run this in Windows. You’re prompted to enter
Enter Encrypted string Without Space and Key. By entering the random string and key we will get the Decrypted text.
As this is a reversing challenge static analysis would be helpful. By using the decryption process We need to find the cipher and look for the encrypted string and key to get the flag.
The decryption algorithm used in this challenge is a variant of the RC4 stream cipher. It takes the ciphertext and key as inputs and returns the decrypted plaintext. The algorithm initializes an array `S` with values from 0 to 255, shuffles the values based on the key, and then uses the shuffled array to decrypt the ciphertext.
After opening in IDA and Looking into Main Function
opening the decrypt function,
Analysing the code
Load i Index and Retrieve S[i] Value:
mov eax, [rbp+0A0h+i] ; Load the value of i into EAX
cdqe ; Sign-extend EAX to RAX
movzx eax, [rbp+rax+0A0h+S] ; Load the value of S[i] into EAX
movzx eax, al ; Zero-extend AL to AX
This sequence loads the current value of the i index, extends it to a full quad-word register, and retrieves the value of S[i] from memory.
Load j Index and Calculate Index for XOR Operation:
mov edx, [rbp+0A0h+j] ; Load the value of j into EDX
lea ecx, [rax+rdx] ; Calculate the index for XOR operation: S[i] + S[j]
These instructions load the value of the j index into EDX and calculate the effective index for the XOR operation by adding S[i] and S[j]
Calculate XOR Operation with Key:
mov eax, [rbp+0A0h+i] ; Load the value of i into EAX
cdq ; Sign-extend EAX to EDX:EAX
idiv [rbp+0A0h+key_length] ; Divide EDX:EAX by key_length
mov eax, edx ; Store the remainder (modulus) in EAX
cdqe ; Sign-extend EAX to RAX
mov rdx, [rbp+0A0h+key] ; Load the address of key into RDX
add rax, rdx ; Add the offset to the base address of key
movzx eax, byte ptr [rax] ; Load the byte value from the key into EAX
movzx eax, al ; Zero-extend AL to AX
These instructions calculate the key index based on the current i index, retrieve the corresponding byte from the key, and store it in EAX.
Update j Index:
lea edx, [rcx+rax] ; Calculate the sum of the previously calculated index and the key value
sar eax, 1Fh ; Arithmetic shift right to get the sign bit into all bits
shr eax, 18h ; Shift right by 24 bits to extract the lowest byte
add edx, eax ; Add the extracted byte to the sum
movzx edx, dl ; Zero-extend DL to EDX
sub edx, eax ; Subtract the sign-extended key byte from EDX
mov eax, edx ; Move the updated value of j into EAX
mov [rbp+0A0h+j], eax ; Store the updated value of j
These instructions update the value of the j index based on the calculated sum and the extracted byte from the key.
Update S[i] and S[j] :
mov eax, [rbp+0A0h+i] ; Load the value of `i` into EAX
cdqe ; Sign-extend EAX to RAX
movzx eax, [rbp+rax+0A0h+S] ; Load the value of `S[i]` into EAX
mov [rbp+0A0h+temp], al ; Store the value of `S[i]` in a temporary variable
mov eax, [rbp+0A0h+j] ; Load the value of `j` into EAX
cdqe ; Sign-extend EAX to RAX
movzx eax, [rbp+0A0h+temp] ; Load the temporary value into EAX
mov edx, [rbp+0A0h+i] ; Load the value of `i` into EDX
movsxd rdx, edx ; Sign-extend EDX to RDX
mov [rbp+rdx+0A0h+S], al ; Store the value of `S[i]` in memory at the correct index
These instructions swap the values of S[i] and S[j] by temporarily storing S[i] , retrieving S[j] , and then updating S[i] and S[j] in memory.
Increment i Index:
add [rbp+0A0h+i], 1 ; Increment the value of i by 1
This instruction increments the value of the i index, preparing for the next iteration of the decryption loop.
Perform XOR Operation with Ciphertext and Update Plaintext:
mov eax, [rbp+0A0h+k] ; Load the value of k into EAX
cdqe ; Sign-extend EAX to RAX
mov rdx, [rbp+0A0h+ciphertext] ; Load the address of ciphertext into RDX
add rax, rdx ; Add the offset to the base address of ciphertext
movzx r8d, byte ptr [rax] ; Load the byte value from the ciphertext into R8D
mov eax, [rbp+0A0h+t] ; Load the temporary value into EAX
cdqe ; Sign-extend EAX to RAX
movzx ecx, [rbp+rax+0A0h+S] ; Load the value of S[t] into ECX
mov eax, [rbp+0A0h+k] ; Load the value of k into EAX
cdqe ; Sign-extend EAX to RAX
mov rdx, [rbp+0A0h+plaintext] ; Load the address of plaintext into RDX
add rax, rdx ; Add the offset to the base address of plaintext
mov edx, r8d ; Move the ciphertext byte value into EDX
xor edx, ecx ; Perform the XOR
In the challenge, we encounter various functions, some of which contain fake flags. However, upon examining the code, we identify that the decryption algorithm being used is RC4. To progress and obtain the correct flag, we must locate the ciphertext and the corresponding key.
Also there was a hint “Search for decryption”. At function “8”. You can see the “Secret” which is the key for decryption as “HAPPYHACKING”
But still we need ciphertext to decrypt the plaintext and find the flag and lets look dig deeper in remaining functions.
By analysing the main function we can infer that it gets the encrypted string as hexadecimal value
By Extracting all the hex values we can get the encrypted string:
F5 DB D6 3E A2 AF B6 41 42 4A A3 D8 1B EE 30 9B 96 59 8A BF 1D 40 3D 51 1B 65 56 93 8E
It is mentioned as without space
F5DBD63EA2AFB641424AA3D81BEE309B96598ABF1D403D511B6556938E
We got the Encrypted string(hex) as
F5DBD63EA2AFB641424AA3D81BEE309B96598ABF1D403D511B6556938E
We already got key as
HAPPYHACKING
Also we can decrypt online:
Finally we found the flag as
NOVA{Revers3_H4ck3r_Unlocked}
Thanks For Reading :)
I hope you learnt something and enjoyed the challenge.Don’t miss out on my upcoming articles! Follow me on Medium for more insightful content. Clap and share this article.
Connect me via LinkedIn, https://www.linkedin.com/in/kishoreram-k/
Best of luck in capturing flags ahead!!!