NOVA CTF {2024} — Mystical Cryptic State Reversal

REVERSE ENGINEERING CHALLENGE

Given Description:
The cipher used remains a mystery, requiring your keen reverse engineering skills. Only those who delve deep, find decryption cunningly, and think beyond the conventional can unravel the secrets within. Approach with curiosity, analyze meticulously, and piece together the puzzle.Are you up for the challenge? Dive in, crack the code, and claim your victory!

Photo by Markus Spiske on Unsplash

Analysing the File:

PE32+ is the Portable Executable 64-bit format. PE files are the standard file format for executables, object code, and Dynamic Link Libraries (DLLs) in Windows operating systems.

Strings:

It seems everything is fake flags.
To gather some context, run this in Windows. You’re prompted to enter
Enter Encrypted string Without Space and Key. By entering the random string and key we will get the Decrypted text.

As this is a reversing challenge static analysis would be helpful. By using the decryption process We need to find the cipher and look for the encrypted string and key to get the flag.
The decryption algorithm used in this challenge is a variant of the RC4 stream cipher. It takes the ciphertext and key as inputs and returns the decrypted plaintext. The algorithm initializes an array `S` with values from 0 to 255, shuffles the values based on the key, and then uses the shuffled array to decrypt the ciphertext.
After opening in IDA and Looking into Main Function

opening the decrypt function,

Analysing the code
Load i Index and Retrieve S[i] Value:

mov     eax, [rbp+0A0h+i]       ; Load the value of i into EAX
cdqe                             ; Sign-extend EAX to RAX
movzx   eax, [rbp+rax+0A0h+S]   ; Load the value of S[i] into EAX
movzx   eax, al                  ; Zero-extend AL to AX

This sequence loads the current value of the i index, extends it to a full quad-word register, and retrieves the value of S[i] from memory.
Load j Index and Calculate Index for XOR Operation:

mov     edx, [rbp+0A0h+j]       ; Load the value of j into EDX
lea     ecx, [rax+rdx]          ; Calculate the index for XOR operation: S[i] +                                    S[j]

These instructions load the value of the j index into EDX and calculate the effective index for the XOR operation by adding S[i] and S[j]
Calculate XOR Operation with Key:

mov     eax, [rbp+0A0h+i]       ; Load the value of i into EAX
cdq                              ; Sign-extend EAX to EDX:EAX
idiv    [rbp+0A0h+key_length]   ; Divide EDX:EAX by key_length
mov     eax, edx                 ; Store the remainder (modulus) in EAX
cdqe                             ; Sign-extend EAX to RAX
mov     rdx, [rbp+0A0h+key]      ; Load the address of key into RDX
add     rax, rdx                 ; Add the offset to the base address of key
movzx   eax, byte ptr [rax]      ; Load the byte value from the key into EAX
movzx   eax, al                  ; Zero-extend AL to AX

These instructions calculate the key index based on the current i index, retrieve the corresponding byte from the key, and store it in EAX.
Update j Index:

lea     edx, [rcx+rax]          ; Calculate the sum of the previously calculated index and the key value
sar     eax, 1Fh                 ; Arithmetic shift right to get the sign bit into all bits
shr     eax, 18h                 ; Shift right by 24 bits to extract the lowest byte
add     edx, eax                 ; Add the extracted byte to the sum
movzx   edx, dl                  ; Zero-extend DL to EDX
sub     edx, eax                 ; Subtract the sign-extended key byte from EDX
mov     eax, edx                 ; Move the updated value of j into EAX
mov     [rbp+0A0h+j], eax        ; Store the updated value of j

These instructions update the value of the j index based on the calculated sum and the extracted byte from the key.
Update S[i] and S[j] :

mov     eax, [rbp+0A0h+i]       ; Load the value of `i` into EAX
cdqe                             ; Sign-extend EAX to RAX
movzx   eax, [rbp+rax+0A0h+S]   ; Load the value of `S[i]` into EAX
mov     [rbp+0A0h+temp], al      ; Store the value of `S[i]` in a temporary variable
mov     eax, [rbp+0A0h+j]       ; Load the value of `j` into EAX
cdqe                             ; Sign-extend EAX to RAX
movzx   eax, [rbp+0A0h+temp]    ; Load the temporary value into EAX
mov     edx, [rbp+0A0h+i]       ; Load the value of `i` into EDX
movsxd  rdx, edx                 ; Sign-extend EDX to RDX
mov     [rbp+rdx+0A0h+S], al     ; Store the value of `S[i]` in memory at the correct index                                     

These instructions swap the values of S[i] and S[j] by temporarily storing S[i] , retrieving S[j] , and then updating S[i] and S[j] in memory.

Increment i Index:

add     [rbp+0A0h+i], 1          ; Increment the value of i by 1

This instruction increments the value of the i index, preparing for the next iteration of the decryption loop.

Perform XOR Operation with Ciphertext and Update Plaintext:

mov     eax, [rbp+0A0h+k]       ; Load the value of k into EAX
cdqe                             ; Sign-extend EAX to RAX
mov     rdx, [rbp+0A0h+ciphertext] ; Load the address of ciphertext into RDX
add     rax, rdx                 ; Add the offset to the base address of                                            ciphertext
movzx   r8d, byte ptr [rax]      ; Load the byte value from the ciphertext into R8D
mov     eax, [rbp+0A0h+t]        ; Load the temporary value into EAX
cdqe                             ; Sign-extend EAX to RAX
movzx   ecx, [rbp+rax+0A0h+S]   ; Load the value of S[t] into ECX
mov     eax, [rbp+0A0h+k]       ; Load the value of k into EAX
cdqe                             ; Sign-extend EAX to RAX
mov     rdx, [rbp+0A0h+plaintext] ; Load the address of plaintext into RDX
add     rax, rdx                 ; Add the offset to the base address of                                            plaintext
mov     edx, r8d                 ; Move the ciphertext byte value into EDX
xor     edx, ecx                 ; Perform the XOR

In the challenge, we encounter various functions, some of which contain fake flags. However, upon examining the code, we identify that the decryption algorithm being used is RC4. To progress and obtain the correct flag, we must locate the ciphertext and the corresponding key.

Also there was a hint “Search for decryption”. At function “8”. You can see the “Secret” which is the key for decryption as “HAPPYHACKING”

But still we need ciphertext to decrypt the plaintext and find the flag and lets look dig deeper in remaining functions.

By analysing the main function we can infer that it gets the encrypted string as hexadecimal value

By Extracting all the hex values we can get the encrypted string:

F5 DB D6 3E A2 AF B6 41 42 4A A3 D8 1B EE 30 9B 96 59 8A BF 1D 40 3D 51 1B 65 56 93 8E

It is mentioned as without space

F5DBD63EA2AFB641424AA3D81BEE309B96598ABF1D403D511B6556938E

We got the Encrypted string(hex) as

F5DBD63EA2AFB641424AA3D81BEE309B96598ABF1D403D511B6556938E

We already got key as

HAPPYHACKING

Also we can decrypt online:

Finally we found the flag as
NOVA{Revers3_H4ck3r_Unlocked}

Thanks For Reading :)

I hope you learnt something and enjoyed the challenge.Don’t miss out on my upcoming articles! Follow me on Medium for more insightful content. Clap and share this article.

Connect me via LinkedIn, https://www.linkedin.com/in/kishoreram-k/

Best of luck in capturing flags ahead!!!