# NOVA CTF {2024} — Mystical Cryptic State Reversal

*REVERSE ENGINEERING CHALLENGE*

**Given Description:**The cipher used remains a mystery, requiring your keen reverse engineering skills. Only those who delve deep, find decryption cunningly, and think beyond the conventional can unravel the secrets within. Approach with curiosity, analyze meticulously, and piece together the puzzle.Are you up for the challenge? Dive in, crack the code, and claim your victory!

**Analysing the File:**

PE32+ is the Portable Executable 64-bit format. PE files are the standard file format for executables, object code, and Dynamic Link Libraries (DLLs) in Windows operating systems.

**Strings:**

It seems everything is fake flags.

To gather some context, run this in Windows. You’re prompted to enter

Enter Encrypted string Without Space and Key. By entering the random string and key we will get the Decrypted text.

As this is a reversing challenge static analysis would be helpful. By using the decryption process We need to find the cipher and look for the encrypted string and key to get the flag.

The decryption algorithm used in this challenge is a variant of the RC4 stream cipher. It takes the ciphertext and key as inputs and returns the decrypted plaintext. The algorithm initializes an array `S` with values from 0 to 255, shuffles the values based on the key, and then uses the shuffled array to decrypt the ciphertext.**After opening in IDA and Looking into Main Function**

opening the decrypt function,

Analysing the code

Load i Index and Retrieve S[i] Value:

`mov eax, [rbp+0A0h+i] ; Load the value of i into EAX`

cdqe ; Sign-extend EAX to RAX

movzx eax, [rbp+rax+0A0h+S] ; Load the value of S[i] into EAX

movzx eax, al ; Zero-extend AL to AX

This sequence loads the current value of the i index, extends it to a full quad-word register, and retrieves the value of S[i] from memory.

Load j Index and Calculate Index for XOR Operation:

`mov edx, [rbp+0A0h+j] ; Load the value of j into EDX`

lea ecx, [rax+rdx] ; Calculate the index for XOR operation: S[i] + S[j]

These instructions load the value of the j index into EDX and calculate the effective index for the XOR operation by adding S[i] and S[j]

Calculate XOR Operation with Key:

`mov eax, [rbp+0A0h+i] ; Load the value of i into EAX`

cdq ; Sign-extend EAX to EDX:EAX

idiv [rbp+0A0h+key_length] ; Divide EDX:EAX by key_length

mov eax, edx ; Store the remainder (modulus) in EAX

cdqe ; Sign-extend EAX to RAX

mov rdx, [rbp+0A0h+key] ; Load the address of key into RDX

add rax, rdx ; Add the offset to the base address of key

movzx eax, byte ptr [rax] ; Load the byte value from the key into EAX

movzx eax, al ; Zero-extend AL to AX

These instructions calculate the key index based on the current i index, retrieve the corresponding byte from the key, and store it in EAX.

Update j Index:

`lea edx, [rcx+rax] ; Calculate the sum of the previously calculated index and the key value`

sar eax, 1Fh ; Arithmetic shift right to get the sign bit into all bits

shr eax, 18h ; Shift right by 24 bits to extract the lowest byte

add edx, eax ; Add the extracted byte to the sum

movzx edx, dl ; Zero-extend DL to EDX

sub edx, eax ; Subtract the sign-extended key byte from EDX

mov eax, edx ; Move the updated value of j into EAX

mov [rbp+0A0h+j], eax ; Store the updated value of j

These instructions update the value of the j index based on the calculated sum and the extracted byte from the key.

Update S[i] and S[j] :

`mov eax, [rbp+0A0h+i] ; Load the value of `i` into EAX`

cdqe ; Sign-extend EAX to RAX

movzx eax, [rbp+rax+0A0h+S] ; Load the value of `S[i]` into EAX

mov [rbp+0A0h+temp], al ; Store the value of `S[i]` in a temporary variable

mov eax, [rbp+0A0h+j] ; Load the value of `j` into EAX

cdqe ; Sign-extend EAX to RAX

movzx eax, [rbp+0A0h+temp] ; Load the temporary value into EAX

mov edx, [rbp+0A0h+i] ; Load the value of `i` into EDX

movsxd rdx, edx ; Sign-extend EDX to RDX

mov [rbp+rdx+0A0h+S], al ; Store the value of `S[i]` in memory at the correct index

These instructions swap the values of S[i] and S[j] by temporarily storing S[i] , retrieving S[j] , and then updating S[i] and S[j] in memory.

Increment i Index:

`add [rbp+0A0h+i], 1 ; Increment the value of i by 1`

This instruction increments the value of the i index, preparing for the next iteration of the decryption loop.

Perform XOR Operation with Ciphertext and Update Plaintext:

`mov eax, [rbp+0A0h+k] ; Load the value of k into EAX`

cdqe ; Sign-extend EAX to RAX

mov rdx, [rbp+0A0h+ciphertext] ; Load the address of ciphertext into RDX

add rax, rdx ; Add the offset to the base address of ciphertext

movzx r8d, byte ptr [rax] ; Load the byte value from the ciphertext into R8D

mov eax, [rbp+0A0h+t] ; Load the temporary value into EAX

cdqe ; Sign-extend EAX to RAX

movzx ecx, [rbp+rax+0A0h+S] ; Load the value of S[t] into ECX

mov eax, [rbp+0A0h+k] ; Load the value of k into EAX

cdqe ; Sign-extend EAX to RAX

mov rdx, [rbp+0A0h+plaintext] ; Load the address of plaintext into RDX

add rax, rdx ; Add the offset to the base address of plaintext

mov edx, r8d ; Move the ciphertext byte value into EDX

xor edx, ecx ; Perform the XOR

In the challenge, we encounter various functions, some of which contain fake flags. However, upon examining the code, we identify that the decryption algorithm being used is RC4. To progress and obtain the correct flag, we must locate the ciphertext and the corresponding key.

Also there was a hint “Search for decryption”. At function “8”. You can see the “Secret” which is the key for decryption as “HAPPYHACKING”

But still we need ciphertext to decrypt the plaintext and find the flag and lets look dig deeper in remaining functions.

By analysing the main function we can infer that it gets the encrypted string as hexadecimal value

**By Extracting all the hex values we can get the encrypted string:**

`F5 DB D6 3E A2 AF B6 41 42 4A A3 D8 1B EE 30 9B 96 59 8A BF 1D 40 3D 51 1B 65 56 93 8E`

It is mentioned as without space

`F5DBD63EA2AFB641424AA3D81BEE309B96598ABF1D403D511B6556938E`

We got the Encrypted string(hex) as

`F5DBD63EA2AFB641424AA3D81BEE309B96598ABF1D403D511B6556938E`

We already got key as

`HAPPYHACKING`

Also we can decrypt online:

Finally we found the flag as

NOVA{Revers3_H4ck3r_Unlocked}

# Thanks For Reading :)

I hope you learnt something and enjoyed the challenge.Don’t miss out on my upcoming articles! *Follow me*** **on Medium for more insightful content. Clap and share this article.

Connect me via LinkedIn, https://www.linkedin.com/in/kishoreram-k/

Best of luck in capturing flags ahead!!!