Recommended Resources for Learning Cryptography: RE Edition

A common question when first reverse engineering ransomware is “what is a good resource for learning cryptography?”. Having an understanding of cryptography is essential when reversing ransomware. Most reverse engineers need to know how to identify the encryption algorithm, be able to follow the key generation, understand key storage and ensure the encryption implementation isn’t flawed. To accomplish these items it is essential to have a good foundational knowledge of cryptography. The following are some recommendations that I have found beneficial on my path to learning cryptography.  

One of the most important skills is having an understanding of how common encryption algorithms work. The best introductory book on cryptography is Understanding Cryptography: A Textbook for Students and Practitioners. It was written in a way that “teaches modern applied cryptography to readers with a technical background but without an education in pure mathematics” (source). The book also covers all modern crypto schemes commonly used. One of the best parts about the book is each chapter has a lecture on YouTube taught by the authors. This format is useful because it reinforces the concepts or adds more details to some of the more difficult topics.

After Understanding Cryptography I’d recommend a non-textbook approach using the cryptopals crypto challenges. It is basically a set of problems that progressively get harder.  You can solve the problems using a programming language of your choice. I have yet to complete the challenges but I’d recommend attempting and solving the first two sets of problems. They introduce you to a lot of foundational concepts that can actually be applied. From what I learned in the first set, I was able to easily crack XOR encrypted executable payloads. I love cryptopals so much that I created a mirror of the site and converted it to markdown so I can easily download everything via git. 

Once a foundational knowledge of cryptography has been established it is useful to see how the algorithms look when compiled. I came across this while I was reversing a family of ransomware and couldn’t correctly decrypt the data. I was able to recover the private RSA key, decrypt the AES key encrypted with the RSA private key and decrypt files using AES in CTR but the after a certain amount of decrypted bytes the data would be corrupted. In response to this I continuously reversed the code, studied AES and all it’s different modes, compiled multiple versions of AES, opened them up in a disassembler and diffed the results but the data was still corrupted. Everything pointed to AES in CTR, eventually I identified that the CTR loop had a off-by-one error and it didn’t matter because (as a colleague pointed out) they also stored the extra byte of the key. It was only when I accounted for the off-by-one error in my decryptor that I was able to successfully decrypt files.  

After this incident whenever I come across a new encryption algorithm that I don’t understand or want to learn more about; I search for references, search for source code, add them to, compile the executables and upload the .exes along with the PDB to a repository named asm-examples. I find the exploration of the disassembled code along with symbols and names from the PDB to be valuable. It aids in being able to quickly identify encryption algorithms and makes the disassembled or decompiled code less intimidating. 

To recap, my goto resources for learning encryption are Understanding Cryptography: A Textbook for Students and Practitioners, cryptopals and comparing compiled binaries to the source code. This isn’t the most in-depth approach to learning cryptography but for supporting malware analysis and reverse engineering ransomware it works well.  

Article Link: Hooked on Mnemonics Worked for Me: Recommended Resources for Learning Cryptography: RE Edition