The Coding Challenge

I finally got what I’ve been asking for, an opportunity to bypass all the crap I’ve complained about, an opportunity to show a potential employer what I can actually do, an opportunity to show what I’m capable of, an opportunity that may not present itself again …. and I fucked it up.  Yeah, you know it’s serious when I use that kind of language.

First let’s go back to my last post, one I’m not exactly proud of, but this blog is about my journey, the good and the bad.  In that mess of negativity and whining, I mentioned one company I applied for whose CEO was quoted as saying that he only cares about what a person is capable of and not they’re background.

In that application I was given an assessment.  One of those aptitude tests that have absolutely nothing to do with programming followed by a personality test.  You know the one, where you’re asked things like “have you ever stolen something” and then asked that same question 5 more times but reworded to try and trip you up.  Well, I never made it past that. I know I KILLED the aptitude portion, which means there was something in my personality assessment they didn’t like.  Man, talk about feeling rejected.  It’s different when it’s personal like that.

So about a month ago I was at a local React meetup and met a person who was working with someone who filled in most of their senior and mid level developers and were about to start on filling in with juniors.  Fast forward almost 2 weeks and I got a call from this person.  He started to talk about the position and I recognized it right away.  He told me the company name and I smiled.  It was the same one I had already been rejected from because of the assessment tests.

I go on to tell him about it and he agreed that those assessments are a ‘crock of shit’ and that even the director at the company was opposed to it, but it was an HR thing.  Well, it just so happened that they figured out a work-around, and he needed people quick.  I’m getting pretty excited at this point, I already know what the company does and would love to be a part of it, and as a bigger bonus, their office is literally less than 2 miles from my home!  In so many ways this was the perfect role for me, and finally, the stars were starting to align!

All I needed to do was take their coding challenge, completely bypassing all the initial crap.  I even felt like the person I was speaking to was rooting for me.  He told me if I knocked it out of the park that I would be ‘golden’ and it would be a big ‘F U’ to that stupid assessment.  I can’t even begin to describe the level of excitement going across the phone.

I get the challenge e-mailed to me, and start digging in to it.  This is going to be fun, I get to finally prove myself, and dive into a topic I haven’t had a chance to before, cryptography!

The challenge provided 3 files, 2 of which were encrypted and a regular plain text file.  At a minimum, I had to write an algorithm that would decrypt the easier of the 2 encrypted files using the plain text file as a base.  As an added bonus, and to really impress, I would of had to also decrypt the harder version, and could create an API and/or a Single Page App (SPA).

So where do I begin? I had no clue, other than the Little Orphan Annie decoder pin from ‘A Christmas Story’ and having recently watched the movie ‘The Imitation Game’ about Alan Turing, I knew absolutely NOTHING about cryptography.  It’s always seemed an interesting field and from what I knew you could spend years trying to master it.

Not knowing anything, I go to a developers best friend, google. I start reading up on some basics of cryptography and really just barely scratch the surface with the different types of ciphers that can be used in a ‘simple substitution’ encryption.  Think ‘A Christmas Story’ here where you simply substitute a letter with another one.  The simple part is that the substitute letter never changes.

At this point I figure the easy encrypted version has to be one of these simpler substitutions.  Especially after looking at the text patterns I could just tell that these letters weren’t changing.

But how do I use the plain text?  The instructions said to use it as a base, but I had no clue what that meant, and didn’t find anything anywhere that talked about ‘using plain text as a base’.  I ended up assuming that the encrypted text was from some part of this plain portion (which was over 5 million characters long).  And since all the spacing, punctuation, and capitalization was preserved, I decided to try and find where the word letter count, spacing, punctuation, etc. matched between the two versions.  Once I found that, I started writing out the letter substitutions.  Things were coming together and I noticed a pattern, which I could instantly see matched a couple of the types of ciphers I read about earlier.

Once I knew the type of cipher used, I was able to write up an algorithm to find it on it’s own.  One requirement of the challenge was to not ‘brute force’ the solution as the cipher could change anytime.  While the algorithm I created could figure out the cipher no matter what the substitutions were, it did have to know what TYPE of cipher is used to work, and in this case one specific type.  So not technically brute force.  Plus I was explained ‘brute force’ meant not hardcoding the cipher as in ‘A’ is always ‘F’, ‘B’ is always ‘C’, etc., and my solution definitely wasn’t that.  The letters could have been substituted with anything and still worked.

With that part done, I decided to start on some sort of SPA.  I decided to write it up using the MEAN stack, leaving out the M since I didn’t need a database for anything.  So it’s all JavaScript built on Node.js with Express.js and Angular.  I was trying to make something that would hopefully impress so I used Angular 1.x since the majority of my experience is with that version.  I didn’t want to waste time trying to use something I had to learn at the same time.

During this time, I got a follow-up call from the person who got me this chance to see how I was coming along.  I had told him and he seemed excited about my progress.

I didn’t have any time limit but it was coming close to a week and I figured I had to get something turned in.  Of course I had a bunch of other things going on so I didn’t actually have all of that time to dedicate to this challenge.  I knew I wasn’t going to have my SPA where I wanted it in the time I allowed myself so I took on the harder encrypted challenge.

My algorithm wasn’t even close to working on this one, which I expected.  And unlike the easy version, whatever this translated to, it wasn’t going to be found within the plain text.  So now I had to rack my brain trying to figure out what to do.  I began searching google for other types of substitution ciphers (even though this was harder it still looked to me like a simple substitution), and stumbled upon a short article about frequency analysis.  Basically what this does is just use the most commonly found letters, words, etc. and starts substituting them.  Based on the odds, this could get you something to work with.

First I started with one letter words, which there are really only 2, ‘I’ and ‘A’.  I found all of the one letter occurrences and started my cipher.  These are actually easy to find because ‘I’ is always capitalized when used, versus ‘A’ which can be either.  This time, I was hard coding, a.k.a ‘brute forcing’, the cipher and using my previous decryption algorithm to start decrypting the text, leaving spaces for the letters I haven’t figured out yet.  Then I looked at 3-letter words, specifically ‘and’ and ‘the’, which are the most common of all of them. I already had the ‘a’ figured out so I could tell which one was which and added those letters to my cipher.  Decrypt again and things were looking good.  Next I looked for twin letters.  There’s not many of those either and ‘LL’ is the most common.  Added that to my cipher, decrypted, and things were starting to look up.  I started looking at other more common things, like after an apostrophe you normally see either a ‘t’ or ‘s’.  I already had the ‘t’ so the ‘s’ was easy to figure out.  Decrypt again and now it became a game of Wheel of Fortune.  Words were coming together and I was feeling just like young Ralphie, licking my lips with excitement and anticipation of figuring out the encryption.

Finally I figured it out, except there was a problem.  Looking at the cipher, there was absolutely no pattern at all to the substitutions.  Nothing to tell me what type of cipher was being used or how it was made.

My personal time limit was nearing an end.  The hard encryption was a bonus anyway so I said I had to brute force it but at least I figured it out on my own.  All the code was up on github and I told him he could submit it.  I was having a lot of fun with this though and told him that I would keep working on it anyway and would be updating github as I continued.

That was a couple weeks ago, and I haven’t heard anything since.  But I kept working on it.  I even got my SPA somewhat presentable and went ahead and deployed it live via AWS EC2.  As soon as I did that I called him up and left him a message to get an update and tell him about my new progress.  I emailed him as well and included the IP address.  That was about a week ago, again, still haven’t heard anything back.

At this point it’s safe to say what I had wasn’t good enough, this job just isn’t going to happen.  I had my chance, the one chance I’ve been begging for, and I blew it.  I didn’t crush this challenge like I believed I could.

I wasn’t lying though about enjoying this challenge, so I kept working on it, trying to figure out how to get a computer to figure out what I did on my own.  Then one night, as I was half asleep, it hit me like a Mack Truck.  Using the plain text as a base didn’t mean it contained the encrypted message (even though it actually did for the easy version).  It was all about the ‘frequency analysis’ I did myself, and what I should have done, is created an algorithm to parse through the plain text and determine the frequency of letters, words, patterns, etc.  Then using those frequencies, have the computer figure out what to start replacing the same way I did.  Then it was a matter of getting the computer to play ‘Wheel of Fortune’ to fill in the gaps.

And now we come to today.  I’m currently working on getting the algorithm to work.  I’m partially there, but it’s a bit ugly with a lot of conditionals.  For the ‘Wheel of Fortune’ part, I plan to use a word or dictionary API to cross-reference the words.  One issue I know is going to come up is made up names, names that aren’t going to be like any word in a dictionary.  But one step at a time for now.

This challenge has turned into a full-blown project for me, one that is going to be on my portfolio.  I have a vision for it and once I make it a reality, it’s going to be impressive.  It might be too late for the original position I was vying for, but I’m going to make sure it helps me with the next.

Before I end this post, I have one last thing I want to say:

He okpe ms lpytw dskp Sjcvmyte!

In case you can’t figure it out, just go to the ‘decrypt your text’ section of my app and use the keyword ‘challenge’.  It’s still in its early stages of development so don’t expect anything too amazing….yet!

Cryptography and Ciphers