Generating SHA hashes for AWS V4 Signatures in Common Lisp

spoiler: ironclad does it like it's not even a thing

For a project I am working on, I needed to generate AWS signatures for API access in common lisp. I haven't searched to look if there are libraries that do this already. Besides, the last couple of weeks I have been discovering golden codebases on GitHub for common lisp but none of them is on quicklisp or ultralisp. So, I don't want to find more and feel bad to have to rewrite these.

AWS has a step-by-step guide if you ever needed to build API access without using the SDK (maybe because you cannot use one or because the SDK does not exist for your language of choice). When I first started reading it, I was thinking - man, I am really shooting myself in the foot with this one.

Anyway, I wrote a bunch of code to do the string processing they have mentioned in the guide. Then I need to generate SHA hash. The obvious choice to do this is ironclad. So I installed ironclad using quicklisp.

ql:quickload :ironclad

Then I used their specific example to verify that ironclad hashes are compatible (or if I have to go down a rabbit hole searching fine-prints and options!). The document presents a hash for an empty string as e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. So I ran this piece of code to see what ironclad does with an empty string. I first created a simple function to call ironclad for a SHA256 hash - following some examples on the interwebs.

(defun my-hash-function (string)
  (ironclad:byte-array-to-hex-string (ironclad:digest-sequence :sha256 (ironclad:ascii-string-to-byte-array string))))

Then C-x C-e to compile the function over SLY. To test, I did this

CL-USER> (my-hash-function "")

Voila! It works and matches the requirements too. So I went ahead to test the second piece of code in the example.

content-type:application/x-www-form-urlencoded; charset=utf-8


which should give me f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59

So I assembled the text as a list of strings, used format to concatenate them and generate the hash using my function. Here's the snippet

(let ((components '("GET" "/" "Action=ListUsers&Version=2010-05-08" "content-type:application/x-www-form-urlencoded; charset=utf-8"
                    "" "x-amz-date:20150830T123600Z" "" "content-type;host;x-amz-date"
  (my-hash-function (format nil "~{~a~%~}" components)))

to get 0f940183b2f8334f2c54304b7118048beb3c0f6682aa6ee08c899f07d61e526e which is quite different from the required value.

The code I wrote looks correct at first but has a pretty big bug. I didn't catch it for a while, though. And I don't have the patience to debug SHA256 algorithms. So I did what I could do best - stare at it and review what am trying to do multiple times. Then I found it. The code I wrote adds an extra newline at the end of the string because of the way I wrote my format call. Peter Siebel to the rescue and I found the fix I needed:

(let ((components '("GET" "/" "Action=ListUsers&Version=2010-05-08" "content-type:application/x-www-form-urlencoded; charset=utf-8"
                    "" "x-amz-date:20150830T123600Z" "" "content-type;host;x-amz-date"
  (my-hash-function (format nil "~{~a~^~%~}" components)))

The ~^ directive in format macro is used to stop the iteration without processing the rest of the directives when the list is fully consumed. Format and Loop are macros that keep on giving. I don't think I know everything I can do with these macros yet.

Anyway, adding that fix solved the problem and my hash generation works perfectly as per the requirement. Now I need to go back and fill in the actual preprocessing and the rest of the steps to get my API access to work.

I'll share more snippets I learn along the way!