Stupid F#: Taking JSON to the SPA 1

Programming should be fun. All you need is a good set of values, some skill, and the right attitude. F# is the most fun I’ve had in a programming language in years. This essay series is about that: having fun. Most books and essays are about writing awesome code. There’s not a lot of material about writing code aweseomely. The code you’ll find here has bugs! Just like your code. Can you find them? If you’re looking for the final version of the code, you can find it in the project’s GitHub page. You can find the story of how all of this started on the series index page.

I’m a weird programmer. I don’t want to program.

I don’t want to use a tool, I don’t want to use a computer, I don’t want to touch a keyboard, I don’t want to write a line of code.

I just want to make things that people want.

Over time, I’ve painfully discovered two things: 1) Programmers are a market and people sell things to them. Bright, shiny things. We love bright, shiny things. We will happily spend our time playing with them and making more bright shiny things for others to play with. 2) Every minute I interact creating and manipulating complex technology increases the risk that the technology becomes more important than the people I’m helping, whether through bugs, useless features, or discussions and passions around things that are only important to me and not the people I’m trying to help.

The process of technology development itself is a risk to creating value.

Ideally? I would meet somebody, they’d say “You know, I really want X”. I would turn to my computer, and just like Old Baldy on Star Trek and say “Make it so”. And it would be so. Just like that. Continuously-tested code running in production that does X. We don’t even need a Level 3 Diagnostic, whatever that was.

It’s never going to work that way, but it’s a dream.

We now have an app that takes data in the form of X=Y and collates by X, where X is a string and Y is an integer. We’ve gradually built our type system to work on the command line and in an old-fashioned CGI html form posting back to the server. The idea is that we gradually increase deployment complexity while exercising our types and feedback loops until we stand-up a full system of the kind we want at the end of the project. This gives us immediate small chunks of work we can test, progress we can make in an our or two at a time, and a system for building out what might be a complex system a little bit at a time.

We might not know exactly what the overall problem is going to be, or how it’s all going to look when we’re done, but we know how to do little chunks of stuff that will get us there. This is the opposite of the way most nerds think, where they start with some vision of the end state and then continue pounding on the keyboard until the computer submits to their will. Lots of fun, but it’s not what we’re doing. Dang it, I know we want a Rails app, we’ll talk about what you want later after I do the important stuff.

If you wanted the technical description of what we’re doing, it’s using the Onion Architecture, Total Programming and Unix Philosophy to incrementally develop an app using pure functional programming that’s deployed using CI/CD/DevOps and continuous testing.

Or you could just say we’re kicking around having some fun with F#. Same thing.

What’s the next little bit of incremental work we can do?

As always, we can spread out one of two ways. We can spread out horizontally, where we add more types and/or make the system do new things it didn’t before. Or we can spread out vertically, where we take the same minimalist stuff and make it work in a more full fashion, from keyboard to production.

We always do vertical first, then continue tweaking and optimizing vertical as we scale out the horizontal. There are also two vertical strategies: start with making it look cool and then make it do stuff — or start with it doing stuff and then make it look cool.

Both of these are equally-important. So in a “real” environment, we’d do both at the same time. (This is a really good place to start pair programming. In the mornings you and your partner work on doing stuff. In the afternoon, making it look cool.)

The next logical step in the “doing stuff” chain is working in JSON on a test harness. JSON has become the XML of the internet; it’s now we move stuff around. We’ll talk about client-side architecture later. For now, let’s get the types right and data moving. We change functional programs from the outside of the onion inwards.

If you guys have been following along with this series, you might expect me to assemble a Json converter from stone knives and bearskins.

You are wrong. Instead I’m going to plug in the excellent NewtonSoft Json.Net library and add three lines of code:

static member FromJson (s:string) = 
        try Some(Newtonsoft.Json.JsonConvert.DeserializeObject<OptionExampleFileLineType>(s))
        with | :? System.Exception as ex ->None
    member self.ToJson()=
        Newtonsoft.Json.JsonConvert.SerializeObject(self,Newtonsoft.Json.Formatting.Indented)    

The only thing that changes is adding new converters on the types. That takes very little code, and I should never have to screw around with it again. That’s why this is a good library. Good tools disappear. You don’t spend time talking about them.

This is also another good time to use options, since we’re working at layer 1 of the onion and we don’t know exactly what we’re getting. Our collection type looks like this:

static member FromJson (s:string) =
        try Newtonsoft.Json.JsonConvert.DeserializeObject<OptionExampleFileLinesType>(s)
        with | :? System.Exception as ex ->{OptionExampleFileLines=[||]}
    member self.ToJson()=
        Newtonsoft.Json.JsonConvert.SerializeObject(self,Newtonsoft.Json.Formatting.Indented)    

As usual, I leave my fully-scoped types in the code until I reach the point that I use the library a lot. Then I’ll put in an open. I want to remind myself later of where this stuff is coming from.

Yikes! After all of this talking about the onion, I forgot to actually code the right way. If we want our app to use JSON, the first thing we do is add a command-line parameter to our console app shim, like so:

type InputFormat = Stream | File | CGI | JSON with
        static member ToList() =
        [Stream;File;CGI;JSON]
        override self.ToString() =
            match self with
            | Stream->"Stream"
            | File->"File"
            | CGI->"Cgi"
            | JSON->"Json"
        static member TryParse(stringToParse:string) =
            match stringToParse.ToUpper() with
                |"STREAM"|"S"->true,InputFormat.Stream
                |"FILE"|"F"->true,InputFormat.File
                |"CGI"|"C"->true,InputFormat.CGI
                |"JSON"|"J"->true,InputFormat.JSON
                |_->(false, InputFormat.File)
        static member Parse(stringToParse:string) =
            match stringToParse.ToUpper() with
                |"STREAM"|"S"->InputFormat.Stream
                |"FILE"|"F"->InputFormat.File
                |"CGI"|"C"->InputFormat.CGI
                |"JSON"|"J"->InputFormat.JSON
                |_->InputFormat.File
                //raise(new System.ArgumentOutOfRangeException("InputFormat","The string value provided for Output format doesn't exist in the enum"))

While I was at it, I found an error in one of the enums which I fixed. I also noticed that updating my parameters for output did _not_ cause an error down the line where I actually input and output, so I checked that out. It was due to my build warning level set too low. You want that sucker on max, so I changed it to 5.

Then I spent ten minutes fixing little stuff here and there. A couple of errors were from my catching a generic exception in the code. You don’t want to do that in production, and eventually as you exercise your types that goes away, but there are still times when you want to catch anything you get, mainly because you don’t know what the heck is going on outside of your onion. In those cases, instead of turning warnings off, you simply add this above the try block to make the compiler happy:

#nowarn "0067"

Then there were a couple of “can’t work with this, you’re accessing a mutable state in a collection” warnings, so we put int a clone.

if cgiVariables |> Array.exists(fun x->x.Key="myInput")
                    then
                        let cb=(cgiVariables.Clone():?>System.Collections.Generic.KeyValuePair<string,string>[])
                        let var=(cb
                                |> Array.find(fun x->x.Key="myInput"))
                        var.Value.Split([|OSNewLine|], System.StringSplitOptions.None) 
                            |> Array.toSeq
 

(This is good because we don’t have enough type annotations lying around. /s)

I finally got around to the error I wanted, which is that I have a new item in my enum that’s not being used anywhere, so I fix the code.

Our smart constructor for the collections class takes a sequence of strings. Hmmm. Remember how we loaded up our collection class?

/// Outside of the onion coming in
let inputStuff (opts:OptionExampleProgramConfig):OptionExampleFileLinesType = 
    let fileIsThere =System.IO.File.Exists(fst opts.inputFile.parameterValue)
    let inputStrings:string seq =
        match opts.inputFormat with
            |Stream->
                readStdIn [] |> List.toSeq
            |File-> 
                System.IO.File.ReadAllLines(fst opts.inputFile.parameterValue)
                    |> Array.toSeq
            |CGI->
                // This is specific to the test html harness form and should go away
                let cgiVariables=processCGISteamIntoVariables() 
                                |> Seq.toArray
                if cgiVariables |> Array.exists(fun x->x.Key="myInput")
                    then
                        let var=(cgiVariables 
                            |> Array.find(fun x->x.Key="myInput")).Value
                        var.Split([|OSNewLine|], System.StringSplitOptions.None) 
                            |> Array.toSeq
                    else Seq.empty
    OptionExampleFileLinesType.FromStrings inputStrings

We loaded up a list of strings, since that was what came from opening the text file and reading it — and it was also what came from reading in from stin. But now, with our JSon parser, all we care about is one big honking string, not a list. We designed expecting a list of strings from a file, but now our requirement is to handle just one string.

You can view this kind of thing as a pain in the neck or as instructional. It’s up to you. I choose to view it as instructional — is there something I’m missing in the architecture that I should consider?

Obviously I need to use just one string, somehow. I could rework the match. Starting down that path, I realize that my category system was whack. The problem here is that “JSON” and “CGI” are data formats, whereas “STREAM” and “FILE” are persistence formats. That one enum was doing too much work. Yikes!

Why didn’t I fix it at the beginning, before I ever started coding? Wouldn’t it be easier then? Wouldn’t one minute of thinking save hours of work downstream?

No. It would not. I used to believe that. I no longer do.

In certain cases, a little thinking prevents much pain later. Those cases are very rare. Instead, what usually happens is that “structure breeds more structure”. That is, once you start looking at the structure of the code as a separate unit from what it does, there’s all kinds of things that look like they also might be a good idea. Things that could prevent you from suffering later on. You’re coding because it looks and feels right, not because it does provably useful stuff.

Remember in functional programming, much more so than OO programming, we’re completely paranoid about managing cognitive load on the programmer. It’s easy as crap to write code that looks like ancient Greek to some poor guy coming in a year later. We can’t do that. (Worst yet, in really good FP you can write code that both looks completely simple, yet is horribly complex under the hood. In many ways this is probably much more cruel to the maintainer.)

Well heck, I don’t even need “FILE”. Everything’s a stream. Remember, our entire app itself is just a giant function — it does one thing, we pipe stuff to and from it as needed, and it can run anywhere without dependency issues. So FILE goes away.

Of course the next logical question is this: Why do we need an inputFile parameter at all?

I honestly can’t think of any reason right now, but my memory sucks. So I’ll comment it out (don’t tell the other consultants!) and then clean up that match statement.

Yay less code! Yay simpler logic! You change a functional app by changing the types first. I did that, and a bunch of stuff changed. The compiler worried about it for me, which is a double win. The inputStuff function collapses down to just this now:

let inputStuff (opts:OptionExampleProgramConfig):OptionExampleFileLinesType = 
    let stringsToProcess= readStdIn []
    match opts.inputFormat with
        |JSON->OptionExampleFileLinesType.FromJsonStrings stringsToProcess
        |Text->OptionExampleFileLinesType.FromStrings stringsToProcess
        |CGI->
            let cgiVariables=processCGIStream stringsToProcess
            if cgiVariables |> Seq.exists(fun x->x.Key="myInput")
                then
                    let varListFromCgiForm=
                        let cgiVarList = cgiVariables |> Seq.map(fun x->x.Key, x.Value)
                        let var=snd (cgiVarList |> Seq.find(fun x->fst x="myInput"))
                        var.Split([|OSNewLine|], System.StringSplitOptions.None) 
                    OptionExampleFileLinesType.FromStrings varListFromCgiForm
                else OptionExampleFileLinesType.FromStrings []

If our entire app is a function, and we only write code that is continuously-tested, where do the tests go?

In the O/S, that’s where. We use unit tests inside to check at compile-time, but we also test at the O/S level to make sure our function can do what it says it can and doesn’t blow up. In good code, the O/S becomes as important to the programmer as the IDE.

We don’t have to start writing a thousand lines of BASH (NOOOOOO!), but we can write up a quick script to check our app.

#!/bin/bash

rm -f TextSample1ToText.testout
rm -f TextSample1ToJson.testout
rm -f CGISample1ToText.testout
rm -f CGISample1ToJson.testout

cat TextSample1.testin | ./stupid1.exe -IF:T -OF:T > TextSample1ToText.testout
cat TextSample1.testin | ./stupid1.exe -IF:T -OF:J > TextSample1ToJson.testout
cat CGISample1.testin | ./stupid1.exe -IF:C -OF:T > CGISample1ToText.testout
cat CGISample1.testin | ./stupid1.exe -IF:C -OF:J > CGISample1ToJson.testout

./filecomp.sh CGISample1ToText.testout CGISample1ToText.desiredout "CGI1ToText Test: PASS\n" "CGI1ToText Test: FAIL\n"
./filecomp.sh CGISample1ToJson.testout CGISample1ToJson.desiredout "CGI1ToJson Test: FAIL\n" "CGI1ToJson Test:: FAIL\n"
./filecomp.sh TextSample1ToText.testout TextSample1ToText.desiredout "Text1ToText Test: PASS\n" "Text1ToText Test: FAIL\n"
./filecomp.sh TextSample1ToJson.testout TextSample1ToJson.desiredout "Text1ToJson Test: PASS\n" "Text1ToJson Test: FAIL\n"

How to run this? When to run it? What to do with it?

Don’t care. We’ll address that in our next hour or two of coding. For now we have a way to make sure the app isn’t blowing up and works with our sample input and output.

O/S tests show we have a problem. Ruh-roh!

Well that ain’t right. Shouldn’t be failing. What’d I do?

My test script was broken (above). The second line had FAIL instead of PASS.

Now would be a time where I would start tweaking out as an architecture astronaut, factoring out all that BASH test code, writing one list that feeds it all…. DRY, you big dummy! (DRY = Don’t Repeat Yourself)

I will not do this. Later, when I mess with this a second or a third time, I’ll factor it. Maybe. We’ll see. I’m working vertically, not horizontally.

My three hours is almost up — it was spread out over ten days this time — but there is a bit of tidying up we can do now before we call this chunk of testable work complete. We can take a look at our main program structure. We’ve exercised it a few times. There should be some lessons to learn.

The first thing is that now we have three functions: inputStuff, doStuff, and outputStuff. For those functions we write tested code and factor it out to other places. Since we’re using an Onion, Total Programming architecture, that pattern isn’t going to change.

I also see that the parameters for those functions aren’t going to change. The input function will take an opts parameter and return the incoming data. doStuff will also take the opts parameter (since it has to know stuff the command-line coder wants) and creates a chunk of processed data. Finally, outputStuff takes and opts parameter and the processed data and returns an int. All together, inputStuff, doStuff, and outputStuff looks like so is pseudo-code:

opts -> opts*incomingData -> opts*processedData -> int

Since that’s always going to be the same, let’s code it up so that the tail of one function feeds the head of the other. We end up with this opts*data pattern where the first parameter is whatever the program configurations are and the second parameter is the data as it moves through the pipeline. It allows our main code to look like this:

let newMain (argv:string[]) doStuffFunction = 
    try
        let opts = loadConfigFromCommandLine argv
        commandLinePrintWhileEnter opts.configBase (opts.printThis)
        opts |> inputStuff |> doStuff |> outputStuff 
    with

Remember, Total Programming and The Onion says we’re always moving through that pipeline, no matter what. In a way, it’s like a little railroad. The train always gets to the end, even if the processed data is an empty set.

This Railway-Oriented Programming is another common functional pattern. We’ll talk about it later. For now we have JSON input and output and tests that verify that the app is running as expected.

In Part 2 we’ll finish putting the app on the server and make a super-simple Single-Page Application (SPA) which uses everything we have so far. We’ve done enough for now. We have our same app which can talk in new ways and is becoming something outside programmers might be able to trust one day. Do that enough times — you’ll end up with a set of tested, composable functions-as-apps that outside programmers can use and compose in hundreds or thousands of ways depending on their needs. People make and use apps the exact same way people make and use functions. It’s turtles all the way up! (grin) Learn one way of solving problems, use it across the entire stack. Cool stuff.

This has been fun! If you’d like the theory for where all of this comes from, I’ve got a book on the foundations of organizing all project information, from user interviews to higher-order functions. You should read it.

You can find the full source code to this app on GitHub


Follow the author on Twitter

August 8, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *