But Will It Play In Peoria? (Running F# as a Plain Vanilla CGI app)

Programming should be fun. All you need is a good set of values, some skill, and the right attitude. F# is the most fun I’ve had in a programming language in years. This essay series is about that: having fun. Most books and essays are about writing awesome code. There’s not a lot of material about writing code aweseomely. The code you’ll find here has bugs! Just like your code. Can you find them? If you’re looking for the final version of the code, you can find it in the project’s GitHub page. You can find the story of how all of this started on the series index page.

We’ve made up this silly simple program that collates and adds up stuff. We started by creating it for the console — with the explicit understanding that it doesn’t matter where it runs. The console is just the first place we stand up our types because it’s the easiest to access. If we can run our types as true microservices we should be good to go wherever they live. We’ve even talked about making a simple web form so we can test our types in a second place.

So how would that work? How do you test something over the web without using a web framework? You said coding like this was supposed to save time. Will it?

Let’s find out what we need to do to trivially interact with any web page.

First thing we need is the ability to output html. We should also be able to output an entire web page. Nothing fancy — we’re not building our own web framework. We just need to do a basic dump of whatever we have for testing/validation purposes.

To modify a pure functional app, we start by changing the type system at the outside of the onion. Functional apps are driven by the type system, and the onion is our gateway into whatever we’re doing. So let’s add a command-line parameter that changes output format. We’ll call it “F”. It can be “text”, “html”, or “webpage”. The default — and error condition — is “text”. Continuing to think about our onion, I don’t care if you use bad parameters, I’ll just send you text. You’ll figure it out.

We change our program configuration type by adding a new type to take the parameter

type OutputFormat = Text | Html | WebPage with
         static member ToList() =
            [Text;Html;WebPage]
         override self.ToString() =
          match self with
            | Text->"Text"
            | Html->"Html"
            | WebPage->"WebPage"
         static member TryParse(stringToParse:string) =
            match stringToParse.ToUpper() with
                |"TEXT"->true,OutputFormat.Text
                |"HTML"->true,OutputFormat.Html
                |"WEBPAGE"->true,OutputFormat.WebPage
                |_->(false, OutputFormat.Text)
         static member Parse(stringToParse:string) =
            match stringToParse.ToUpper() with
                |"TEXT"->OutputFormat.Text
                |"HTML"->OutputFormat.Html
                |"WEBPAGE"->OutputFormat.WebPage
                |_->OutputFormat.Text
                //raise(new System.ArgumentOutOfRangeException("OutputFormat","The string value provided for Output format doesn't exist in the enum"))
    [<NoComparison>]
    type OptionExampleProgramConfig =
        {
            configBase:ConfigBase
            inputFile:ConfigEntry<FileParm>
            outputFormat:OutputFormat
        }
        member this.printThis() =
            printfn "EasyAMConfig Parameters Provided"
            this.configBase.verbose.printVal
            this.inputFile.printVal
            printfn "%s" (string this.outputFormat)

I’ve written what I consider to be a “fully-loaded” enum: one that’s able to talk easily to the outside world. I can look at it like a list. It can translate easily to string and back. I might have done this using reflection, but that would suck. How this enum works is an app decision. I don’t want no generic stuff here.

The compiler immediately tells me: hey bub! You’ve got a new item in your program config type that doesn’t exist in the rest of the program!

Cool. That’s what I want, a compiler looking out for me. So I add a new default type in case there’s nothing provided.

let defaultOutputFormat =
        createNewConfigEntry "F" "Output Format (Optional"
            [|"/F:<TEXT|HTML|WEBPAGE> -> type of output desired""Defaults to TEXT"|]
            OutputFormat.Text

I still have the error since I haven’t actually stuffed the type, just made a default. So I stuff the type. (second-to-last line)

let loadConfigFromCommandLine (args:string []):OptionExampleProgramConfig =
        if args.Length>0 && (args.[0]="?"||args.[0]="/?"||args.[0]="-?"||args.[0]="--?"||args.[0]="help"||args.[0]="/help"||args.[0]="-help"||args.[0]="--help"then raise (UserNeedsHelp args.[0]) else
        let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
        let newConfigBase = {defaultBaseOptions with verbose=newVerbosity}
        let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
        let newInputFile = ConfigEntry<_>.populateValueFromCommandLine(defaultInputFile, args)
        let newOutputFormat = ConfigEntry<_>.populateValueFromCommandLine(defaultOutputFormat, args)
        {configBase = newConfigBase; inputFile=newInputFile}

Well crap. Now it’s telling me something about my generic ConfigEntry<_> class Oh yeah. There’s custom logic with taking a new kind of input. Let’s go fix that. All this involves is looking at one overloaded method that takes one type of enum and copying/modifying it for our type). While it may look complicated, brain load factor is still very low.

static member populateValueFromCommandLine ((defaultConfig:ConfigEntry<OutputFormat>), (args:string[])):ConfigEntry<OutputFormat>  =
                let parmValue = getValuePartOfMostRelevantCommandLineMatch args defaultConfig.commandLineParameterSymbol
                let newVal=if parmValue.IsNone then defaultConfig.parameterValue else
                            let tp=OutputFormat.TryParse parmValue.Value
                            if fst tp=true then snd tp else defaultConfig.parameterValue
                defaultConfig.swapInNewValue newVal

I had to move my OutputFormat type up above my command line stuff. That’s because you can’t use things until you’ve defined them. Dang computers! (Shakes cane at sky)

Well heck if there isn’t still an error. Oh yeah! I still haven’t created the type. If Ir remember correctly, that’s the last line of our load configuration paramters function.

let loadConfigFromCommandLine (args:string []):OptionExampleProgramConfig =
        if args.Length>0 && (args.[0]="?"||args.[0]="/?"||args.[0]="-?"||args.[0]="--?"||args.[0]="help"||args.[0]="/help"||args.[0]="-help"||args.[0]="--help"then raise (UserNeedsHelp args.[0]) else
        let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
        let newConfigBase = {defaultBaseOptions with verbose=newVerbosity}
        let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
        let newInputFile = ConfigEntry<_>.populateValueFromCommandLine(defaultInputFile, args)
        let newOutputFormat = ConfigEntry<_>.populateValueFromCommandLine(defaultOutputFormat, args)
        {configBase = newConfigBase; inputFile=newInputFile; outputFormat=newOutputFormat.parameterValue}

Compiles fine. Looking at my watch, I’m 45 minutes in and now I have an opts variable that contains my shiny new OutputFormat type. Yay! (I’ve also blogged during this time).

Now I just need to add some kind of code to create some dummy output and test it. Let’s pop over to the doStuff function. Remember, doStuff is where we start adding complexity. Then we move that complexity out to the type system by way of functions, then shared utility functions, then types.

We’ll begin spinning off our complexity that will end up in types somewhere with a simple match at the end of doStuff.

let doStuff (opts:OptionExampleProgramConfig) =
    let fileIsThere =System.IO.File.Exists(fst opts.inputFile.parameterValue)
    let keyValuesFromFile = 
        if fileIsThere = false 
        then 
            Seq.initInfinite (fun _ -> System.Console.ReadLine())
            |> convertLinesIfPossibleToKVPair
        else 
            System.IO.File.ReadAllNameValueLines (fst opts.inputFile.parameterValue)
    // we want kv lines with alpha for key and number for value
    let optionLines = OptionExampleFileLines.FromStringKVCollection keyValuesFromFile
    match opts.outputFormat with
        |Html->printfn "Hey there"
        |_->groupAndSum optionLines |> Seq.iter(fun x->printf "%s" (string x))

Does it work? It does not! Over in our newMain function, we’re supposed to add a print statement for new configuration entries. So let’s add that.

Help system: upgraded

But when we try to run the code, even just by itself, we get this spectacular crash:

Holy bunch of stuff, Batman!

My hour for this morning is almost up, but let’s see what we can do with this. Hmmm. Looks like FSharp is complaining about OptimizedClosures and the Generic KeyValuePair. I haven’t done anything with KV pairs. What the heck is going on? Is it trying to read my command-line parameter as data? Nope. It fails with no parameters. In that case, it should be pulling in the default file.

Let’s run it like we did last time.

I am sad

I’m beginning to think it’s nothing we did. It’s just deflicted. Something must have changed. If only we had some way to keep track of changes to our programs! If only there was some way of keeping track of versions of code!

Walking through with the debugger, it’s the groupAndSumKV function that’s crapping out. And it was solid. (Long rant here about how I should have set up testing in the last essay. But it was only a educational prototype! Frack! That good coding practices stuff sure is annoying, right?)

let groupAndSumKV (optionLines:seq<System.Collections.Generic.KeyValuePair<string,int>>) =
    optionLines
    |> Seq.groupBy(fun x->x.Key) 
    |> Seq.sortBy fst
    |> Seq.map(fun x->
        let sumOfGroupedData=snd x |> Seq.sumBy(fun x->x.Value)
        (fst x, sumOfGroupedData)
        )

Nothing to do now but to fix it — and put testing in even if it means taking an hour or two screwing around with dependencies. Yuck.

let groupAndSumKV (optionLines:seq<System.Collections.Generic.KeyValuePair<string,int>>) =
    let oTemp = optionLines |> Seq.toArray
    let var1= oTemp |> Array.groupBy(fun x->x.Key) 
    let var2 = var1 |> Array.sortBy fst
    let var3 = var2 |> Array.map(fun x->
        let sumOfGroupedData=snd x |> Array.sumBy(fun x->x.Value)
        (fst x, sumOfGroupedData)
        )
    var3 

Well, if we’re going to hack, dang it, let’s hack. First we switch over to Array from Seq. Sequences are great and awesome, but god help you if you’ve to find a bug. I run the code again and it’s fine.

Ha! The error wasn’t there at all. Freaking sequences. So let’s back that change out. Hack up the next function.

let groupAndSum (optionLines:seq<NameNumberPairType>) =
    let newOptionLines = optionLines |> Seq.toArray
    let var1 =newOptionLines |> Array.map(fun x->x.ToKVPair) 
    let var2 = var1   |> groupAndSumKV |>Seq.toArray
    let var3= var2 |> Array.map(fun x->OptionExampleFileLine.FromKVPairString)
    var3

Stepping through the debugger (while kicking myself yet again for not adding testing), we see this:

Ruh-roh Raggy

Well that ain’t right. It’s that dang “OptionExampleFileLine.FromKVPairString” that’s not resolving to a value. Instead it’s just giving me the function address. I set a debugger on that, run it.

And nothing happens. Doh! That was the problem all along, Doofus! It didn’t run that function. Instead it just return the method. (Second line from the bottom in the code above has no parameter passed to it).

type OptionExampleFileLine = private NameNumberPair of NameNumberPairType with
        static member FromKVPairString (kv:System.Collections.Generic.KeyValuePair<string,int>) =
            NameNumberPairType.FromKVPair kv
        static member FromNameAndNumber (name:string) (number:int) =
            {Name=name; Number=number}

I had a really good laugh at this point

    let var3= var2 |> Array.map(fun x->OptionExampleFileLine.FromKVPairString x)

Ya dude. It needs a variable to work. Otherwise you’re just mapping whatever you have to an array of function addresses — which is what we got. Fixed that. Now I had the wrong type. So I fixed that

let groupAndSum (optionLines:seq<NameNumberPairType>) =
    optionLines 
        |> Seq.map(fun x->x.ToKVPair) 
        |> groupAndSumKV
        |> Seq.map(fun x->
            OptionExampleFileLine.FromKVPairString 
                (System.Collections.Generic.KeyValuePair<string,int>(fst x, snd x))
            )

And it runs okay. Confession: I had made a small (Ha!) refactor to some code after our last essay. What harm could it cause?

So I spent 45 minutes this morning blogging and doing this cool command-line type judo. Then I spent an hour screwing around for no reason at all except the fact that I’m a messy programmer.

That’s what harm it could cause. Always put in automated tests, my friends. They’re your safety net — and they prevent just this kind of thing from happening. It can be funny when you waste an hour. Not so much when a team stalls for a week.

That was easy. Kinda.

Lunch time.

I didn’t get back to this until after dinner, and the first thing I did was grab myself by the lapels and shake the living tar out of me. If this is going to be an ongoing exercise instead of just a blog entry or two, I need to treat it as a real app. That means file structure, version control, testing, and whatever else comes along with it. (Although I’ll stop short of true DevOps, since even though it’s an app, it’s an educational app, not a production app. I always put adding structure off as long as possible — but for some of this stuff, it’s been long enough already. Time to get a little more serious.

So I created a new F# console project. This time I called it “stupid1” instead of that old faithful “ConsoleApp”. I tried to add FsUnit, my favorite testing framework.

This was as unpleasant as I thought it would be. The package manager refused to add it. My target .NET version was too high. Although I could add NUnit. So I went to GitHub and got the FsUnit files directly to build them myself.

That didn’t work because they were using a different build system. Yay 17 different versions of the same stuff written different ways. I got as far as beginning to change out my build system and I realized What the hell am I doing?. I don’t need this! Let those guys figure out how to get their code working with my tools.

Two hours shot. At the end I decided just to use NUnit.

Two hours.

I call this “framework/tool noise” and it’s all-too-common. I teach and work with a lot of tech teams. While I’m there, I listen to their conversations. Are they talking about making users extra happy? Or are they talking about how to get WhizBang 7.3 working with RocketLauncher 3.6?

Most of the time it’s not the users. Between stupid processes and tools that end up owning the teams instead of the teams owning them, most teams spend more time not trying to help people and instead mired in organizational cruft.

Look, I don’t care what kinds of tools or frameworks you use. I don’t care what kinds of processes you use. I don’t even care if you test or not. When you’re kicking around like we have been, it’s all good. If your code has no economic value? It can’t have technical debt, and even a master cabinetmaker should be able to kick out a whistle now and then to have fun. She can also spend a month making the perfect cabinet, if that’s her thing. Coding ain’t religion — or at least it shouldn’t be.

But once you start making stuff that you want people to value, all of that other stuff matters. Tests ensure value, both value of the code and value to the user. Processes are supposed to the minimal amount of overhead necessary to deliver that value. And tools are supposed to disappear as your mind engages directly with the user’s world.

Over the next couple of days, I spent several hours — the most time I’ve spent in one task since we started this journey — setting everything up. No new functionality, just setting stuff up. We now have a project with eight files:

Program.fs
ScratchPad.fs
AppTests.fs
AppUtils.fs
AppTypes.fs
SystemTests.fs
SystemUtils.fs
SystemTypes.fs

Is this some magic system I’ve created and now I’m going to sell you a certification? No. I just made it up. I know from experience that I do a lot of code reuse. Some of that code I want to reuse everywhere. That goes in the bottom three “System” prefixed files. Some of the code I want to reuse in whatever family of apps I’m currently creating. That goes in the files prefixed with “App”. Finally I want a scratchpad for playing around with stuff before I decide to use it. That’s ScratchPad. Nothing goes in Program. Program is an artifact of our decision to initially deploy as a console app, not a necessary part of the system we’re creating.

Tests, strong types, and shared files will allow this set of files to scale across most all solution spaces, no matter the platform. For different platforms you’d just need to add some more stuff. Eventually you may end up with separate Persist and Lenses files. Or not.This is enough for now. It’s probably about 100 times the crap that we need. But we’ve started mentally organizing how we want our solution files structured, and we didn’t go too far overboard doing it. That’s the real work, even if we have to change it up a bunch of times later on.

Once I sorted all of my old code into the correct buckets, I took a look at the outside of my onion. In SystemTypes, I have some code that takes a filename and tries to get a sequence of key/value pairs out of it. I had to rewrite it a bit to get testing working, which is cool. Making your code testable makes your code better. The new code looks like this.

let lineContainsADelimiter (delimiter:char) (line:string) = line.Contains(string delimiter)
let lineOnlyHasTwoPieces (delimiter:char) (line:string) = line.Split([|delimiter|]).Length=2
let splitLineIfPossibleIntoTwoPieces (delimiter:char) (line:string)  =
    if lineContainsADelimiter delimiter line && lineOnlyHasTwoPieces delimiter line
        then
            let split=line.Split([|delimiter|])
            Some (split.[0],split.[1])
        else None
let convertLinesIfPossibleToKVPair (stringSequence:string seq) =
    stringSequence
        |> Seq.map(fun x->
            splitLineIfPossibleIntoTwoPieces '=' x
            )
        |> Seq.choose id 
        |> Seq.map(fun x->
            new System.Collections.Generic.KeyValuePair<string,string>(fst x, snd x)
            )
type System.IO.File with    
    static member ReadAllNameValueLines s =
    let textLines = 
        try
            System.IO.File.ReadAllLines s |> Array.toSeq
        with
            | :? System.Exception as ex ->Seq.empty
    convertLinesIfPossibleToKVPair textLines

Then I wrote up some system tests, like so:

/// Create a dummy file in the OS and return a .NET FileInfo object. Used as a mock for testing
let getFakeFileName() = 
    let tempColl = (new System.CodeDom.Compiler.TempFileCollection(System.AppDomain.CurrentDomain.BaseDirectory, false))
    tempColl.AddExtension("fakefileextension") |> ignore
    let rndPrefix = System.IO.Path.GetRandomFileName()
    let tempFileName = System.IO.Path.Combine(System.AppDomain.CurrentDomain.BaseDirectory, (rndPrefix + "_tester.bsx"))
    tempFileName
[<Test>]
let ``FILE: ReadAllNameValueLines with bad file returns empty sequence``() = 
    let badFileName = getFakeFileName()
    let ret = System.IO.File.ReadAllNameValueLines badFileName
    Assert.AreEqual(0, (ret |> Seq.length))
[<Test>]
let ``convertLinesIfPossibleToKVPair: Empty sequence returns empty sequence``() = 
    let ret=convertLinesIfPossibleToKVPair Seq.empty
    Assert.AreEqual(0, (ret |> Seq.length))
let initialTestFile = 
    seq [
        "a=9"
        ;"b=8"
        ;"c=10"
        ;"a=4"
        ;"b=4"
        ;"c=11"
        ;"#$%^"
        ;""
        ;"a=9"
        ;"   "
        ;"   asdf"
        ]
[<Test>]
let ``convertLinesIfPossibleToKVPair: Ignore bad lines``() = 
    let ret= convertLinesIfPossibleToKVPair initialTestFile
    Assert.AreEqual(7, (ret |> Seq.length))

I have a new function, getFakeFileName, which is kind of neat. Where’d I get that? I reused it. I’m already using it in a few other projects. I’ve also reproduced the initial test file.

Now we’ve verified the outside of the onion works. Bad files? You get empty Sequences. Lines don’t parse? They’re skipped. Then I wrote a couple of TypeTests. Does the main app type load up correctly? Does the core of the application, groupAndSum, work right for the initial file?

[<Test>]
let ``OptionExampleFileLines: Initial test file``() = 
    let ret=OptionExampleFileLines.FromStrings initialTestFile |> Seq.toArray
    Assert.AreEqual(7, ret.Length)
    Assert.AreEqual({Name="a";Number=9}, ret.[0])
    Assert.AreEqual({Name="a";Number=9}, ret.[6])
[<Test>]
let ``groupAndSum: Initial test file``() = 
    let input=OptionExampleFileLines.FromStrings initialTestFile |> Seq.toArray
    let ret=groupAndSum input |> Seq.toArray
    Assert.AreEqual(3, ret.Length)
    Assert.AreEqual({Name="a";Number=22}, ret.[0])
    Assert.AreEqual({Name="b";Number=12}, ret.[1])
    Assert.AreEqual({Name="c";Number=21}, ret.[2])

In many ways, the horse is already out of the barn here, so we’re playing catch-up. We’ll do what all teams should do when starting testing: we’ll test what we know that works, we’ll test the failure paths we’re already aware of, and every time something breaks we’ll write a new test before fixing it. That’s an easy way to start, and along with TDD and other good coding practices? It’ll clean up your code in no time.

Note that I’m still focusing on pulling that “thin string” through whatever I’m doing. There are three main feedback loops in coding. Maybe we’ll talk about them later. But I’m always managing those feedback loops first, what the code does second. Even if I knew we had three other data types and a lot more functionality, once we have an app that needs testing we need to manage the testing feedback loop before anything else. Test-code-compile-green lights

Finally I checked the code into GitHub. You can find it at DanielBMarkham/stupid1

This was a little bit of micro-work that started off in one direction but ended up somewhere else! That’s cool. That’s the way it goes sometimes. I’m still taking little chunks of a couple of hours here and there and doing something useful with them.

When I used to code a lot in C++, coming in cold to an application might mean sometimes spending an entire morning flailing around in the debugger getting my bearings and getting back in the groove of the code. I don’t do that anymore.

In Part 2 I finish up the changes and show the app running from an Apache server. Things always take longer than you think!


Follow the author on Twitter

July 22, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *