Programming In F# Like A Stupid Person

I program in F# like a stupid person, and I’m okay with that.

Ignorant people don’t know something. There’s no shame in being ignorant. Stupid people don’t want to know. Or they know and purposely ignore what they know. Nobody wants to be stupid.

I do. I have my own way of coding in F# that is not the accepted way, and I’m okay with that.

When people describe all the problems they have with microservices, I ask them: are you defaulting to text streams? Is your code more than 100LOC? Is your executable the same as a pure functional method?

Most of the time the answer is “no”, so I know they’re not doing true microservices. They’re probably using some kind of library hooked into a message queue or relational database. They’ve bought a buzzword, not a concept.

Vasily Kirichenko was kind enough to factor out my options problem. I got a tweet: “This is what all idiomatic F# looks like”

Make it look like this?

That’s wonderful, and it looks gorgeous, but I’m much more interested in workflow than I am beauty. Would my workflow have evolved towards this? Probably not, as I was demonstrating using types across projects to handle dataflow. But it might have. Beats me. I’m not going to get worked up about whether my code looks idiomatic or not or way or another. (I love his code, though. Nice!)

I am able to look at it and reason about it. Sweet.

Why would I evolve towards strong-typing around the incoming data? Am I adding complexity that’s not warranted? Why? To me? That’s a cool question to ask.

But as far as looking at a sample piece of code, even if it were in a different style the question would still remain: could I look at it and reason about it? Same question would be true in C++, or Haskell. That’s my only criteria for beauty. How am I able to engage with it?

I have big variable names. I use temporary variables in my code. Why? Because I am forgetful. Big names help me remember, reason about code when I’m coming in cold. Sometimes they go away. Sometimes not. I don’t worry about it one way or the other.

Please understand, I’m not saying anybody else is wrong. There are a damned ton of better programmers in F# than I am. That’s neat. I love the fact that I have things to learn from them.

I read about various F# concepts. I get out my IDE and play around with them. I learn new things. I don’t always use them, but I learn. It is enjoyable.

I read books like the excellent “Domain Modeling Made Functional” by Scott Wlaschin, a book I consider to be one of the most important programming books in the last few years, and play around with the ideas in my code. Looks like the Model-Driven Development guys are back in action! Woot! Fun times ahead, especially for coders in F#.

I have fun. I play. I learn. I enjoy myself.

I was talking to a friend about Domain Modeling, event storming, and the rest of it a couple of weeks ago. He told me he loved the idea and it was the only way he wanted to code — but how to prevent the Average Joe programmer he worked with from screwing things up?

This did not sound like fun, playing, learning, or enjoying yourself.

I hate to say it, but as a programmer who’s coded in a dozen different languages, in all kinds of situations, programming actually bores the hell out of me. “So much of it is wiring things up” one non-coder observed, and they were right. For whatever reason, the tech community seems insistent on developing coding situations where there’s a lot of grinding. I’ve seen people build and deploy thousands of lines of code to get something 3 lines of code can do. And that’s not unusual. That’s actually the norm. Hell, I might have been doing it in my example code showing off options. Or not. I’m not sure — but I want to know.

What does interest me is the people part of that. Why do we create such overblown systems? Why are some things more popular than others? What new concepts are entering the community that might get traction over the coming years?

When I saw the first cross-compiler, F# to Javascript, I knew we were on-board for quite a ride with this language!

When I code in F#, I don’t put on my mathematics hat. I don’t put on my architects hat, and I’m not building a giant model railroad. I am not looking for some aesthetic of perfection, clean, or nice.

I put on my explorer’s hat. My party hat. We’re going to have a blast, guys! Look at all of what we can do! Let’s go learn stuff!

When I get out F# and start coding, it’s a happy time. An adventure awaits! I code in F# because coding in F# makes me a better and happier programmer.

I program in F# like a stupid person, and I’m okay with that.

July 19, 2018  1 Comment

Analysis Still Exists, Whether You Understand It Or Not

Almost got into an argument with my friend Tim yesterday. That’s weird because both Tim and I aren’t the kind of people who argue very much. We like everybody.

Tim was talking about how important it was to have a specialized skillset. You can’t just throw developers around here or there and expect them to do well!

I replied that solving people’s problems was the important thing. Not skillsets. Skills were important, sure, but everything else hinged off of making stuff people want.

We talked for a while online and came to the real problem: companies that refuse to let developers stay in any one piece of code for extended periods of time, instead shuffling them off to random other pieces of code — sometimes in a completely different tier or subsystem — and expecting them to drop in and thrive immediately. After all, they said, coding is coding, right?

Yeah, that’s not going to work. So who came up with this plan?

Architects. Architects who don’t code, the most dreaded form of architect.

If you’ve read Info-Ops, you know that all structure is derivative. That is, structure should only exist to satisfy some behavior and/or supplementals. People used to show me things like a database diagram. Is this any good? How the heck would I know? Could you tell me what it does? Can you tell me the constraints it operates under? Then we can test it out to see if it’s any good.

All structure exists because it has to behave in certain ways and must operate under certain constraints. The only question is whether or not those behaviors and constraints are explicit or implied. Sometimes they’re explicit: “The system must reconcile accounts overnight using a batch process and COBOL/JCL”. Sometimes they’re fuzzy/implied: “We always use Docker images here. They told you that, right?”

The problem with architects is this: without being part of delivering value, the behavior and constraints they put on any system they touch is by necessity theoretical. They just don’t know. Businesses and technology deployment are complex processes. There’s no way in heck you could picture it all in your head in such a way that maps up to reality. Instead it’s always a learning process — even for folks who have been doing it for years.

Worse than that, there’s no feedback loop. So you’ve got people creating tools, processes, and structures based on how pleasing they find them mentally. And they’re never wrong. Instead, it’s always the dang developers who can’t be trusted to code their way out of a paper bag.

That’s no good for anybody. All it does is create hard feelings, conflict, and poor-performing organizations.

But what about this drop-the-developer in anywhere idea? Surely a professional could work anywhere, right? This is where Tim and I crossed horns.

From the book, Analysis is a series of professional conversations among a group of people that reaches an implied consensus. Done well, it creates a shared mental model. It happens all the time, whether it’s two people in a break room or a group of people doing formal analysis meetings. Analysis is a natural part of being sentient. Two-year-olds do it.

Most of analysis happens sub-rosa, that is, you don’t see it. Non-verbal cues, tone of voice, body language, inflection — dozens of human elements come into play when groups of people talk about things. It’s not the same as filling out a form. It doesn’t work that way.

So while for Tim it was the technical angle of moving people around, what was actually happening was that these companies were denying that analysis existed. Coding was coding and had nothing to do with context. In fact, coding is solving problems for people and is entirely about context.

Over time, working with the same people and the same problem area, the natural analysis process builds up a huge library of things everybody knows and agrees on, whether it’s ever stated explicitly or not. You get a feel for why this feature is there, what might help it be better, and why these other attempts failed.

That analysis turns out to be a huge accelerator in making things happen. And when it’s not present, it’s a huge hindrance. Ever walk into a new IT shop and try to solve a problem? You can spend hours just getting the environment right. There’s a hell of a lot that goes into setting up a problem in the right way so that it can be solved. It’s much more work than the actual coding.

These companies that felt that a programmer was a programmer. They were all fungible. Many times people are guilty of the robot myth. If I spend my career programming computers and they do what I tell them to, I should be able to program groups of people and they should do what I tell them to also. If not, I must be providing them the wrong instructions. That is, people are robots.

Tim was emphasizing the tech nature of different environments. Everybody can’t be good at everything! I was emphasizing the business nature of different environments. You gotta know the problem. In fact, it’s both. Good things happen when people with deep skills engage in continuous quality and long-running conversations. They understand the business better, they start to match their skills up with the problem, and they think of things nobody else would have.

And when there’s some technical issue, like what sort of tools or platform to use for a new rollout, who would be better equipped to make those decisions than technical folks who were soaked in the problems of the user?

Oddly enough, by trying to architect the entire system the architects had removed any actual architecture from going on! It was an architecture-free zone. More like a museum of modern art — this looks nice, this worked well at BigCorpX, this always is cool when it’s configured like so. Everything was based on the aesthetic tastes of a select few.

Analysis doesn’t go anywhere. It exists and carries on whether or not you acknowledge it or not. Ignore it and it will completely destroy your company.


If you’re interested in how all structure is derivative, and how analysis happens and how to have better professional conversations, you should read Info-Ops. It explains the “magic” works behind-the-scenes when cool tech gets built. If you’ve ever wondered what “emergent” means, how to know you’re having professional conversations, or how to get the maximum value out of the minimum amount of tools, Info-Ops explains it.

July 18, 2018  1 Comment

Failure Is Not An Option, Revenge Of The Nerds

[you’re spending] a lot of time thinking about a problem for which functional programming already has an idiomatic solution: the monad. [This] feels very much like taking object-oriented code and thinking and translating it into a functional language, instead of using the techniques offered by functional programming…

Where are the tests? Surely you’re not telling people to code without tests, are you?

(This is the third in a three-part series about how I think and the workflow I use
to code F# programs. Part1 Part2)

I started this essay series with a question about the appropriate use of options. This led to a discussion of The Onion, Total Programming, and the Unix Philosophy. These are the three principles I use to guide my functional coding. I also code for readability and understanding. I actively avoid anything that might look “clever”. We end up doing things like Higher-Order Functions (HOFs), monads, and the like, but we end up doing them once the code reaches a point it no longer needs any maintenance.

What the code is supposed to do, and my ability to understand it, drives the structure. I never pick up something that I think is cool and slap it into my solution. Instead, I prove out that it’s needed, then add it in. Focus on the user and code maintainer. Always.

Here’s the Onion, the sample data file we will process, and what the output looks like. It’s brutally simple.

The Onion is love, peace, and happiness. Come to love it.

Our data

All it does is group by letter and sum

I’ve done a tiny bit of refactoring since we last spoke. This is so we can pick up the story where it gets interesting. If you’ll recall, I came into this with a home-grown libary I use for command-line parsing. My “main” function had a bunch of boilerplate stuff that basically caught any exceptions and called “doStuff” (It’s my code, I get to name things what I want to.) Since that was boilerplate, I factored that out to a Utils.fs file.

[<EntryPoint>]
let main argv = newMain argv doStuff

Then I broke a personal rule and took the code we used to read key/value pairs from a file and genericized it, sticking it in the System.IO.File class where it belongs. (I would normally want to use this code several different times to prove it was solid enough to move out.) That kind of stuff is at layer 1 of the onion.

type System.IO.File with
        static member ReadAllNameValueLines s =
            try
                let textLines = System.IO.File.ReadAllLines s
                let lineContainsAnEquals (line:string) = line.Contains("=")
                let lineOnlyHasTwoPieces (line:string) = line.Split([|'='|]).Length=2
                textLines
                    |> Array.filter(fun x->lineContainsAnEquals x && lineOnlyHasTwoPieces x)
                    |> Array.map(fun x->
                        let split=x.Split([|'='|])
                        new System.Collections.Generic.KeyValuePair<string,string>(split.[0],split.[1]))
            with
                | :? System.Exception as ex ->[||]

Finally, we used smart constructors to protect our types and handle moving to layer 2 of the onion. When finished, our smart constructors look like this:

type NameNumberPairType = { Name:string;Number:int}
    type OptionExampleFileLine = private NameNumberPair of NameNumberPairType with
        /// Send it a KV pair with a name and a number (both strings) makes into a strong type
        static member FromKVPairString (kv:System.Collections.Generic.KeyValuePair<string,string>) =
            try
                {Name=kv.Key; Number=System.Int32.Parse kv.Value}
            with | :? System.Exception as ex->
                failwith "This should never happen"
    type OptionExampleFileLines = private OptionExampleFileLines of OptionExampleFileLine[] with
        static member FromKVCollection (keyValueCollection:System.Collections.Generic.KeyValuePair<string,string>[]) =
            /// Process anything with alpha=number, ignore the rest
            let doesKVWorkForUs (pair:System.Collections.Generic.KeyValuePair<string,string>) =
                (fst (System.Int32.TryParse pair.Key) = false)
                && (fst (System.Int32.TryParse pair.Value) = true)
            let optionLines = keyValueCollection 
                            |> Array.filter(fun x->doesKVWorkForUs x)
                            |> Array.map(fun x->OptionExampleFileLine.FromKVPairString x)
            optionLines

So what’s in Program.fs? Not a lot.

let doStuff (opts:OptionExampleProgramConfig) =
    let fileIsThere =System.IO.File.Exists(fst opts.inputFile.parameterValue)
    if fileIsThere = false then () else
    let keyValuesFromFile = 
        System.IO.File.ReadAllNameValueLines (fst opts.inputFile.parameterValue)
    // we want kv lines with alpha for key and number for value
    let optionLines = OptionExampleFileLines.FromKVCollection keyValuesFromFile
    let groupAndSum = optionLines 
                    |> Array.groupBy(fun x->x.Name) 
                    |> Array.sortBy fst
                    |> Array.map(fun x->
                        let sumOfGroupedData=snd x |> Array.sumBy(fun x->x.Number)
                        (fst x, sumOfGroupedData)
                        )
    groupAndSum |> Array.iter(fun x->printfn "%s=%i" (fst x) (snd x))
    ()

We check to see if the file is there. If not, we bail out. Otherwise we use out File extension to read whatever lines it can find in. Our smart constructor turns them into a strong type. Then we do a group and sum and print it out.

But wait! Couldn’t we factor out some more, end up with just a series of pipes?

Yes, we could. And we’re not. Instead we’re going to ask ourselves: where is the code at? Is it in the pipes you see in doStuff? They look kind of trivial. It’s not in the command-line stuff, that’s generic console app stuff. Where’s the code? It’s in our smart constructors.

Types are how we control pure functional programs. If the type system is solid, it doesn’t matter where your code appears or how it’s used.

We also have three files for code right now

Types.fs - where program control happens
Program.fs - where changes initially get coded before being factored out
Utils.fs - where factored out code lives before being pushed out to the rest of the system

I’ve worked on what I consider to be large codebases, and I’ve never had to add more than 3-4 files to this list. (SystemTypes, ProgramTypes, SystemUtils, ProgramUtils, Persist, Lenses, and Program). Larger, more complex systems get broken up into true microservices that share some of these files. I don’t necessarily deploy in microservices, but what I deploy in and the scaffolding I use to create and test my types are two entirely different things.

So we’re about four hours in and the code is already starting to “disappear” — get absorbed up into the various layers where it’s supposed to go. Now’s the time to think about new requirements.

The first change we need is to be able to accept a stream of stuff as input, not just a filename. This is because we’re sticking to our Unix Philosophy: if we can handle text streams, we can handle anything.

Well crap. Remember how I complained so much about moving that code out into a type extension too early? Yep. Came back and bit me. Turns out I need to take out the part that tries to make a value kv pair from the part that reads the file — I need to reuse the kv pair stuff. I also need to switch over to sequences. List and Arrays are great, but if you’re taking stuff from another program, you never know how much you have. It’s always better to process as you go. With the other formats, you’re asking .NET to load them all up first. So now our incoming code looks like this:

let lineContainsAnEquals (line:string) = line.Contains("=")
    let lineOnlyHasTwoPieces (line:string) = line.Split([|'='|]).Length=2
    let convertLinesIfPossibleToKVPair (stringSequence:string seq) =
        try
            stringSequence
                |> Seq.filter(fun x->lineContainsAnEquals x && lineOnlyHasTwoPieces x)
                |> Seq.map(fun x->
                    let split=x.Split([|'='|])
                    new System.Collections.Generic.KeyValuePair<string,string>(split.[0],split.[1]))
        with
            | :? System.Exception as ex ->Seq.empty
    type System.IO.File with
        static member ReadAllNameValueLines s =
            try
                let textLines = System.IO.File.ReadAllLines s
                convertLinesIfPossibleToKVPair textLines

And switching over to sequences and moving the logic around a bit, our doStuff code looks like this:

let doStuff (opts:OptionExampleProgramConfig) =
    let fileIsThere =System.IO.File.Exists(fst opts.inputFile.parameterValue)
    let keyValuesFromFile = 
        if fileIsThere = false 
        then 
            Seq.initInfinite (fun _ -> System.Console.ReadLine())
            |> convertLinesIfPossibleToKVPair
        else 
            System.IO.File.ReadAllNameValueLines (fst opts.inputFile.parameterValue)
    // we want kv lines with alpha for key and number for value
    let optionLines = OptionExampleFileLines.FromKVCollection keyValuesFromFile
    let groupAndSum = optionLines 
                    |> Seq.groupBy(fun x->x.Name) 
                    |> Seq.sortBy fst
                    |> Seq.map(fun x->
                        let sumOfGroupedData=snd x |> Seq.sumBy(fun x->x.Number)
                        (fst x, sumOfGroupedData)
                        )
    groupAndSum |> Seq.iter(fun x->printfn "%s=%i" (fst x) (snd x))
    ()

Does it work?

It works! Yay!

Over lunch I thought about a couple of other optimizations that make sense right now. First, now that the program can both take a stream input and provide a stream output, it’s time to declare a canonical output. We’ll do that by moving our printf to the type definition. (We’re not outputting anything, so it’s okay in the type definition. We’re simply defining a standard way it appears in string format)

type NameNumberPairType = { Name:string;Number:int} with
         override self.ToString() =
            self.Name + "=" + self.Number.ToString()

Next the “groupAndSum” chunk of code sure looks like it should be generic. I won’t go that far, but I’ll at least pull it out from doStuff.

/// Takes a name=value collection and sums by name
let groupAndSum (optionLines:seq<NameNumberPairType>) =
    optionLines
    |> Seq.groupBy(fun x->x.Name) 
    |> Seq.sortBy fst
    |> Seq.map(fun x->
        let sumOfGroupedData=snd x |> Seq.sumBy(fun x->x.Number)
        (fst x, sumOfGroupedData)
        )

Note how it was a variable assignment — and now the same basic code is a function. First-class functions for the win!

Now that we have a canonical format, what about inputting lines? We don’t have a requirement for that right now, so we wouldn’t code it. Remember YAGNI, You Ain’t Going To Need It. But for purposes of illustration — let’s assume we have one.

type NameNumberPairType = { Name:string;Number:int} with
         override self.ToString() =
            self.Name + "=" + self.Number.ToString()
         static member FromString (s:string) = 
            if lineContainsAnEquals s && lineOnlyHasTwoPieces s
                then
                    let split=s.Split([|'='|])
                    let tryParse = System.Int32.TryParse(split.[1])
                    if (fst tryParse) = false 
                        then None
                        else Some {Name=split.[0]; Number=snd tryParse}
                else None

And here’s our FromString method again, back from the dead. Also, more importantly, here’s the option type my friend said he used when using smart constructors. How does the read flow work with the FromString smart constructor?

static member FromStrings (strings:seq<string>) =
            strings |> Seq.map(fun x-> NameNumberPairType.FromString x) |> Seq.choose id

Pretty simple stuff. Note that I’m using the collection type, OptionExampleFileLines, to manage the typed OptionExampleFileLine with a NameNumberPairType inside of it. The string parsing, which has been following us around like a small puppy, finally goes home to live there, where it belongs. The NameNumberPairType can’t always succeed, but throwing an error is whack, so it just returns an option. Then we sort it all out in the Seq.choose

Everything’s private. I can only make new items through the smart constructors, and they protect the rest of the program. The “plural” format handles the collection of the line items, so it has methods to translate vectors around. There’s still more factoring to be done, but there always is, so let’s move on.

Our types code now looks like this:

type NameNumberPairType = { Name:string;Number:int} with
         override self.ToString() =
            self.Name + "=" + self.Number.ToString()
         static member FromString (s:string) = 
            if lineContainsAnEquals s && lineOnlyHasTwoPieces s
                then
                    let split=s.Split([|'='|])
                    let tryParse = System.Int32.TryParse(split.[1])
                    if (fst tryParse) = false 
                        then None
                        else Some {Name=split.[0]; Number=snd tryParse}
                else None
         member self.ToKVPair = 
            System.Collections.Generic.KeyValuePair<string,int>(self.Name, self.Number)
         static member FromKVPair 
            (pair:System.Collections.Generic.KeyValuePair<string,int>) =
                {Name=pair.Key;Number=pair.Value}
    type OptionExampleFileLine = private NameNumberPair of NameNumberPairType with
        static member FromKVPairString (kv:System.Collections.Generic.KeyValuePair<string,int>) =
            NameNumberPairType.FromKVPair kv
        static member FromNameAndNumber (name:string) (number:int) =
            {Name=name; Number=number}
    type OptionExampleFileLines = private OptionExampleFileLines of OptionExampleFileLine[] with
        static member FromStringKVCollection 
            (keyValueCollection:System.Collections.Generic.KeyValuePair<string,string> seq) =
            /// Process anything with alpha=number, ignore the rest
            let tryParsingKeyIntoAnInteger 
                (pair:System.Collections.Generic.KeyValuePair<string,string>)
                    =System.Int32.TryParse pair.Key
            let tryParsingValueIntoAnInteger 
                (pair:System.Collections.Generic.KeyValuePair<string,string>)
                    =System.Int32.TryParse pair.Value
            let doesKVWorkForUs (pair:System.Collections.Generic.KeyValuePair<string,string>) =
                (fst (tryParsingKeyIntoAnInteger pair) = false)
                && (fst (tryParsingValueIntoAnInteger pair) = true)
            let optionLines = keyValueCollection 
                            |> Seq.filter(fun x->doesKVWorkForUs x)
                            |> Seq.map(fun x->OptionExampleFileLine.FromNameAndNumber
                                                x.Key (snd (tryParsingValueIntoAnInteger x)))
            optionLines
        static member FromTypedCollection 
            (keyValueCollection:System.Collections.Generic.KeyValuePair<string,int> seq) =
                keyValueCollection |> Seq.map(fun x->NameNumberPairType.FromKVPair x)
        static member FromStrings (strings:seq<string>) =
            strings |> Seq.map(fun x-> NameNumberPairType.FromString x) |> Seq.choose id

“OptionExampleFileLine” should probably disappear, since it doesn’t look like it does much anymore. We’ll save that for our next round of cleanup.

But wait a minute, Markham! The boss yells. This is supposed to be a web page!

Oh. Needs to work in a web page? Ok boss. Is there much to making this a web page? Let’s do it and find out.

<!DOCTYPE html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta charset="utf-8" />
    <title>Harness to test NameNumber collation</title>
</head>
<body>
    <p>
        Enter your name and numbers here<br />
        Use the format NAME=VAL
    </p>
    <form id="myForm" action="http://myserver.com/cgi-bin/myapp" method="get">
        <textarea name="myInput" id="myInput"></textarea><br/>
        <button type="submit">SUBMIT</button>
    </form>
</body>
</html>

First we create a web harness. This one is actually a little fancy. It has a HEAD tag and all kinds of stuff. I’d normally just pound out an HTML tag and the form. What? No Angular? No TypeScript? No JQuery? Bootstrap? This thing looks like crap!

Yikes! Don’t tell the UI/UX guys!

Whenever I’m in a system of any complexity at all, it’s not unusual for me to have several pieces of scaffolding lying around, pieces use to prop up and test other pieces. This is one of those. All it does is take some lines entered by the user and send it off to a CGI program — our own little console app we’ve been developing.

This is because you should always separate the functionality from the look-and-feel. If you create harnesses with the same control names as your final output, you can use them to test your code — even pass acceptance tests — without having to spend your day pushing pixels around. Make it work, then make it pretty. Never try to do both at the same time.

I can’t walk you through the next step because it’s highly-dependent on the stack you’re using. But if you’re using Apache and CGI (instead of ASP.NET), you just place your console app in the right directory, then make sure the permissions are correct, then add code for reading the environment variables. Finally, you can just do a Console.WriteLine (or printfn) to send a new page back out to the client. Configuring it is more work than doing it. Maybe that’s something we can play around with in a future article.

Key question: do we need to change our types any?

We do not. We will need to add some layer 1 stuff, mainly around working with form data from a web page, but there’s no significant new functionality. It just works. And over time, if the system leads us in this direction, perhaps we build out a reusable few functions for interacting with web forms. If it’s actually a web app, then we’ll make a bunch of framework decisions like ASP.NET, Vue.js, and the rest of it. But none of that is important right now. YAGNI. We’ve got types to exercise. We can paint the Sistine Chapel once we get our paints mixed.

And now we get to monads. Monads are beyond the scope of this essay, but the question was a valid one: isn’t this just a long, roundabout way to re-create monads?

Yes, in the same way that it might be a long, roundabout way to create a web app. Are we building a duck? I don’t know. It might be a duck. Let’s stick this other foot on and see where the beak goes. It might be a dinosaur. We’ll know when we finish. I know that sucks for a lot of folks, we want to know the exact answer, but that’s the nature of this job. The more we think we know up front, the more time we end up jerking everybody around in details that turn out not to matter.

This is incremental type development. While this represents a morning and afternoon of kicking around in F#, there have been a lot of changes to our types as we’ve learned more and exercised them in code. We’ve started with something that looks ugly and it slowly begins to look more and more like tight code. That’s the way it’s supposed to work. Although functional programming is based on category theory, programming ain’t calculus. It’s messy. If you’re working with people it should be messy. You had better have a way to deal with that messiness.

So sure, maybe we end up with monads. Or not. If we do, we’ll end up there after our types are rock solid — and not before. Complex architectures can be very brittle. That’s another reason to grow them incrementally.

What about tests? I was going to add in some FsUnit tests at this point, but I ran into dependency hell so I bailed out. The key thing to realize when talking about testing using this development format is that the executable is the function. That is, we’re developing executables that look and feel just like F# functions. We pipe things to them, they pipe things to other programs. They have parameters, and aside from the output of the executable, they don’t change the state of the underlying system. Microservices FTW!

So our tests should go in two places: first, to use our FromString(s) methods to exercise our smart constructor type safety. In TDD, we probably should have done this first. Second, we should use tools in the O/S to check on the safety and operation of our executable as it runs. It’s development, it’s operations. It’s DevOps.

By developing in this way, what are the scaling limitations for this code? It should be easy to see that they’re basically limitless. I could run a million of these executables on servers all over the world and assemble the results — without much additional code at all. Scalability comes naturally. When you see this happen? And then you realize that you have dozens of different deployment options based on what kind of device you’re deploying into? You never want to go back to coding the old way. In fact you feel kind of sad for people who are struggling with a bunch of architectural cruft in other ways of coding when they don’t need to.

What about interoperability? How do you prevent the output from one program from crashing the input of another?

By sharing the code. Unless you’re building Microsoft Excel, you shouldn’t have more than a few dozen types in your system no matter how complex it is. And these types need exercising! We want them out there, hanging out with the other types, telling jokes and getting into trouble. So share the code and tests, how they read and write their canonical form, how their constructors work, and so on. You get better types. Also nothing crashes. (There’s a discussion here about versioning for another day). Your tests should exercise your types, so you always have a safety net.

Does this way of coding take longer? Hell no! It looks that way, sure. What actually happens is that the initial steps are slower, but then you and the team build up a mental library of tools as you scale out the type system. After the first week or two you’ll be ahead of other teams programming in other ways. (What usually happens to those guys is that they start off really quickly, since they have so many tools that do so many happy-path things so easily. But then they drift off the happy path, and they’re left with the worst of both world: lots of cognitive overhead from various tools suites, and a type system that’s needs a lot of work.)

I find that as long as I manage complexity by “Coding F# like a stupid person”, I can do useful F# coding as a hobby, whenever I’ve got a few hours here or there. It actually becomes more of a fun adventure than work. With more complex environments, it might take a day or two to get my head in the game and start making a difference. That kind of overhead wears on a coder and a team after a while.

Fun times. Now that we’ve got this out of the way, next time we chat we can build on this and go exploring somewhere else. Maybe web forms, maybe asynch, maybe WinForms, maybe monadic parsers. Who knows? Adventure awaits.

(There’s a question about whether this works with event storming and/or DDD. I don’t see a conflict at all — aside from workflow. Event storming and DDD should drive a backlog, which drives incremental type development. In this fashion things can change and the workflow still keep humming along, i.e. continuous flow.)


I hate to do a crummy commercial, but this essay series is huge and I really haven’t explained some critical things. If you’re interested in why I choose the approach of looking at F# coding in terms of behavior, value, and flow instead of types, you should read my Info-Ops book. Programming is one of many forms of project information. Once you learn how to organize all of your project information, not only will you be a better programmer, your paperwork and BS reports and meetings will decrease. Plus you’ll have more fun and make better stuff.

Daniel Markham sucks at programming, but he still loves it. What he’s good at is helping groups of technical people make stuff people want. He teaches them to do that using an understanding of value creation, maximizing time with users, minimal tool and paperwork overhead, and good technical practices, including things like ATDD and TDD. He’s been a fan of F# since it first came out.

July 18, 2018  Leave a comment

Failure Is Not An Option, The Sequel

(This is the second in a three-part series about how I think and the workflow I use to code F# programs.)

Last time we covered options, the entryway into any functional program. We talked about a bunch of other stuff, the things that lead us to develop software one way or another. But we really didn’t get to the “end”.

And that’s been buggin’ me. So let’s keep going.

So far we just do some housekeeping and load in a string array. If you look at our first “Application Type”, you can see that:

type OptionExampleFileLines = string[]

Now let’s go about the process of actually making something. In TDD, the first thing we do is write the smallest valuable test we could. The tests drive out the design. In FP, the first thing we do is write the smallest valuable type constraint we can. The type system makes the code work correctly no matter how it’s assembled. (Total Functional Programming. Remember?)

We know we only want pairs of alpha=number in our incoming data. So let’s add an item type with a smart constructor that only takes that.

type NameNumberPairType = { Name:string;Number:int}
type OptionExampleFileLine = private NameNumberPair of NameNumberPairType with
    /// Takes a string in the form of <name>=<numericvalue>
    /// And makes into a strong type
    static member FromString (s:string) =
        try
            let stringSplit = s.Split([|'='|])
            let name = stringSplit.[0]
            let numberString = stringSplit.[2]
            let numberVal = System.Int32.Parse numberString
            {Name=name; Number=numberVal}
        with | :? System.Exception as ex->
            failwith "This should never happen"
type OptionExampleFileLines = OptionExampleFileLine[]

Note that I just slam whatever I’ve got into the field member variables, not caring if the conversions are going to work or not. Why is that? It’s because this Smart Constructor’s job (This is called a Smart Constructor, by the way) is to protect the type, not the protect the program. It may sound like the same thing, but it’s not.

Reading a file, eliminating empty lines, sorting out only those lines in the name=value format? Those are generic functions. That’s stuff I can use over and over again. They have no business in my type constructor. My type constructor should take that — and only that. But it shouldn’t be a file or text processor. (In fact, maybe I should change the constructor to take the pairs. But it’s good enough for now.)

Ah crap. Now that I’ve said it, I gotta code it.

/// Takes a name and value
    /// And makes into a strong type
    static member FromNameAndNumberValue (name:string) (numberVal:int) =
        try
            {Name=name; Number=numberVal}
        with | :? System.Exception as ex->
            failwith "This should never happen"

Ok. So now we’re beginning to define our type and build an impregnable fort around it. We’ve built out our spec a bit. Let’s drop back into level 2 of our onion and wire things up a bit.

let filterFileStringListIntoLinesWithNameEqualValueFormat fileName =
    try
        let textLines = System.IO.File.ReadAllLines fileName
        let textLinesWithoutEquals = textLines |> Array.filter(fun x->
            x.Contains("=")
            )
        let textLinesWithEqualsAndWithoutAValueOnTheEnd = textLinesWithoutEquals |> Array.filter(fun x->
            let splitText = x.Split([|'='|])
            splitText.Length<3
            || fst (System.Int64.TryParse splitText.[2]) = false
            )
        textLinesWithEqualsAndWithoutAValueOnTheEnd
    with
        | :? System.Exception as ex ->
            printf "I am loading the file to process. I should never fail here, just return an empty array"
            [||]
 
let doStuff (opts:OptionExampleProgramConfig) =
    let linesToProcess=filterFileStringListIntoLinesWithNameEqualValueFormat (fst opts.inputFile.parameterValue)
    let optionLines = linesToProcess |> Array.map(fun x->OptionExampleFileLine.FromString x)
    ()

I’ve renamed my function to indicate that it’s job is to process files into line arrays. It’s in the outer layer of the onion, remember? I’ve also added a line in doStuff to make the correct, typed array. The only thing left is just to do the work of collating and summing. Now we get to do some cool pipe stuff.

let doStuff (opts:OptionExampleProgramConfig) =
    let linesToProcess=filterFileStringListIntoLinesWithNameEqualValueFormat (fst opts.inputFile.parameterValue)
    let optionLines = linesToProcess |> Array.map(fun x->OptionExampleFileLine.FromString x)
    let groupAndSum = optionLines 
                    |> Array.groupBy(fun x->x.Name) 
                    |> Array.sortBy fst
                    |> Array.map(fun x->
                        let sumOfGroupedData=snd x |> Array.sumBy(fun x->x.Number)
                        (fst x, sumOfGroupedData)
                        )
    ()

I always love doing pipes — and well-constructed type system can turn your entire program into just a bunch of pipes. That’s really cool. I’m always amazed at how pure functional code just “collapses” into very small but powerful bits. It’s amazing.

But I also like playing with ferrets. I don’t let it get in the way of making things people want. In this case, I was able to hack out the pipes using only one temporary variable, “sumOfGroupedData”. You might be able to do better than that, or you might need more temporary variables. Either way is fine. This is new code you’re going to have to look at over and over again. As long as you can come back in three months and make sense of it? Do what works for you. (Also it needs to easily make sense to other folks on your team.)

The crazy thing about FP is that I might be able to hack the entire thing out with some clever types and pipes — all in a single pass.

But they do not pay me to be clever. They pay me to make things people want. And people don’t want code that’s clever. The maintenance costs are too high. That doesn’t mean I shouldn’t simplify, collapse things, tidy up, eliminate complexity, or remove unnecessary detail. I just do that incrementally, not all at once.

Cool. We’ve added enough of the onion to get to the cool bits that actually do the work. Yay! Took 2-3 hours to set the dang thing up this morning, during which I wrote the first essay. It’s taken 2 hours this evening for me to close out the first round of the “how do I make stuff” workflow.

And now an Agile question. I work with teams all the time to get their backlogs set up and help them get rocking on making cool software. One of the things I tell them is that they should code as if their funding might be cut off at any moment. I also tell them that their user stories shouldn’t take them more than a couple of hours or maybe half-a-day to complete.

They smile at me, but I can tell that many times they do not believe me. How can that be? We’re writing FizzlePop 3.0 here, bub! It’s complex. It’s intricate. It’s connected to everything else. We must deliver it all at once or nothing at all. It simply doesn’t work any other way!

Ah, but yes it does. We’ve been doing it. That’s the entire point of what we’re doing here: incrementally building stuff without making a mess. By concentrating on flow, the onion, Unix Philosophy, and Total Programming, the rest of it works out.

Let’s say we run out of money. What of value have we done?

Ok. Here you go boss.

groupAndSum |> Array.iter(fun x->printfn "Name: %s Total: %i" (fst x) (snd x))

And we’ll test it.

Ha! It crashed and burned. I step through the debugger. The first problem is that it can’t find the file. That’s some simple .NET System.IO stuff. I also had my option parameter backwards. Instead of “None”, it should have returned Some FilInfo for the file to be processed. That way there’s no work to do with IO later on, deeper in the code. So I fixed those two things.

let defaultFullFileName = System.IO.Path.Combine([|System.AppDomain.CurrentDomain.BaseDirectory; "OptionEssayExampleFile.txt"|])
let defaultInputFile = 
    createNewConfigEntry "I" "Input File (Optional)" 
        [|"/I:<filename> -> full name of the file to use for input."|]
        (defaultFullFileName, Some(System.IO.FileInfo(defaultFullFileName)))

Then my logic was messed up down at the bottom. I added variables to make it clear and fixed it

[<EntryPoint>]
let main argv = 
    try
        let opts = loadConfigFromCommandLine argv                
        commandLinePrintWhileEnter opts.configBase (opts.printThis)
        let fullFileName=fst opts.inputFile.parameterValue
        let fileIsThere =System.IO.File.Exists(fullFileName)
        let theFileOptionHasAValue=(snd opts.inputFile.parameterValue).IsSome
        let inputFileExists = theFileOptionHasAValue && fileIsThere
        if inputFileExists
         then
            (doStuff opts)
            0
         else
            0

Note that all of that crap shouldn’t be down in Main(). The pattern here is to code a little bit in doStuff, then factor it out into my layers. We’ll leave it in main for now.

Crashed again. This time it was an off-by-one problem in the string array indexing. I made that mistake in a couple of places, the filter on the incoming data and the smart constructor.

let numberString = stringSplit.[1]

There we go

For my TDD and Domain-Driven friends, I hear you all screaming right now. What the heck, Daniel! This is just amateur slash-and-burn coding! This is an awful, terrible, no-good, bad example to be setting to show people how to code! You are a bad teacher and should be sent into the forest with only a slide rule and COBOL compiler!

Yes. And No.

The goal at first is to pull a “skinny thread” through the system. This is now complete. I have a framework and a flow that returns some kind of value. Nothing from here on out will be added without first creating a safety net of either tests or types. And we will continue to tidy up as we go along. Fear not, my Total Programming friends! I am on your side. I am also on the side of my TDD friends. Peace be unto you! The only place any of us might differ is in the workflow I’m using. I’m teaching emergent design using both Total Programming and TDD in pure FP. I apologize if it looks unseemly, but this is a tour inside my brain and dang, it gets ugly in there at times! There’s also some testing we’ll do in a way I’ve bet you’ve never thought about. But that’s for later.

But I have not lost the faith, nor strayed from the One True Way.

At this time, if I were writing a production app, I’d be standing up the DevOps pipeline too. I’d be trying to pull a skinny thread through it, from code to production. And it would feel the same way. Environment stand-ups always feel clunky for some reason. You shouldn’t get stuck, but there’s always a bit of churn as you orient yourself to the particulars of where you are. Every project is different. You’re supposed to do some serious thinking (and perhaps thrashing) as you stand up the walls, build-out the roof, turn the lights on, and start making music.

But once you have a place to work, the workflow here is important. We never write code that does anything without first writing code that makes sure we’re doing the right thing. Our preference is to write types. Never test for a constraint that a type will handle for you. But if we can’t figure out how to stick it in the type system, we’ll write a test first. Then we write the code. Always.

When we wrote the smart constructor, we locked-in that our app would always work. Delivering the data to that constructor, the first layer of our onion, is a different matter entirely. It failed because of a few dumb-programmer errors. But the type caught it. Just like a test would.

The only thing I would add to the TDD workflow is that I’m in no hurry to refactor and clean things up. That’s because I don’t know where anything goes until I use it a lot. I find that if I make initial decisions about creating modules and classes and such? All I’m doing is adding cognitive overhead and sometimes creating a mental model that doesn’t match the actual problem I’m solving. Instead it just matches what I think is cool. That’s no good. So I’m not afraid of leaving things lying about until I figure out where they should live. Remember, I went for a long time before I finally realized I had a real, live reusable class. It had to be proven to me through reusing the same code in groups over several programs. I’ll write tight code. Yep. I just won’t start organizing it until the problem drives out what the best organization should be.

Let’s clean up a bit. That stuff in Main bugs me, the incoming data flow bugs me, and I don’t like the printed output. Let me fix the output first.

groupAndSum |> Array.iter(fun x->printfn "%s=%i" (fst x) (snd x))

Why’s that better? Because now the output of the program can also serve as the input. It’s the same stuff, only in a summary form. It should have the same format. Next let’s tighten up the incoming data flow.

type System.IO.File with
    static member ReadAllNameValueLines s =
        try
            let textLines = System.IO.File.ReadAllLines s
            let lineContainsAnEquals (line:string) = line.Contains("=")
            let lineOnlyHasTwoPieces (line:string) = line.Split([|'='|]).Length=2
            textLines
                |> Array.filter(fun x->lineContainsAnEquals x && lineOnlyHasTwoPieces x)
                |> Array.map(fun x->
                    let split=x.Split([|'='|])
                    new System.Collections.Generic.KeyValuePair<string,string>(split.[0],split.[1]))
        with
            | :? System.Exception as ex ->[||]

Now it’s starting to look like the F# code you might see in a textbook. I broke my own rule and moved this out to the .NET type system just to show that this work is totally about the generic aspect of moving data around, layer 1. In real code, I’d write a function, I’d re-use it, and over time it would naturally migrate out to be a type extension. It really doesn’t belong anywhere else. I also return a key-value pair, which is the generic thing we’re looking for in the file.

Exceptions should be surprises, not stuff you expect. Using the onion metaphor, you end up with two kinds of surprises. On the outside of the onion, where it touches the world, I usually eat exceptions. So what if there’s no file? Just return an empty array. It’s nothing to blow up about. (Missing data downstream is another matter entirely, but that’s a matter for another kind of test.) “Inside” the system, however, if something goes wrong I want to blow up. That’s because I’m supposed to be locking the system down so that it never blows up.

Now what? Could I add more methods to handle file sections? Strongly type the data pairs coming in from the file? Add some related methods to System.IO.File?

Yikes no! This is more of that top-down, structural thinking happening. It’s almost impossible to get rid of. I’ll add a piece of structure in one place, then suddenly I have an idea about four other places that also need structure added to be “correct”. I’ve changed from focusing on data flow that creates value to an architecture astronaut, determining where I’d like the piano and couch in this code. Maybe moving it around a bit until it all just looks right.

Let’s not do that. Instead, let’s continue the cleanup. The function “filterFileStringListIntoLinesWithNameEqualValueFormat”, which won the award for longest function name ever, goes away completely now. It’s all handled in that type extension. Now that we’ve cleaned up the outer layer to return a key-value pair of strings, our doStuff method changes to handle the new incoming data. (We changed a type. That change then causes us to change the code.)

let doStuff (opts:OptionExampleProgramConfig) =
    let keyValuesFromFile = System.IO.File.ReadAllNameValueLines (fst opts.inputFile.parameterValue)
    // we want kv lines with alpha for key and number for value
    let doesKVWorkForUs (pair:System.Collections.Generic.KeyValuePair<string,string>) =
        (fst (System.Int32.TryParse pair.Key) = false) && (fst (System.Int32.TryParse pair.Value) = true)
    let optionLines = keyValuesFromFile 
                    |> Array.filter(fun x->doesKVWorkForUs x)
                    |> Array.map(fun x->OptionExampleFileLine.FromKVPairStrings x)
    let groupAndSum = optionLines 
                    |> Array.groupBy(fun x->x.Name) 
                    |> Array.sortBy fst
                    |> Array.map(fun x->
                        let sumOfGroupedData=snd x |> Array.sumBy(fun x->x.Number)
                        (fst x, sumOfGroupedData)
                        )
    groupAndSum |> Array.iter(fun x->printfn "%s=%i" (fst x) (snd x))
    ()

That stupid “FromString” smart constructor goes away. What was I thinking? The constructors look like this now.

type NameNumberPairType = { Name:string;Number:int}
type OptionExampleFileLine = private NameNumberPair of NameNumberPairType with
    /// Takes a name and value and makes into a strong type
    static member FromNameAndNumberValue (name:string) (numberVal:int) =
        try
            {Name=name; Number=numberVal}
        with | :? System.Exception as ex->
            failwith "This should never happen"
    /// Send it a KV pair with a name and a number (both strings) makes into a strong type
    static member FromKVPairStrings (kv:System.Collections.Generic.KeyValuePair<string,string>) =
        try
            {Name=kv.Key; Number=System.Int32.Parse kv.Value}
        with | :? System.Exception as ex->
            failwith "This should never happen"
type OptionExampleFileLines = OptionExampleFileLine[]

Should I tighten those constructors up even more? Surely there’s more optimization to be done here, right?

Yes, but the key thing is that the constructors continue to protect the type. No matter what they look like, or how ugly they are, they’re doing their job of protecting the type. So we can clean them up more at our leisure. Premature refactoring is a smell, at least for me. Functional Programming ain’t OO.

Oh yeah. I forgot about the clutter in main. Tuns out I was over-checking stuff. All I needed to check was whether or not the file existed — and that should go into doStuff, not main. If the file doesn’t exist, it’s application-level decision what to do, and main is a layer 1 interface to the OS. Remember the pattern is to code in doStuff, then factor out, then clean up — adjusting types and/or testing before any change in behavior.

Finally we move all of that command-line crapola and our application type out to a Types module. Why? Because all of that is truly reusable. I’ll be doing command line stuff as long as I’m writing F# code. That should be written and forgotten. The application type, “OptionExampleFileLine”, is really where all of the important stuff goes — but it’s also supposed to be completely reusable. That’s the beautiful thing about Total Programming with a strong type system. I should be able to create a type that never fails or hangs, then I can put that type anywhere. Over time I can test the heck out of it in various scenarios and applications — and the code always goes right there, in Types, to be reused by anybody who needs it.

We’re not really writing a program. This console thing we’re doing is a prop. What we’re doing is beginning to develop a rock-solid type system for this problem that can be reused wherever we need it.

I’ve put the final code below. doStuff has about a dozen lines of code which feels about right. I find in true microservices that follow the Unix Philosophy you don’t end up with a lot of code. Most of it is reusable boilerplate stuff. Right now, doStuff it is the true layer 3 of our onion. We’ll continue to add level 3 stuff there, then push the generic part of the work into layers 4 and 5 and the IO part of the work to layers 1 and 2. This will keep doStuff small and our library of resuable code more and more useful (and tested).

Next time we’ll talk about adding functionality and moving to a new platform. We’ll do some TDD. One of the many things I love about this kind of coding is how you can write once and use everywhere.

Once you get the math part of dealing with functions as first-class citizens, and if you’re willing to suffer through the pain of getting the freaking whitespace wrong and trying to figure that out, F# really isn’t that tough. In fact, it’s fun. When you add in the philosophical stuff I’ve talked about in these essays and follow that, the code really writes itself. It’s been a very long time since I’ve worked in such an easy, fun environment where the compiled code mostly just works. It may sound weird, but I don’t consider programming in F# to be work. Instead it’s like putting a puzzle together. Fun and relaxing.

let doStuff (opts:OptionExampleProgramConfig) =
    let fileIsThere =System.IO.File.Exists(fst opts.inputFile.parameterValue)
    if fileIsThere = false then () else
    let keyValuesFromFile = System.IO.File.ReadAllNameValueLines (fst opts.inputFile.parameterValue)
    // we want kv lines with alpha for key and number for value
    let doesKVWorkForUs (pair:System.Collections.Generic.KeyValuePair<string,string>) =
        (fst (System.Int32.TryParse pair.Key) = false) && (fst (System.Int32.TryParse pair.Value) = true)
    let optionLines = keyValuesFromFile 
                    |> Array.filter(fun x->doesKVWorkForUs x)
                    |> Array.map(fun x->OptionExampleFileLine.FromKVPairStrings x)
    let groupAndSum = optionLines 
                    |> Array.groupBy(fun x->x.Name) 
                    |> Array.sortBy fst
                    |> Array.map(fun x->
                        let sumOfGroupedData=snd x |> Array.sumBy(fun x->x.Number)
                        (fst x, sumOfGroupedData)
                        )
    groupAndSum |> Array.iter(fun x->printfn "%s=%i" (fst x) (snd x))
    ()
 
[<EntryPoint>]
let main argv = 
    try
        let opts = loadConfigFromCommandLine argv                
        commandLinePrintWhileEnter opts.configBase (opts.printThis)
        doStuff opts
        0
    with
        | :? UserNeedsHelp as hex ->
            printfn "%s: %s" defaultBaseOptions.programName hex.Data0
            printfn "========================"
            printfn "Command Line Options:"
            // Manually list program config entries here 
            defaultInputFile.printHelp
            0
        | :? System.Exception as ex ->
            System.Console.WriteLine ("Program terminated abnormally " + ex.Message)
            System.Console.WriteLine (ex.StackTrace)
            if ex.InnerException = null
                then
                    1
                else
                    System.Console.WriteLine("---   Inner Exception   ---")
                    System.Console.WriteLine (ex.InnerException.Message)
                    System.Console.WriteLine (ex.InnerException.StackTrace)
                    1

I hate to do a crummy commercial, but this essay series is huge and I really haven’t explained some critical things. If you’re interested in why I choose the approach of looking at F# coding in terms of behavior, value, and flow instead of types, you should read my Info-Ops book. Programming is one of many forms of project information. Once you learn how to organize all of your project information, not only will you be a better programmer, your paperwork and BS reports and meetings will decrease. Plus you’ll have more fun and make better stuff.

Daniel Markham sucks at programming, but he still loves it. What he’s good at is helping groups of technical people make stuff people want. He teaches them to do that using an understanding of value creation, maximizing time with users, minimal tool and paperwork overhead, and good technical practices, including things like ATDD and TDD. He’s been a fan of F# since it first came out.

July 13, 2018  Leave a comment

Failure Is Not An Option (In F#)

Why all the hate on Option types? You said you wouldn’t pass them from a function. Why not? What’s wrong with options?[1]

(This is the first in a three-part series about how I think and the workflow I use to code F# programs.)

There’s nothing wrong with options. I love options. I use options all of the time. Options are my friend.

I just use them in the right place. Understanding where the right place is? That’s the purpose of this essay.

But first we gotta talk about the Unix Philosophy and Total Programming. Or at least talk enough about them that the option stuff makes sense. Warning: I am going to vastly over-simplify a bunch of stuff so that this essay isn’t 40,000 words long.

When I started writing true microservices, I learned a lot of things I wouldn’t have learned otherwise. Microservices need to run at a certain time. They need to work with one another. They need to use the same data types and storage/transfer mechanism. (Some folks use a database for this but there are all kinds of problems with that. I will not go into them here.) Heck, they need to stop running — no matter what happens.

Most of the things I learned rested on a weak version of the Unix Philosophy and Total Programming. Instead of my trying to convince you that the things I learned were good, I’ll just point you to those.

The Unix Philosophy. The usual summary of the Unix Philosophy goes like this:

  • Write programs that do one thing and do it well.
  • Write programs to work together.
  • Write programs to handle text streams, because that is a universal interface.

There’s a lot more in the linked article. I have a much simpler working definition: Make it work like the unix command “ls”. That is, when I type “ls” into a linux prompt? It just works. I can’t think of any time it’s failed. It always does something. And I can join ls together with a bunch of other linux commands to do useful things. It’s just one program, but when combined with other programs it becomes tremendously more valuable than it is by itself. It does one thing, does it well, never fails, and infinitely connects to other programs to make more useful stuff than I could have imagined while building it.

That’s what I want in my code. Tiny pieces of rock-solid stuff that I can assemble later into various things I might need without having to write (much) code.

Total Programming. This is another rabbit hole you can dive down if you have a lot of time on your hands. The dumbed-down version goes like this: your program has to provably stop running at some point. When you start it up, you have to know — without a doubt — that it is going to stop. Once again, it doesn’t have to do anything. It just can’t hang.

This spins off into all kinds of type and category theory stuff. I am not a Computer Science person. I’m just some old fart that likes to code, so the dumb version is enough for me. There looks like some cool stuff there, though, if a person wanted to study up on it. Where you end up is mathematically creating a type system that is deterministic. Put differently, you stick all the rules, flows, validation, and the rest of it into the type system so that it is impossible for the program not to complete in some way. That’s where we’re headed. (But that’s not where we’re starting)

We’re are not laser-focused on the type system. Instead, like we said in the last essay, we’re always focusing on output. Behavior. Flow. What are you doing for me? Not structure. Structure is always derivative — but that’s material from the Info-Ops book and too much to go into here. For now, if we can remember never to focus on structure, to instead do things that “force” ourselves to create structure based on other constraints? We’ll go far.

But wait! What the heck are you going on about, Daniel?!? Console apps? Text streams? I don’t want to write no stinking console apps! Linux commands? What the heck? I’m doing modern web development. It’s not the dark ages anymore. This command-line crap doesn’t look like it has anything at all to do with my day-to-day work. What’s next, building our own C compiler out of coconuts?

It’s a fair point, but misguided. The purpose is to structure your code as if it were a command-line app. Not that it’s deployed that way. I would even make it work as a command-line app for testing purposes, no matter where it went. This philosophy, along with the onion, will tell you where everything goes, no matter what kind of architecture you’re using. Like we said in the last essay. If you’re doing it right, the architecture doesn’t matter. Carpenters don’t spend all day staring at their hammer.

Remember the onion?

This is not about command-line apps. It’s about how good programs are constructed. How the pieces go to live in various places is not relevant here. In fact, if it’s relevant, you’re probably focusing on the wrong thing. The command-line just provides the simplest and easiest-to-use first platform for the code to live on. I can take an appropriately-structured program from the command-line and run it anywhere, on dozens of platforms. All I need to do is add some shell code, the grunt-work stuff in layer 1. (Which ends up being reusable. Yay!) And if I can’t? Then I haven’t structured the program correctly.

This is the entire point of programming, right? Write stuff once, then use it in a bunch of places. I’m lazy. I want to get the maximum value for the minimum amount of work. Don’t you?

Let’s make up some dummy example to walk through this. Let’s say I have a text file that’s supposed to have lines where there’s a value which equals something, like this:

Our data

The job is simple. Group together things by letter and total up the numbers. Then display the results to the console The first thing I do is create a program that takes one command-line parameter for an input file. I’ll use that boilerplate stuff I wrote years ago.

open Utils
/// Command-line parameters for this particular (OptionExample) program
[<NoComparison>]
type OptionExampleProgramConfig =
    {
        configBase:ConfigBase
        inputFile:ConfigEntry<FileParm>
    }
    member this.printThis() =
        printfn "OptionExample Parameters Provided"
        this.configBase.verbose.printVal
        this.inputFile.printVal
let programHelp = [|"This is an example program for talking about option types."|]
let defaultBaseOptions = createNewBaseOptions "optionExample" "Does some thing with some stuff" programHelp defaultVerbosity
let defaultInputFile = 
    createNewConfigEntry "I" "Input File (Optional)" 
        [|"/I:<filename> -> full name of the file to use for input."|]
        ("OptionEssayExampleFile.txt", Option<System.IO.FileInfo>.None)
let loadConfigFromCommandLine (args:string []):OptionExampleProgramConfig =
    if args.Length>0 && (args.[0]="?"||args.[0]="/?"||args.[0]="-?"||args.[0]="--?"||args.[0]="help"||args.[0]="/help"||args.[0]="-help"||args.[0]="--help"then raise (UserNeedsHelp args.[0]) else
    let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
    let newConfigBase = {defaultBaseOptions with verbose=newVerbosity}
    let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
    let newInputFile = ConfigEntry<_>.populateValueFromCommandLine(defaultInputFile, args)
    {configBase = newConfigBase; inputFile=newInputFile}
 
let doStuff (opts:OptionExampleProgramConfig) =
    ()
[<EntryPoint>]
let main argv = 
    try
        let opts = loadConfigFromCommandLine argv                
        commandLinePrintWhileEnter opts.configBase (opts.printThis)
        doStuff opts
        0 // remember to return an integer exit code
    with
        | :? UserNeedsHelp as hex ->
            printfn "%s: %s" defaultBaseOptions.programName hex.Data0
            printfn "========================"
            printfn "Command Line Options:"
            // Manually list program config entries here 
            defaultInputFile.printHelp
            0
        | :? System.Exception as ex ->
            System.Console.WriteLine ("Program terminated abnormally " + ex.Message)
            System.Console.WriteLine (ex.StackTrace)
            if ex.InnerException = null
                then
                    1
                else
                    System.Console.WriteLine("---   Inner Exception   ---")
                    System.Console.WriteLine (ex.InnerException.Message)
                    System.Console.WriteLine (ex.InnerException.StackTrace)
                    1

This took me an hour! Why? Is my toolkit overblown? Do I not know anything at all about coding? Am I a moron? (Please don’t answer that last question). No. I had indented the first line one way and the rest of the code another. It took me 10-20 minutes to put the code in. The next 40 minutes I spent trying to figure out why my code wasn’t working.

My code was working. The spacing was off. Sometimes I say very unkind things about F#.

Looks like it’s working

And look! We’re not even a third of the way into the boilerplate code and there’s an Option type. It’s this line:

("OptionEssayExampleFile.txt", Option.None)

Why do I need an option when all I’m doing is getting the input file? Because the input file might not exist. Why not just fail? Because that’s not the job of this code. This code just gets command-line parameters for whatever programs might need them. The programs themselves may fail — or not. That’s a program decision, not a decision for this library.

I’m working the outside of the onion, the part where my application touches the rest of the world. The rest of the world has unknowns and empty values! So the option type accurately reflects what might happen when I interact.

What’s next? Well, I have my parameters loading up, and I know I’m working from the outside of the onion inwards. What if the input file doesn’t exist?

That’s a decision that’s not part of the outer layer. It’s part of layer 2. At this point I consume the option and decide what I want to do. Maybe I provide default data. Maybe I just go away. It varies — so it’s not an outside layer question.

let inputFileDoesntExist = 
            (snd opts.inputFile.parameterValue).IsNone
            || (System.IO.File.Exists(fst opts.inputFile.parameterValue) = false)
        if inputFileDoesntExist
         then
            0
         else
            (doStuff opts)
            0

Now I’m transitioning from the nature of the outside world to how I want my program to run. And I want my program to run like a clock. No muss, no fuss. When I type in “ls” on the linux command line, it works. That’s what I want.

I’ve found that allowing option types past level 2 is basically a way of deferring important decisions until I’m in the middle of doing something else. This complicates things and is always a bad idea. I’m working on this other thing. Why the heck should I be concerned right now about whether the file is there or not? You get four or five option types floating around? They could take a simple five-line method and turn it into a 40-line logic monstrosity. That’s no fun. And it leads to crappy, muddled code with mixed responsibilities.

What’s the next step? Well I’m not going to do anything if there’s no file. What if there’s a file with spaces? Or bad lines? Or lines without the name-equals-number format?

Now I’m fully in level 2. I have successfully interacted with the outside world. I have some hunk of stuff in my hand that I have to do something with. Now I need to decide on how to clean, filter, sort, or replace data I don’t like. I’m transforming the outside world data into my application data.

There’s no right or wrong answer here. It’s up to you and your app. But you have to decide. Once we leave level 2, failure is not an option. That is, you only have stuff that you know you can process. So let’s add a little more code around our “doStuff” function. (It’s very important to use names that describe things. Ha!)

type OptionExampleFileLines = string[]
let makeStringListToProcess fileName  :OptionExampleFileLines=
    try
        let textLines = System.IO.File.ReadAllLines fileName
        let textLinesWithoutEquals = textLines |> Array.filter(fun x->
            x.Contains("=")
            )
        let textLinesWithEqualsAndWithoutAValueOnTheEnd = textLinesWithoutEquals |> Array.filter(fun x->
            let splitText = x.Split([|'='|])
            splitText.Length<3
            || fst (System.Int64.TryParse splitText.[2]) = false
            )
        textLinesWithEqualsAndWithoutAValueOnTheEnd
    with
        | :? System.Exception as ex ->
            printf "I am loading the file to process. I should never fail here, just return an empty array"
            [||]
let doStuff (opts:OptionExampleProgramConfig) =
    let linesToProcess=makeStringListToProcess (fst opts.inputFile.parameterValue)
    ()

A few things to notice. First, I’ve added a type,OptionExampleFileLines. It doesn’t have a lot around it, but the day is still young. We’re just getting started. At level two we’re translating into our application types — so we need application types.

When I mentioned I was going to write about option types today, a friend said “I use them in smart constructors”

Smart constructors are a way to control how types are created so that you have more control over being sure the type isn’t going to blow up later on. (Apologies if I missed some details here.)

The crazy thing is, we’re saying the same thing. My friend is saying, “Look! You can make constructors such that you always know your types will run on your application. You have tight control, and by adding it to the type system you’re creating a program that cannot fail. Whereas I am saying, “Look! Once you begin interacting with the outside world, you’ll get messy things like null values and bad data. The first thing you have to do is add code to make sure your program cannot blow up.”

This is one of these things where you could end up violently agreeing. What you have to know is 1) It’s the same goal, and 2) You don’t have to choose one or the other. In fact, use both! There’s cleaning data to protect the program from the outside world, and then there’s cleaning data to protect the type system from bad data. Write some code to clean the data in general, then figure out where it should go.

After all, once I leave level 2, I want strongly typed data that I know won’t blow up. We’re headed the same way, the only difference is that my friend is looking at it from a type perspective and I’m looking at it from a data flow perspective. Remember! I’m always focusing on output. What are you doing for me? That drives structure, not the other way around.

The other thing to notice are my huge names and wordy code. Couldn’t I collapse that? Isn’t functional programming always supposed to look like “foo |> bar |> foobar”?

Yes and no. Functional programming can look dang near like anything you want it to. The computer doesn’t care. The important thing is whether or not you can look at a piece of code and immediately understand what it’s doing. When I visited my TDD guru friends Bob and James, one of the problems I noticed was that as good programmers, they’d almost immediately start refactoring, collapsing stuff, making the code cleaner.

That was a bad idea, because with FP, it all kinda collapses into nothingness. I end up losing track of what I’m doing. What’s the code supposed to do for people? Instead I’m taking some function and making it disappear. (It’s a nice trick. It just doesn’t help me reason about usefulness or not. Instead I’m wallowing around in how cool FP is.) Could I take that “makeStringListToProcess” code and make smaller? How about adding the program type right there, have it return a a name/value collection? Move it off to a generic function that takes any file and only returns the name/value parts of it?

The collapsing/refactoring game can go on almost forever, pushing both outward towards generic IO functions, inwards towards new language additions, and up the type chain to a more structure program type system.

These are all wonderful and great things, and I’ll be pushing the hell out of this code — once it starts doing something useful. Then I’ll use what it has to do (that’s useful) as a guideline for what to clean up first and how to clean it up. (Pushing as much as possible into the type system before you start is how you get Domain-Driven coding). Until then, however, I want to read what I’m doing in nice, human language. I especially want to think through all the outer onion issues around process data. I am a distracted, busy, forgetful, lazy programmer. A month from now I don’t want to load up the IDE and see something that looks like Klingon. Instead, I’ll refactor as I go and over time it all works out the same.

Finally, why not carry options into layer 3? What if I wanted to take the lines that had non-numeric values on them and output them to another file? What if I wanted to write a report for numeric entries and another for alpha entries?

This is where I got hung up a lot as an OO guy. What I was doing was focusing on the structure instead of the behavior. “I have this structure to read these files I want to do four or five things with. So I’ll keep the structure and just add in branches to do the other stuff.”

Nope nope nope nope nope. Remember the Unix Philosophy. “Write programs that do one thing and do it well” What I was doing was trying to be lazy and force re-use by taking the same structure and making it do multiple things. A divided house cannot stand, and it’s enough to do one thing and do it well. Then do the next thing.

And surprise! We get re-use, just like we wanted! We just get it by using shared libraries that we develop over time. In fact, I have never seen code resuability work so well as I have using F#, and each time I re-use it, it keeps getting better, because I developing and factoring various functions based on several real-world behaviors they have to support.

That’s extremely cool.

Option types are great, and there are multiple ways of looking at programming that are all valid: type-driven, flow, constraint-driven, test-driven, and so on. At the end of the day, however, everybody’s trying to do the same thing. So when you see somebody doing something one way talk about something, you can betcha the same thing happens when you’re doing it another way. Coding is coding. The important thing is not to forget any of this important stuff simply because you’ve decided to use one method of coding over another.

Most of all, have fun! And make stuff people want.


I hate to do a crummy commercial, but this essay is already huge and I really haven’t explained some critical things. If you’re interested in why I choose the approach of looking at F# coding in terms of behavior, value, and flow instead of types, you should read my Info-Ops book. Programming is one of many forms of project information. Once you learn how to organize all of your project information, not only will you be a better programmer, your paperwork and BS reports and meetings will decrease. Plus you’ll have more fun and make better stuff.

Daniel Markham sucks at programming, but he still loves it. What he’s good at is helping groups of technical people make stuff people want. He teaches them to do that using an understanding of value creation, maximizing time with users, minimal tool and paperwork overhead, and good technical practices, including things like ATDD and TDD. He’s been a fan of F# since it first came out.

1: This is paraphrased from a comment on a previous essay.

July 12, 2018  Leave a comment

Project Management Charts I Have Known

This is an elephant

I love charts and graphs. I remember one of the coolest programs I had for DOS was “Harvard Graphics”, which just created graphs. Fun times.

But then I became an independent business owner writing software for people. And then a contractor, tech lead, architect, and architect lead. Finally, I was doing some technical PM work! Yay! More graphs! Back then, you weren’t nothin’ if you didn’t have a GANTT chart to show somebody.

Anybody remember what GANTT stands for? (Nothing. It’s named after Henry L. Gantt.)

GANTT charts were really all I needed until people started wanting to do “formal” Agile — that’s Agile with all the names, roles, rituals, and such. (Back before agile was Agile, everybody worked, everybody helped one another, and we had stuff we promised the client we would do every week. Obviously this was too simple?)

Once Agile came along, everybody was talking about Scrum, Sprints, and Burn-down charts. Who can forget this guy?

An old friend, the sprint burndown

The burndown worked on a simple yet brilliant principle: if every day everybody guesses how much they have left, you can spot problems in complex work way ahead of the time you would if you interviewed each person separately to try to figure out what was going on. And when the guesses reach zero? There’s no more work to be had. Individually the guesses may be crap, but over many people, the overall numbers tend to be useful — even though they may not mean anything! The change is what you’re looking for, along with the linear regression that shows where “done” is.

It’s not engineering or a science. It’s a hack. But done correctly, it works. Plus it’s a nice little graph.

A personal favorite

Assuming the team and nature of the work remains the same, there are two major things that change in any project: how much stuff you have left to do and how good you are at doing stuff. The burn-up captures both of those. It’s a personal favorite, although I don’t see it in the wild as much as I’d like to. It’s not difficult to do, it involves guesses as we’ve said before, and over time it eventually starts being as dead-accurate as if it were a math problem. (Part of this might be the team psyching itself out that the chart accurately reflects where things are — but that’s a story for another day.)

I was teaching a coaching team many years ago and had a young coach bring me this graph that one of his Scrum Masters had put together.

The Cross-Fire

These guys were once every morning guessing how many hours they had left to do, then the SM was assembling it all into a graph of “How much we guess we have left”, “How much we budgeted”, “Actual hours we used”, and “How much budgeted we have remaining”

Wow! That’s a bunch of lines! It was a matrixed environment. That means that while there were teams, each person on the team was also on several other teams at the same time. Sounds crazy? It is. But it gives people the impression that there are a bunch of projects all being worked on at once without folks having to have difficult conversations about what’s important and what’s not.

(Each person also had multiple managers they reported to. One for each project, one for their department, and one “people manager”. Did I mention that these people needed to be agile in a big way? It was great. Once we showed up, the first thing they wanted to do was make a complicated set of diagrams and instruction manuals for exactly how to be agile. But I digress.)

I would never want a team to standardize something like this, but I liked what happened here. This team kept having a problem with delivering, even though they had 3-week sprints and could have just changed how much they promised to do and everything would be fine. But somehow they kept disappointing the customer and nobody knew why.

So the SM did this chart, which over several sprints showed that nobody worked much on the project the first week, just a little the second week, and then they tried to cram it all in on the last week. This was probably because everybody was busy on other projects doing the same thing.

They did a chart for a few sprints, figured out what was wrong, then stopped doing the chart.

I’m going to skip a bunch of Statistical Process Control charts — man, do those guys love charts! But I have to show this one.

The great and wonderful CFD

The (somewhat) new guys on the agile block are the Kanban folks. They take out the guessing entirely and just break stuff into really small pieces and track how fast the pieces get done. With large enough numbers, it’s the same difference. The CFD diagram is probably the key diagram to get all of the PM goodness you want without having to ask people to estimate anything.

There are also more program-level charts, showing how dependencies are managed and how complex releases happen. I’m scoping those out too — there are lot of PMs that do projects. Not as many that do programs. Some of it actually scales out nicely. If you can do a project burn-up, you can do a program burn-up. Same goes for CFDs.

But it’s time we talk about the elephant in the room. Or rather, the elephant in this post.

What is the most important and useful graph on this page?

It’s the elephant.

That’s because none of these graphs are useful at all. They’re just graphs. Pieces of paper with images on them. At least the elephant is honest about it. Do these graphs tell you something? They might — but as a manager, you’re not the person who needs to know stuff. The team is. And handing them pieces of paper — or showing something on a screen — probably won’t get much attention or buy-in.

Management is about people. If you say you’re a manager, I want to see your mouth moving, your ears listening, and your body in close proximity to the people you’re trying to help. After all, the primary goal of management is the elimination of obstacles. (Secondary is the coordination with outside constraints). If you’re not physically participating in important conversations, you’re not managing. You may be dictating. But you’re not managing.

Here’s somebody who’s managing.

Drawing. With a pencil. Oh the pain!

Every day when the team meets, after their five-minute standup, Shaunte draws a couple of graphs; whatever the team is interested in tracking that sprint. They always do burn-down because somebody is always asking “Are you done yet?” and that isn’t important enough to interrupt their work. They add in more as-needed, depending on the problems they’re working.

This is called publish-subscribe. If a bunch of people want something from me, instead of each of them bringing me into a meeting, I publish it in one spot. Then “subscribers” can come and get the information as needed. Saves a ton of time.

The team stands there while the graphs are updated. After all, they are the ones that own the data. The purpose of the graphs is to save time later on. It’s their graph, not hers. Sometimes they rotate around who does the graphs.

“Well!” I hear you say. “Certainly these graphs should be put into a tool! We have all kinds of great tools to handle this kind of grunt work! People can come online and see the graphs from anywhere! And the graphs are much nicer too! None of this crappy schoolhouse craft show.”

But the purpose of PM graphs aren’t to create data to move around, they’re to assist in management — conversations. Project Management is not bits and bytes. It’s people. Yes, you do reports that have these kinds of things in them — but that’s not the work. The graph alone is worthless because you don’t have the context and can’t engage in the conversation. In fact, it’s worse than worthless because it gives you the impression that you know something now that you didn’t before. If you’re four-steps removed from the team that’s fine. You don’t have to know a lot. If you’re only a step or two? It’s not. Do your job.

These graphs are hand-created because the team has to own and talk about them as they evolve. That’s their purpose. You don’t make a graph so that somebody else can read it and swoop into the team to announce all the things they’ve done wrong. You make a graph so that as you update it, the people affected can learn more and have better conversations about what’s happening.

They’re physically put on the wall because the team owns this. It isn’t some report you get when you push a button. It’s the replacement for a bunch of meetings. If you’re interested in how the team is doing? Come by and look at the chart. Got questions? Cool! There’s the team. You now have data plus context plus access to the people you need to do your job. This also saves you time. In fact, the first thing you should do as a program manager or higher is a “Wall walk” every morning. Before folks get there, walk around and let each team tell you how it’s going. Then you can pick and choose where to go listen and help out based on the need they report

Isn’t that a lot better than a management-level meeting where a bunch of folks look at charts made by some tool that don’t mean anything and are a hassle for everybody to update?

I’ve known a lot of Project Management charts in my time. But I’ve only known four tools that I’ve seen consistently kick ass over and over again: your feet, your eyes, your ears, and good conversations. You use them in that order.

July 11, 2018  Leave a comment

Teachers Of Functional Programming: Stop Driving Me Crazy With Math Problems!

“Actually, I’m starting not to like functional programming…if I see one more Fibonacci or factorial coding example in an F# tutorial or textbook, I’m going to stab the next nerd I find”

I feel this pain. When I learned F#, I’d start reading books and the first thing I saw was stuff that looked like this:

let rec fact x =
    if x < 1 then 1
    else x * fact (x - 1)

Hey look! Now I can do factorials!

Who the heck wants to do factorials?

Like most OO programmers, I struggled a lot learning F# and functional programming. I kept at it, slogging through a bunch of books with a lot of math problems disguised as coding. I had been programming for a long time, and I’ve never coded a math problem. People rarely come to business programmers and ask them to do math.

And then I would talk to some super-smart people, like the guys at google. I’d ask them what their code does, and I’d get weirdest answer.

Well really it just does some matrix math and assembles the results.

Now this code may be running on 100,000 servers worldwide, it may have fault-protection and do all sorts of wondrous things, but to them it’s all about some kind of math problem.

What was I missing? It didn’t make sense to me. Sure, I got the math coding part. I understood how to do it. But when was I going to start form validation? Business logic coding? Important stuff?

So I jumped into my first project. It was a common, plain windows application. I had written something like this in C, C++, VB, C#, maybe a couple of other languages. Using F#, I coded up some windows.

I decided to add in some mailbox stuff, which is kinda cool. And then I went functional-reactive, which sort of fit naturally. Then I wanted to do some other stuff…..

It was a mess. I ended up with code that looked like this. (I may have actually copied this from somewhere. Beats me. It’s been a while)

///<remarks>Worker bee class for doing long-running threaded work</remarks>
    type AsyncWorker<'T>(jobs: seq<Async<'T>>) =
        /// <summary>Capture the synchronization context to allow us to
        /// raise events back on the GUI thread</summary>
        let syncContext =
            let x = System.Threading.SynchronizationContext.Current
            if x = null then new System.Threading.SynchronizationContext() else x
        let cancellationCapability = new CancellationTokenSource()
         
        // Each of these lines declares an F# event that we can raise
        let allCompleted    = new Event<'T[]>()
        let error           = new Event<System.Exception>()

I had a disk utilities class, a class for writing html, a class for handling gating. I had a class for everything that I thought was important.

It’s not that I didn’t get it working — I did. The problem was that I was spending all of my time being an OO programmer in a functional world. My thinking was not in alignment with the tool I was using. I was interested in class libraries that could do cool stuff, which class libraries I would need to create, which code went in which class. In my mind I was assembling a large structure with a lot of little building blocks. Some of them I made on my own. Some I got elsewhere.

I did that a few times. Heck, I’d probably still be doing that if I hadn’t decided that I wanted to learn startups more than I wanted to learn functional programming. To learn startups, I started writing various web pages — no programming if I could get away with it. In fact, many times I would compete with myself to see if I could take a weekend and write something up on the web that would make money. Anything. The programming part didn’t matter. Just make something useful.

I did all kinds of cool stuff. Eventually I landed on serverless web applications a few years ahead of everybody else. Fun times.

But with those serverless applications, eventually I wanted to add functionality. How would I do that as directly as possible? Pick up a framework?

That’d be tough to do in a weekend, and I was all about making stuff people want and not losing focus on that. So why not just write an F# app that runs on the server as a plain-old CGI app? It’d just be an F# console app for linux. It’d read the header info of the page POSTing or GETing the data, then write html directly back out to the client. This is about as simple as I can possibly make it.

I wrote that up and it worked fine. It was a very small program! But it did what I wanted and I never had to maintain it. It just worked.

I did that a few times with various apps. I started to notice a pattern emerging. First I would “clean up” the incoming data: make sure it was valid, ignore bad data, take care of any option types, make it as correct as possible. Then I would process it. And the processing would always end up being some kind of simple sorting, matching, or recursing through various data structures.

In short, a math problem.

The light was beginning to come on. As I moved to microservices, I saw the pattern again: clean the data, take care of the housekeeping. By the time you finish cleaning things up the problem is trivial. My microservices never got more than around 100 lines of code — and they did very useful things! With very little maintenance!

I found that I needed the strongest types on the “outside” of the program, where it touched the rest of the world. But those types were all about I/O and data movement. As I moved inward, I wrote more and more generic code. This code I captured and put in a utilities library. I also refused to make a class until I was absolutely sure it was doing something useful, that is, that I had persisted state and methods that belonged together and could be used across several applications.

I went for a long time before I had good reason to make a class.

/// Parameterized type to allow command-line argument processing without a lot of extra coder work
    /// Instantiate the type with the type of value you want. Make a default entry in case nothing is found
    /// Then call the populate method. Will pull from args and return a val and args with the found value (if any consumed)
    type ConfigEntry<'A> =
        {
            commandLineParameterSymbol:string
            commandLineParameterName:string
            parameterHelpText:string[]
            parameterValue:'A
        } with

I finally found some code that belonged in a class! It was a class to handle a problem I would have over and over again: taking care of the argument list passed in from the command-line. I made a nice generic type that took care of what I wanted where the programmer using the type could make it all work with just a few dozen lines of code.

// Create a type that will handle your program config
    // Use configBase to handle common stuff
    // Then put whatever you want, inheriting from ConfigEntry
    [<NoComparison>]
    type MyAppConfig =
        {
            ConfigBase:ConfigBase
            InputFile:ConfigEntry<FileParm>
            OutputFile:ConfigEntry<FileParm>
        }
        member this.printThis() =
            printfn "MyProgram Parameters Provided"
            this.ConfigBase.verbose.printVal
            this.InputFile.printVal
            printfn "Input File Exists: %b" this.InputFile.parameterValue.FileInfoOption.IsSome
            this.OutputFile.printVal
            printfn "Output File Exists: %b" this.OutputFile.parameterValue.FileInfoOption.IsSome
    // Add any help text you want
    let programHelp = [|
                        "Here's some program help."
                        ;"and some more.. as much as you want to provide,"
                        |]
    // Add in default values
    let defaultBaseOptions = createNewBaseOptions "myapp" "gets new links from a site list" programHelp defaultVerbosity
    let defaultInputFileName="myAppInput.json"
    let defaultInputFileExists=System.IO.File.Exists(defaultInputFileName)
    let defaultInputFileInfo = if defaultInputFileExists then Some (new System.IO.FileInfo(defaultInputFileName)) else option.None
    let defaultInputFile= createNewConfigEntry "I" "Input File (Optional)" [|"/I:<fullName> -> full name of the file having program input."|] ({FileName=defaultInputFileName; FileInfoOption=defaultInputFileInfo})
    let defaultOutputFileName="linkLibrary.json"
    let defaultOutputFileExists=System.IO.File.Exists(defaultOutputFileName)
    let defaultOutputFileInfo = if defaultOutputFileExists then Some (new System.IO.FileInfo(defaultOutputFileName)) else option.None
    let defaultOutputFile = createNewConfigEntry "O" "Output File (Optional)" [|"/O:<fullName> -> full name of the file where program output will be deployed."|] ({FileName=defaultOutputFileName; FileInfoOption=defaultOutputFileInfo})
    // Do the actual loading
    // This returns back an MyAppConfig structure
    // to be used by the caller
    let loadConfigFromCommandLine (args:string []):MyAppConfig =
        if args.Length>0 && (args.[0]="?"||args.[0]="/?"||args.[0]="-?"||args.[0]="--?"||args.[0]="help"||args.[0]="/help"||args.[0]="-help"||args.[0]="--help"then raise (UserNeedsHelp args.[0]) else
        let newVerbosity =ConfigEntry<_>.populateValueFromCommandLine(defaultVerbosity, args)
        let newInputFile = ConfigEntry<_>.populateValueFromCommandLine(defaultInputFile, args)
        let newOutputFile = ConfigEntry<_>.populateValueFromCommandLine(defaultOutputFile, args)
        let newConfigBase = {defaultBaseOptions with verbose=newVerbosity}
        { 
            ConfigBase = newConfigBase
            InputFile=newInputFile
            OutputFile=newOutputFile
        }

That’s a bit more setup than I like, but it’s all fairly straightforward. You should be able to read the comments and figure out what it does. It’s easy-to-use, and really? It’s not part of the app. It’s a thing that handles getting args from the command line. It’s become a standard part of my toolkit. I don’t count it as part of the code I write. I instantiate the type with a few things…and it just works.

This way of working outside-in is called an “Onion Architecture”. People have been doing it for years. Done correctly, it prevents the program from failing. No matter what, the program runs — which is exactly what you want in a true Unix-Philosophy microservices approach. (There’s a long discussion to be had about the Unix Philosophy that we’ll save for another day)

Welcome to the Onion Architecture. Much to come about this in later essays

And I finally figured out why everybody kept trying to give me math problems when I was learning! Functional programming to me is focused on output, not structure. What are you doing for me? Everything revolves around that, and it should be the driver of any coding or architectural decision you make. By the time the type system kept out bad data and controlled program flow, the only major thing that was left was understanding simple semantics and how first-class functions were an entirely different animal from the methods in C# that I had been used to. And math is probably the easiest way of understanding that.

They weren’t trying to teach me math. They were trying to teach me function construction and composition. FP is based on math, and it was the easiest thing they had handy to use as an illustration. Otherwise, to do the “real” stuff I was asking about, the authors would have had to guide me through the entire onion process. That’s too much overhead for talking about something simple like recursion or pattern-matching.

The reason those guys at google said their code just added some matrices is that the rest of it doesn’t matter. That outside-of-the-onion stuff is just grunt work, most of which you can reuse (as opposed to the “reuse” promised to us in OO, which never panned out.) In the diagram above, it’s level 2 and 3 where all the “interesting” stuff happens. In fact, it’s mostly level 3 — and it’s mostly not that much stuff. And just like in OO, once you start thinking the right way, with the grain of the language, the problem kind of “falls apart” and becomes trivial. It’s a matrix problem. The only difference is that it falls apart in an entirely different way in FP than OO.

July 8, 2018  Leave a comment

Function Overloading Five Ways in F#

Ever want to overload a function in F# like you do in other languages? So you have a function “foo” that sometimes might take a string, sometimes an int? You write two different functions with the same name that take different parameters, then you just call it. The compiler figures out which one to use.

Yeah, that’s not going to work in F#

I’ll explain why, and how to fix it, but my real purpose in today’s post is to talk about programmer workflow. When you’re learning F# and you get hung up, what should you do?

Yesterday I decided to start writing every now and then about F#. For my first post, I included a few little code snippets I’ve picked up over the years. I also wanted to do function overloading. People say you can’t do it, but you can! I just knew it!

I was wrong. It didn’t work like I thought it would. In fact, I couldn’t get it to work easily at all. So I had to find out why.

First thing I did was find the easiest cut-and-paste answer I could, like any good programmer. It uses a type and static methods. Turns out you can override methods inside a type.

// Function overloading five ways
// First way. Overload a type member with static methods
type FooType =
    static member foo(x:int) = printfn "The int is %d" x
    static member foo(x:string) = printfn "The string is %s" x
FooType.foo 9
FooType.foo "weasels"

That kinda-sorta worked, but it sucked having to always stick that type name on the front of all of the calls. While I’d use it in a pinch, it just didn’t feel right. So I looked around some more.

As it turns out, you can use the “inline” operator to point to a static method on a type. Then the compiler figures out which of those methods to call. This gets you pretty close to something that’s function overloading.

// Second way. Overload a type member and use inline to resolve to it
type FooInt = { thing: int } with 
    static member foo (x:FooInt) = printfn "The int is %d" x.thing
type FooString = { label: string } with
    static member foo (x:FooString) = printfn "The string is %s" x.label
let inline foo (x:^t) =
    (^t: (static member foo: ^t -> unit) (x))
foo { thing = 98 } 
{ label = "Car" } |> foo 

Well heck. Now we don’t have to worry about the carrying the type name around everywhere. but we have to instantiate some stupid dummy-type first just to get the inline magic working. I still don’t like that. Isn’t there some way to use the inline function to do the overloading, but then have the type and such constructed behind the scenes? Let’s try this one.

// Third way. Ugly! Figure it out at runtime
// The problem is that not only will it take
// our parameters, it'll take _any_ parameters.
// Ouch!
let Bar (x:obj) =
    match x with
        | :? string as x->FooType.foo x
        | :? int as x->FooType.foo x
        |_->failwith "We don't do that"
Bar 98
Bar "cats"

The usage looks perfect. Just type in the function name and throw a parameter at it, just like you’d expect. But heck, you can throw anything at that function. There’s no safety at all. The only way to fix that is to return an option. And I don’t want to return options from functions unless I’m being held hostage at gunpoint. It’s a sign of my not-solving a problem and just foisting it on somebody else. That’s a code smell — and I’m not releasing anything into production that has that smell if I can help it.

Ok. Can I check for the type in the inline function, perhaps in this case only taking strings and ints, the types I know to work?

Looking around, I figured out how to check for a subtype. I couldn’t out how to test for one of multiple subtypes. We’re venturing here into type programming, where you’re writing code that controls which types you’ll let do which things. You can look at it as an extremely advanced form of polymorphism.

What we need, I think, is called a “typeclass”, but F# doesn’t have those. We could check and see if a method is present, like we did in our second example.

// Bonus fourth way. Vastly more ugly!
// Flag the type that it's okay, then check during compile
// and reference using inline. Uglier, but safer
type FooBar = {o:obj} with
    static member bar (x:FooBar) =
        match x.o with
            | :? string as x->FooType.foo x
            | :? int as x->FooType.foo x
            |_->failwith "We should never, ever get here"
type System.String with static member DoesFoo()=true
type System.Int32 with static member DoesFoo()=true
let inline barfoo (x:'t when 'a:(static member DoesFoo:unit->bool)) =
    {o=x} |> FooBar.bar 
barfoo 6
barfoo "elephants"

Well that’s a big old hunk-o-fun. You know you’re having fun coding when you start changing the way the system types work. I would get out a flamethrower, but I don’t have one handy.

Way ugly. But look! Now we have type safety and a usage pattern that looks like it should. I guess this is probably the best compromise. Overload all you want inside a nested type, then use an inline call to handle the dispatch and mark your handled types for function safety. After all, this is meant for custom types. You shouldn’t be using with string and int as shown here.

Is there no way to get rid of this nested type/static nonsense? How about Active Patterns? I do some googling around and play around a bit.

// Play with some ideas
let Fun1(x)=printfn "s"
let Fun2(x)=printfn "q"
let Pick1 (|Fun1|Fun2|) (x:int)() = (if x<10 then Fun1 else Fun2)()
let Pick2 (x:int) = ((Pick1 x) x)()
Pick2 4 //outputs "s"
Pick2 15 // outputs "q"

That’s pretty cool. Still ugly as crap, but no nested types. Instead Active Patterns handle the switching. I can use them instead of inlining, but I’m back to the dynamic type problem: to make overloading work, I’m forced to take an obj type and then figure out what to do with it. I could fix that with my type annotation trick from example four.

Meh. Here’s what I have so far:

// Fifth way
// Combine what we have so far:
// Overload all you want, just add type in at end of func name
let FooFooString(x) = printfn "The string is %s" x
let FooFooInt(x) = printfn "The int is %d" x
// then make an active pattern to manage it that takes an obj
let FooFooAP (|FooFooString|FooFooInt|) (x:obj)() =
        match x with
            | :? string as x->FooFooString(x)
            | :? int as x->FooFooInt(x)
            |_->failwith "Back to dynamic typing problem, but without inlines"
// finally wrap it. I must be doing something wrong to have to use
// so much syntactic junk
let FooFoo (x:obj) = ((FooFooAP x) x)()
FooFoo 42
FooFoo "dasdf"

Wow, that’s some ugly syntax, isn’t it? I had to write a second function just to wrap the first one. In fact, I probably missed some things here. There has to be a way to clean that up.

And here’s a good place to stop. After researching and playing around with the code, I now have a better understanding of what’s going on. F# has type inference, which means that it’s always trying to figure out the appropriate types for you, instead of your having to put types on everything like you do in C or C#.

Type inference does not play well with function overloading. As soon as you type the first “foo” into the code, the compiler figures out what types it has to take. It’s locked in. If you type another “foo” in with a different signature, it doesn’t match the first. The type inference system blows up. You can scope this out to a static member inside a nested type in your code — probably because F# keeps track of more stuff when you explicitly and statically add a method to your own type.

Because F# tries to figure out all the type stuff behind the scenes for you, you’re never going to do function overloading like you would in other languages. That’s why the answer will always involve “moving up” and horsing around with the type system itself. Now you know.

It used to be that the Program.fs file was a module, kind of a hidden module. I screwed up thinking it was easy because in my mind I thought I could add static methods there. But it doesn’t work like that. Instead, after a couple of hours of poking around, I now have several options if I want to overload. And that’s plenty.

Who’s got time to chase down this stuff? I’ve got code to get out the door! I hear you. This is the kind of thing you do to “sharpen the saw”, make yourself better understand how the tool you’re using works. When I’m in a production environment, it’s the kind of thing I might do once or twice a week: take 2-3 hours and dig in to figure out how things work. I wouldn’t consider myself a professional otherwise.

Note the difference here. There are programmers who want to be uber-nerds. They’ll learn everything possible about DoodleSquat 7.1 and they’ll be able to tell you the difference between 7.0 and 7.1. Then here are “hack and slash” programmers (we’ve all been there), that are just looking to google something quickly, copy-and-paste, and move on.

Here we’re picking our battles, making sure every week there is at least one battle to fight, and then diving deep enough to understand the issues and be able to make several reasonable suggestions for solutions, then moving on. It’s not that I’m nerdy or not nerdy, I’m just nerdy enough.

And remember: F# is a mixed-mode language. If you come from an OO world and get stuck, fall back to your OO skills, just translate it into F#. Then schedule a “clean up” where you go pure functional, for later in the week, after you get the work done. Because it’s mixed-mode, F# teaches you good functional programming, it doesn’t mandate it.

I love writing in F# because it teaches me to be a better programmer. If you don’t let it teach you, you should probably just stick with whatever you’re doing now.

Thanks for hanging out! If you’re interested in watching Uncle Bob (Clean Code), James Grenning (Embedded TDD), and myself horse around with learning F#, check out these videos.

July 6, 2018  6 Comments

F# Tips

What the heck is the deal with the Microsoft RegEx object? It says it returns a MatchCollection, but it doesn’t look like any collection I’ve ever seen.

As I understand it, the MS regex object goes back — way back. Way back before folks stuck enumerators on things. So it’s a collection. Kinda. I don’t like System Type extensions willy-nilly, but this looked to me like a good place for one. So I do this:

    type System.Text.RegularExpressions.MatchCollection with
        member this.toSeq =
            seq {for i = 0 to this.Count - 1 do yield this.[i]}
        member this.toArray =
            [|for i = 0 to this.Count - 1 do yield this.[i] |]

I have an array and I’d like to write a function that picks a random item from it. How do I do that?

This is another type extension, and it shows the danger of writing type extensions. In this case, the extension assumes that the array is not empty. I could have written around that by returning an option type or something, but then I’d kind of defeat the purpose of writing the extension in the first place.

    type 'a ``[]`` with         
        member x.randomItem = 
            let rnd = new System.Random()
            let idx = rnd.Next(0,x.Length)
            x.[idx]
    type System.Random with
        /// Generates an infinite sequence of random numbers within the given range.
        member this.GetValues(minValue, maxValue) =
            Seq.initInfinite (fun _ -> this.Next(minValue, maxValue))

I also included how to get an infinite sequence of random numbers.

I hope you’ve enjoyed these tips!

July 5, 2018  Leave a comment

Top Five Reasons You’re Wrong About Needing a Large Backlog

Most of these happen because people are confused about what, exactly, a backlog should be in an Agile environment.

The future will be awesome! And really messy too. Does it have to be that way?

  • But we have a lot of work to do! – You are confusing activity with value. Backlogs measure value, not activity.

    The old way to do project management was to use a Work-Breakdown Structure (WBS). It took a lot of work and broke it into smaller pieces which would be assigned to individual team members to accomplish. This was great if one person was able to completely understand everything required to breakdown the work, but that is not true in technology development. In fact, the reason you hire people instead of robots is that you can’t predict how they will solve certain problems. So you give them the problems and let them work it out. Instead of a Work-Breakdown Structure (WBS), you need to create a Goal-Breakdown Structure (GBS). If that sounds like the same thing just with different words? Then you don’t get it.

    Goals are tests that describe value, not actions. They are the problems you are giving people to solve. So a backlog is a series of progressively-described tests. When the tests pass, the goals are met. How the tests get to the passing state is up to the workers. Your job as backlog owner is to create tests with more and more detail the closer you get to actually doing the work, not pre-digest work. Remember: the backlog is about useful things you’re providing the users, not useful things you’re doing. If your focus is on yourself, it’s in the wrong place.

  • Without a huge backlog, we’ll lose track of important details! – You make this mistake when everything is a blade of grass, but nothing is the lawn. It stems from the belief that lots of tiny things have more value than one big thing. If your backlog is goals, not tasks, then goals have natural hierarchies, right? A lot of detail about those goals goes higher up in the hierarchy so it doesn’t have to get repeated everywhere. (In fact, if you think about it, the only extra information needed for the backlog item should be the last few bits of information to make this goal different from similar ones under the same parent. The rest of the information is already in place. DRY.)

    You can store as much detail about anything you like, just not in the backlog. That’s not what it’s for. People stick all kinds of crap into their backlogs: database tables, to-do lists, design notes.

    Another way of looking at it is this: the backlog exists to easily let the people-solving-the-problems work with the people-they’re-trying-to-help to make sure they’re solving the right problems in the right order. Once it becomes incomprehensible because of size, it ceases to fulfill its mission. Instead it becomes an artifact, a deliverable, a paperwork tiger, an obstacle, not a facilitator to rapid value delivery. Backlogs, like all the rest of your tools, are supposed to make you go faster, not slower.

  • How else can I keep track of what people are doing? – I don’t know. Maybe ask them? A lot of teams do a daily standup where they talk about how the value delivery is going. (Poor teams focus on “Important stuff I did yesterday”. Good teams focus on “I am trying to deliver this value and I could use some help”.) Hang out there and listen to what’s said.

    There are a lot of Agile folks who will wave their hands around and say something like “Stop being a micro-manager”, but I get it: somebody should be looking at how big teams are, where teams should be formed or shut-down, who should go to which division, and so on. There is a people management skill that’s required in businesses of any size. It’s just backlogs aren’t the tool to do people management. There’s a recurring theme here: the minute you stop focusing on value delivery and start focusing on activity, you end up with a lot of activity and very little value delivery.

    So whatever your management needs are, you should handle them in some other way. (Which is outside the scope of this essay)

  • But we need to coordinate at a low level with Team X! – In a lot of corporate environments, the work is both pre-digested for the teams and then split out among several “expert” teams.

    Wow, does that create a mess. Nobody can do anything without coordinating with another team. It’s all one big hunk of work.

    At some point, your job is either to create value or go down a to-do list. If they’re giving you a to-do list, do the to-do list. You will find no value in this essay. But remember: if your job can be done by robots? It will be done by robots. Having a job where everything can effectively be broken up 3 months ahead of time by one really smart person should be setting off alarm bells. Something is wrong somewhere.

  • But we have this huge list of bugs/defects/change requests/feature requests/…! – Yes. I understand. You have this big list of stuff, right? And you also have this tool that handles big lists of stuff! How cool is that?

    Aside from the fact that you’ve destroyed the entire purpose of a backlog by making it incomprehensible en toto, here are a couple of things to consider.

    Perhaps a big list of anything is just going to lead to endless boring meetings, loss of morale, and a feeling that things are hopeless. Perhaps big lists of things fit naturally into a simple and easy-to-understand hierarchy along with everything else. Perhaps the only things you should be worried about are the things you’re working on and the things you’re getting ready to work on — the rest is useless overhead. Perhaps the entire purpose of a managing a backlog is to prevent this very scenario.

In 1963, a U.S. President stated that he wanted to see, in that decade, a man walking on the moon. Did he deliver the specifications to the Saturn V moon rocket? Of course not. Do you think every week he received details on how hard the guidance team down in Huntsville was doing? Of course not. He stated a test, a high-level goal he wanted accomplished. He received periodic updates as new goals were accomplished: man living in orbit, capsules docking, and so forth. People followed along as each of these sub-goals were met, knowing that with each new capability the moon came closer.

Backlogs can be easy, intuitive, fun, and positive motivators for a team! Yay! Or they can be a micro-management death march. It all depends on what you’re willing to learn — and what you’re willing to un-learn.


Interested in learning more? I wrote the book on effective information management in your project. As it turns out, looking at your processes and workflows in terms of information management can help you figure out where things are going wrong. It’s called “Info-Ops”.

July 4, 2018  Leave a comment

« older posts