Stupid F#: Taking JSON to the SPA 2

Programming should be fun. All you need is a good set of values, some skill, and the right attitude. F# is the most fun I’ve had in a programming language in years. This essay series is about that: having fun. Most books and essays are about writing awesome code. There’s not a lot of material about writing code aweseomely. The code you’ll find here has bugs! Just like your code. Can you find them? If you’re looking for the final version of the code, you can find it in the project’s GitHub page. You can find the story of how all of this started on the series index page.

We’ve got a type system that correlates and adds numbers which we’ve coded up using a console app to read and write JSON. Next we’ll make a Single-Page Application out of it. A SPA.

It’s helpful to remember what we’re doing. Looking at our tests helps. Automated tests that execute and pass are the specs for any program. I look at our tests and realize that I forgot to, well, read and write JSON.

Doh! So let’s fix that. What do we want for sample JSON code? What’s our current JSON output look like, anyway?

{
    "groupAndSum": [
        {
            "ToKVPair": {
                "Key""a",
                "Value": 12
            }
        },
        {
            "ToKVPair": {
                "Key""b",
                "Value": 4
            }
        },
        {
            "ToKVPair": {
                "Key""c",
                "Value": 10
            }
        }
    ],
        "TEST": [
            {
                "ToKVPair": {
                    "Key""a",
                    "Value": 12
                }
            },
            {
                "ToKVPair": {
                    "Key""b",
                    "Value": 4
                }
            },
            {
                "ToKVPair": {
                    "Key""c",
                    "Value": 10
                }
            }
        ]
}

Well heck, that kinda looks like crap. The entire point of JSON, aside from being everywhere, is that you can read it. Yes, I can read this, but it’s noisy as frack. Why is that?

First, it looks like there are two objects here embedded in our top object, “groupandsum” and “TEST”. What the heck is up with that?

They look like this on the collection class:

member self.groupAndSum =
        self.OptionExampleFileLines 
            |> Seq.map(fun x->x.ToKVPair) 
            |> groupAndSumKV
            |> Seq.map(fun x->
                OptionExampleFileLineType.FromKVPairString 
                    (System.Collections.Generic.KeyValuePair<string,int>(fst x, snd x))
                )
    member self.TEST = self.OptionExampleFileLines

Without the () at the end, F# treats these as immutable values it can resolve and use wherever it wants without having to go back to the type. Well dang, that’s pretty much the same as a contained object or a field, so that’s the way it gets treated. (F# is what’s called an “eager evaluator”, which is a story for another day.) If we fix it by making them both true functions, we get this when we run it from the command line:

$ cat CGISample1.testin | ./stupid1.exe -IF:C -OF:J
{}

Nothing! Ha! Showed us. Now we got bupkis. That’s because there’s no public access to the underlying array, which is just what we wanted when we built the type but not exactly what we need now. (For C++ folks, this might be a good place to use friends.) By poking right into the private OptionExampleFileLines, we add an attribute to tell those nice Newtonsoft people that this private field is what we want them to serialize and deserialize. We do the same with the individual item type. We also name the private data in the individual option type so that the Json reads better.

type NameIntPairType = {Name:string;Number:int}
type OptionExampleFileLineType = 
    private {[<JsonPropertyAttribute>] NameIntPair:NameIntPairType} with

Adding the attribute to the private member on both the collection and individual private fields makes it all work out nice and pretty.

$ cat CGISample1.testin | ./stupid1.exe -IF:C -OF:J
{
  "OptionExampleFileLines": [
    {
      "NameIntPair": {
        "Name": "a",
        "Number": 12
      }
    },
    {
      "NameIntPair": {
        "Name": "b",
        "Number": 4
      }
    },
    {
      "NameIntPair": {
        "Name": "c",
        "Number": 10
      }
    }
  ]
}

Same question as before: is this human-readable and easy to understand? Mostly. I can see using this code on the client-side. Good enough for now. Notice how as we incrementally add tests and exercise the type, the structure of what goes where and how things are named and used changes. Is good. This is what we want to see.

Now we take the Json output and create a sample test file for it, then add it to the BASH O/S testing scripts, just like before. We’ll be sure to include all kinds of bad data. After all, JSon coming from the client could be most anything.

{
    "OptionExampleFileLines": [
        {
            "NameIntPair": {
                "Name": "a",
                "Number": 5
            }
        },
        {
            "NameIntPair": {
                "Name": "b",
                "Number": 3.14159
            }
        },
        {
            "NameIntPair": {
                "Name": "c",
                "Number": "the cow goes moo"
            }
        },

(And so on)

As expected, this created another error, since the data was purposefully bad. This led me down a six hour rabbit hole, spread out over several days, where I learned more than I ever wanted to know about the NewtonSoft Json library

This might be a good time to think about who is in charge in this relationship between myself and my Json library.

Why was so much time wasted? Is parsing that complex? No. Is the library too complex or broken in some way? No.

It took a long time because of one of the “features” of OO: encapsulation and data hiding. Combine that with simple-sounding English names and it’s a mess. What happens is that the programmer reads the plain English, uses the API in a way that seems terribly obvious and simple — and gets bad results. But it looks like it should work, dang it! So they poke a little more. Still looks like it should work. Still bad results. Repeat and rinse for several hours, the answer always appearing as if it’s just another tweak or two away.

This is how you get an internet full of coders googling around for the magic sauce to make whatever they’re working on right now work — instead of actually trying to understand what they’re doing. I love the library, but frankly I don’t care how it’s put together. I want my goal met. That’s it. So there’s lots of googling and thrashing. Much wasted time.

As coders, we use a lot of complex stuff. That’s the gig. That’s always been the gig. If somebody had told me the Json job might take half a day or more of my programming time? I’d spend an hour or two reading as much as I could before I started. But instead? The “happy path” worked like a charm and took five seconds to do. I was hooked. Anything just a little off the happy path looked like it should be just as easy — and might take days, hours or be completely impossible. I don’t know. You don’t know. It’s done when it’s done.

So it’s not the NewtonSoft guys or this particular library. It’s everything. Everywhere. It is the logical nature of large-scale OO adoption. This is why it’s so critically-important to keep an eye on your time usage with various abstraction layers. OO gives you nice titles to put stuff under. It also hides implementation. The assumption, which is rarely true, is that the categories the programmer put things under will naturally match up with the categories the caller puts things in.

Long and short of it, the library wanted stronger coupling with the application than I was comfortable giving. My policy outside the onion is to fight against coupling with anything. So I had to build “shim”, a fake piece of structure who’s primary purpose is to prop up another, useful piece of structure.

// A type shim to decouple from Json as much as possible
// We don't care what's in the Json as much as we care
// that we can process whatever we can and ignore the rest
// MUCH different than strongly-type db-type stuff cf Onion
type NameIntPairJson = {Name:string;Number:string}
type OptionExampleFileLinesJson ={OptionExampleFileLines:NameIntPairJson[]}

Then when I read it, I convert over, just like when I’m reading CGI forms or anything else.

static member FromJsonString (s:string) =
        let JsonShim=Newtonsoft.Json.JsonConvert.DeserializeObject<OptionExampleFileLinesJson>(s)
        let JsonStrings = JsonShim.OptionExampleFileLines |> Array.map(fun x->x.Name + "=" + x.Number + OSNewLine)
        OptionExampleFileLinesType.FromStrings(JsonStrings)        

Note that I take the Json, parse it as strings, then re-assemble it back into the “a=x” format everything uses. Then I use the smart constructor FromStrings. There’s a lot about this code that feels crappy. Much refactoring needed. But let’s get the tests in place first, the refactor all we want to. Our Json format now looks like this:

{"OptionExampleFileLines":[
  {
    "Name": "a",
    "Number": "5"
  },
  {
    "Name": "b",
    "Number": "3.14159"
  },

…and so forth

So we take the sample data and make a .desiredout file in both text and json formats. We know the answer is correct now, so we can use working data as our first test case. We run the script and have failures. Doh! We’ve changed our Json format. So we fix that.

Now we should be in a cool spot: our app is tested and works (mostly) like a function. So we should able to pipe the results from one place to another, like so:

cat CGISample1.testin | ./stupid1.exe -IF:C -OF:T | ./stupid1.exe -IF:T -OF:J

This takes the CGI test file, sends it to our app to convert it to text format. It sends the results of that to our app again, this time converting from text the first time we ran the app to JSON.

That doesn’t work, though, because whoever wrote .NET Console engine doesn’t close their pipes properly. You can try doing a Console.In.Close() at the end of your app (be sure to also close Error and Out), but I didn’t find that it worked. (I suppose it works in linux. We’ll find out later.) So I was forced to use temp files for piping, like this:

cat CGISample1.testin | ./stupid1.exe -IF:C -OF:T > pipetemp1
cat pipetemp1 | ./stupid1.exe -IF:T -OF:J > CGI2Text2Json1Sample.testout
rm -f pipetemp1

This is an extraordinary amount of butt-uglinesss that I was totally unprepared to see. Sadly, however, this is programming: half-finished little bits-o-stuff here and there you have to glue together with bailing wire.

Speaking of half-finished stuff, our time is up for this chunk of work. In doing these essays, my goal is to work in small chunks of work, making useful things happen in each chunk.

Up ahead is plugging up the SPA, which looks like it might involve another set of oddball things to deal with. So we’re going to have a Part 3 whether we want it or not.

I would consider this chunk of work a failure. It didn’t look too far to go from what we had to a SPA, but as it turns out the infrastructure itself is the biggest part of the hurdle. This is a pattern we’ve seen over and over again. We fight our tools, we fight the infrastructure, we fight the generic, happy-path ideas that good people have about how our work should happen. Welcome to coding, my friends!

But failures happen, and part of the reason for having such small chunks of work is to be able to spot failures early and keep an eye on whether or not you’re churning or actually delivering value. This time around we were churning. Stuff happens.

So what am I currently worried about as we finish this little chunk? The BASH stuff needs refactoring. No, we’re not making a BASH testing framework, but a small shell script is fine. And our script is reaching the point where it needs to be cleaned up.

I’m not sure how to hook Vue.js into what we have so far. Never done it before. So it’ll be fun! Coding is also adventure, and learning new intellectual stuff is some of the best adventures you can possibly have. This will be something to read about over the next few days while traveling and doing other things.

Why Vue? Because it looks like the current coolness. I hear good things about working with Vue components in large-scale systems — things I don’t hear about with the competing models. Plus I’ve worked with the other models. Ugly stuff. Typically I’ll walk into a shop where two people understand what’s going on and ten people are following along, coding bits and pieces, doing the best they can. Not a fun place.

F# is fun, though, and to keep it fun, we’ll do one more of this series and then take a look at writing a “real” app, one that could appear on app store somewhere. And we’ll do it one tiny chunk of testable useful work at a time, like always.

Thanks for hanging out! Fun times ahead.


Follow the author on Twitter

August 13, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *