Stupid F#: Taking JSON to the SPA 3

Programming should be fun. All you need is a good set of values, some skill, and the right attitude. F# is the most fun I’ve had in a programming language in years. This essay series is about that: having fun. Most books and essays are about writing awesome code. There’s not a lot of material about writing code aweseomely. The code you’ll find here has bugs! Just like your code. Can you find them? If you’re looking for the final version of the code, you can find it in the project’s GitHub page. You can find the story of how all of this started on the series index page.

We’ve coded a console/command-line app that works with text, cgi forms, and Json. We have automated tests for that app, both in the code and in an O/S batch file. Now can we make it work with Json?

The first thing we do is take the original CGI form and copy it to a new html file that uses Bootstrap and Vue. This is a good time to add the Vue DevTools as a Chrome extension. I’ll also throw in some boilerplate stuff I used on another page on this site, things like Google Analytics and Font Awesome. (I know I’m breaking my rule about not adding something unless I need it, but this should be a quick, in-and-out hunk of code.)

Now we have a “blank” slate to start working in. If you want to see the finished code, you should get it from GitHub.

<!DOCTYPE html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>The Stupid App in a SPA setting</title>
    <meta name="description" content="A little page with Json, Vue, and Bootstrap">
    <meta name="author" content="Daniel B. Markham">
    <link rel="icon" type="image/gif" href="images/onair.gif">
    <link rel="stylesheet" href="css/bootstrap.min.css">
    <link rel="stylesheet" href="css/sticky-footer-navbar.css">
    <link rel="stylesheet" href="css/fa-svg-with-js.css">
    <link rel="stylesheet" href="css/main.css">
    <style>
        body {
            background-imageurl('images/woof.png');
            background-sizecover;
        }
    </style>
</head>
<body>
 
    <header>
        <h1>The Stupid App, Now With Json!</h1>
    </header>
    <!-- Begin page content -->
    <main class="container">
        <div class="row">
            <div class="col-xs-12 col-md-12 col-lg-12 text-center">
                <p>
                    Enter your name and numbers here<br />
                    Use the format NAME=VAL
                </p>
            </div>
        </div>
        <div class="row">
            <div class="col-xs-12 col-md-12 col-lg-12 text-center">
                <form id="myForm" action="http://info-ops.org/cgi-bin/st.cmd" method="post">
                    <textarea name="myInput" id="myInput"></textarea><br />
                    <button type="submit">SUBMIT</button>
                </form>
            </div>
        </div>
    </main>
 
    <script src="js/jquery-3.2.1.min.js"></script>
    <script src="js/popper.js"></script>
    <script src="js/fontawesome-all.min.js"></script>
    <script src="js/bootstrap.min.js"></script>
    <script src="js/vue.js"></script>
    <script src="js/main.js" type=""></script>
    <script async src="https://www.googletagmanager.com/gtag/js?id=UA-XXXXXXXXX-1"></script>
</body>
</html>

That’s a lot of stuff for a starting point. Web programming somehow went off-the-rails early on and has yet to recover. It is what it is.

Next we’ll add in some standard HTML controls to add new lines of data. At the top let’s have a list of the processed data updated from the server. Next we’ll have a list of the user’s data. It will have an ‘X’ on the right of each item to delete bad rows. Finally an entry box and an “Add” button. Could we have done a real-time grid-like interface? Sure. But why? The point right now is to exercise the types back on the app, not impress folks with slick UIs.

This is also the time to switch over to SublimeText to do our page editing. It’ll work with WinSCP to keep the page updated. If we continue this series, and we do a lot more web programming, we’ll update our tools accordingly, bit-by-bit depending on need. Right now all we need is to be able to edit and auto-update one web page on one server. As a pro, you should be able to scale your working toolset up and down depending on need, right?

First thing we do is throw some static data on the page to make sure everything, all of the “wires” are connected correctly. Make an array and hook Vue.js to both it and a table. Here’s that result:

It’s not the end to world hunger, but it’s working

Next we take some of the sample Json output we have (isn’t it nice to already have Json output done?) and paste it in the web page, to make sure our bindings are correct. We’ll be returning back this data from the server.

That’s easy. We just stick the data in as a mock. Next we’re going to need to get an actual file from somewhere, so we take some of our output and make a test.json file on the server.

We’ll need to load that file — which will happen asynchronously as far we’re concerned.

At this point I had a little internal debate about tools. Is it time to go full REST with a big bunch of cool new stuff? The reason I started thinking about plugging in huge frameworks was because it was painful — much more painful than I felt it should be — to load a simple json file into my page. I attributed it to being cold. I lost some skills. I kept poking around until I figured it out.

I decided against a huge load of tools, opting instead for this nice Axios library which handles interacting with external data through the use of promises. Axios is new to me, so new learning opportunity ahead. Yay!

This was important because I always want to be able to load dummy files from a server into my webapp, no matter what other tools I may ever use. Plus promises are the way to go. And the bare-bones browser data loading stuff is arcane. Data loading is maturing. It’s coming up on a decade old. The foundational tools are starting to get into place the same way JQuery got into place way back in the dark ages. These foundational tools are the ones you want to focus your learning on.

I added the min version of Axios to my list of Javascript libraries to load, then updated my Vue object that handles the result like so:

        var resultModel = new Vue({
            el:'#resultData',
            data: {
              "OptionExampleFileLines": [
                {
                  "NameIntPair": {
                    "Name": "dummy",
                    "Number": -1
                  }
                }
              ]
            },
            created: function(){
                this.getTestJson();
            },
            methods: {
              getTestJson: function() {
                axios.get("https://info-ops.org/test.json")
                  .then(response => {this.OptionExampleFileLines = response.data.OptionExampleFileLines})
              }
          }
        })

On Vue’s “created” method, we call the getTestJson method we just wrote, loading in the sample file from the server and binding to our data. Vue takes care of the rest. (During this I learned some cool stuff about Vue. I am much impressed.)

There are only two things left to do: adding the code to allow the user to enter their data and calling our CGI app we wrote previously to grab the result. First let’s let the user add data.

As you can see in the screen shot above, I’ve decided on a text box and a button in the middle of the page, which is about as simple as you can make this. (Remember, we do functionality first, then look-and-feel). The user will enter in data in the format of NAME=NUMBER, just like always. First thing we’ll do is prompt them with the appropriate HTML decorations.

This is just standard Bootstrap stuff. You can find code like it in their examples page

<div class="row">
    <div class="col-xs-12 col-md-12 col-lg-12 text-center">
        <form id="addStuff">
            <input id="newNameValueField" type="text" class="form-control" aria-describedby="nnvHelp" placeholder="New name=value" />
            <small id="nnvHelp" class="form-text text-muted">Name can be any alpha. Number must be an integer, eg "dog food=7".</small>
            <input id="newNameValueSubmitButton" type="button" value="Add This" class="form=control" />
        </form>
    </div>
</div>

(Sidebar: I’m not happy with Sublime’s lack of ability to export a chunk of html to the blog. Perhaps there’s a plugin I’m mssing. Oh well. Somebody else’s problem. I think I’ll switch to something else, like VsCode, as soon as possible.)

This is all turning into pretty standard Javascript stuff, the kind any junior-level developer should be able to do. We write an onClick handler for the button and hook it to a function. Double yay! We also hook the form’s onSubmit event to that function. Inside the function, which we’ll call “addNewItem”, we bounce a regex against whatever the user enters, either adding to the “input” list and calling…something, or just deleting whatever they put in. Finally we clear the input box, set the focus back there, and return false so nothing else happens.

I lost about 30 minutes here, but it was all just silly stuff. I haven’t done JS in a while. I put parens around Js fields (aren’t we supposed to call them like F#? Doh!). I forgot the signature to make sure your Js functions get both the event and sender. That kind of thing. Nothing to write home about, just the usual PITA. At the end we get something like this:

function addNewItem(sender, event) {
    var inputValidationRegex = /^.+=[0-9]+$/;
    var stuffTheUserEntered = $("#newNameValueField").val();
            //alert(stuffTheUserEntered);
            if (null != stuffTheUserEntered.match(inputValidationRegex)) {
        if (1 == stuffTheUserEntered.match(inputValidationRegex).length) {
            alert('looks like we made it');
                    ///
                    /// DO SOMETHING COOL HERE!
                    ///
                }
    }
    $("#newNameValueField").val("");
    $('#newNameValueField').focus();
    event.preventDefault();
    event.stopPropagation();
    return false;
}

As you know by now, I’m happy to use long variable names, and yes, it’s fine to split things up into highly-verbose pieces. I’ll also throw an alert in while coding something up, if only for the few minutes it takes to load the page a couple of times. What can I say? I have no shame. (smile). I’m more interested in getting things running than posturing with my code. Craftsmanship rocks. Ego doesn’t. Everybody’s a hack at heart, and in the end all code looks the same to the machine.

What’s the “cool stuff” look like? First we split the string into two pieces. Then we stuff an item and add it to our incoming array. Once again, all standard stuff. Picking up a library you have to have and flailing around for an hour to get it to work? All part of the job. Picking up plain Javascript or basic OO or FP code? It should come really easily to you. If not, something’s wrong in your environment. Or your training.

First let’s load up the grid with new items. We’ll add an array.pop() in the “created” method of each model. That’ll delete our dummy data (which was necessary for data binding, I think. Beats me.) In the “incomingData” model, the place where people enter stuff, it looks like this:

            created: function(){
                this.$data.OptionExampleFileLines.pop();
            }

So now we have a single-page app that allows people to load up these little strings. We’re checking to make sure the strings are valid. We’re also loading a sample json file as dummy output. On the server side, we have an app we’ve developed to tally up these little name=value strings. Now we hook up the last wire, connecting the form to the CGI app.

Well dang, if you’ve been playing along with the home game, you realize that we’ve already called the CGI app from the server. We did that a couple of essays ago.

So we’ve already done this. If you’ll remember, we made a shell file on the server that the form called. It was “st.sh” and looked like this:

#!/bin/bash

mono stupid1.exe -IF:C -OF:w 

It takes a CGI Form GET and returns a little web page with the result, using the appropriate program parameters to handle the kind of input and output we want. So we’ll just make a new shell file that doesn’t care what we send but returns data. Then we’ll see if we can load that data. That file, ss.sh, looks like this, and lives back on the server.

#!/bin/bash

cat ./test.cgi | (mono stupid1.exe -IF:C -OF:J) 

We add in some Axios code to do the POST. This is basically just a copy-and-paste from a cheat sheet on Axios that I downloaded.

                    axios.post('https://www.info-ops.org/cgi-bin/ss.sh', {
                        dummy:''
                    })
                        .then(function(response){
                        console.log(response);    
                        })
                        .catch(function (error) {
                        console.log(error);
                    })

When we call this, we get a fail! Shows us for running so fast! Taking at a look at the Chrome debugger, it’s HTTP code 500.

Something happened. You figure it out.

This is about useless. So we go to the Apache error logs for this site where we find….

[Thu Aug 23 08:46:48.326797 2018] [cgid:error] [pid 28087:tid 139857582073600] [client 172.243.134.182:65087] malformed header from script 'ss.sh': Bad header: a=12

Bad header? Oh yeah, we just can’t return data directly back from the server, can we? Clients (and the server engine) expect headers to come first. Computers are so silly about details. Geesh.

At this point, I’m asking myself “Do I really want a new type of output format that’s JSON, just with headers? Or is the process of adding standard headers something that’s not related to my app at all?” Should I add another OF (Output Format) type to my command-line parameters to handle returning Json data over Apache to an app?

Frack that’s a lot of coupling, yet headers are important. I’m thinking that headers are not part of my app. Part of my project? Absolutely. This is why they call it DevOps. The responsibility is still mine. It just doesn’t go in the F# code. So, for right now, I hack it in BASH, making my line of code look like this:

echo -e "Content-Type: text/html\r\n\r\n$(cat ./test.cgi | (mono stupid1.exe -IF:C -OF:J) )"

Important lesson here: this is the type of thing that you’d completely miss if you weren’t taking your time standing up your stack. This is how teams full of really smart people end up creating good-intentioned abominations of all that is holy that make users want to hunt them down and hurt them. They’ve got some rock-solid, kick-ass C# code, but nobody knows what the hell the headers are or what they’re supposed to be. They’re just busy being code monkeys, not people who solve problems for other people using tech. (There is a lot here related to headers and performance that we won’t go over. This little chunk of value delivery covered a dozen topic areas just like this.)

Then, because things were going too well, I spend ten minutes getting all the dependencies in place over on the linux box. DevOps again. This is stuff you set up once in the pipeline, tweak, and then it always just works. (Well, kinda. Everything we touch is complicated)

We also ran into a CORS issue. CORS stands for Cross-Origin Resource Sharing. I know this because Google. That required another tweak to our one-line BASH script. Then it blew up yet again! But it blew up in a new way. This time in the new Axios library we’ve been playing around with. Woo hoo! I’ll just keep walking the stack until the thing submits to my will.

Check out this error in the Chrome console:

error = Error: Network Error at createError (https://info-ops.org/js/axios.js:1196:16) at XMLHttpRequest.handleError (https://info-ops.org/js/axios.js:1050:15)...

There’s nothing like “Error: Network Error at createError” to really explain things to a programmer. So we load up the full version of Axios, instead of the min version, open the Chrome debugger, and take a look. E-gads! My nice new shiny library has turned on me! This is what you get when you pick up tools. As Darth Vader said once, the deal has changed. Let’s pray it doesn’t change further.

I enjoy this kind of thing a lot when I’m doing it on-purpose. Nothing like poking around under the hood of somebody else’s code, figuring out how it all works. Different story entirely when I’m on a deadline. This is why I rant so much. It’s not because I don’t like new stuff. Hell I love new stuff. It’s because I want to be in control over when these productivity hits come up. One or two a week is fun. Seven in a day with a Product Owner on my back to deliver? Screw that. I refuse to ever do that again. This risk needs to be aggressively managed in our kind of work.

This began an adventure that played out over several hours and a few days in the evenings after work, mostly involving CORS, Apache, and CORB. The basic problem is that folks don’t like other sites using RESTful services on their sites. CORS and it’s brother CORB are set up to “pre-screen” calls to the server (and results) to make sure everything is okay.

All of this is fairly recent, occurring in the last 2-3 years. CORS makes sure that only apps that are approved can call my web programs. CORB makes sure that we’re not passing around the wrong kind of data. CORS controls access. CORB controls reads. It all seems rather simple but it’s the kind of extra bit of detail developers need to pick up as they code. You don’t need to be an expert in it, but you need to know what it is, how to set it up initially, and who to call to teach you how to set it up and maintain it in production. (Because you are responsible for the entire stack, right?) For now, we got it working. For those keeping count, the “Yay!” count is now 2.5.

My BASH ended up changing to this, which makes a temp file as it runs so I can inspect it for debugging. For quick-and-dirty projects, debugging also becomes an issue, since you really don’t need to configure and entire debugging environment yet. Just like the CORS thing, this is another heads-up that if we continue with this app, there’s work ahead setting up something serious for debugging. Heck, full-stack, full-pipeline debugging is most always a crapload more complicated than whatever the coding is. One line of BASH is getting off lucky.

echo -e "Content-Type: application/json\r\n\r\n$(mono stupid1.exe -IF:J -OF:J | tee /var/www/info-ops/public_html/cgi-bin/temp2)"

The workflow pattern in all of this is the same: we write tests, we hack it, the tests pass, we repeat, we standardize our hacks, repeat, we optimize our standardizations. It’s all the same whether it’s F# code or Operations scripting. We also add a technology or two at a time to the mix so that we can play around and learn stuff as we go along. Little bits. Baby steps.

The F# code changed very little, mainly around our new way of doing serialization.

For an F# series, there sure isn’t much F# here, right? I think the biggest change in our codebase was this:

ype NameIntPairType = {Name:string;Number:int}
// type shim
type NameIntPairTypeJson = {Name:string;Number:string} with
    member self.ToTextLine() = self.Name + "=" + self.Number + OSNewLine
type OptionExampleFileLineTypeJson={[<JsonPropertyAttribute>] NameIntPair:NameIntPairTypeJson} with
    member self.ToTextLine = self.NameIntPair.ToTextLine
type OptionExampleFileLinesJson ={[<JsonPropertyAttribute>] OptionExampleFileLines:OptionExampleFileLineTypeJson[]}

Interestingly, the thing that changed was the thing we thought we’d already done: Json persistence. That’s because for us the phrase “Json persistence” means one thing when you’re talking about on-disk, something else when it’s over the wire. Libraries will never be able to make app-level decisions.

There’s very little F# coding or pain around making the app work while delivering hits little chunk of value. It’s almost completely infrastructure related, and we had time to play around with new stuff and have fun. This is another pattern to look for: push the pain to the front, get the type system in place for basic functionality, stretch out the app to cover the entire DevOps chain, then add new functionality a tiny bit at a time, beefing out the pipeline and stack with each increment. Done this way, it hurts a lot at first, getting damned simple and fun towards the end. That’s how you know you’re doing it right: pain first, then fun. If you’re having fun at the very beginning of a project (aside from goofing off with the team and users, of course), you’re doing it backwards.

I spent quite a bit of time in the Chrome debugger playing around with Axios and getting Apache configured right. Also spent some time remembering Apache config stuff and I had fun learning about CORS and Axios. And sure enough: library or not, there I was down in the weeds with all the browser data stuff at the lowest-level anyway. All of that is great. Those are skills I can use later. (Aside from Axios perhaps. Time will tell for that library. Good luck guys!)

There was a ton of stuff I learned related to my specific configuration that I won’t share here, like network logging and OS configuration. But back to coding.

The last coding steps we have are to take away the dummy data I was loading in the result section, leaving the code commented out. The entire Vue.js model for the results looks like this now. Note no templates, no modules, nothing but a few lines of simple and readable JS.

var resultModel = new Vue({
    el: '#resultData',
    data: {
        "OptionExampleFileLines": [
            {
                "NameIntPair": {
                    "Name""dummy",
                    "Number": -1
                }
            }
        ]
    },
    created: function () {
        this.getTestJson();
        this.$data.OptionExampleFileLines.pop();
    },
    methods: {
        getTestJson: function () {
            axios.get("/test.json")
                    //.then(response => {this.OptionExampleFileLines = response.data.OptionExampleFileLines})
                }
    }
})
function addNewItem(sender, event) {
    var inputValidationRegex = /^.+=[0-9]+$/;
    var stuffTheUserEntered = $("#newNameValueField").val();
    if (null != stuffTheUserEntered.match(inputValidationRegex)) {
        if (1 == stuffTheUserEntered.match(inputValidationRegex).length) {
            var newName = stuffTheUserEntered.split("=")[0];
            var newNumber = stuffTheUserEntered.split("=")[1];
                    //alert(newName + "=" + newNumber);
                    var newItem = {
                "NameIntPair": {
                    "Name": newName,
                    "Number": newNumber
                }
            };
            incomingModel.$data.OptionExampleFileLines.push(newItem);
            axios.post('/cgi-bin/ss.sh', incomingModel.$data)
                .then(function (response) {
                    console.log(response);
                    resultModel.$data.OptionExampleFileLines = response.data.OptionExampleFileLines;
                })
                .catch(function (error) {
                    console.log(error);
                })
        }
    }
    $("#newNameValueField").val("");
    $('#newNameValueField').focus();
    event.preventDefault();
    event.stopPropagation();
    return false;
}

And our SPA looks like this:

Remember that there’s a GitHub link at the top of the page for the project. I’ll leave as an exercise to the reader to implement the delete button. It should be extremely trivial.

With our new SPA installed, I’m finishing up this part of the series. We started with a toy problem and took it from REPL to SPA. Now I need to decide whether to pick up a beefier problem and do a longer series where we build a bigger system out: web frameworks, our pipeline, a build server, Ansible, and so forth.

All-in-all, this series represents about a week or a week-and-a-half of programming time split up over a month or so, the kind of stuff you’d do when you first get rolling on a project. I did it all the while getting distracted for hours or days on end, going to meetings, and generally doing all kinds of other annoying stuff besides actually programming. The phone rang constantly. My kids asked to borrow money. I don’t know about you, but that’s the way I always end up coding. It’s always like this — unless I take drastic measures. When you code like this it looks very different from “classroom” coding!

Thanks for taking the time to tag along on this adventure! I’ve been doing a ketogenic diet lately. Maybe an app that tracks carb intake, activity, and allows for custom recipes would be fun? I’d love to do some stuff with user management, FAKE, maybe Fable, OAuth, and various app store submissions. Don’t know. Golly there’s a ton of stuff to play with. (smile)

Doh! Almost forgot. Here’s the page in case you want to play around with it. I have no idea if it will scale or not. (That means no, it will not scale)

If you’re interested in how I ended up with this way of coding — and the foundation to things like Extreme Programming, the Onion Architecture, and the Unix Philosophy, you should read my book on information management while building stuff for folks. It’ll change your career for the better. Plus? Tips on badger farming.


Follow the author on Twitter

August 24, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *