JohnVonNeumann

My end-to-end Git Workflow

2020-03-10T00:00:00+11:00

From master branch, to make sure you have a clean copy of your codebase, first, ensure you're on master:

    git branch

The output should be similar to this, with the "" next to master*:

    aaa1-lint-makefiles
    \* master
    setup-terraform-automation
    test-yamllint-ci-step

If your asterisk is positioned next to a different branch, you can return to the master branch with:

    git checkout master

Now that you are on master, you can branch into a feature branch for local development:

    git checkout -b {project_code}{issue_number}-short-summary

You will be prompted with your new branch name, otherwise you can re-run the first step to ensure you're now on the correct branch. Now you can continue your development.

When you have made an edit to a file, you will need to stage the change, this indicates to git that you wish to package it into a patch and introduce it to the codebase. Firstly though, we can check what changes are available for staging by running:

    git status

This will give us output detailing the current state of our local environment, showing a potential combination of tracked and untracked files.

Untracked files are files which have not been checked into our version control system. For example, if you have just created a new file, and have not committed that file into the code base.

Tracked files are files that have been checked into the version control system.

In this scenario, as we have not added anything, the above command will return:

    On branch aaa-test-example-branch
    nothing to commit, working tree clean

Lets say we add a file to the repository, we wish to commit this file, first, we perform some work on the file.

    touch example_file.py

We now verify that the file is in the repository and git is aware of it.

    git status

The output of which is:

    On branch aaa-test-example-branch
    Untracked files:
    (use "git add <file>..." to include in what will be committed)

    example_file.py

Now that we've confirmed git is aware of our file, we can stage it. With the name of your file, run:

    git add example_file.py

We can now check whether the file has been staged with:

    git status

You will receive output similar to this, indicating that the file is now ready to be committed, if this is the first time you have committed a particular file, it will state that it is a new file::

    Changes to be committed:
    (use "git reset HEAD <file>..." to unstage)

    new file:   example.py

You can now commit the change. Committing something is basically to package it into a change, and add it as a piece of work, consider it git's primary data unit, repositories are built up of hundreds or thousands of commits, each telling part of a story of a project.

    git commit

This will spawn a new window in the text editor specified in your git settings, allowing you to leave a message to go along with the commit.

Commit messages should help tell a story, the git log will live for the length of the solution, and should enable other engineers to understand how the code was written and in some cases, why it was written the way it was. A good git log helps engineers create a better solution, by providing them with valuable context to utilise in their day to day engineering.

A good structure to use for commit messages is:

{UPDATE, DELETE, CREATE} - A verb to describe at a high level what you are doing in this commit, for example you may be adding an entirely new file, so you may choose to start your message with "CREATE".
The file name you are impacting - This should be the file name of the file being worked upon, we may be creating the new python file, example_file.py so we can set this to "example_file.py"
A dash to separate the header of the message from the description to come.
A short description to tell the reader from a high level what/why you are doing. For example, "example script for git how-to blog". Ideally, this should be kept to under 78 characters, otherwise it will wrap in the ggit log and be harder for engineers to read.
Finally, underneath this, we add an extended explanation, after performing a double carriage return, this extended message will not show in the short log, only in the full log, so we can be more elaborate in our explanation of the change here, ultimately we wish to detail why we are performing the change, any interesting context around this change (Were their issues, Is this a workaround for a bug in a vendor software?), be detailed in your message. For example: "In order to explain the example git workflow in the how to guide, I had to create an example file with an add/commit workflow along with a simple way to understand the best way to set out a commit message. This change will illustrate an example that will assist developers craft better commit messages."

This leaves us with a final commit message of:

    CREATE example_file.py -  example for git how-to

    In order to explain the example git workflow in the how to guide, I
    had to create an example file with an add/commit workflow along with
    a simple way to understand the best way to set out a commit message.
    This change will illustrate an example that will assist developers
    craft better commit messages.

Type in your message, ensuring that you keep your subject line shorter than 50 characters and your body lines shorter than 72 characters otherwise the comments will not line wrap and people reading the git log will have an awkward time reading.

Now that we have written out our commit, we can save it:

:wq

Our output will look something like this:

    [aaa-test-example-branch 6b20c8a] CREATE example_file.py -  example for git how-to
     1 file changed, 0 insertions(+), 0 deletions(-)
     create mode 100644 example_file.py

We can now check whether our commit was created with another call to status, if the commit has been created, we will not receive any additional output than we did from our original call to git status at the start of the guide:

    git status

However, if we check the git log, we will see our commit at the HEAD of the chain:

    git log

Returning:

    commit 6b20c8aada7ae3c97fed086ab00e86d0312224fe (HEAD -> aaa-test-example-branch)
    Author: JohnVonNeumann <louiswillcock@gmail.com>
    Date:   Thu Oct 17 14:31:50 2019 +1100

    CREATE example_file.py -  example for git how-to

    In order to explain the example git workflow in the how to guide, I
    had to create an example file with an add/commit workflow along with
    a simple way to understand the best way to set out a commit message.
    This change will illustrate an example that will assist developers
    craft better commit messages.

This shows that we have indeed committed our change, the only issue is that it only exists on our local repository, in order to tell our source of truth about it, the remote; we must push the code.

    git push origin branch-name

Now that our change is in our remote branch, in order to have this change integrated into the master branch we must issue a merge request or a pull request depending on the platform you're currently using to support git.

This will typically be done via the GUI, so we can navigate to there to issue the merge request, you can search online to find out how to do this.

By this stage, it will be assumed that you have received the correct number of approvals for your branch and new code, so you can proceed to get the thing merged.

We position ourselves onto the local branch we are working on, use the processes mentioned up above, in this example we are going to assume that inside the master branch there are commits that have been merged since we started developing our code, as this will be the trickiest situation to deal with.

Once you are located on the branch, you can pull the changes from master into your branch, which will add changes and place them onto your branch, effectively removing the need for dirty merge commits that clog up the git log.

    git pull --rebase origin master

We can assert that the changes have been pulled into our branch with our handy friend git log.

Finally, now that the changes from remote master are in your local branch, you can push them upto your remote branch, however, as you have pulled in changes from elsewhere and performed a rebase, your remote branch will complain that the tip of your local branch is behind the remote. This is incorrect, we are ahead, so we need to force the changes upon the remote.

    git push origin branch-name --force-with-lease

Finally, you can either migrate back to the GUI and use the button to merge the code, or you can do it from your terminal if you have the privileges to merge code.

    git fetch origin
    git checkout master
    git merge branch-name
    git push origin master

Repository Design

2019-08-04T00:00:00+10:00

Changing the way we think about repository UX.

Since I started working in software, and more specifically, "DevOps", I have had to work with a smorgasbord of irregular repositories. They have varied in every way you can imagine, much of the time with relative impunity.

No one likes working with repositories like this. They are dirty and downright depressing in some instances, to move from a "clean" repository with good quality CI/CD, automation and quality control, into an old and crusty repository, without these things, feels wrong. I will make the generalisation that when people work in one of these "forsaken" repositories, they tend to work to a much lower level of quality than they might usually, because the state of the repository, does not inspire the need to work to a high level of quality. This is a problem.

Broken Windows Theory

The broken windows theory is a criminological theory that states that visible signs of crime, anti-social behaviour, and civil disorder create an urban environment that encourages further crime and disorder, including serious crimes. The theory suggests that policing methods that target minor crimes such as vandalism, public drinking, and fare evasion help to create an atmosphere of order and lawfulness, thereby preventing more serious crimes.

I believe that the broken windows theory exists within software. I believe that human operators; software engineers, will instinctively checkout of good quality work if the environment they are working in has "visible signs of crime, anti-social behaviour, and civil disorder". It's a fairly understandable reaction.

The reasons for such "lawlessness" are enumerate, timelines, resources, laziness and general bad work, lack of knowledge/ignorance to better ways of doing work, lack of good quality tooling, etc. There are a host of bogeyman that can be used to blame this situation on.

Programming has had a definition of "good" for a while now

There is so much content out there surrounding general software engineering and what constitutes good quality code that in many ways, we have developed a natural sense for what can be considered good quality. We "walk" into a codebase, and can see the hallmarks of greatness, and conversely, the hallmarks of a dumpster fire. This is endemic, it occurs without us considering it, we look at a codebase and say "this is garbage", we have developed a keen sense for garbagism and what its ugly little offspring look like. This is good.

Programming actually has a number of frameworks to support good quality code creation, and these frameworks are the logical backing that provides us our quick reflex "garbage identification". Extreme Programming, 12 Factor Application Design, Clean Code. All great examples of the work that has been put into creating good quality, well backed theory, to support the creation of good quality software engineering output.

So where is ours?

Despite programmers (I mean this in the traditional Operations-Development sense) having had this structure and overall definition of "good work" for a relatively large amount of time, the DevOps space in many ways seems to struggle with the idea of taking some of this and relaying it into our own space.

We still struggle to (from my experience) standardise what a "good repository" should look like, with deltas in implementation occurring across organisations everywhere. This problem is only amplified with the shift to SaaS based git hosting, where new repositories cost nothing to create and more organisations move to multi-repository set-ups.

Personally, I am tired of having to enter a repository and add a gitignore with the basic ignores, add a CI file that'll largely look the same within the same language ecosystem, and I'm definitely tired of having to keep either a mental check-list of repository actions or consult a README file, then copy pasta the directives across, hoping they work.

Enter "Repository Design"

My thoughts around how to fix this issue have started to fall under my own mental banner of "Repository Design", which I use as a way of thinking about what the best practice for a repository should be and how we as developers can go about designing good quality repositories.

I personally believe that the repository should be viewed in many ways as a product, a shippable that is self contained and can be quantified in terms of its quality. As we encounter a heavier push towards automation/infrastructure as code, it is important to highlight the delivery mechanism for this code, and having a publicly understood "API" for a repository, should help push us towards better automation. I also believe that repository automation is an area ripe for improvement, with massive gains to be achieved through simple adjustments in repository work flows.

An important distinction of Repository Design is the focus on the overall User Experience of a repository.

How easily can a repository be used?

In many cases where large scale automation is done well, many people shy away from using the end product to its fullest potential what can be extremely complicated automation if they feel they either don't understand it or do not possess the skills to fix fuck ups if they occur. In many ways, a defining feature of good quality automation is that it is typically low maintenance once it is in place. You've already automated the task, so in many cases, the automation may sit chugging along for months, sometimes years, untouched, before human eyes gaze over it again. In my experience, the easiest way to detract value and usage from your otherwise perfect automation solutions is to not pay close enough attention to the very real human operators that will be using it.

As a simple example of the type of high level goals of Repository Design consider your local development environment:

It is most likely a 16GB+ machine, with plenty of free compute and zero network latency (to yourself), yet we consistently push all menial linting tasks off your local environment and onto a remote CI/CD server. How much time is lost in the process of creating the commits, sending them to your git hosting provider, having the build picked up, initialisation of a build agent, then it fails because you used a tab instead of a space.

Repository Design does not seek to remove CI/CD or anything of the kind, it simply seeks to allow developers to use the resources they have, properly, to enable better workflows for everyone involved. We wish to keep all the tasks that can be done locally, within the current CI/CD process, but also bring what we can internal, to quicken the feedback loops of development.

No need for new tooling

The best thing about this situation is that in a large amount of situations, we have no real need for newer tooling.

Repository Action/API Automation - Makefiles
Automated Code Edit Automation - Githooks
Stable, repeatable linting - Docker with Linters

This list is to continue being fleshed out, but for the time being I'll leave this where it is.

Older tooling, re-purposed with additional wrapping can provide great solutions to existing, boring problems, that for the most part are theoretically solved, but not in an automated fashion.

In some cases, it's not even that you need complicated automation, but a tool that generated simple basic default gitignore files would be a godsend, how many times have you gone into a repository littered with garbage like .DS_Store files, that a coworker hasn't put in the .gitignore but have now been committed into the repository? I know I've fixed this numerous times. And the main issue with this?

It's fucking boring.

No one likes doing stuff like this, but if you are an engineer with high standards, eventually, with enough time spent in one repository, these bugs will get under your skin and you'll want to fix them.

So you do fix it. And the world is good. Then you go to another repository...

Conclusion

I'll be putting together some ideas/tooling around this in the months to come, I think these issues have bounced around inside my head long enough now that I've finally decided to automate these garbage tasks so I can focus on personally higher value work, this is what automation is about, moving humans towards higher value work.

I've been building a simple tool I'm calling pstart, and will have a simple "beta" release available soon when I have around 3 of the core features delivered. It's a simple tool that will allow for contextual repository management, regardless of the stage of life the repository currently exists in. I hope for it to be a way for automation folks (and everyone else) to take back some of their jobs, and usher in a more sensible way of thinking about repositories. As usual, it'll be opinionated, but it will also be Open Source. So feel free to make a pull. It's unit tested and all that, so I intend for it to be a good quality production.

Keep an eye on my Github at: https://github.com/JohnVonNeumann

Cheers for reading.