Why TDD, Vim, and Python?

A lot of network engineers are learning programming, and this is great! There is no better feeling than scripting a massive firewall change and seeing it go off successfully. The focus of development efforts for network programmability can vary from person to person and company to company. But the thing I found that pulls people into writing scripts is the joy they experience when they overcome a challenge by learning something new. And this is a great way to begin any journey! As you progress, there are some valuable practices and skills you should add to your repertoire. Today we will be discussing test-driven development and text editing with Vim, using most everyone’s favorite programming language: Python.

Why TDD?

Even though I have been coding for close to five years, test-driven development (TDD) is a new practice for me. The first scripts that I wrote ignited an interest in learning to code to help with tasks at my job. As I developed more code, I started to get into the habit of testing every method I wrote by hand. I would run through multiple manual test iterations trying to break my code to make sure it worked before I would run it in production. One day, a colleague and friend of mine, knowing I was taking courses, had asked me what I knew about TDD and if I could help explain its usefulness and demonstrate how to do it. This caught me flatfooted. In all the programming classes I had taken to that point, I had never really encountered TDD. The conclusion I came to, incorrectly at the time, was that it is a practice some people do, but felt it was a bit redundant.

Oh, how wrong I was. It was not until I encountered a professor that was highly critical of the code the class submitted for projects (in a good way!) that I began to see the value in TDD. He told the class, “Always write your test data and expected results before writing any code.” This was intimidating, especially as I started to get his feedback on some early assignments. This was the first professor I encountered that was not just asking me to make something functional, but to make software following good development practices. Fortunately, I had sat across an amazing software engineer at a previous job. We chatted a lot and I knew if I was to get to the heart of what TDD is I could ask him.

My friend told me, “Regarding TDD, I think it’s one of those skills if you learn well once in life, it sticks with you forever, but it can be easy to build bad habits.” He went on to recommend some resources, but also said something that stuck with me. He told me, “To start out with the functional interface of what you are trying to test.” Why this statement resonated was due to my experience with Java interfaces. An interface is simply a contract, you expect something to go in and you expect something to come out. The code in between the input and output is entirely fungible. When I first learned about “programming to an interface” it seemed unnecessary, but I soon found I preferred interfaces over inheritance.

The practice of TDD makes you thoughtful. Even before any code hits the IDE or text editor, you must consider your implementation. Now, something I should have made clear earlier is the difference between unit testing and integration testing. Unit testing, which I have made a public repository in GitHub for, is used to verify pieces of code will produce the desired result. And those pieces of code are the functions/methods that I use to generate output from input. So why go through this practice when you believe the functions you write are rock solid? Well, in just my first foray with TDD I found a fallacy in my function logic.

I thought hard about what I should test first and recalled a Shell sort implementation I did for a class. For those of you not familiar with sorting algorithms, Wikipedia gives a solid overview. Now, a lot of sorting algorithms are found as existing functions in high-level languages. However, it is entirely possible to write these on your own and implement different algorithms based on your requirements. So now that I knew the first function I wanted to write, I started considering my test. Basically, I want to take an unsorted list and sort it in ascending order. The assertion is, once my function completes, I am left with a sorted list of integers. And an easy to test to write this was!

But, what value did I just gain by doing this? I mean, I’ve implemented a Shell sort before, and I know it works so did I just create more work for myself for the sake of it? Not at all. That function, shell_sort() could just as easily been called unstable_sort() and I would be free to implement any of the unstable sorting algorithms in this method. That means I can replace the entire body of code in my function and have a test to instantly validate (or assert) that I correctly implemented my algorithm. Now that’s cooking with fire! It also dawned on me that code is easy to break. Imagine you are doing a ‘search and replace’ in a large body of code. You figure you put the right search criteria in and instead of checking those 16 instances of the pattern you found, you just go ahead and replace all. What if you hit a piece of code you shouldn’t have? Worse yet, what if that change propagates up to the default branch and into production? Wouldn’t you want to know if something you did had an unintended consequence?

There are dangers to TDD though. I found this out in the second function I added for data munging. I basically wanted to deduplicate list entries and just return a sorted list with no repeated values. My original assertion is to make sure the list contains unique entries. I thought this should be enough. And I reused a piece of code I wrote for deduplicating IP addresses in a list. It worked! But I saw I was using a pre-canned function (called reverse) and I wanted to write my own code to implement this. What I ended up with was a list of unique entries, but I also entirely deleted the duplicate entries! This is bad. My test would pass since it met the uniqueness criteria, but I created bad data. Making matters even worse, it had inconsistent results on sorted and unsorted lists. Just imagine tracking this bug down if this was buried in tens of thousands of lines of code and your unit testing giving you false positives!

Well, this is where that exploratory testing comes in handy. I saw my unit testing was passing, but my results were not what I wanted. This pushed me back into really thinking about what I should be testing for my deduplication method. I included testing logic for making sure I removed duplicate values but retained a single value for maintaining it in the sorted list. I was able to now see my function was failing the test I had given it! I went back to my original implementation and saw that it had worked just fine! Looking back on that project I wrote the deduplication method for, I honestly believe I just got lucky. I remember Googling what I needed and slapping it together and moving on. It is frightening realizing things you used in production environments were successful not out of true understanding, but a mix of luck and enough knowledge to be dangerous.

I write all this to answer the questions that were posed to me by my colleague and friend, “What do I know about TDD? And can I explain its usefulness and demonstrate how to do it?” I am still early in this process of learning TDD and I plan to develop this skill more, so the next time I am asked I can not only answer these kind of questions, but field code that I know, with confidence, will work.

Why Vim?

IDE’s are great! They can alert you to errors in your code, help you import the right libraries or packages, redirect you to documentation, facilitate refactoring, and more. So, if an IDE can do all these really great things, why use Vim? Well, the crux of code development is being able to write, edit, and save text. And the simplest way to do work with text is with a text editor. And what better to learn than the successor to Vi? I had heard stories of people using Vi/Vim to develop, often citing the speed at which they can develop code, but I was skeptical at first. Having needed to use Vi on numerous occasions before, I found it frustrating and confounding. A simple open, edit, copy/paste, and save seemed like a major accomplishment. I was content to live in my bubble for a bit until one day I decided to check out Vimtutor after seeing something about it in a blog post. Who would have thought a simple tutorial could make Vim accessible for a first time user? After several hours of going through the tutorial, Vim was no longer intimidating, and I felt more comfortable writing code with it.

Despite gaining confidence in using Vim, I still found myself consistently using an IDE, whether it was for Java, Python, PowerShell, etc. I never really stopped to consider just how much help an IDE provided until I challenged myself to complete all projects in a class using Vim. Initially it was slow going, and I was thankful the assignments were not overly difficult. I found I would write some code then immediately compile and run it to check for errors. Again, I was practicing unit testing without even knowing what it was called. When I was coding in an IDE I could see through highlighting and error detection that my code seemed decent and I would just keep moving on to the next block, never testing until much later. And I would have some great fun debugging! However, with Vim I was constantly moving between writing and testing since my safety net had been removed. But, like most habits if you do not stick with it, it disappears. After the class ended, I slipped back into only using an IDE.

I may not have been a Vim enthusiast after that experiment, but I had learned to appreciate text editing more. I think one of the most valuable things about learning is picking up ideas and giving them an opportunity to marinate. When I set out to answer my friend’s questions about TDD I had no intention of using Vim, but something just clicked. I had done a form of TDD without even realizing it thanks to stepping away from the IDE. The error checking was no longer built-in, it was up to me, the developer, to do it all. Furthermore, text editing forces you to recall what you have learned about a language since there is no context sensitive help to get you to where you want to go. And that’s how Vim got thrown into the mix!

Why Python, TDD, and Vim?

TDD puts you in the mindset of figuring out your functional interface, what you expect to put into your function and what you expect to get out. Writing without an IDE forces you to recall what you have learned about the tools you plan to use. Vim also keeps your hands on the keyboard and off the mouse. Finally, any language would suffice. Python just felt more accessible and the lessons learned here can be carried forward to Java, C, and well, any programming language for that matter!