EP 032 Arduino Unit – Testing with Arduino

Join Mike as he speaks with Dr. Warren MacEvoy about Test Driven Development using ArduinoUnit.

Transcript:

Dr. MacEvoy:
All right. I’m a professor and one class that I teach is software engineering. It’s a capstone class for people in computer science. I was trying to summarize something about what it is to be good at this as a field, or what is the engineering part of this? One, I think, key thing is, I think I’d call it being less nervous. I think a lot of good practices that come in the area of building really anything has to do with being more comfortable or less nervous in the process of building that. It’s fine if like working with, I don’t know, your boy scout club or whatever is, and you make something and it falls over and you try it and you do it, and it doesn’t quite work. The go-kart that falls apart, is an okay experience. It’s actually really valuable to get started on how do you make something?

But I would hope that in most people’s minds that hopefully that’s not what say Ford uses to build a car. You would hope that you’re pretty consistently going to have a safe, reliable vehicle when you go to some big place like that and ask for a car and you get one. It’s mostly going to work. Of course it’s a complex system, it’s not like it’s a perfect thing, but there’s a certain amount of comfort from the point of view of you using it, that side as a consumer, using that thing. But you can made it into that a little bit and say, well, hopefully the engineers that are at work doing that aren’t nervous every day about, man, is this car going to blow up today? Is it all going to fall apart? Does their process allow the people that are building things to be less nervous? What are the things that are in the process of making new things that help you be less nervous, be more confident that the thing you’re making really is going to solve the problem that you want it to solve?

Testing is one of those big parts of that. Nobody I think should try to put a car altogether without having ever say, tested the motor before they put it in the car. If you just put everything in a car and just hope that when you turn the ignition on, it’s going to work, the likelihood of that all working out just goes way down. You can go [inaudible 00:02:36] into that. The engine, before they put the engine together, hopefully they tested all the bolts and the wires and all that stuff.

The process of building reliable things is part of what hopefully keeps you from having a heart attack at a young age if you’re an engineer. If all the time when you’re building stuff, you don’t know how it’s going to work out and what’s going to happen, and whether things would go south, that’s a pretty high level of stress that you experience all the time. If you didn’t feel like, oh yeah, this worked 400 times before probably going to work the 400 and first time. And so how do you build that confidence? How do you feel like, okay, you’re building an embedded system? Embedded, I guess is an interesting word to me, it just means inside of something else.

Apparently, if you’re an embedded system, you’re trying to build something to sort of, it is going to be the motor or something like that inside of something else. Therefore, maybe it should be reliable. If you went to the store and you bought a carburetor or something like that, you probably want it to be one that will work by the time you put it in something else. And so how do you make it so that, well, so that the people watch your thing are pretty happy about when they get it, it works? And then on the other side of it is, while you’re making that thing, how comfortable do you want to be that, oh yeah, it’s going to work, like it’s going to do the things you want it to do? And so that’s testing, I think. I mean, it’s a big part of it. Big part of your assessment.

There’s a couple of other things like version control. That’s super important. But we’re focusing on, well, what are the swords in your box of tools to fight the monsters of software development? Testing is a pretty mighty sword from that point of view of feeling like, yeah, it’s going to do all the things you wanted it to do by the time you’re done. And so there’s a couple of different ways you can call it, but I was trying to think about this as another philosophy thing I was thinking about [inaudible 00:04:36]. What is a test? I think a pretty good answer for a test, the test is anything you do to some component, that is somehow a lot less expensive than the actual thing.

To like an extreme example would be an interplanetary, satellite launch thing. The actual device landing on Mars is both ridiculously expensive, and a very one-off thing. Also it’s not just money that is a stake in those sorts of things, it’s just reputation. If NASA sent a lot of stuff that crashed all the time, NASA would stop existing. The funding of NASA would just disappear if they weren’t pretty consistently providing positive results. If their version of solving stuff was just build another one and throw it at Mars, that would maybe cost a lot of money, but in the end the reputation cost in their case might be more than the financial cost of a failure. That’s an example of where the actual event is really expensive and therefore when you go look at NASA and the subcontractors of NASA that built stuff, tests look really expensive, but they’re trivially expensive compared to the cost of the actual event.

Dr. MacEvoy:
The cost of a dry run on one of these solid rocket motors or something like that, must be in the millions of dollars. As they put one together and find some safe place to shut it off and put all the instrumentation around it and do all that stuff. Yeah, there’s a lot of money they just blew out the back door basically when they lift that off. But that’s a lot less than having one of these rockets blow up on a watch bag. In embedded systems, or really in testing in general, I think what you’re after is like, okay, maybe identify how expensive, what’s at risk for you when it’s to put out there? I tried to start with, it’s not just money or something like that. It’s not just that. It’s, I think from a personal wellbeing point of view, so how nervous do you want your life to be?

If you’re handing things over to your boss and you’re not sure if it’s going to work every week, then that’s probably not a good situation. Living in this state of nervousness all the time, I guess is a human expense. In software, one of the great divisions of what software is compared to most of the other fields of engineering, is the cost of trying things out is usually really low. Hitting compile, most of the time is pretty cheap. Tests have to be something where there’s very little at risk when you run them. Hopefully that means they’re smaller, they don’t somehow bet the firm that these are going to work before they run.

And so in that way, it really covers a lot of ground. Really, it just depends on how much confidence do you want in how something works? For a med system that could mean the lab test or the bench test where you take the entire system with all the hooked up electronics, but don’t wire in some high power component that if it breaks, that’s a $10,000 loss or something like that. Just make sure that all the rest of it works before that integration happens. That could be a test. Tests, what are the good tests? Dry runs of anything is a pretty good test.

Getting back to the word, embedded, an interesting thing about Arduinos and that world is, in a bigger system if you’re writing software on a laptop or something like that, a lot of times you have a lot of resources to look at it. There’s things like debuggers. You can step through the code. You can do a lot of print statements even if that’s old school, it works and log stuff out to log directories and ask [inaudible 00:09:00] to trace how things went. In some sense because the software is running in a pretty big system and that big system has a lot of resources, then you can, I guess, it’s stick to words, you can instrument. The word instrument means to [inaudible 00:09:16]. Putting a pressure sensor or something like that on something to get some data out of it of how things are working.

You can take a piece of software and you can instrument it so you’re going to see how it’s going. How much memories are using, where’s it getting to? Instrumenting in bigger systems like stuff that you could build on a laptop or on a server somewhere, is way easier than instrumenting in these tiny systems. Basically because they’re tiny. There’s not room for regular logging or some giant log file that you can go back and look at, that’s living on the system. A lot of times the system is responding to events that are going so quickly that if you spent the extra time to do the logging, you couldn’t actually accomplish the task that you wanted to do. You can’t sit there and spit out a bunch of log events while you’re trying to do some [inaudible 00:10:15] pulse with modulation or something like that on some line that’s supposed to be going at a megahertz.

Well, you can’t log that like it’s not a reasonable thing to think you could do. A typical Arduino has 64 kilobytes of flash and have like two kilobytes or eight kilobytes of RAM maybe. It’s just not a big space. And so you just have enough room to solve a problem that you need to solve. You can’t, or it’s hard anyways, to add anything else that instruments what’s going on in there if it’s not going well.

Tests, back to tests. Just before we’re talking about these programming contest, problems, one tests, or is it called a black box test where you don’t look inside or something, you just use it on the outside. In the embedded system world, that might be something like take off the robot arm because you don’t trust that it’s going to work correctly, but just on a test, all the rest of the stuff or hookup a robotic arm that’s really cheap, compared to the expensive one and make sure that it’s doing the right things before you hook up the one that’s really expensive.

That would be a version of a black box test of the little controller because you’re not changing anything inside the controller, you’re doing something on the outside and seeing if it’s working the right way. Not being able to inspect things on the inside is a good idea from the point of view of, it’s going to work, but it’s also just very obscure. If something goes wrong, if that server arm or something does something wild, it’s hard to tell why. It just something bad happened. If you were a mechanic, then that’d be like, throw that board away and try a new one. But if you’re the person building that board, it’s not very helpful because you know there’s something wrong in there, but you know the next one’s going to be just as wrong as the last one until you figure out what’s in there that made it do the wrong thing. And so black box testing I think is useful, but what’s called white box testing is to try to somehow get information from the inside of that device and discover what’s going on.

What kinds of tests? Arduinos are tiny. They can’t hold very much stuff and so the goal of this ArduinoUnit library is to be correspondingly tiny and try to use this as a smaller footprint, not a no as possible. You can always do less, and that is a perfectly good answer in some cases. Arduinos have LEDs, and certainly I think a good part of testing is blinking the LED so you can sit there and like, oh yeah, I guess it got to that [inaudible 00:13:11] routine, because it’s flashing twice now. That is a version of testing. It’s like looking inside that box and you could do that without any testing official environment at all, but just appreciate that, okay, I can create my version of video lights. And so then, then as you write the code, you have to make sure that you hopefully consistently put the lights on in some way so that you can see a little bit about what’s going on inside as the system is working.

What ArduinoUnit tries to do is mimic other bigger instrumentation libraries, but in a tiny way. See, Google Test is a library for C, C++. I guess, for your audience, I don’t know if they would know they don’t make it, they don’t advertise it very well, but Arduinos, the language that you’re writing in is C++. It has a very custom set of libraries in ArduinoUnit to.. Sorry, Arduinos have a quite custom set of libraries that you’re interacting with, but the core language that you have is this thing called C++. And so in the bigger context, like the laptop, if you wanted to write unit tests and have a friendly environment for writing those kinds of tests, one reasonably popular choice is something called Google Test. What it creates are, I said they look like functions. You write tests, the word test in caps because it’s important apparently. And then you just write two words, it reminds you what things you’re testing.

It’s usually the component and then the property, that component you’re testing. So sticking to what we had for the test robotic arm, reach for coffee, or something like that, the test that you want to try out. And then some curly brackets in what things you want to have workout. You write that kind of a test down, and then you say, well you initiate the movement or something like that. And then somehow monitor that it’s working the way you wanted it to. And so you could write a test in a Google Test framework that looks something like that. Why is that different from just a plain old function? One answer is, it’s not. What makes those useful, is it lets you quickly filter and say, well, I want to run all the arm tests or maybe I want to skip all the arm tests. Now I want to run all the arm tests or I want to only want to run this one test, the one that’s failing so I can repeatedly check until I fix it and it’s working the right way.

These test frameworks tend to be good at reporting. They give you a nice report of what’s going well and what’s not going well on the system, and selectivity. As a developer if you realize something’s going wrong, then it’s nice to be able to zero in on just that one thing until you’ve made it better. And so it’s nice to be able to filter down to, just run this one test, not the other ones. I want to figure out why that is not working. ArduinoUnit, has that stuff going on at the microcontroller level. I guess so the word unit in a lot of these testing frameworks actually has a meaning. If you say unit testing, I’ll say the whole field has a certain amount of religion to it. If you have say certain words, people get very upset if you use them the wrong way. I’ll say ArduinoUnit uses the word unit mostly because of salesmanship. It’s the same reason why JavaScript was native JavaScript. JavaScript has nothing to do with Java, it’s just that Java was really popular when JavaScript came out, so they decided to call it JavaScript.

I think ArduinoUnit did the same thing. It’s like, okay, everybody touched on unit testing, so we should call it ArduinoUnit, then people will know that you can do unit testing with it. You can. You can always test anything else you want to in there too. It really doesn’t care about what kinds of things you want to test. ArduinoUnit, well, you can call a unit anything you want, so test this thing and this component, and then a bit of a difference between the Google Test stuff and what ArduinoUnit does is, you can have tests that are run once tests, which is a typical Google Test kind of thing. Like run this thing, you’d expect this output. But in embedded systems, I think a good point of view of med systems is they’re supposed to run forever. Typically the idea is you power them on and they just keep going. That means they can break at any time too. You just go.

In ArduinoUnit there’s tests to label test, and then there’s testing and testing are the ones that are just repeatedly run. If you have something that you want to test on a more or less continuous basis, you call those testings, and the other tests are the ones that you want to do maybe one set boot up time to see that things are going the way you want, and then let them go. They’re all filterable, there’s a little tools in there to choose which ones you want to run. You can run all of them or not [inaudible 00:18:34]. It has a little meta things about grouping the tests together. You can test how other tests have gone in the setup to figure out, well what are some little philosophical ones?

Embedded systems. What makes a reliable thing? Here’s a little philosophy. Reliable things are things that work correctly if you use them right, and will tell you that they’re broken if they’re broken. A thing that is a reliable thing, will somehow have a green light when it’s configured right and everything’s okay. There’s also some red light, they will tell you there’s something wrong and it’s not going to work for you. When you’re talking about tests and reliability when you’re building embedded systems, I think it’s easy to think, well, I want to make sure that all the good things work, but reliable things worry about all the bad things too. There’s a saying, or I guess a fact in computer science, about 70% of the code that gets written, gets written for the error cases. Especially people that are young coders, they tend to be happy when the green light case works.

It’s like, oh, I made it all work when all the things happen. But in fact, you’re only maybe 20% done at that point. Now you have to worry about, what do you do when all the things happened that you wished didn’t but will happen anyways? What happens when you have a brownout? What happens when you lose a connection, like a serial port drops and resets? All those things tend to be pretty significant amounts of effort to get right, and will be a lot of part of what it means to make a reliable thing.

ArduinoUnit tries to create the situations where you can build those kinds of tests and both check the green lights and the red lights about how they work. Actually it’s more than just like a test one thing, you can have ArduinoUnit be in the living system and those testing components can just sit there and monitor the system and tell you, oh yeah. That’s a good place where you can wire in the red lights so someone on the outside can say, oh, there’s something wrong right now, it’s not doing the right thing. Have you had any experience with like testing environments or anything like that?

Mike:
I have. I’ve used like JUnit before. It’s probably the one I’m most familiar. And then I’ve used Python unittests. And so I’m familiar with those testing frameworks and I think where I think I’m struggling the most to wrap my head around ArduinoUnit, is when I think of a unit test, I think of taking a function or a method and then testing the logic in that function or method. With ArduinoUnit, what I’m not sure I understand, and again, this is just my naivety here shining through my ignorance is, with ArduinoUnit, are we writing tests for logic or are we writing tests for the functional output of the system? For example, we’re making sure that we’re checking that digital pin 12 is actually on, kind of thing, versus maybe the logic in the system. I guess that’s where I’m trying to figure out. I don’t know if this is the right terminology again, would it be like a functional test thing, versa a logic unit test? Does that question even makes sense? But that’s where my brain has been a little confused.

Dr. MacEvoy:
No, that makes great sense. I think, I’ll say ArduinoUnit is non-denominational that way, it can do both kinds of tests. You can write a test that checks to see if some function is giving the proper outputs. It’s more of what I would call an internal test. More of a strictly unit test when you think of like JUnit or Python unittests, that kind of stuff. Those kinds of tests you can absolutely write using ArduinoUnit and they can be, and they should be part of your testing framework. I think a pretty good point of view to take is, you’re going to have a custom firmware set up like a different, like a project, is really just tests the tests. It test all the unit tests so you can run all those to validate the correctness of those implementations, and then maybe not have those tests in your true deployment. You just skip them. You can leave those off and back to just the requirements of tiny systems so you can avoid the memory footprint of those tests, if you didn’t want to have them in there.

But you can have them if you want to. And then if it is one of those tests that a certain pin is high, you can write that too in there, that sounds like more one of those sorts of testing cases. Like when things are running, this pin should be at a certain level. But you can also write that as more of a functional test where try to do some conversion, wait a certain amount of time. At a certain point, you should see this expressed on an input pan or something like that. You could write that as a little bracket test and it will run that for you. You can check to see where the pins are in the right levels and [inaudible 00:24:16]. You could get like a pass or fail out of those sorts of tests as well.

I guess another version of it’s non-denominationess. This is actually more beta, but if you look at the latest version of our community unit on GitHub, and I have to say, I have not contributed to this in a couple of years, I’ve been working on it. But it also will work where you can run it, what I’d call, in vitro. You can run it off of the Arduino. You can run your unit tests, not even on the system. If you’re back to those functional tests, the one when you’re talking about there like the JUnit stuff, you can run those on the development platform instead of on deployment platform. And so you can actually pretest the correctness of components before those components end up in the device using ArduinoUnit.

It has what’s called a mocking layer. They can mock some of the core pieces of the Arduino environment and run that way. It also has a simulator basically. That’s none of my writing, I got somebody else’s, but you can simulate out Arduino Mega on the full-on platform and then run things that way. You don’t have to actually deploy it into a system to test. I don’t know if that’s… Does that help a little bit?

Michael:
No, it does. It does. I have just a couple more just logistical questions about it. Let’s say somebody listening to the show wanted to check out our ArduinoUnit, they wanted to play around with something. They download the library, they install the library on their Arduino IDE, and let’s say they open up the Arduino IDE, and then they just write. Let’s just say they just write a simple, I don’t know, a simple function that’s going to calculate temperature and from a temperature sensor that they happen to hook up and it’s going to spit out like Celsius, Fahrenheit and Kelvin or something like that. They want to write a test that just makes sure that their logic and their calculation is right or something like that. They would just include the library in there, and then they would write a test? They would just write a test and set up, is that… The tests are going to get called. I guess I’m just curious, where do the tests get called? Your tests are going to be right there in your code, right? I mean-

Dr. MacEvoy:
I don’t know if you notice this, I feel like such a lead question that I’m not sure that you I’m smiling cause [inaudible 00:26:53]. If you go to the GitHub repository, so you just search for ArduinoUnit, I think that would be enough in GitHub, I guess. If you add GitHub [inaudible 00:27:04] GitHub and ArduinoUnit, it’s [Mackey Murdoch 00:27:08] is actually the original developer. I’m very much the primary developer that thing now, but it’s hosted off of his GitHub account. There’s a guidebook.md file. It’s a little markdown file. It’s not that long, it’s 20 pages that really talks about testing, and in particular it’s just funny because it uses an example of a temperature sensor of, how would you write that as a set of tasks? What would you do?

And so I would say that would definitely be a good place to start, is to go look at that guidebook and see what it has in it about some of the philosophy and the details of what you can do to, to use it. It does go pretty far into different ways you can use the library to test different things and in different ways. From the functional tests, the more trial stuff, traditional unit testing ideas, to different kinds of power on self tests kind of tests, and what you would write and where it would go. But to leave it to the listener, the core idea is, you write the tests as little functions, essentially similar to the Google Tests, you write tests in lowercase. Now that’s the only difference. You give them names, and then in brackets you describe what the tests are.

The main options are tests and testing. Tests would be one that runs once, testings are ones that run repeatedly. In the setup phase, basically what you do is you wire in which tests you actually want to run, and by default it’s all of them, but there’s some simple filter rules to change that if you want to. The other important thing is to wire in, where do you want the reporting to go. It defaults to the serial port, but if you’re using a Mega or some other situation, you can point it to basically any print style device. It’ll send the data out that way. It can be a lot of things, but you can set that up.

And then in the loop part of your program there’s just one magic thing you call test::run. And in there, it will run all the tests. It will repeatedly run the testing tests as well for you in that same environment. Some, I guess, convenient things maybe to think about for your audience, it does it in a pretty efficient way, so basically it doesn’t just go through the entire list and repeatedly checked to see if there they’ve passed or failed. As tests complete, there’s fewer and fewer things that happen in the test run thing yet. If you excluded all the tests, it would be essentially almost a no op step that happens in a test run. It’s implemented in a quite efficient way, I would say, about how that works.

As each of the tests take themselves off the list because they’ve resolved themselves, they are no longer query, there’s no CPU overhead or anything like that, that goes into trying those out. If you have a testing environment, then those will continue to use up some of your cycles to be executed. The guideline or the guidebook, sorry, does explain a bit of how to make that lower if you didn’t want to do it every loop. You can trim it back and say, well, I’ll read 250 milliseconds. I want to run a test, I don’t want to run it every cycle. That’s described in the guidebook, how to trim it back. That’s a bit of the overview thing. Definitely guidebook in the GitHub repo is just a markdown. It goes, I think, quite a ways into philosophy and details of how [inaudible 00:30:57].

Michael:
All right. Yeah, we’ll make sure to link to that. That’s in the Arduino… Is that in the ArduinoUnit?

Dr. MacEvoy:
Yes. I think you just see it as a link yeah, [inaudible 00:31:05] drops right there, and then look for a guidebook.md in that list, and then that’s a hopefully great starting point for how to. Just the read me, the absolute hello world read me is the read me file. I’d say start from the top, just the read me file in there, and then the next step is the guidebook for where to go next, for how you could really use that library.

Michael:
I’m just fascinated by this. The testing that I’ve done, I absolutely love. I’m one of those drink the Kool-Aid guys when it comes to a TDD, test-driven development, for anybody out there wondering what that might mean. I’m just excited to try it with Arduino development. What do you say to the folks listening out there? Maybe somebody listening to this, they’re still trying to wrap their head around testing. They’re like, okay, I get the idea that what would I build should be reliable and that kind of thing, but come on, it’s such a small system. Do I really need to bother with it? Part of me, I can see that. In some regard, if somebody’s maybe not building something professionally, maybe they’re just… I’m curious to get your thoughts there.

I guess I would say before I hand it over to just get your thoughts on that, I would say there’s something about doing TDD that can help you be a better developer in general. And so for some times just learning that practice is a good thing. That’s where I come from. I don’t know. That’s what I’ve been thinking about as I digest, but I don’t know. I mean, is it… I don’t know. For something small, is it [inaudible 00:32:51].

Dr. MacEvoy:
Right. You’re like sharpening a pencil, how hard are you going to work at making all this infrastructure around it? Let me start by just agreeing with you. If it’s simple, it’s simple. Maybe that is a unit you’re willing to throw away if it doesn’t work. What’s some thoughts in there? One is, independent of testing, just like, back to that, what makes you nervous or frustrated? If you’re trying to build something and you realize you’re spending a lot of time trying to figure out what’s wrong, that’s a good sign that you may be wanting to up your game. The thing that is fine, if you’re feeling like you’re developing and you’re comfortable, and the people you’re working with are comfortable with what you’re handing over, then maybe it’s a good solution.

But if you’re sitting in there like, man, why in the world is this not working right now? Then test. There’s something wrong with how you’re doing it if you just like stare at that box and wonder what’s going on in there. And so I think I think it’s that. If it’s a little thing and you can get away with a couple of debug statements in the LED, then you’re writing tests. If you’re deciding, okay, that yellow light when it goes on, that tells me something. That’s a test. I mean, it doesn’t have to be an official library. If you’re putting together the things that help you feel like it’s reliable, you’re testing. The tools, or I guess walking in that path, it’s just realizing that I don’t know what’s going wrong and it’s keeping me from doing the thing I want to do. I’m sitting here, like, I don’t know, being the muse, reading my code again and again, trying to find, what is wrong with this logic?

Because you can’t step through it anymore. It’s over there. And so how do you figure out whether it’s right or not? And so I think that’s where the testing comes in, is when it feels like it’s just, you’re not happy anymore. You’re just frustrated. It’s like, oh. And then when you do find out, it’s like, oh, it’s so stupid, this is this little thing. Well, chances are, it’s because you didn’t have the pieces tested enough to know it was broken. How about the next version is that stack of Legos thing where you build stuff and it all falls over and then you don’t know which Lego to look at, because you don’t know which one’s the broken one. That’s a good sign that maybe unit testing is the way you should be thinking. How you know all those pieces are right before you put things together.

If something doesn’t work and the only thing you have to do is start at line of code one, get to line of code n and try to figure out what is wrong, then, well, you probably could do better at how you wrote that in a way that each of the pieces, your trust, your nervousness, has gone way down on all these other parts. And then there’s this one part that may be if you got to write more tests for right, or apparently maybe that’s where the problem’s at. Tests, back to that first thing about trying to be less nervous, you sit in there pretty nervous about what you’re doing. If you feel like that actually is a pretty good spot to be as an engineer, is if you’re sitting there feeling nervous, then try to sit back and think about why is that person sitting there nervous? What are the things that you could do that’s different, that would make it work out?

And so, yeah, testing and version control, I know this one is not about version control, but I think testing is version control to those big pieces of software development that everybody should know about when you’re trying to make something work.

Michael:
Absolutely. I think that is a great place to call it. Dr. [MacEvoy 00:36:47], thank you so much for your time. You’re really generous and I appreciate it so much.

Dr. MacEvoy:
It’s a pleasure. I’ve been great to talk about this. I’m glad it’s as popular as it has been. I do hope that a bunch of software developers and working on business systems, start to drink the Kool-Aid and learn that they actually enjoy it, it actually makes for better stuff, both for themselves and the things that they’re trying to make.

Michael:
Cool. Fantastic.