More on Generated Tests

Shortly after posting this post about using dlls to run generated code to make sure it’s actually generated correctly, I realized that, unfortunately, that approach doesn’t work 90% of the time, and actually doesn’t work 100% of the time if you’re using C++, unless you find a work around to import objects. If you’re using STL (especially as parameters to functions) though, you’re kind of screwed when trying to implement your unit tests in C#.

So what to do? Again we have the same problems, but I’ve come up with a different approach. In the case of this generated code I can deem it “correct” if it conforms to the following properties:

1. It calls the correct functions in an external library with the correct parameters.
2. It puts data received from the external library into the correct place

I decided to test these two different ways, but that will show me not only that my generator created the correct code, but that it also created the correct unit tests for that code. Basically here was my end solution.

1. Code a Mock Object To Impersonate the External Library. In my case, the mock object is a mock connection to a database that always returns one row of data with fixed values. The fixed values are dependent on type, which makes it easier to track down if I’ve accidentally put the wrong data in an address. In addition, it logs ALL of the functions called to it, which I store for later use.
2. Generate Unit Tests using the same generator as the code generator. The unit tests initialize the objects using the mock connection. The objects then do what they’re supposed to do and then check to make sure they get the right data back. At the end of the test, the test outputs the calls to the mock connection object to XML (something coded into the mock object) to a predefined folder.
3. Code the C# Unit Tests. In this case, the unit tests generate code based off of a known value which has all of the features our generator supports. It then builds the generated code and its unit tests and runs them (actually as part of the post build step). If the unit tests fail, there’s something wrong in either the generated code or the generated tests. If they succeed, I go the extra mile to compare the output XML to files I’ve written (or at least verified) using Microsoft’s XMLDiffPatch library. This library was *also* how I checked that my Mock Object was outputing the XML correctly.

So why XML over just doing a diff on the code files? Again, all I care about are the calls to the external library. So long as they happen, I don’t care what the code internally looks like, and I actually may want (or need) to change the code to optimize things further on down the line, so comparisons against the code text is our of the questions. XML is nice because you can set the diff program to ignore whitespace and order, which I need.

So yeah, hopefully you find that helpful. By the way, I promise to go back to writing about design sometime in the future. I’m just finding a lot of this code stuff really exciting (and I can talk about it now, which is nice) so I figured… why not? Hopefully people are finding this interesting? I'd love to hear if anyone has any feedback on the shift in focus over the past few weeks.