We think a lot about testing. Both
because we want to feel more comfortable changing things, and because we
realize that many, many developers depend on Unity being stable and
reliable. And that’s something we want to continuously get better at.
We have previously blogged about how we do high-level testing from C#, but over time we have come to realize that we also need much more low-level, much faster, and much more fine-grained tests. The thing is, though, that our C++ codebase is rather hostile to any such test. Testing a huge C++ codebase with lots of legacy code is tough.
But… we’ve now made it a little easier.
After looking around and realizing that no-one quite solves the problem the way we need to (what TypeMock does comes closest but it’s Windows-only so not of much use to us), we sat down and pondered what we could do. Obviously, we can’t rewrite our whole codebase for the sake of tests. Whatever we were to come up with would have to work with where we are and with how we write C++ code.
So we thought “code patching!” Turns out we had just developed this nifty framework to do live code patching for the sake of profiling. But the problem is many platforms don’t allow runtime code patching, so that rules out that option.
The proposition is actually pretty simple: once you have a mechanism to hook into any function, you can build a whole framework on top that reprograms those functions for whatever is needed in tests.
So first step: hooks. Goes like this:
It’s simple. Does nothing by default (and actually compiles to nothing in builds we ship) but when tests grab the hook, it turns into magic. There’s a bit of intrusiveness but it’s very modest.
Next step is to make the hook go live during a test:
This will look up the hook based on the function signature and then grab the hook for as long as the fake is in scope.
So finally, we can program the hook:
From here, you can do all the usual mocking like redirecting the call to your own function, observing arguments and return values, and running the original function in a controlled way. And it’s straightforward to support instance methods in addition to free functions. In fact, the system makes it easy to even replace entire classes.
We have previously blogged about how we do high-level testing from C#, but over time we have come to realize that we also need much more low-level, much faster, and much more fine-grained tests. The thing is, though, that our C++ codebase is rather hostile to any such test. Testing a huge C++ codebase with lots of legacy code is tough.
But… we’ve now made it a little easier.
After looking around and realizing that no-one quite solves the problem the way we need to (what TypeMock does comes closest but it’s Windows-only so not of much use to us), we sat down and pondered what we could do. Obviously, we can’t rewrite our whole codebase for the sake of tests. Whatever we were to come up with would have to work with where we are and with how we write C++ code.
Hijacking…
This made us realize that the one thing we want is the ability to hijack any function anywhere in our tests. Take it over, mock it, fake it, stub it, rule it. We’re okay with function X calling function Y directly, as long as tests can hijack the thing and turn function Y into whatever they want.So we thought “code patching!” Turns out we had just developed this nifty framework to do live code patching for the sake of profiling. But the problem is many platforms don’t allow runtime code patching, so that rules out that option.
… At Compile-Time …
So then we thought “no problem, we patch at compile time!” Tricky! Some guys have done it, like the Bullseye guys whose tech we use for C++ code coverage monitoring, but it’s still really quite involved. And we have many different compilers across a variety of platforms so… no. And our builds are already quite complicated. So that one’s out as well.… With Macros
With C++, if you have nothing else, you always have macros and templates. So I sat down, got my < and > keys ready and wrote us some templates sprinkled with some macro bits.The proposition is actually pretty simple: once you have a mechanism to hook into any function, you can build a whole framework on top that reprograms those functions for whatever is needed in tests.
So first step: hooks. Goes like this:
It’s simple. Does nothing by default (and actually compiles to nothing in builds we ship) but when tests grab the hook, it turns into magic. There’s a bit of intrusiveness but it’s very modest.
Next step is to make the hook go live during a test:
This will look up the hook based on the function signature and then grab the hook for as long as the fake is in scope.
So finally, we can program the hook:
From here, you can do all the usual mocking like redirecting the call to your own function, observing arguments and return values, and running the original function in a controlled way. And it’s straightforward to support instance methods in addition to free functions. In fact, the system makes it easy to even replace entire classes.
No comments:
Post a Comment
comment