What should we build next? At Blueleaf, we have a process called progressive fidelity testing to try to answer this and several related questions. Does anyone care? What do they care about? Does our design solve that problem? The idea is to test a feature from concept to implementation beginning with the simplest, lowest effort version that will give us data about user behavior with the feature or concept. We conduct tests right inside the application as well as outside to help us make sure we’re addressing a problem that people care about and that our implementation lives up to our users expectations.
Does anybody care? A simple example of this progressive fidelity testing comes from when we were designing our user reporting. We made some assumptions about what people “needed”.
For example, when looking at their account balances over time, users absolutely needed the to be able to choose different historical time periods: a week, a month, a year. And since those are all rolling time frames, they also must be able to create a custom time frame.
It turns out that with the type of data we’re working with and the kind of aggressive simplification we’re delivering, custom time frames are time-consuming to build. How important is a custom timeframe to our users? We tested it. We built the selector to include the “Custom” choice.

But when you click on it you get a simple message:

We turn this click data into votes for the feature. It turns out only 6.5% of users who visit this page voted for custom timeframes. So we’ve held off building it. That saved a couple of days of engineering effort, an additional half day of arguing about including the feature, and gave us a clear signal to make a decision.
There are lots of folks who’ve written about doing this and just letting the user hit a 404 error. While I think it teaches you the same things, it’s lazy and unnecessarily tells a user that they’re a lab rat. Blueleaf is a financial application, so trust matters, and adding this little “We know you’re a person and appreciate you” message helps users feel good about the site and takes about 30 seconds to implement. Just do it.
What do they care about? In another case, we were trying to understand what kind of financial help our users want and how to make it available in our application. Logging a click just doesn’t tell us all that much. We want to know exactly what they expected, not just that they voted for that feature. Because people may need different kinds of financial help when looking at different types of data, we added a button to every page in the application:

When users clicked on the button, they got this:

It’s a simple single survey question. With it we’ve learned a few things. 1) People want to know a lot different kinds of things and get information in lots of different ways and 2) Oh yeah, our current users aren’t really that interested in having Blueleaf answering these questions. There are lots of potential reasons for that and we’re digging in qualitatively to understand it better. There is one particularly interesting thing about this feature area. Most people we spoke with or surveyed suggested it would be valuable to them. It was also a very popular suggestion while we were fundraising.
So our customer interviews were diametrically opposed to the behavior of actual users.
The plural of anecdote is not data. Test, test test.
Does our design solve that problem? (The one they know and care about) Outside of the Blueleaf application, we use a number of other tools like Verifyapp.com to help zero in on the right implementation details. In conjunction with typical paper prototyping and show-and-tell style interviews, these kind of tools that can get you a powerful amount of detailed feedback quickly. In particular, we test for what people understand about our design, do they connect it to the problem we verified, does it meet our design goals?
In one case, we were redesigning a key application page and we were convinced that it had to be graphical. Finance = lots of pretty graphs and charts, right? So we launched a test on
Verifyapp. We showed the following screen (It was full size and full detail):

We asked people what the screen was showing and what they felt when they looked at it. After a few hundred responses it became clear that it failed our core objectives of being simple and understandable. In fact, not one person associated those words or the right problem with this design. That is despite in-person review sessions where people said, “this is great”. It turns out two forces were at work. First: people, particularly when dealing with financial information, have a need to appear that they understand, even when they don’t. No one wants to look dumb. People want to appear to know what they think they’re supposed to know. Interviews create a kind of subtle pressure that makes that behavior more pronounced. Second: people were reacting to some of our unique data on the page, which is admittedly cool, but targeted at solving a different problem. The page design failed. Back to the drawing board.
We ran through that same process two more times and finally removed the cool info from the page, simplified it and dropped the graph entirely. Data on the 3
rd round design showed it solved the intended problems and was simple, obvious and easy to understand. Bingo! Cool data isn’t so cool when it makes the page less useful. Total time on the iterations on this page, including gathering, collecting, and analyzing feedback from almost 800 people, was under one week and required absolutely
No Engineering Time. We implemented the final design in a couple of days and it’s received lots of positive feedback from our current users and importantly, it gets used.
Personal finance is a space that is filled with “shoulds” and “must haves”. “You can’t be a real personal finance application without building this or that feature.” Lots of domains and individual visions are polluted with these beliefs. They are BS, a form head trash. That trash creates an incredible pressure to build additional features in the name of progress. It’s not progress. Make real progress and build something that people want.