Visual Studio Exception Assistant
01 Background
Visual Studio is fully-featured integrated development environment (IDE) for Windows, Android, iOS, web, and cloud. Visual Studio provides the most powerful and feature complete development environment, bar none. Its complexity is intimidating and the framework it’s built on presents challenges akin to moving mountains just to make the slightest change to any of its components. Visual Studio has 4 million users with 1.5 billion revenue per year.
An exception is an indication of an error state that occurs while a program is being executed. A simple example of exception will be: if you divide 5 by 0, you will get “DivideByZeroException”. There are hundreds or even thousands of different exceptions. Dealing with Exceptions is a regular part of the developer daily routine. The developers need to find out why they have these exceptions and how to fix them. Visual Studio exception experience includes exception setting and exception assistant.
COMPANY
Microsoft
TIME
2016
ROLE
Lead Designer
How important is Exception Experience?
- 15 Million Exceptions are Broken on Every Month
- 510k+ users break on exceptions in one month
- Dialog pops up 28 times/month on average
02 Exception Setting Experience Improvement
The users can use the Exception Settings window to specify which exceptions (or sets of exceptions) will cause the debugger to break, and at which point you want it to break. The users can add or delete exceptions, or specify exceptions to break on.
Previously when the users wanted to configure exception settings in Visual Studio, he would have to go to a modal, slow-opening, hard-to-search Exceptions Dialog. It’s not search friendly. Since it’s a dialog, the users cannot see their code side by side with this dialog. There are a lot of issues. So in Visual Studio 2015, I worked with PM and the engineer team together and made a big improvement to Exception Settings experience, which you can find below.
Before
After
Postive feedback from customers
03 Customer problems
We got enormous positive feedback from the improvements of Exception setting in Visual Studio 2015. Meanwhile, we also got a lot of feature requests from our customers from all channels. The team was encouraged by previous success to make improvements to Exception Assistant too. The images below shows customer requests and one design proposal from PM to address customers’ need.
The design proposal from PM is a low cost and easy way to address customer’s need. But is this the approach we want to take? Not necessary. We could do better. We interviewed a lot customers both via inviting them to our usability lab and online survey. The below animation shows well how our customers felt about Exception Assistant experience in certain frustrated situation.
The problem for Exception Assistant
- Too much information on the UI. Useful information deeply embedded
- The Troubleshooting tips found are largely not helpful
- 5 million views of Troubleshooting Exception topics
- The average rating of the top troubleshooting topics is terrible (1.3)
- Too large and non-resizable
- It shows up in random locations
- The users using native code still got 20 years old Exception Dialog
What did we do
We also analyzed how we got the current exception experience today, what we did in the past, etc. The original Exception Dialog was built in 1998. In 2005, there was a big improvement to update the Exception Dialog to Exception Assistant we had. Due to resource limitation, only the customers who wrote managed code, such as C#, got the Exception Assistant experience. The customers who use native code such as C, C++, etc. still got the old Exception Dialog. Since 2005, not much changes have been made into the Exception Assistant. Every time we got customers’ request, we added a few controls in the UI. If we followed the design proposal from the PM, we would end up with the same path and make the Exception Assistant even more crowded.
Exception Dialog in 1998
Exception Assistant in 2005
Exception Assistant in 2015
04 Design goal
“Replacing both Exception Assistant and Exception Dialog with a simple, usable & delightful experience”
05 Sketching and Wireframing
At this point, all team members agreed with me that we should do something better instead of simply adding a few controls to existing Exception Assistant UI. We want to created a simple, usable and delightful experience for all our users. The team members including designer, researcher, PM and engineer had brainstorming meetings for future exception experience. The below are some paper sketching and wireframing at the very beginning.
Brainstorming meeting notes
Paper sketch
Wireframing for early ideas
The below are some early design ideas I tried during this stage. The design iterations below focus on trying different way to present information to our users. I also did a lot of design iterations on the content we present to our users.
06 Final Design
The below are before and after for Exception Assistant experience. We are designing for simplicity. We want to make the core experience simpler for the majority of usage, while retaining support for all the corner cases and less frequent usage scenarios. If we compare this experience with Exception Assistant showing earlier, we can see that more than 70% of the information have being removed.
In the final design, we fully remove the trouble shooting tips section. We bold the most important information. The three hyperlinks are most useful actions that the users will need. We remove the whole trouble shooting tips section, and replace it with the “Search…” link. When clicking on the search, we will automatically search on Bing, Microsoft Search Engine. We not only search for the name of the Exception. But also automatically put the project type, programing language and other key words to search to help users get the best search result.
Moment of delight
A moment of delight is created when a user’s expectation is surpassed. For example when a user gets a new release of a product, and they discover that a thing that always used to be a pain to do now suddenly just works. Moments of delight can appear magical to a user, but they are very much a thing that can be specifically designed into a product. So we want to design “moment of delight” for our users. As mentioned earlier, there are hundreds of different Exception types. We want to find a few important exception types and design the moment of delight for such scenarios.
Null Reference Exception
- 188,000 users breaking on this type per month
- Dialog pops up 9 times/month
- 2000+ votes on Visual Studio voices
In previous release, the users can only find that “A.b().c().d().getData().createArray()[getIndex()]” is null. Any item in this expression can actually be null and be the root cause of this exception. The users need to spend a lot of effort in order to find out which specific part is null. In this new Exception experience, we will tell the user exactly what is null and bold the null expression.
Inner Exception
Inner Exception is another good example which we can create Moment of Delight for our customers. Because the code structure can be nested, when you catch an exception, the original cause of the exception might be several levels deep. You can get to the original exception by looking at the InnerException property of each exception, working your way down until you find one that has a null value for InnerException.
We first show the current exception, which is “ArgumentException”. Then we present the inner exception right below. The users no longer need to go to details… and find out the inner exception. More important is that the users know immediately that they have Inner Exception. In previous experience, even seasoned developers may waste some time until they realize that they have an inner exception which is the root cause. When there are multiple levels’ inner experience, we show the inner most exception by default.
Redlines
After we finalized the design, I also created final deliverables like redlines and shared with the engineers. I scheduled a meeting with the engineering team to walk them through the design and answer any questions they had. I also reviewed the ongoing implementation regularly to ensure the right experience is being built.
07 Result
On average, the new design is 5 times simpler than previous experience. The design proposal was well received by both outside customers and internal stakeholders, and got the VP’s endorsement too. This feature has been demoed during Build Conference 2016 as one of the most important improvement in this release of Visual Studio. The design goal has been achieved.
What should be improved?
After releasing the feature, the biggest customer request is to add a Pin function to the new exception helper. They want to be able to pin the exception helper to any place they like while they navigate and browse to other page of their code. So, we updated our UI to add this function.
