Design Critique
Stakeholder Feedback
Engineer Feedback
Product Strategy
Product Design
User Interview
Customer Interview
Sketches
User Testing
Success Metrics
B2B2C Design
Competitive Analysis
Content Design
User Interface Design
Duo Security (Cisco) provides two factor authentication service to businesses and their end users. Its in-app mobile authentication, Duo Push, is its most popular approval method because it conveniently allows users to login with a tap. However we’ve learned from customers that users fails at phishing push tests that their IT Admin runs without really looking at the data. Companies are concerned that their employees aren't being careful when approving identity verification requests. This leads to customers turning off the ability to authenticate with Push Notifications.
The More Secure Push project's final product was a newly designed, "step up" authentication method requiring additional QR Code/passcode verification, which fully quelled the Push Phishing concerns of all 4 customers interviewed. A working proof of concept was also engineered to get further buy-ins and for additional testings in natural contexts.
Throughout the project, I worked closely with the Product Manager Intern, Hari, to ensure the designs consider Duo's, product's and users' perspectives.
I led the interview scripts for initial customer research and spearheaded the analysis efforts. I also led all design efforts, such as competitive analysis, design iterations and feedback, success metrics, usability testing and analysis, and final design handoff.
Figma
Mural
Google Analytics
Product Manager Intern
May - Aug 2021
"I have considered disabling Duo Pushes for my users within my organization" - Duo's customer
The More Secure Push project aims to combat the issue of Push Phishing, which involves an extremely sophisticated attack where a bad actor has obtained a user’s primary credentials first, then attempt to get the users to approve a fraudulent push sent by the bad actor. A scenario could be a bad actor sending authentication requests randomly to the user, hoping the user will approve the push.
From talking to the internal Duo employees and customers, we learned that Push Phishing is an issue that has been in the back of people’s minds since 2016. In fact, through our customer interviews, we learned that there has been instances where customers have disabled, or considered disabling, Duo Push within their organizations. This can be extremely consequential as Duo Push is Duo's most popular authentication method.
How can we design a more secure authentication experience to mitigate the risks associated with Push Phishing for customers and protect users from approving fraudulent pushes?
While Duo provides mitigation strategies focusing on user education and blocking access based on locations for customers with higher tier plans, some customers felt that they are not adequate solutions to prevent Push Phishing attacks. They believe that users are often habituated to approve pushes without reading the details.
The More Secure Push project's final product was a"step up" authentication method that requires users to verify using another layer of passcode or QR code, which fully quelled the Push Phishing concerns of all 4 customers interviewed.
.gif)
Skip to Interactive productThrough background research and interviewing internal Duo employees, we learned that customers have voiced concerns related to Push Phishing and "Push Fatigue" (habituation), which occurs when users are so used to the act of approving pushes that they are habituated to just automatically approve it, without paying attention to the request.
In fact, the issue was voted on 30+ times in the Product Catalog. While the issue may not be an immediately urgent one, it is believed to be an important one. As one Customer Support Engineer puts it, “If this did escalate and there were some kind of major credential issue, would that escalate, accelerate this? are we beating the problem if we fix it? or are we fixing a small bucket problem?”
Why is Push Phishing an important issue to the customers? How did customers know it was a problem?
Through the customer interviews, we aim to understand customers' concerns related to the issue and the context in which Push Phishing occurs. We interviewed a total of 14 customers and 4 Customer Solutions Managers, who served as our proxy customers.
The key questions asked in the interviews are:
1. How does a trustworthy authentication request look like to you?
2. Has there been any instances in your organization where Push Phishing is observed? Can you tell me about it?
3. What actions have you taken to detect or combat Phishing Pushes?
Customers need to be able to trust that Duo Push is providing a balance of security for their organizations and convenience to their users.
• Customers interviewed rarely observed legitimate push phishing attempts
While 6 out of 14 customers voiced concerns with Push Phishing, most of them had concerns due to their red team tests.
• .. but customers are not confident to say it didn’t happen because it's so hard to detect, and pushes are rarely monitored proactively
11 of 14 (almost 80%) customers interviewed review authentication activities reactively, only when users report something as fraudulent. This means that even if Push Phishing did happen, customers may have not been aware about it.
• Customers wants a more proactive way to protect against Push Phishing
2 out of the 2 customers who experienced legit Push Phishing attacks only discovered them after the fact through user reporting and 3rd party email anomaly detection service. To better protect against Push Phishing, there needs to be more front door protection to make authentication more secure. As one customer puts it, “Would have never discovered the push fraud if that one user didn't report it as fraudulent”
5 out of 6 customers who had concerns with Push Phishing believe that users do not pay attention to the details of the authentication requests
• Push phishing has led customers to disable push, or consider doing so
1 out of 6 customers who experienced Push Phishing has attempted to disable Pushes in his organization. There has been another instance where Push is fully disabled because of Push Phishing concern
• Customers believe that Push Phishing is a user education issue, and it occurs mainly because users do not pay attention to auth details
5 out of 6 customers have concerns with Push Phishing believe that users simply do not pay attention to the details. Some believe users are unaware of when they should get a push or not. And, some believes Lees have approved push phishing just because they want it to stop. This supports our initial assumption that Push Fatigue/habituation plays a factor in Push Phishing.
Thanks to Duo's mature design environment, the well established personas helped us understand foundational goals and pain points of our customers and users, which allowed us to focus on the problem at hand. Here are the two main personas used for the project, as defined by the research team.
Lee (primary persona) represents the primary users of Duo. She just wants to get to her applications. She does care about security, but it can be hard to keep up.
For this reason, she often equates security with friction:
- frequent (as opposed to one-time) logins
- Context switching between devices and apps
- Potential ill effects of delayed authentication
Gary (secondary persona) represents the IT admins who are generally in charge of maintaining a company’s Duo-related systems, and he often works with other teams or individuals to do his job.
The ecosystem that Gary manages includes:
- Applications
- Outages
- Events
- End Users
- Vendors
- Devices
Gary is a security analyst at an organization, who is in charge of implementation of Duo. In his day to day life, he tries to ensure the best security posture within his users and organization, and maintaining Duo-related systems. Duo is a convenient way for him to keep his organizations safe while ensuring users are able to do their daily work. It also mitigates Gary’s security concerns: even if his user's primary credentials are stolen, Duo protects the hacker from passing the secondary authentication.
However, because of his penetration tests, he occasionally worries about the possibility of push phishing attacks because he believes his users often don’t pay attention to the authentication requests and sometimes don’t know proper push etiquette, such as the user should only receive and approve a push if it’s initiated by him or her. He understands that push phishing is a human and universal MFA problem, and has considered disabling push to combat the issue. He wishes there is a technical mitigation Duo can provide to quell his concerns, since one open door can lead to severe consequences. He is willing to trade off user experience for users and applications that contain more sensitive information, but also would like to ensure any changes created do not lead to help desk tickets or productivity hindrance.
From our customer interviews, we identified that Push Phishing is an important risk that can potentially lead to customers disabling Pushes. We also learned that customers are interested in having the option to have a "More Secure Authentication" method. This leads to our design goals for our project:
1. Should quell our Gary's concerns with Push Phishing attacks (random pushes sent scenario)
2. Lees should easily complete their logins even with the added friction.
3. Lees should understand what to do in a random push attack scenario without needing Gary’s help.
I began the design phase by researching what other competitors are doing for their “step up approach”, analyzing what worked and didn’t during the project's previous attempt, and reading academic papers on warning designs.
This information helped me ideate solutions for how a more secure push auth experience should look like. Which you can see in the images above, where I broke the ideas into three main points: before the usual authentication request shows up, during the screen, or after the approval happens.
Within these points, I explored ideas ranging from pop up warnings before the actual approval screen to opinionated button design and adding an additional step after approval.
I then narrowed the ideas down by gathering feedback from the Trust Monitor PM, who's also working on Push Phishing mitigation, engineers, and the Product Designer for adaptive authentication.
Through the feedback session, I learned that it makes the most sense to have the additional friction added to when an approval is actually accepted. The two main reasons that contribute to this are:
(1) From Trust Monitor's Product roadmap perspective, the most accurate signal for suspicious login occurs when the approve/deny actions happen. Adding the friction after will minimize the false positive rates.
(2) From authentication's perspective, if the friction were to be added as an additional method of authentication, users will only encounter the friction if they approve the regular request, instead of for every request up front.
After stakeholder feedback, I created wireframes for 3 ideas of friction that occur AFTER the approval happens:
(1) QR Code (# on phone versus prompt)
(2) One Time Passcode
(3) MFA Options
I then held design critiques with more Duonaunts, which included content designer, more Product Designers, engineers, project stakeholders, and app security team.
The feedback helped ensure that the design options I end up testing not only help mitigate push phishing and security concerns, but also consider Lee’s experience, contents, security, technical feasibility, and how it may impact other team’s roadmap.
(1) Offering more options expand the ability to breach through and requires more organizational support and engineering efforts
(2) Both QR Code and Passcode offers similar value with security and engineering efforts since there are existing use of them already
(3) Verbiage such as "authenticate" may be too technical, make sure to use simple languages for design
These led to my final decision to test the QR code and passcode designs.
Thanks to Duo's mature design environment, the existing design style guide allowed me to easily design a hi-fi prototype for the test plan.
Before I began crafting the test plan, I leveraged Google's HEART framework to decide on the important user-centered metrics for our solutions: user happiness and task success. In this context, Lees need to easily navigate the new design without needing help or getting frustrated and confused.
Success here looks like...
(1) Rating of 5 on ease of use scale
(2) 100% task success on clicking the right option
(Rating of 5 oconfidence with decisions
While testing with just 6 users is not enough for statistically significant results, focusing on these two metrics and the "whys" allow us to gain deeper insights into user behaviors and attitudes.
We conducted 6 usability tests with Lees with the goals to understand if Lees can make sense of what to do in a random push attack scenario with the designs, or complete the login with the additional frictions, without needing Gary’s help.
Some of the questions we explored are:
(i) What gives Lees the confidence to approve a push? Mental model?
(ii) What is easy, hard, or confusing with the experience?
(iii) Which experience is preferred and why?
To achieve our goal, we had Lees perform 5 key activities
(i) Login to account with Duo Push's new design Mobile 4.0/Universal Prompt
(ii) Approve request without reading carefully, and ask what to do after seeing the design in a Push Phishing simulation
(iii) Login with additional QR code friction
(iii) Login with additional Passcode friction
(v) Preference Test
Through the usability tests, we identified several key insights:
(1) The new deigns have clear instructions that allowed all Lees to complete the login quickly without trouble
All Lees rated the tasks as easy (4) or extremely easy (5) because the instructions are clear. They understood that "if I was not trying to log in, and because I could not type in the code anywhere, I was allowed to say i'm not logging in”
(2) The additional screen gets Lees out of auto-pilot and realize that they are not logging in
Lees understood the designs is an additional safeguard in place "This is interesting. If you make a mistake, you are not totally in trouble.. this is a good second step verification"
(3) 4 out of 6 Lees preferred the QR code method & mentioned it leads to “less chance for mistakes”
“Scanning is easier because I'm trying to login from my phone and i just point at the QR code instead of sitting and typing the numbers down... If I am busy and the passcode pops up, what if I type the code wrong?”
(4) QR Code can lead to frustrations if Lee’s camera is delayed
1 out of the 6 Lees preferred the passcode for this reason, and 1 had no preference
(5) Lees voiced that suspicious alert made their “heart beat race”
“If you really are the one trying to login, and you see that for the first time and are not used to it, it’s going to give you heart beat race”
At the end of the project, we successfully achieved our goals set for the overall project and the design goals:
(1) 4 out of 4 Garys interviewed agreed that the designs solve their Push Phishing concerns
(2) All Lees had 100% task success rates completing logins across the two designs
(3) Developed a proof of concept for the passcode design with engineers
For the final design, we incorporated several changes to address the pain points discovered from the user tests:
(1) Changing verbiage of "Suspicious Login Detected" to “Additional Verification Needed”
From the user test finding and stakeholder feedback, we learned that the wording may cause frustrations as users felt their heart beat race from the warning and confused about why the request is suspicious. Thus, a more neutral content will be critical to ensure users are not alerted negatively.
(2) Adding an alternative route to "enter a verification code" for the QR code idea
Due to the nature of prototypes, it's difficult to test the impact of "phone delays" or technical issues. Thus, there needs to be an alternative way in which users can authenticate.
(3) Both designs solve the issue, but testing in natural environments is needed to gather behavioral data on the frictions.
The final recommendation was to continue testing with more users to gather statistically significant quantitative data and make fully informed decision on which design to pursue. Test with a more diverse user group and understand how the experience of the designs are like in a natural environment.
• The complexity of designing for enterprise: it’s not just about the users
Designing at Duo is about B2B2C design -- we are designing for our end users, but we are selling the product to our customers (the admins). The experience was truly one of a kind, and I thoroughly enjoyed thinking through this complex ecosystem of our target “personas”. It requires designers to critically think about all sorts of constraints and risks with the designs to pursue!
• Representing data accurately: quantitative data versus qualitative, when to use % and when not to
Throughout the internship, I received countless thought-provoking feedback from my peers. One of them is on the importance of how we present our findings, especially when the sample size may not be big enough to be statistically significant. Specifically, at what point can we use percentages and at what point do we use them as anecdotal findings. The conversation encouraged me to dive deeper into validity of data, and at what point can we make a strong, informed claim.
• Success metrics are crucial to a good design research plan
Through leveraging Google's HEART framework, I developed skills to critically analyze what "success" would look like for the design, and how it can be incorporated in the user tests. It allowed me to make effective decisions and consider how to track improvements for future iterations. It also guided me to reflect on my goal -- am I looking to test engagement? happiness? retention? task success? -- and ensure the testing plan is as successful as possible.
.gif)
.gif)