Under the Consumer Duty, the FCA expects firms to put the delivery of good customer outcomes at the heart of their business model and, critically, be able to evidence the outcomes achieved. Fundamental to delivering this evidence is customer outcomes testing. Alongside demonstrating regulatory compliance, tracking customer outcomes can offer a number of other benefits.
In this article, we propose four guiding principles firms must embrace to make a success of customer outcomes testing (COT).
- COT must be built on a solid understanding of your business.
- COT implementation is an iterative process.
- COT requires a mindset shift and patience to deliver success.
- COT should be driven by timely, forward-looking management information (MI) that is proactively considered and acted upon.
What is outcomes testing (and what it isn't)
A COT framework should measure whether a customer's product, service or related interaction has performed in a way that the customer would reasonably expect, and whether the risk of any potential for financial detriment or material distress or inconvenience has been minimised.
Some firms misinterpret (or overly rely on) outputs such as quality checking scores or customer satisfaction scores as a means of determining whether customers receive good outcomes. However, Consumer Outcomes Testing (COT), Quality Checking (QC) and Customer Satisfaction (CS) are not the same and meet different objectives. QC and CS only test certain elements of the journey through certain lenses and do not assess the full end-to-end picture. As illustrated below, there are a number of ways COT differs from QC and CS assurance processes:
Whilst customer satisfaction monitoring provides valuable information, information asymmetries between customers and firms mean that customers may not realise, until much further down the line, (or not at all) that they haven't received a good outcome. QC, by design, does not put the firm in a position to fully consider a customer's individual circumstances and how they may relate to a good outcome — nor whether the process itself would have delivered a good outcome at all.
Case study
The customer makes a phone call to their bank because they want to take out a personal loan for a holiday.
The customer service agent conducts the sales process over the phone and appropriately follows the script, inputs the customer's details accurately and emails the correct sales documentation to the customer. The case is selected for QC and is rated a Pass.
The customer receives the loan two days later and can book their holiday. They are very happy. When asked to complete a customer survey, they state they were very satisfied with the service.
However, following the sales call, the back-end system which completes the sale and underwriting of the loan had a system glitch whereby the customer's affordability was incorrectly overstated, meaning the customer received a loan that was unaffordable. Three months later, the customer falls into arrears, incurs an arrears fee and has adverse data applied to their credit file. The case is selected for outcomes testing and is rated a Poor Outcome.
Importantly, implementing COT does not mean decommissioning QC as it still provides valuable insight. This insight into the performance of staff should be used to support performance management, training and competency requirements. An existing approach led by COT may result in QA and QC programmes being able to be scaled back, releasing valuable resource to re-focus on outcomes testing work.
Four COT guiding principles to embrace
1. COT must be built on a solid understanding of your business
To understand the outcomes customers are receiving, firms first need to understand their business and have defined what good outcomes means to them. It may sound obvious, but a robust outcomes testing approach can't be built without first having:
- a strong understanding of each product, service or customer process,
- the target market they are designed for,
- the customer journey involved in buying, using and exiting them, and
- what a good outcome looks like for each product, service and stage of the customer journey.
Together, these factors form the foundation of the outcomes testing framework and MI requirements.
The Consumer Duty has a clear focus on each of these factors. Therefore, firms should make use of the implementation phase to scope and articulate what good outcomes look like for different cohorts of customers at key stages of each product lifecycle (as well as how they will be tested). The key here will be an appreciation that what constitutes a good outcome for one cohort of customers may not always be the same for another. Tailoring analysis to this effect will be vital as without having a firm understanding or definition of what 'good' looks like it will not be clear what to test against or how to calibrate outcomes.
2. Implementing COT is an iterative process
Implementing an outcomes testing framework will take time and require constant revision. For example, there may be areas where testing is required but the IT or business capability is not yet available to deliver the optimal MI. However, this should not delay the delivery of the framework (and use of tactical MI). Instead, a phased approach can be adopted and a more sophisticated solution implemented in a later phase (likely to be post July 2023). Equally, the framework should be dynamic/reactive with touchpoints built in for reviewing the model against emerging social, economic, organisational and regulatory issues.
3. COT requires a change of mindset and patience to deliver success
Outcomes testing looks at more variables relating to product, process, systems and communication than QC, and assesses them in granular detail. This means it will find problems which may culminate in poor customer outcomes. Typically, firms will notice a drop in quality scores when COT frameworks are first implemented. This is because legacy issues, competency weaknesses and full understanding of journeys across teams/divisions/companies are unearthed.
Additionally, the use of more forward-looking indicators rather than lagging ones in outcomes testing mean an increase in issues should be expected. Consequently, firms should be prepared to embrace a material drop in previously high assurance levels which, without context and education, may trigger alarm amongst the governance fora. Educating the business on the outcomes testing programme, how it differs from existing approaches and what insight it is providing should be viewed as a key part of its delivery and vital to its success.
The FCA has cited that the transition to Consumer Duty may require a change in mindset for some firms. For example, accepting that more issues may be found, and improvements can be made. This mindset shift will be required at all levels within the business. During the design and development stage, serious consideration should also be given to how the insight from outcomes testing MI will be extracted, presented, acted upon as well as the appropriate resource needed to conduct this. Harnessing the evolving benefits of outcomes testing MI will be an ongoing challenge.
4. COT should be driven by timely, forward-looking MI that is proactively considered and acted upon
An effective outcomes testing framework can deliver more benefits than solely addressing the need for regulatory compliance.
Responsive and Timely — MI should be proactively considered and more forward-looking. This provides the opportunity to address issues before they evolve or crystalise at an individual, customer cohort or product target market level. If insights are acted upon in a timely and appropriate manner, this could lead to commercial and financial benefits. For example, by negating the need for costly, large scale retrospective remediation mitigation, reduced adverse regulatory scrutiny and protection from reputational damage.
Identifies opportunities — Outcomes testing doesn't just identify poor customer outcomes. It can also be used as an opportunity to identify process efficiency improvements, technological solutions, training needs, cost savings, product simplification opportunities as well as new market opportunities based on customer circumstances. For example, MI may reveal an area of unmet need that the firm may wish to explore in future product design processes. Equally it can provide insights into products and services that are not operating or delivering as expected and indicate the need for improvements.
Synergy with FCA approach — The FCA uses an outcomes testing framework when undertaking its supervisory activity. Therefore, looking at conduct risk through a similar lens provides the opportunity to align perspectives as well as helping to pre-empt broader regulatory intervention. It will also provide a commonality of understanding when evidencing compliance with regulatory expectations.
Conclusion and Next Steps
To be successful, an outcomes testing framework needs to be appropriately scoped, structured, resourced and integrated across all relevant business operations and departments. There are a number of areas specific to each firm and its business model that will need to be carefully considered. Sample sizes, classifying outcomes, identifying and acting on root causes, and the interaction with the wider conduct risk framework are some of the more critical components.
However, over and above these, a firm's culture and mindset will be critical. From the top down, outcomes testing must be adopted by the first line of defence and understood and championed as an opportunity to continuously improve customer outcomes. This will deliver a solid foundation upon which to implement (and benefit from the opportunities arising from) the FCA's Consumer Duty.
In conclusion, the design and delivery of an effective outcomes testing framework is not straightforward and requires careful tailoring to the firm and its customers. However, alongside the challenges, its implementation can be rewarding. When starting on this journey, firms would be wise to consider the following questions:
- How well do you know your customers and products, what good outcomes look like for them and how they have they been captured?
- Does your firm have the right mindset upon which to develop, deliver and operate an outcomes testing approach?
- How will buy-in for, and understanding of, the wealth of information an effective framework will deliver and the consequential impacts be achieved?
- How will the benefits provided by the insights of COT be unlocked?
How can KPMG support?
Understanding, testing and evidencing customer outcomes is an opportunity to improve customer engagement and trust and potentially business performance.
KPMG in the UK has specialists with experience supporting firms with the development and implementation of outcomes testing. If you would like to discuss outcomes testing or our experience of designing and deploying a outcomes testing framework, or any other Consumer Duty component, please get in touch below.
Related content
