ESG Ratings and Data: How to Make Sense of Disagreement

This client alert, part of a series focused on ESG disclosure and regulatory developments, should be read together with our ESG Lexicon, available here, which provides definitions of some of the key terms used in ESG reports, disclosures and regulations.

Key Takeaways

• Providers of ESG ratings play an important role in how markets assess the ESG performance of companies. However, their use of myriad methodologies and data, including aggregate E, S and G scores, for instance, can produce conflicting results and market confusion.
• Ratings discrepancies have important consequences for both investors and corporate managers; each must take steps to address potential ratings inaccuracies.
• When investors use ESG ratings, it is important that they thoroughly understand the ratings methodology and data collection process to ensure the ratings they use are meeting their expectations.
• Companies would benefit from actively monitoring and, where possible, managing their ESG ratings and the raw ESG data being collected about them in order to identify inaccuracies and improve the ESG information available to investors.

Executive Summary

The ESG reporting ecosystem is rapidly evolving and is made up of an ever-increasing number of frameworks, standards and providers of ESG research and ratings. Our prior client alerts have provided an overview of leading ESG frameworks and standards as well as related harmonization initiatives (available here and here, respectively). ESG ratings can be an important resource as investors seek to interpret and compare ESG information. The number of ESG ratings providers expanded in recent years, but unfortunately this has not led to greater market clarity, because the resulting ratings often are inconsistent or even conflicting, and also compete with one another. In this client memorandum, we consider the variations in methodologies and data that drive the divergence across ESG ratings, as well as the steps companies and investors can take to mitigate misinformation.

Growing Investor Demand for ESG Data

Investor demand for ESG-driven investment is increasing, which has led to a related demand for ESG information and data from which to evaluate investment options. Over 3,000 organizations with over $103 trillion in assets under management (AUM) are now signatories to the UN Principles for Responsible Investing (“PRI”).^[1] In the United States alone, capital inflows into ESG/sustainable-labeled funds exceeded $21 billion in 2019, a nearly fourfold increase from 2018.^[2] Net inflows into ESG funds available to US investors totaled $30.7 billion through the first three quarters of 2020, representing a roughly 50% increase from 2019.^[3] Investors are working to more thoroughly integrate ESG into all parts of their investment processes, from fund formation to due diligence to portfolio management. In response to growing investor demand for more ESG data points that are comparable across a broad universe of companies, a number of organizations have undertaken efforts to both increase and standardize ESG disclosures, which currently are largely voluntary.^[4] While there has been a significant focus on improving the level and quality of ESG disclosures by companies, investors and other stakeholders predominantly assess ESG performance through research and ratings provided by third parties. These third parties collect ESG data from a variety of sources and transform these data into company-specific ESG ratings.

There are currently at least 125 organizations providing ESG ratings and research.^[5] While many are niche players that focus on specific ESG issues or geographies and use novel rating methodologies, major data providers and credit rating agencies are also entering this space. As Bloomberg stated in relation to S&P Global Inc.’s acquisition of RobecoSAM’s ESG ratings business in late 2019, “[c]redit rating companies are muscling their way into the burgeoning world of responsible investing, purchasing smaller outfits that provide [ESG] scores.”^[6] The most commonly used ESG ratings include: Morgan Stanley Capital International (MSCI); Institutional Shareholder Services Inc. (ISS); RobecoSAM; Sustainalytics; RepRisk AG; and Refinitiv (formerly Asset4).

ESG Rating Disagreement

One of the major challenges for investors and other stakeholders attempting to use ratings to measure ESG performance is that ESG ratings often conflict with one another. As a result, rated companies often struggle to interpret ESG rating reports in the context of their own internal assessment of the company’s ESG performance, and investors are left unsure of which ratings can be relied upon.

Divergences in ratings are commonplace. One study by researchers from the MIT Sloan School of Management found the correlations between six of the major ESG ratings were on average 0.54, and ranged between 0.38 and 0.71.^[7] Divergences in scope of analysis (e.g., analyzing different segments of the value chain, such as supply chain analysis vs. internal operations) and  specific issues measured (e.g., employee training vs. internal promotion used to measure human capital development) have been shown to explain the majority of variation across ratings.^[8]

Moreover, ESG ratings can be challenging to interpret by their nature of aggregating a variety of issues. Combined ESG ratings (i.e., the final ratings reported by ESG rating agencies) also take into account social and governance factors, which are equally broad topics that similarly lend themselves to varying interpretations of scope and measurement categories. For example, a company might excel in worker health and safety but demonstrate poor performance with respect to the board’s risk management oversight or diversity and inclusion. Critics (including former SEC Chairman Jay Clayton) suggest combined E, S and G scores may create “aggregate confusion” or may be over-inclusive and imprecise.^[9] Positive performance in one area does not negate poor performance in another – a point that combined ratings might obscure.

Even when one adjusts for differing measurement methodologies, it is difficult to find consistency across ratings.^[10]  Commentators believe that the cause is not only the approaches to ratings, but also the quality of the underlying data used. Due to the lack of mandatory ESG reporting standards, providers of ESG ratings use not only data disclosed voluntarily by the company, but also data from third-party sources to evaluate ESG issues identified as relevant by the ESG rating agency. For example, if a company does not report water use data, an agency may use data from water utilities near the company’s known operational sites to estimate the water used by the company at those locations.

Paradoxically, greater company disclosure of ESG data is associated with greater disagreements across ratings.^[11] One study found that in its sample set of companies with some of the most robust public environmental disclosures, the environmental components of MSCI, RobecoSAM and Sustainalytics had correlations ranging from just 0.31 to 0.46.^[12] Researchers at Harvard University and the University of Oregon recently offered an example of what this looks like in practice:

[W]hen Workday Inc. significantly increased its ESG disclosure for fiscal year 2015 by adopting the new G4 Global Reporting Initiative guidelines, its sustainability report increased from 54 to 98 pages. Its ESG disclosure score from Bloomberg increased significantly for both the environmental and social pillars. However, disagreement increased as its Thomson Reuter’s rating for both environmental and social issues increased, MSCI’s environmental rating decreased and its social rating increased, and Sustainalytics left its environmental rating unchanged while its social rating increased. . .. The rating disagreements can arise because more disclosure is more likely to lead to rating agencies using different metrics in assessing a firm’s ESG performance.^[13]

What else is causing these problems with ESG data? Research suggests a few answers:

• Inconsistencies in the form of arguably similar data are commonplace. A random sample of 50 Fortune 500 companies’ employee health and safety related disclosures found these companies reported employee health and safety metrics more than 20 different ways in their sustainability reports (e.g., number of accidents with fatal consequences, occupational injury rate-related fatalities, and/or lost-time incident frequency rate).^[14] These metrics each attempt to assess a company’s health and safety performance, yet their differences make determining which metrics best capture good company performance challenging. It is unclear if differing sets of metrics can be used to produce comparable assessments of companies’ employee health and safety performance. Those who prepare ESG ratings are, therefore, left to determine for themselves which metrics best capture good employee health and safety performance and how to aggregate this data. Their conclusions may vary.
• ESG data is unaudited. Significant data omissions, unsubstantiated claims and inaccurate figures can be difficult to identify within sustainability reports.^[15]
• Actual company disclosure does not always match what is reported by ESG data providers, suggesting they may impute ESG data.^[16] Different imputation methods can produce wildly different results, threatening the validity of any analysis that uses the data. As such, Harvard Business School researchers found systemic disagreements and frequent errors in environmental data sourced from two providers.^[17]
• ESG metrics predominantly measure whether organizations have specific policies or engage in certain activities, rather than the impacts of such policies or activities. A study of 1,700 “S” indicators found 92% assessed whether the organization conducted audits, risk assessments or training; participated in membership organizations or other collaborations; or engaged stakeholders (e.g., consider the metric “does the company have a Diversity & Inclusion Policy?”). Only 8% measured the actual effects or impacts of these policies and activities (e.g., consider the metric “percent women in management positions at the company”).^[18] Ratings based on binary responses as to whether a given company policy exists may be practical, since this information is easy to measure and generally available, but may not offer investors a clear picture of company performance.

Implications for Investors and Companies

Inconsistencies among ESG ratings can sow confusion regarding a company’s ESG performance, which in turn can impact stock performance.^[19] Moreover, research findings prescribe caution to investors using ESG ratings in their investment decisions as a result of “an evident lack in the convergence of ESG measurement concepts.”^[20] Ratings discrepancies therefore have important consequences for both investors and corporate managers. More and more investors now aim to counter this confusion through their own proprietary ratings and assessments based on analysis of raw ESG data, and they should thoroughly vet the data they collect and analyze in order to ensure its reliability. Similarly, when investors do use an ESG rating, it is important that they thoroughly understand the rating methodology and data collection process. Not all ESG ratings convey the same set of information, and, as such, investors should ensure the ratings they use are meeting their expectations. Companies must also take steps to protect themselves from ratings inaccuracies. As a first step, companies should actively monitor their current ESG ratings and develop an approach to engage with ESG rating agencies to ensure an accurate assessment of the company’s ESG performance. This includes confirming that ESG rating agencies are using correct data for their analysis. Armed with that knowledge and the publicly available ESG data, companies will be better positioned to engage directly with ESG rating providers to address perceived rating inaccuracies and, even more importantly, to be able to respond to investor questions regarding any particular rating or perceived deficiency. 

                                                                                                                                     *       *       *

[1]      See UN PRI, “About the PRI,” available here.

[2]      Jon Hale, “The ESG Fund Universe is Rapidly Expanding,” Morningstar (March 9, 2020), available here.

[3]      Morningstar, “Global Sustainable Fund Flows: Q3 2020 in Review” (October 2020), available here.

[4]      For example, two recent initiatives – one by leading ESG standard-setters, such as the Global Reporting Initiative (GRI) and Sustainability Accounting and Standards Board (SASB), and another by the World Economic Forum and leaders of the “Big Four” accounting firms – attempt to unify the ESG reporting ecosystem and call on companies to expand their voluntary ESG disclosures. For more on these initiatives, see our prior client alert here.

[5]      Mark Kramer, Nina Jais, Erin Sullivan, Carina Wendel, Kerry Rodriguez, Carlo Papa, Carlo Napoli and Filippo Forti, “Where ESG Ratings Fail: The Case for New Metrics,” Institutional Investor (September 7, 2020), available here.

[6]      David Caleb Mutua and Jacqueline Poh, “These Agencies Want to Check Who’s Naughty and Who’s Nice,” Bloomberg (November 22, 2019), available here.

[7]      Florian Berg, Julian F. Koelbel, and Roberto Rigobon, “Aggregate confusion: the divergence of ESG ratings,” (May 2020), available here.

[8]      Id.

[9]      One critic of combined ESG ratings, chairman of the Securities and Exchange Commission Jay Clayton, recently stated: “I have not seen circumstances where combining an analysis of E, S and G together, across a broad range of companies, for example with a ‘rating’ or ‘score’, particularly a single rating or score, would facilitate meaningful investment analysis that was not significantly over-inclusive and imprecise,” Chris Flood, “SEC chair warns of risks tied to ESG ratings,” FT (May 28, 2020), available here.

[10]     Aaron K. Chatterji, Rodolphe Durand, David I. Levine, and Samuel Touboul, “Do ratings of firms converge? Implications for managers, investors and strategy researchers,” Strategic Management Journal 37, no. 8 (August 2016) 1597-1614., available here.

[11]     Sakis Kotsantonis and George Serafeim, “Four things no one will tell you about ESG data,” Journal of Applied Corporate Finance 31, no. 2 (July 2019), 50-58, available here.

[12]     David Freiberg, D. G. Park, George Serafeim, and Rob Zochowski, “Corporate Environmental Impact: Measurement, Data and Information,” Harvard Business School Accounting & Management Unit Working Paper 20-098 (October 2020), available here.

[13]     Dane Christensen, George Serafeim, and Anywhere Sikochi, “Why is Corporate Virtue in the Eye of The Beholder? The Case of ESG Ratings,” Harvard Business School Working Paper 20-084 (September 2019), available here.

[14]     Kotsantonis and Serafeim, “Four things no one will tell you about ESG data,” Journal of Applied Corporate Finance 31 (2) (July 2019). Read more here.

[15]     Dinah Koehler and Eric Hespenheide, “Disclosure of Long-Term Business Value: What Matters?,” Deloitte University Press (2013), available here.

[16]     Kotsantonis and Serafeim, “Four things no one will tell you about ESG data,” Journal of Applied Corporate Finance 31 (2) (July 2019). Read more here.

[17]     David Freiberg, D. G. Park, George Serafeim, and Rob Zochowski, "Corporate Environmental Impact: Measurement, Data and Information," Harvard Business School Accounting & Management Unit Working Paper 20-098 (October 2020), available here.

[18]     Casey O’Connor and Sarah Labowitz, “Putting the ‘S’ in ESG: measuring human rights performance for investors,” NYU Center for Business and Human Rights (March 2017), available here.

[19]     Rajna Gibson, Philipp Krueger, Nadine Riand, and Peter Steffen Schmidt, “ESG rating disagreement and stock returns,” (June 2020), available here.

[20]     Gregor Dorfleitner, Gerhard Halbritter, and Mai Nguyen, “Measuring the level and risk of corporate responsibility – An empirical comparison of different ESG rating approaches,” Journal of Asset Management 16, no. 7 (September 2015) 450-466, available here.