Commission Effectiveness: Is It Measurable?

If you can’t measure it, you can’t manage it.
What’s measured, improves.

— Peter Drucker


… [I]f I have quick access to key metrics every day, my creativity stays within certain bounds—my ideas all center on how to achieve our goals.
— Paul Allen

*   *   *

A commissioner asked recently, “By what metrics can I assess my commission’s performance?” 

Commissions are realizing that their legacy practices—processing retail utility filings and intervening in federal cases—are too reactive.  They wish to display more purposefulness, more decisiveness, more independence from the stakeholders, more leadership.  How can they plot their progress?  First, some thoughts on metrics, arranged by three major objectives.  Then, some critiques of “traditional” metrics; and finally, some thoughts on whether the term “metrics” is even useful.

Clarify and Embed the Commission’s Regulatory Purposes

Metrics must reflect purpose.  Regulation’s purpose is to align private behavior with public interest.  The regulator must establish performance standards for utilities, then enforce those standards.  Setting standards requires multiple value judgments.  Two examples:  (1) What is the appropriate performance standard—average, superior, state-of-the-art, better-than-thy-neighbor?  (2) Where there are vectors in tension, like economic efficiency versus affordability, economic development versus environmental sensitivity, speedy decisions versus participatory procedures, what are the right resultants?  These conflicts call for courage, in the form of decisiveness.  Being everything to everyone—what Garry Wills calls “omnidirectional placation”—precludes purposefulness.

Metrics:  Has the commission made the necessary value judgments?  If you asked each of your commission’s employees (California has 800) to state the commission’s regulatory purpose, performance standards, and tension resolutions, would the answers be consistent?  Does the commission have clear expectations for each utility’s performance, in terms of cost-effectiveness and innovation?  Does each utility’s performance meet those expectations?  Would that utility’s employees all answer the “purpose” question consistently?  

Identify the Regulatory Challenges

Gas and electric challenges include climate change, energy efficiency, rate design, and regional power supply planning.  Water industry challenges include small water company finances, compliance with federal water quality standards, the prospects for regionalization and consolidation of scattered service territories, and the reality that Earth’s water supply is fixed while its population is growing.  Telecommunications challenges include redefining the responsibilities of the “carrier of last resort,” ensuring the cost-effective spreading of broadband, resolving the persistent scuffles between incumbents and new entrants, and getting candid (i.e., factual) about the competitiveness and price efficiency of “deregulated” markets.

Added to those industry-specific challenges are those cutting across all sectors:  market structure (Do we want competition or monopolies?); corporate structure (Do we want leveraged buyouts, conglomerates, and non-integrated asset collections; or do we want local companies focused on providing essential services?); infrastructure sufficiency (Will we pay for our own wear and tear, or will we bequeath those costs to our offspring?); rate design (Is it time to remove the average-cost mask, exposing the true hourly cost of consumption?); technological advances (Who should fund the demonstration projects that produce both dry holes and breakthroughs?); and workforce replacement and modernization (Where do we find the next generation of pole planters, tree trimmers, and substation designers?).

Metrics:  Has your commission identified the main challenges?  Has it assigned resources to them?

Shape the Regulatory Infrastructure to Meet the Challenges 

Regulatory excellence requires a regulatory infrastructure.  There are five parts to that infrastructure, each with its own metrics.

1.  Authority to act:  Prehistoric regulatory statutes need systematic modernization, not piecemeal revision.  Enacted a century ago, then layered with amendments responding to the latest urgencies and opportunities, our statutes have gaps, redundancies, and asymmetries that weaken regulatory authority. 

Metrics:  Given the challenges described above, do we have the statutory authority we need to produce the performance we need?  Are our relations with legislators productive?  Do legislators look to the commission first?

2.  Information necessary to act:  Information reaches regulators unevenly:  if they ask, if they have time to ask, if they think to ask, if someone chooses to supply it, if the information supplied is truthful and evenhanded. 

Metrics:  Do we have the information, and the information-gathering processes and resources, necessary to make good decisions?  Do the utilities have a duty to inform?  Does the commission learn of its options sufficiently early to make decisions timely and carefully?  Do they fulfill that duty timely and objectively?  Is there trust?

3.  Expertise necessary to act:  Traditional commission expertises are silo-based:  accountants, economists, engineers, lawyers; electric, gas, telecomm, water; rate cases, mergers, service quality investigations.  These expertises remain essential, but there is a risk of hammers seeing only nails.  The new challenges are multidisciplinary and long-term.

Metric:  Do our commissioners and staff have skills in multidisciplinary thinking and multi-year strategizing?

4.  Decisionmaking leading to action:  Many legacy procedures emphasize the reactive—awaiting utility filings at the state level, intervening as supplicants in federal venues.  Reactivity risks prioritization-by-parties.  Utilities initiate proceedings to advance business objectives.  That’s understandable and acceptable.  But the sum of these entrepreneurial efforts does not equal public-interest policy.  Reactivity makes regulators captive.

Marshall McLuhan (Understanding Media:  The Extensions of Man (1964)) offers an analogy.  If “medium is the message,” then procedure is the policy.  The context (e.g., cinema, television, comic book, traditional book) defines the intensity of a person’s participation.  The procedure (e.g., adjudication, investigation, rulemaking, cross examination, multi-party panels, informal discussions, “collaboratives,” commission-directed submissions) defines the level of commission alertness—and the quality of its leadership.

Metrics:  Policymaking is both proactive and reactive.  Are we getting the mix right?  Do the present procedures and workloads allow the commission to lead, or only to react?  Do our proceedings focus on the commission’s priorities or the stakeholder’s priorities?  Are we disposing of dockets or producing performance?

5.  Continuous re-evaluation:  Effective regulation is continuously self-critical:  It examines and re-examines its infrastructure to ensure the best fit of purposes, authorities, information, expertises, and procedures to the challenges at hand. 

Metrics:  Have we defined regulatory excellence?  Are there re-evaluation criteria and procedures, a process for making mid-term corrections?  When a commission approves a merger based on specified expectations of benefits, does a later commission assess the results?

Other Metrics:  Useful and Not-So-Useful

I have identified above three main commission goals—clarify purposes, identify challenges, shape infrastructure—and several metrics by which to assess progress.  Here are more metrics that together test achievement of these goals.

Advocates’ performance:  Useful.  Do the advocates appearing before regulators address commission priorities or their own objectives?  Do they offer balanced positions that apply their expertise to the commission’s needs, or do they insist on their clients’ advancement regardless of context?  Do they use facts and logic, or adjectives and adverbs?  Do they advance propositions that are absolute and unprovable?  (See Mark Twain:  “An adjective habit, or a wordy, diffuse, flowery habit, once fastened upon a person, is as hard to get rid of as any other vice.”  Letter to D. W. Bowser, 3/20/1880.)

Commissioner and staff time allocation:  Useful.  How do we spend a majority of our time:  Reading objective material or reading stakeholder advocacy?  Meeting with experts or meeting with parties?

Docket disposition:  Not so useful.  “We process cases expeditiously” is basic competence; it goes without saying.  If we were evaluating a young musician for admission to Juilliard, we would not say, “She plays in tune.”

Rate levels:  Not so useful.  “Our rates are lower than comparable states” tells us nothing about performance.  Are they lower because commission policies produce better utility performance, or because the utility has deferred maintenance by shifting costs to future ratepayers and future commissioners?

*   *   *

We now can see the inadequacy of the term “metrics.”  The term connotes quantifiability.  But none of the useful metrics are quantifiable.  A commission is not a consulting firm (metrics: billable hours and quarterly profits); regulation is not an election contest (metrics:  I lowered taxes, you increased the deficit).  Even in the worlds of consulting and campaigning, real public value is unrelated to these common quantities.  Measurement of value is necessary, but the currency of value remains elusive.  Let’s keep thinking.