Five Minutes…With Don Stacks

Posted on February 4, 2011 by admin

This is David Geddes, chair of the Institute for Public Relations Commission on Measurement and Evaluation. I am also vice president of research and development at evolve24 in Saint Louis, Missouri.

Today I am talking with Professor Don Stacks, Professor of Public Relations and Associate Dean for Research in the School of Communication at the University of Miami. Don also a member of the IPR Commission on Measurement and Evaluation, and an IPR Trustee.

Don, the International Public Relations Research Conference takes place March 9-13 in Miami. Tell us the history and objectives of this conference.

IPRRC began 15 years ago when a group of senior professionals and academics got together and decided it was time for a conference that focused on public relations research and practice. Its first three meetings were at the University of Maryland, and since then IPRRC has met in Miami. When the conference moved to Miami we decided to focus the discussion on public relations research, pedagogy, and theory, and a new conference format was adopted.

The format of this conference is distinctive … I describe it as speed dating meets academia. How does it work?

We are interested in good ideas and how authors go through the research process. First, we accept papers based on abstracts, and when papers are presented in Miami, we sometimes find either no results or different results — something you don’t see at most conferences where hypothesized or significant findings are the prerequisite to being included. Second, IPRRC is focused on discussing, not lecturing; therefore, each presentation is presented in round table format. Participants choose where to sit and when to leave a round table (or, not leave and hear the presentation again). The idea is to get people talking to each other. Thus far it has worked quite well. There are six presentations going on at the same time — but each presentation is limited to 15 minutes. This means that participants must choose which presentations to engage in and which not to engage in. Abstracts of all presentations are in the IPRRC Program.

I will be attending the conference for the sixth time. Why would you say that a PR practitioner should consider this conference?

Several reasons. First, it is cutting edge research and thought. Second, the top academics are attending and presenting. Third, the professional often asks questions the academic hasn’t thought of (like “so what?”). And fourth, also in attendance and presenting are some of the best young graduate students –students who will be leading the way in the future. In addition, the conference Proceedings add to the published Body of Knowledge.

How can we keep up with the IPRRC?

All we do is found on the Institute for Public Relations website www.instituteforpr.org and will be on a complementary IPRRC website under construction. Some top social media researchers and measurement people will be tweeting from the conference.

Posted in Barcelona Principles, Business Outcomes, Dashboards & KPIs, Market Mix Modeling, Media Analysis/Metrics, Reputation Management, Search Analytics, Social Media ROI, Stakeholder Metrics, Survey Research, Web Analytics | Leave a comment

Advancing PR Research by Dr. David Geddes

Posted on January 10, 2011 by admin

The mission of the Institute for Public Relations (IPR) – developing the science beneath the art of public relations™ ‑ remains the guiding principle of the IPR Commission on Public Relations Measurement and Evaluation in turbulent yet exciting times for the field of research.

Public relations and communications research, advertising research, and marketing research face common challenges to deliver business insights in changing marketplaces where our stakeholder audiences, publics, and customers have access to and use new communications tools.

Our goal is to advance best practices and principles in public relations research, advocate for research, and generate new knowledge to support the field. As researchers, we are comfortable with the tried and tested methods of qualitative and quantitative research. We understand focus group dynamics, probability sampling, significance tests, factor analysis, cluster analysis, and multiple regression. We have white papers and case studies. Can we develop reasoned approaches to measurement and evaluation in the exciting world of social media?

So, for 2011, how will we continue our mission (with the caveat that we are having a strategic planning meeting in January to develop these and other themes)?

  • Following a successful 2010, the IPR Commission will continue publishing white papers and research. We will be organizing and improving access to our existing papers on IPR’s new website.
  • We’re eager to receive your contributions about social media research for our social media hub, www.iprmeasure.org. Please contact me at david.geddes@evolve24.com to contribute.
  • I will be writing a regular blog based on conversations with experts in public relations measurement from corporations, non-profits, agencies, and universities. If you are interested in joining in a short conversation, please let me know via email david.geddes@evolve24.com.
  • The International Public Relations Resarch Conference in Miami on March 9-12 ) and the IPR Summit on Measurement in September will assemble a formidable group of experts for serious discussion and some fun. Join us this year. For more information, see the Events section of the IPR website…
  • IPR and the International Association for Measurement and Evaluation of Communication (AMEC) are co-organizing the 3rd European Summit on Measurement to be held inLisbon in June 2011.
  • We will be hosting webinar events in 2011 addressing key issues in our field. These will be especially valuable for those who cannot attend our conferences.
  • Our new members – Professor Shannon Bowen of Syracuse University, Jackie Matthews of General Motors, and Ruth Pestana of Hill & Knowlton – bring a wealth of additional experience.
  • Through collaboration with other organizations including the PRSA, AMEC andothers, we look to advance the practice of research and measurement in public relations, leveraging the different strengths of each organization in a common cause.

I am honored to chair the IPR Commission on Measurement and Evaluation for 2011.
Commission members and past Commission chairs have left me with a rich legacy on which to build.

David Geddes
2011 Chair
IPR Commission on Public Relations Measurement and Evaluation
david.geddes@evolve24.com

Posted in Barcelona Principles, Business Outcomes, Media Analysis/Metrics, Reputation Management, Search Analytics, Social Media ROI, Survey Research, Web Analytics | Leave a comment

Advancing Measurement and Evaluation of PR by Pauline Draper-Watts

Posted on December 14, 2010 by admin

As I reflect on my two years of chairing the Institute for Public Relations Commission on Measurement and Evaluation, I appreciate the rich legacy passed on by preceding chairs and the amazing team who tirelessly give their time and expertise to further the quest for good research and measurement. I feel privileged to work with many of the foremost thought leaders in our field who are passionate about research and measurement.

In the past two years we have seen two major initiatives.

The IPR Commission shaped the recommended measurement standards as part of the Public Relations Society of America (PRSA) initiative to promote the Business Case for Public Relations. The Commission also worked with the PRSA to compile an extensive database of measurement resources many of which came from IPR papers that can be found on our website.

The second major initiative has been the Barcelona Declaration of Measurement Principles that was presented, debated and voted on at the 2nd European Summit on Measurement organized by the Association for Measurement and Evaluation of Communication (AMEC) and the Institute for Public Relations. In addition to the IPR Commission, AMEC and the PRSA we were joined by the Global Alliance for Public Relations and Communication Management and the International Communications Consultancy Organization. We saw more than 200 delegates from 33 countries entering into the discussion and providing feedback.

The 7 Barcelona Principles are:

1. Importance of Goal Setting and Measurement

2. Measuring the Effect on Outcomes is Preferred to Measuring Outputs

3. The Effect on Business Results Can and Should Be Measured Where Possible

4. Media Measurement Requires Quantity and Quality

5. AVEs are not the Value of Public Relations

6. Social Media Can and Should be Measured

7. Transparency and Replicability are Paramount to Sound Measurement.

The principles will again be addressed in Lisbon, Portugal when AMEC and IPR host the 3rd European Summit on Measurement from June 8-10, 2011. Like the two previous Summits, we will continue to use this European Summit to establish the basic philosophy behind a set of standard practices for proving the value of public relations to business.

Perhaps the most controversial of these principles is the one associated with AVEs.

Within the commission we have had a year-long task force following up on our October 2009 vote to reject the term, concept and practice of Advertising Value Equivalent. This resulted in a paper which concluded that there was no evidence to suggest that advertising and editorial space hold equivalent value. While use of AVE is a common practice, this does not justify that practice as appropriate. AVEs are not a proxy for measuring value or return-on-investment for public relations.

Following on from the Summit, several members of the commission have been involved with the AMEC task force to look at validated metrics to replace AVEs. A further task force has been set up to consider Social Media metrics. Both of these task forces presented at the IPR Measurement Summit in October (see www.iprmeasure.org) and at the AMEC/PRSA London Summit last month.

Throughout my term we have also published several white papers on a range of topics. I will mention one here and encourage you to explore all of the IPR papers online.. We produced an update to Guidelines for Setting Measurable PR Objectives which aligns well with Principle 1 of the Barcelona Principles. The paper has been translated into Chinese and published in the December edition of the China International Public Relations Association’s journal PR Magazine.

As always we have supported the International Public Relations Research Conference in Miami and the Summit on Measurement both of which always prove to be a stimulating time both within the presentation settings and in offline discussion.

I am delighted that Dr David Geddes, Vice President of Research & Development at evovle24, a Maritz Research company, will be our next chair. He has done an excellent job of steering the publications committee and is excited to be taking on leadership of the commission. David has creative, exciting plans for this coming year and I will leave him to talk about them in a future post.

Thank you for the opportunity to serve the IPR Commission and advance measurement and evaluation in our profession.

Pauline Draper-Watts

pauline.draper@wattsconnect.org

Posted in Barcelona Principles, Business Outcomes | Leave a comment

“AVE” Is Not a Proxy for Measuring ROI of Public Relations

Posted on October 11, 2010 by admin

Hot off the press: The Institute for PR Measurement Commission formally voted Friday on a new position paper that concludes there is no evidence to suggest that editorial space obtained through public relations and advertising hold equivalent value. The report follows on last year’s vote by the Commission to “condemn the name, concept and practice of ad value equivalency,” which helped lead to development of the Barcelona Principles ratified in summer 2010. The new position paper formally and publicly outlines what’s wrong with AVEs and how practitioners can better measure the value of public relations.

“It’s the right time for a leading voice in public relations research and measurement to reject this practice,” said Robert W. Grupp, President and CEO of the Institute for Public Relations. “The use of AVEs has distracted the industry from more valid measures of public relations’ impact on business goals and objectives.”

“AVE is not a proxy for measuring the return-on-investment of public relations,” wrote Dr. Brad L. Rawlins, who chaired an IPR Commission Task Force on the subject. Dr. Rawlins also is chair of the Department of Communications at Brigham Young University. “Even more problematic is the use of AVE to represent a public relations outcome, and a meaningful measure to represent a financial return on investment. This obfuscating practice often prevents or misdirects focus from quantifying the more meaningful outcomes of public relations.”

Check out the news release and position paper below, and spread the news…

IPR News Release: Measurement Commission Position on AVE 08-11-2010

View more documents from Institute for PR Measurement Commission.
IPR Measurement Commission: Position on AVE 10-08-2010

View more documents from Institute for PR Measurement Commission.

Posted in Barcelona Principles, Business Outcomes, Media Analysis/Metrics, Social Media ROI | Leave a comment

Barcelona, Meet Portsmouth — Let the Comments Begin

Posted on October 10, 2010 by admin

Building on this summer’s historic Barcelona Principles ratification, the IPR Measurement Summit hosted a session Thursday afternoon that outlined recommendations from two AMEC task forces that were established to address practical questions about putting the principles into action:

1) What are the “validated metrics” to replace ad value equivalency (AVEs)?
2) How do you get started in measuring social media, and what are the definitions of relevant metrics?

The task forces were established by AMEC’s US Agency Research Leaders Group chaired by David Rockland of Ketchum, and were led by Ruth Pestana of Hill & Knowlton and Tim Marklein of Weber Shandwick. Thursday’s session was led by Peter Wengryn of VMS, Mike Daniels of Report International (filling in for David and Ruth) and Tim Marklein.

IPR Measurement Summit — Barcelona Principles Taskforce Recommendations
View more presentations from Institute for PR Measurement Commission.

Validated Metrics to Replace AVEs
The first task force objective was to provide clear direction on how to develop metrics that adhere to the Barcelona principles. The resulting “validated metrics frameworks” are detailed enough to speak to the customary communications outputs, outtakes and outcomes, at various stages along the traditional “marketing funnel”: awareness, knowledge/understanding, interest/consideration, intent/support/preference, and action.

To aide in putting the guidelines into practice, the task force defined the three communications phases for evaluation as: “Public Relations Activity” – the metrics reflecting the process of producing or disseminating communications products or messages; the “Intermediary Effect” – metrics that result when the target audience receives the communications and which play a significant part in what drives the ultimate impact of the public relations activity; and the “Target Audience Effect” – the action-driven outcome metrics obtained through explicit quantifiable results of the PR activities.

To help practitioners meet the challenge of demonstrating the impact of PR activities to their marketing colleagues, we have matched the communications phases with each stage of the marketing funnel and proposed metrics at each level. This should allow for developing meaningful metrics for those outside of the PR discipline which can be clearly linked to the PR objectives.

Finally, the tailored guidelines have been prepared for seven key areas of communications: Brand and Product Marketing, Reputation Building, Issues Advocacy and Support, Employee Engagement, Investor Relations, Crisis and Issues Management, and Social/Community Engagement.  Each section outlines some specific metrics for that communications area, but naturally there is some overlap. As all PR programs need customized measurement, the task forces encourage practitioners to combine metrics as needed to suit the details of every communications program.

The grids included in this presentation have been developed as a framework through which to think about possible metrics for individual PR programs. They are not intended to be comprehensive and therefore do not include every possible metric — they are merely a starting point to identify appropriate metrics based on the communications stage and business/marketing objective.

Replacing AVEs
As part of the “validated metrics frameworks,” the task force outlined several alternative and more meaningful metrics to replace advertising value equivalency — giving clear guidance to build on Barcelona Principle #5 that rejects AVEs as a measure of the value of public relations.

For practitioners that were using AVEs as the sole metric for evaluating PR success, the task force said to “stop” — there is no “single metric” for measuring PR success. Public relations is a broad discipline that requires multiple metrics tied to well-defined objectives. The “right” metrics will vary by results achieved. Please use the validated metrics guidelines to help determine what’s most appropriate for your program.

For practitioners who were using AVEs to provide comparative media costs for PR in relation to other marketing disciplines, there are several validated metrics that can be appropriately used for evaluating earned media results to paid media results: earned impressions (measures potential reach based on media impressions data); earned CPM (cost per thousand impressions – measures efficiency of earning media coverage, provides comparison to efficiency of other marketing vehicles); GRP (gross rating points – measures reach against percent of total population); TRP (target rating points – measures reach against percent of targeted population); etc.

For practioners who were using AVEs to provide a dollar/euro/yen or other financial denomination for PR results, there are several validated metrics that can be used appropriately to measure public relations in financial terms (where demonstrable): total value of sales/sales leads/revenue generated by PR activities; PR activities contribution to sales/sales leads/revenue (often via marketing mix analysis); cost savings due to PR activities (reduced customer complaints, etc.); increased target market size due to expanded mindshare; increase or decrease in market capitalization; etc.

Social Media Measurement
The second task force was chartered to build on Barcelona Principle #6, which outlined that “social media can and should be measured.” Based on that foundation, the task force explored key issues for practically implementing social media measurement, and emerged with four key recommendations and a series of supporting insights:

1) Focus on outcomes: This builds on the core concept behind the Barcelona Principles and other smart measurement best practices laid out over years by the Institute for PR, AMEC and other organizations. For social media, that means defining outcomes and goals in advance of implementing a social media program — and the task force recommended two documents from PRSA and Altimeter Group that provide good menus of potential outcomes for social media programs. The task force also outlined that these outcomes can’t be defined or managed purely within PR or communications silos, and that the outcomes will likely span multiple business goals.

2) Starter Set of KPIs: To help practitioners who aren’t yet measuring social media, the task force outlined a set of key performance indicators to get started — including basic quantitative data tracked via social media monitoring and web analytics, as well as more qualitative metrics that offer insight and context into the conversations and communities relevant to a brand. The task force also developed a validated metrics framework for “social/community engagement” to put the KPIs into context of the communications stages and business/marketing goals.

3) Influence Rating/Ranking: As PR practitioners engage with customers, influencers and media online, the question of how to assess who is influential has become one of the most significant areas of exploration — sometimes addressed via data and other times via “gut feel” or other ad-hoc evaluation. Unfortunately, much of the data available today confuses “popularity” as influence, and fails to recognize that influence is domain-dependent and often client-specific. The task force recommends that practitioners not rely strictly on the generic tools — and instead should create their own custom criteria that reflects influence as a multi-level, offline and online concept that can change over time based on commentary, behavior and audience.

4) Content Sourcing: Last but not least, the task force tackled the fundamental and often overlooked challenge of how content is sourced, filtered and aggregated within social media measurement. Recognizing that “not all sources are created equal,” the task force outlined several key questions that practitioners should ask before adopting social media monitoring or measurement tools. Given the pervasiveness of data quality and consistency issues, the task force also encouraged practitioners to invest more time in quality control, filtering and “looking under the covers” to ensure they are measuring what they want to measure. The task force also encouraged vendors to be more transparent and realistic in their marketing, so practitioners can know what they are getting — and equally important, what they’re not getting.

Open Comment and Consultation Period
AMEC has published the post-Barcelona recommendations via the AMEC site, this IPR Measurement Commission hub, and via PRSA (send comments to christina.darnowski@prsa.org). The recommendations will be open for comment until October 21, at which point the recommendations will be refined and edited through a consultation process with key organizations including CIPR, Global Alliance, PRCA, ICCO, PRSA and others. The final recommendations will then be presented and discussed November 17 at an AMEC/PRSA Workshop in London with participants from the IPR Measurement Commission, PRCA and CIPR. Tell us what you think…

[NOTE: Users of these Guidelines may find it useful to refer to the Dictionary of Public Relations Measurement and Research by IPR Measurement Commissioners Don Stacks and David Michaelson.]

Posted in Barcelona Principles, Business Outcomes, Media Analysis/Metrics, Search Analytics, Social Media ROI, Stakeholder Metrics, Uncategorized, Web Analytics | Leave a comment

“An In-Depth Look at What’s Really Happening in Social Media Measurement”

Posted on October 8, 2010 by admin

Closing off the conference today is a power-packed panel hosted by Katie Paine of KD Paine & Partners, exploring the latest insights and front-line best practices in social media measurement. Here are the highlights from Katie and panelists Diane Lennox from SAS, Margot Sinclair Savell from Weber Shandwick, Richard Binhammer from Dell and K.C. Brown from Cision:

  • SAS: Huge challenge due to 16 different market segments and audiences, and a company name that’s hard to isolate in search. Large, successful company and “no one knows who we are” competing against all the big names. Transitioning to a social media culture where everyone is a publisher and content officer. Expanded traditional media metrics and “top tier lists” to broader analytics and filtered lists of influencers. Who else is talking? What are they talking about? Are they talking about other things? And frankly, does share of discussion still matter? More data and complexity, but it’s “really the same game.”
  • Weber Shandwick: Got started in measurement as a national journalist in Canada, counting how many stories I wrote versus competitors. Joined Weber Shandwick and started doing media measurement and competitive analysis. Automation is great to a degree, but it’s “still really important to read.” Account teams in early days of social media would often ask: “Just tell us what the metrics are, we don’t care about the comments.” Diverse client base and each one has different measurement needs. Always starts with “why” — what are the goals. Online + offline = inline. Focused on “inline analytics” and developed Digital Media Scorecard, which can be flexible to address any client need and any channel. Use different tools for listening, engagement, research, measurement. What is the a-ha moment? What’s actionable?
  • Dell: Social media is a “journey” — keeps it in perspective. Need to test and experiment, then figure out how to deploy and scale. “Fail fast if you’re going to fail.” Social media is rich in data and metrics. Can track back to dell.com to evaluate traffic and sales. Moving beyond share of voice and tone — looking more at how information is shared, how conversations move, and what’s the result of action. Ratings and reviews on dell.com proved successful based on metrics, and are now being more broadly deployed. Focused on operationalizing social media to become a factor in the business — pushing the analytics into each business unit for decision making (listen, learn, engage). “We’re still on the journey, and no one’s completed the journey.”
  • Cision: Measurement people are a very small minority “on the mountaintop” speaking in a shared language. Cision has the challenge of working across a large client base who all want social media measurement, but they fall into three camps: 1) Give me the clips; 2) Give me the charts; or 3) Help me understand, give me the insights I need to make sound business decisions. Our answer to everything we do for the third camp is “custom, custom, custom.” Have to measure in depth across paid, owned and earned media. “You can’t just own the positives about you — you need to own the negatives about you as well.” We try to understand what’s being said, who’s saying it and where/when it’s being said.
  • What’s the #1 thing you’ve learned? Need to customize — there’s no single metric. Have to understand the business. For the first time, we as communicators can draw a causal effect. Critical to “look behind the numbers” and evaluate both qualitative and quantitative. “Question everything — and drive your vendors insane.” If you don’t understand the metric, it’s not useful. You get what you pay for — time and money. Get the right people in the room upfront to ensure you’re measuring the things that matter to the business.
  • You can’t create a “viral video” or a “viral link.” But videos and links can “go viral.” Look at the traffic patterns and who’s influencing who. Need to think through all the connections and explore richer, more accurate and business-savvy metrics. Clients want to look at “what’s the crossover?” Surveys show that 90% of media are getting story ideas from social media.
  • What mistakes are made? Sometimes you get so close to the data that you forget the main purpose of why you’re doing it. You have to experiment — fail fast and learn from it. Need to set expectations for the client upfront that the metrics and reporting mechanisms will need to change — don’t get locked into one approach.
  • How’d you get budget? Didn’t always have budget upfront but did it proactively to stay relevant. Shouldn’t be a budget issue — metrics are part of running business. Best practice is 10-15% of the program budget for monitoring and measurement, though the reality is 5-10%. As the program improves, the volume will grow.
  • Key finding: Dell has 500 execs on daily listening report disttibution list and 1700 execs trained on dashboard inc CMO & CEO.
  • Dell’s important learning: If you listen to negative sentiment, learn from it, and engage — the sentiment changes. This is a medium that evolves. Engagement matters. Act on the data, don’t just monitor it. Sentiment changes — and your business changes.
  • Final thoughts: Vast majority of clients don’t get so much coverage, should be able to read and analyze it without overly complicated analysis. That’s good enough for many of them. Volume still matters to a lot of people but categorize it — share of voice by topic, or share of desirable coverage, etc. Qualitative assessment of influence is important, not just how many fans or followers you have. Need to put all the metrics in context.
  • Best recommendation of the day thanks to K.C. from Cision: If you don’t have enough money to measure it, leave a blank in your report: “Awaiting funding.” State what’s missing, don’t wait for perfection and show that you have desire to do it right!

Posted in Business Outcomes, Media Analysis/Metrics, Search Analytics, Social Media ROI, Stakeholder Metrics, Web Analytics | Leave a comment

“Building a Business Case for Employee Engagement”

Posted on October 8, 2010 by admin

We’re kicking off day two here in sunny (!) Portsmouth with an in-depth case study on employee engagement from the Jack Felton Golden Ruler Award winner for 2010. Panelists include client Kevin Olp from Northwestern Mutual, agency leaders Patricia Bayerlein and Morgan Marzek from Gagen MacDonald, and Dr. Palmer Morrell-Samuels from Employee Motivation and Performance Assessment. Here are the highlights:

  • Northwestern Mutual culture – Long-term employees that appreciate stability, linear thinkers, don’t like surprises, family oriented, nurturing and caring environment (free lunch every day and “Mother Mutual” mindset), left-brain bias (actuaries and attorneys), fact-based decisions.
  • Northwestern Mutual had incredibly motivated sales force out in the field, working with clients, commission based – not as strong motivational factors for the people in the home office. Younger employees need to be engaged differently — may have different motivations from traditional Northwestern Mutual staff.
  • Many different definitions of “engagement.” Gagen MacDonald resonated with us because they talked about how employee engagement isn’t about “making people listen while they work” – it’s about real productivity.
  • Challenge: build a business case for employee engagement that will secure budget; must stand up to analytical scrutiny by demonstrating measurable impact on the business.
  • Outlined a research path: define scope and objectives; conduct research; connect to bigger picture; pilot engagement approach; codify and market internally; develop 2010 plan and budget.
  • Research overview: external literature review to identify other studies about behaviors and motivations that lead to good employee engagement; leader intake sessions to understand priorities and potential focus areas for the pilot; linkage analysis to evaluate current survey tools and business metrics.
  • Literature review highlights: identified proven drivers and proven outcomes of engagement that tied employee engagement to profit – reviewed 150+ surveys and reports to “stand on the shoulders” of existing research – and then needed to translate that to the specific corporate culture.
  • Leadership intake interviews with all Northwestern Mutual management about what’s important to them – don’t boil the ocean; outlined strengths and challenges in the organization; identified groups that were “ripe for change” to be part of the pilot.
  • Linkage analysis highlights: identified a proxy metric to focus pilot efforts on productivity; isolated behaviors from the employee survey — “Fact Based Leadership“; and then tested the link between Fact Based Leadership and productivity (percent of paid hours actually worked) – what’s the linkage between employee behaviors and profit.
  • Test 1: Demonstrated significant correlation between Fact Based Leadership and productivity with existing Northwestern Mutual data. Test 2: Demonstrated predictive value that Fact Based Leadership drove increase in paid hours worked. Test 3: Used exclusion tests to show that other factors (“open communication,” “knowledge of company strategy/goals” and others) don’t show the same correlation or predictive value. Test 4: Tested “dose dependence” that shows greater doses lead to greater responses.
  • Key factors of Fact Based Leadership based on existing research led to hypothesis for pilot = intellectual challenge; good information from leaders; feedback that is constructive, supportive and helpful; pragmatic metrics that are verifiably useful.
  • Pilot phase: Conducted four pilots in three work areas over a 13-week period to test the hypothesis. Control groups and experiment groups with “intervention” activities on Fact Based Leadership including kickoff training session, manager resource site, weekly study groups and 1:1 coaching. Training was for the leaders — and the measurement was done with the employees in those groups.
  • Learned that each group in pilot study had varying levels of business metrics and goals for employees. Led to development of metrics confidence model (from level 1-5): Survey measures, productivity measures, quality measures, financial contribution measures, strategy map measures.
  • Pilot results: During 13-week period, control group scores dropped 1.4 points in Employee Motivation Index; experiment group scores increased 2.4 points. The difference of 3.8%, while small, is highly statistically significant and quite promising given the short duration of the pilot study.
  • Surprise: Executives who saw the pilot results wanted to know as much about the 1.4% decrease as they did about the 2.4% decrease. “Why’s that going down?!”
  • In each group, we had some unpredictable results: In one group, cases closed per employee per month increased dramatically for experiment group and decreased significantly for control group.
  • Bottom line: Proved through surveys and control groups that employee motivation and engagement drove productivity increases.

Posted in Business Outcomes, Stakeholder Metrics, Survey Research | Leave a comment

“Influence: What It Is & How to Measure It”

Posted on October 7, 2010 by admin

Today’s closing session is all about influence — one of the most interesting areas of discussion in social media measurement, and equally important to traditional media and public relations. Here are some of the highlights, thanks to moderator David Geddes from Evolve24 and panelists Anne Fenice from Yahoo!, Jason Forget from GE Energy, and Zach Hofer-Shall from Forrester Research:

  • “Lot of talking” in the industry about influence, but “not a lot of thinking.”
  • Quick search over the past few weeks resulted in 6.5 million references to “social media” and “influence.” Not a new topic of study, but today’s market discussions don’t reference the historical science in the field — including, for example, “the two step model” of communications developed by Paul Lazarsfeld.
  • Is it the influencer or is it the audience? Interesting academic studies about which matters more, including Duncan Watts at Yahoo and Bernardo Huberman at HP Labs.
  • GE Energy example: Marketing team would say customers are most influential; communications team would add regulators, policy makers, energy-related blogs, greentech media and other subject-matter experts — plus employees.
  • Yahoo example: “Influencing” is creating active supporters of our brand, and “influencers” are those active supporters. More than 660M customers plus all the employees. Social media is decentralized, so influencers vary by organization, topic and program goals. If it’s about raising exposure, number of followers matters. If it’s about increasing engagement on a topic, different metrics come into play.
  • Forrester: Every company I’ve talked to has a different way to do it, and every vendor has a different way to calculate it. Many different uses — marketing, customer support, public relations, etc. Tracking number of Twitter followers is “pretty ridiculous.” Need to look at both online and offline influence.
  • Has influence changed over time? Forrester says influence is driven by the same practices, now just applied to new channels. Yahoo believes things have changed – no longer “pushing information to people.” Customers like to engage and share. GE Energy: “Influence is influence — it doesn’t matter what the channel is.”
  • GE Energy: “We have a terrible propensity to over-react to social media conversations. Don’t always see the forest for the trees.”
  • What’s the shelf life of an influencer list? It’s a “revolving door” or “dynamic concept” that changes from situation to situation. May be a little more stable in B2B than consumer.
  • How do you manage it? Forrester: “Because we have all this new social media data, a lot is possible. That doesn’t mean it’s practical.” GE Energy: “Stake out your story” and then engage whomever you need to engage, even if it changes over time.
  • Forrester study breaks down different engagement patterns for people in social media. “Creators” are not necessarily the most influential — even ”spectators” have influence because they talk to people offline. New report coming out breaking down the influence level for each engagement patterns. There’s a “long tail of influence” — much more than a list of the top 10 people.
  • Influence derives from the creator, the curator, the content, the audience, etc. Yahoo views all its communicators as “content officers.” Just because you create content, though, doesn’t mean you have influence in that arena. It depends a lot on frequency and quality of content.
  • Need to “step outside your list” to find other influencers. Conduct stakeholder analysis, reach out to new people, ask customers, listen and learn.
  • Forrester: The more you segment, the more the list will be stable — but the more complex it will be. Segmentation is done different ways — often based on tiering, sometimes based on personas. Same set of data can be used for research, pitching and measurement stages.
  • Future development: How can you predict the influencers that haven’t emerged yet? What are the “risk factors” or “emotional triggers” that could create the next firestorm?
  • Forrester’s best practices: The more data you can combine, the better. Pull from social media monitoring, web analytics, CRM, offline data, etc. Start with an ideal, scale back to what you have access to, and then build up to the ideal over time.
  • Final thoughts: Learn before you interact. Hire someone to track and measure influence. It takes time — and you have to find the time. Refresh your lists often.

Now we’re off to lobster at Katie’s house…

Posted in Business Outcomes, Media Analysis/Metrics, Social Media ROI, Stakeholder Metrics, Web Analytics | Leave a comment

State Department: “Global Digital Communications Measurement: Getting It Right”

Posted on October 7, 2010 by admin

Up second this morning is a valuable case study (in progress) from a noteworthy and rare-to-share source: U.S. Department of State. Thanks to Cherreka Montgomery from the State Department and both Xiaoyan Zhao and Holly Heline Jarrell from GfK for sharing these highlights from their “Electronic Media Engagement” project:

  • Goals of the “ambitious” project: evaluate the department’s Web 2.0 activities to communicate U.S. policy messages, culture, society and values — while tracking knowledge, behaviors and attitudes among foreign audiences (eight countries).
  • Program reflects all of the Barcelona Principles, especially #1 “Importance of Goal Setting and Measurement” and #6 “Social Media Can and Should be Measured.”
  • Need to understand the audience in order to set goals. Can measure a lot online, but many of the tools miss the parameters — need traditional research to help.
  • Measuring social media globally is “daunting” and complicated — languages, cultures and different communities are important for real context.
  • State Department has built a “logic model” for performance measurement of public diplomacy. It’s used to integrate strategic planning, metrics and tactics, and also to teach people internally about the linkage between the Department’s mission and public diplomacy resources, programs and results.
  • Team has 100 different performance measures — 75 of which are output oriented. Long-term outcome measure is reducing the level of anti-American sentiment among key foreign audiences. Especially focused on emerging audiences, not just traditional elites.
  • Starting from scratch — there are no domestic policy models that can be applied, need to invent new models for public diplomacy.
  • Research design is focused on tracking number of conversations or interactions, and what impact “interventions” via digital campaigns have over time. Can we influence behavior? How strong is the effect? How long does it last?
  • Conducting market research (focus groups, structured interviews, infrastructure analysis, web conversation tracking and network analysis) to understand audiences, user demographics, online behaviors, key influencers, online platform usage, opinion trends and message reach across eight targeted countries in nine languages. Data is used to fuel communications planning, measurement, resource planning and needs assessment.
  • In-depth effort that integrates four research partners with different skills: GfK, Morningside Analytics, Crimson Hexagon and Office Remedies Inc.
  • Program assumption: Constant, repeat engagement in real time will drive positive change in foreign communities. Depends on us being credible, trustworthy.
  • Snapshot of insights from Indonesia research: Baseline favorability is low — U.S. recognized as best partner in global affairs, but falls off the preferred list in all the other areas surveyed. Some attitudes are contradictory and potentially fluid. Local Internet users are talking about social issues via social networks, and the Internet reaches well beyond the elite group.
  • Indonesia research led to three segmented audiences for targeting: “humanists” with diverse interests and high civic influence; “civic-minded” people who are community-oriented, connected and active; and “light-hearted” people who are pop-culture immersed and web 2.0 savvy.
  • Topics that have the highest impact on U.S. favorability within Indonesia: governance, environmental record, and diverse and open society. That insight translated to three engagement themes: democratic development, environment and education. Further research is then done within each theme to evaluate online conversations. In education, for example, the #1 online discussion driver is sharing personal experiences, and #2 is educational opportunity.
  • Indonesia research is impacting strategy for key State Department platforms: Embassy web site, Twitter posts, blogger engagement, Facebook fan page (largest U.S. Embassy fan page in world), etc.
  • Interesting fact: Indonesia has the world’s largest Facebook user population. You won’t find that stat in “The Social Network” – just here!
  • For those clients who haven’t fully invested in communications measurement, take note: The State Department has a formal “evaluation and measurement unit” as part of the Office of Policy, Planning and Resources for Public Diplomacy and Public Affairs.

Posted in Barcelona Principles, Media Analysis/Metrics, Reputation Management, Social Media ROI, Stakeholder Metrics, Survey Research, Web Analytics | Leave a comment

GM: “Building Relationships One Car at a Time”

Posted on October 7, 2010 by admin

We just kicked off the official proceedings with an inspiring case study from GM. Plus for those in Portsmouth, we’ll have an opportunity to check out GM’s latest cars on the way to tonight’s Lobster Fest (the benefits of being in person!). For those who aren’t, here’s a recap of this morning’s highlights, thanks to Jackie Matthews from GM, Dan McGinn from TMG Strategies and Rex Repass of RL Repass & Partners:

  • GM has deep research experience looking at all aspects of communications. Not just opinion research — mixed with media analysis, case studies, behavioral analysis, census and economic data, cultural studies. Insights come from many sources. “There’s not one silver bullet for research.”
  • Example: 16 year olds don’t care about cars — they care about their phone — cars used to be about freedom and design, but freedom isn’t “new” for today’s 16 year olds.
  • Opinions are shaped largely outside the news — you need “deep listening.”
  • Five key lessons from GM and other research: 1) Trust in institutions obliterated; 2) Most dangerous position is biggest/best/first/most; 3) Loyalty shifts are accelerated and magnified; 4) The new consumer test for institutions — are you “partner” or “predator”; 5) Consumers want respect in three dimensions — personal, time and money.
  • Before the storm about GM: 67% of consumers have favorable view of GM; imports capture 48% of U.S. market share; 66% of 18-34 year olds enthused about GM Volt technology (about an idea — more than a car).
  • Research insights about non-GM customer challenge (young professional women): stigma associated with U.S. brands; aversion to visiting all dealerships; perception that American cars are not a “smart buy”; reluctance to consider specific models.
  • Insights about non-GM customer opportunity (young professional women): Would look/test outside of dealership; story of quality and design comeback impactful; open to package of benefits (“total confidence”); interested in perspective of peers.
  • Research insights fueled a “cars-to-people grassroots” effort. Conducted focus groups, in-depth interviews and surveys. Tested 70 concepts and narrowed to 5 categories. Cross-checked concepts and reactions with dealers and key staff. Focused on “modern family” audience.
  • GM tested in-person exposure to Chevy vehicles, with pre/post surveys — 30% more impressed after experience with cars. Even stronger among recent GM buyers.
  • Grassroots demonstrations to introduce new vehicles — live car experience, partnered with local organizations, interns in t-shirts created “safe” no-pressure environment.
  • GM’s grassroot program principles: Fun, hassle-free, convenient, relevant. Hyper-local initiatives — including “drive through finals” to get food for busy college students. Extended vehicle loans to influencers (all kinds) with no strings attached. Pre/post surveys showed brand impression boost by 48 points and purchase intention up 33 points.
  • Broader impact than grassroots results: GM favorability ratings went from 18% Jun’09 to 37% May’10, and from 47% negatives in Jun’09 to 27% May’10 one year after restructuring. Surveys showed 48% “great deal or fair amount of progress” — they’re willing to give GM a shot. Increased exposure and consideration, sales TBD.
  • Important to remember: You need to have the product, not just the communication. Chevy Equinox is GM’s best product launch — can’t make enough of them, driven largely by word of mouth. Re-launched Camaro, which “day one outsold the Mustang.” GM was confident it had first-class products — just needed to re-engage consumers.
  • Another important success factor: PR joined at the hip with advertising and marketing teams from the start, grounded in smart research. “We have scads of data” and are now listening to it more closely — using research to drive our business.

Posted in Business Outcomes, Media Analysis/Metrics, Reputation Management, Social Media ROI, Stakeholder Metrics, Survey Research | 1 Comment