Community of Practice Evaluation following Work, Connect, and Learn


This post is part of a case study on the development of a Community of Practice (COP) for Maintenance and Engineering teams at Coca-Cola Amatil.  A previous post outlined the COP evaluation strategy.  This post summarises evaluation following completion of the five-week Work, Connect, and Learn (WCL) program.

Network value creation

Network value creation

This evaluation examines:

  1. Increase in networks (potential value)
  2. Engagement with work and Community (potential and applied value)
  3. Opportunities for community value creation

Data gathering methods used were:

  • Pre and post program surveys sent to all 200 (approximately) Community members.  115 people responded to the pre-program survey and 78 to the post-program survey.
  • Data from monitoring Community SharePoint site

Community Demographics

Community members are from nine operational sites and two head office locations in Australia and New Zealand.  Job role and age distribution are shown in the tables below.  The geographic and age distribution of respondents was similar between the two surveys.  The percentage of trades-people who responded to the post-program survey declined compared to the pre-program survey.  This is consistent with feedback about barriers to entry for this group to take part in the online community.

  % Program Respondents
Job Role Pre-Program Post-Program
Tradesperson – Fitter (performs hands-on maintenance and repair of mechanical equipment) 27.4% 23.1%
Tradesperson – Electrician (performs hands-on maintenance and repair of electrical equipment) 24.8% 16.7%
Maintenance – Other (e.g. Coordinator, Planner, Manager – plan and manage maintenance tasks and resources) 17.7% 28.2%
Engineer (production line design, project manage changes to production equipment) 12.4% 19.2%
Other 17.6% 12.8%
  % Program Respondents
Age Group Pre-Program Post-Program
< 30 years 10.5% 10.3%
31-40 16.7% 20.5%
41-50 34.2% 37.2%
> 50 38.6% 32.1%

Increase in Networks

Completed Online Profile

By default, all employees have a brief personal profile in SharePoint and contact details in Lync (now Skype For Business).  We also set up a contact directory on the Community site, organised by work location and job role.  People were asked to update their profile with details such as experience, past projects, and interests.  Profiles are included in SharePoint search results, so these details make it easier to find and connect with relevant people.  As an entry level networking activity, updating a profile is an important step in community participation.

31% of respondents updated their SharePoint profile during the program.  This increased members with complete profiles to 40%, against a target of 80%.

Interaction with People at other Locations

Unfortunately, we are unable to gather any network analysis data from SharePoint or Lync.  We asked about the interaction between Community members in different locations using SharePoint and Lync.   We compared the number of people respondents interacted with in the four weeks before each survey.  WCL webinars were excluded from the data.

The graph below shows two key shifts:

  • Approximately 20% increase from no interactions to 1-5 interactions
  • Approximately 6% increase from 6-10 interactions to 11-20 interactions

Maint cop interaction

We asked respondents to list up to five people they had interacted with at other sites in the previous two weeks.  However, we lacked an effective tool or method to analyse this data.

Interaction across sites increased during the WCL program.  Sustaining and building interaction would require effort.

Community Engagement

Site Newsfeed

The Maintenance and Engineering Community used an existing SharePoint site.  General updates and transient chat could be posted on the newsfeed.  However, the newsfeed was rarely used before WCL.

An early WCL activity was for everyone to follow the SharePoint site.  Following a site ensures that site newsfeed posts appear in your personal newsfeed.  SharePoint does not ‘push’ notifications of newsfeed activity outside of the newsfeed itself.  This means that the only way a person will be aware of newsfeed posts is if they check their feed.  The graph below shows how often respondents checked their feed.  The number of respondents who never check their feed dropped from 62% to 27%.  Those checking at least once a week rose from 18% to 48%.  There was a slight increase in the people who check their feed daily from 9% to 13%.

Maint cop feed check

Discussion Forum

Two discussion forums were added to the site: one for the WCL program, and a second for ongoing Community use.  During WCL, we gradually moved activities from the program forum to the Community forum.  We encouraged people to use the Community forum to share knowledge, solve problems and collaborate on improvements.

Forum posts do not appear in the SharePoint newsfeed.  An alert can be set up on a forum to receive email updates of activity either immediately, daily or weekly.  WCL participants were shown how to set up an alert and asked to set one up on the forum.  At the end of the program only 30 people (approximately 14% of the group) had set up an alert.  However, only 30% advised that they ‘never’ check the forum.  This indicates that most are visiting the forum without being prompted by email alerts.

Activity on the Community forum was analysed.  The count excluded activity on the WCL program forum and by program facilitators.We counted the number of questions, likes and replies, and the number of active individuals.  There were 115 interactions from 23 individuals, representing 11% of the Community population.

Participation rates are consistent with the 1-9-90 rule which is a positive start.  A small number of community champions are emerging.

Barriers to Community Engagement

The survey listed a set of activities and asked respondents who had not done at least two why they had not been more active.  The table below shows frequency of different responses.

MaintCOP Barriers

Respondents identified the key barriers to community engagement as:

  • Time – finding the time to do activities
  • Skills – not being sure how to use SharePoint and/or Lync
  • Need – not having identified a need to engage
  • Technology – Inadequate access to computer or mobile device
  • Who would be interested? – Uncertainty about what they can contribute and who would be interested in their contribution

Opportunities for Community Value Creation

Two open-ended questions gathered views on how participation in the Community could add value.  The questions focused on improving business results.  The were also phrased so that the answers reflected personal pain points and opportunities.

Q1: What do you see as the biggest opportunity to improve work practices and maintenance results at your site?

Key themes in responses were:

  • Access to information
  • Communication and collaboration between production plants
  • Relationship and collaboration across departments at a local level
  • Improving troubleshooting and speed to resolve equipment faults
  • Time / workload, improving workflow
  • Standard processes, setting standards, accountability
  • Maintenance planning
  • Improving technical knowledge
  • Innovation

Q2: How do you think the Community of Practice could help you with this opportunity?

Key themes in responses were:

  • Drawing on everyone’s experience
  • Allowing information to be shared
  • Ease of communication with others
  • Having a greater number of people to ‘bounce’ ideas, solutions and improvements
  • Use forums to ask questions and access feedback/experiences from other sites
  • Learning from mistakes and successes of others
  • Not reinventing the wheel
  • Alignment to common goals through interaction in new ways
  • Training on technical skills

I am actually writing this post seven months after the WCL program.  This gives me the benefit of knowing what has happened in the intervening period.  I recall being positive immediately following the WCL program. The WCL program had helped us to launch the Maintenance and Engineering Community of Practice.  Participants understood how a Community of Practice could create value. The interaction between people in different locations had increased, and community engagement was growing.  We could build on this with strong community facilitation.  We had some barriers to address, particularly if we wanted to enable the trades-people to take part.  There were also opportunities.  Community Champions were emerging.  The Community had identified improvement opportunities that we could build activity around.

The National Engineering and Maintenance Managers had a deeper understanding of tacit knowledge.  They had a stronger appreciation of the value of networks and potential contribution of a Community of Practice.  Our next step was to support them to develop a strong plan to build and sustain the community.  We engaged Helen Blunden  of Activate Learning Solutions to provide coaching on Community facilitation.

,

  1. #1 by Bruno Winck on November 16, 2015 - 9:08 pm

    It’s interesting because you take the time to go into every details and background information.

    I understand the challenges you faced to collect data and arrange the experience as you wished given the technical constraints. That would be another debate.

    What I’m not able to understand is the role of tradepeople. From my experience in bottling lines engineering and maintenance, I don’t where they fit unless you are selling them. This makes me feel I missed something in the profile of users. I though they were mostly maintenance agents.

    Maybe a few lines on the job profiles of each group would help to make sense?

    • #2 by Michelle Ockers on November 17, 2015 - 6:16 am

      Thanks for the feedback Bruno. I added a short description of each job role in the table. The standard enterprise SharePoint installation in my organisation makes it difficult to gather any but the most basic data, and as an site owner I am unable to install any add-ins to assist with this. It appears that we will expand our use of Communities of Practice in the coming year, and our IT department is introducing new templates for some community interactions – so there may be an opportunity to improve on this.

    • #3 by Bruno Winck on November 17, 2015 - 9:00 am

      Got it tradeperson is what we call a technician. Someone good in his craft, with some formal education around 3 years and experience in his technical domain.

      I’m surprised that videos on the point of work don’t seem to be a favorite solution for communication.

      Oh, just realize that just precisely shared one https://t.co/tefbUQScbp The last guy mentions a schedule with videos so something but in place.

      Good work Michelle!

(will not be published)