Background and Methodology | Key Findings | Recommendations

This Executive Summary provides an overview of an impact evaluation conducted on behalf of Alamosa PCS by the Learning Alliance. A full report is available for those interested.

Background and Methodology
The Learning Alliance’s Success Case Evaluation method is based on the assumption that training and selection tool impact is always variable, and depends in large part not only on how much was learned, but how learning was subsequently put to use in manager-related applications on the job. Further, impact of training and selection tools is always heavily dependent on critical factors in the work environment of learners; some of these factors will encourage application and impact, and others will inhibit application and reduce impact. The evaluation is structured to identify the very best results that Audition, Conductor, and the Chally tools are accomplishing (and why), so that learnings from successes can be identified and leveraged for greater impact in future initiatives.

^ top

Key Findings
  1. Significant, positive business impact has been generated.
    2. 146 of the 173 survey respondents (84%) reported that these initiatives had either a “Very High” or a “High” level of business impact. Specific business impact was found in the following areas:

    • More than doubling of sales volume in an Indirect Alamosa PCS account
    • Reductions by more than 1/2 the turnover of staff in a retail outlet
    • More effective hires who perform more quickly and with better fit for Alamosa
    • Increased sales results due to a current employee being more effectively coached
    • High performing employees retained

    The estimated ROI of these initiatives is 3.6 to 1 ($810,000/$225,000). In all cases, the investigators have used conservative figures so as not to over inflate the ROI figure.

  2. Significant business impact is being “left on the table.”
    3. 24 individuals of 173 participants (only 13% of the survey respondents) reported “very low” or “low” business impact as a result of their participation. It can be assumed that these participants are producing little or no business impact. The Recommendations section of this report contains methods for correcting a large portion of this “low” impact finding.

  3. Immediate application and manager support were the two key factors most closely linked to impact.
    4. When correlating the relationship between manager involvement and use of the impact maps and positive business outcomes, 93% of the 30 “very high” impact respondents indicated that the impact mapping and manager discussion was “helpful” or “very helpful.” Conversely, 90% of the “low” or “very low” impact respondents indicated that the impact maps and manager conversation were only “somewhat,” “little”, or “no” help

  4. Audition and Conductor has powerfully changed the candidate interviewing process
    5. All managers interviewed, who had hired someone since their participation in these processes, indicated that they had dramatically changed their interviewing questions and processes based on what they learned in these courses.

  5. Conductor skills are having a significantly positive impact on both new and current Alamosa staff
    6. Most managers, including those without staff turnover issues, are using the Conductor skills to motivate and more effectively manage their current staff. Managers could easily cite how they have taken time to analyze the differences in people, personalities, and styles, and have adjusted their leadership and communication to those differences. Many managers reported more satisfied, enthusiastic, and committed current staff, with fewer internal conflicts, and greater teamwork

  6. Chally selection tools have strong believers and some skeptics.
    7. The investigators believe that the following facts are operating in regard to the Chally tools:

    1. Since Alamosa managers have been hiring for a very long time, primarily on intuition, it will take time for them to adjust to a more scientific approach.
    2. It will be nearly impossible for Alamosa to find universal acceptance of the tools through the “getting burned” process.
    3. A perceived corporate mandate forbidding the hiring of a “not recommended” candidate may seem overly harsh to managers trying to make a transition to the Chally tools.

  7. Conclusive evidence showing that these initiatives are having a long-term, positive impact on retention is not yet available, but during this initial post-training period, retention improved 10%.
    8. It appears to the investigators that a number of intermediate milestones on the way to accomplishing that goal are in place (e.g., better behavioral interviewing, open ended questions, the use of the Chally tools, the use of Conductor skills with new and current staff, etc.) Long-term retention data will confirm this trend.
^ top

Recommendations
Continue using the Audition and Conductor learning and Chally tools with the following suggestions for increasing impact:
  1. Provide follow up support, reinforcement training, and encouragement for use of the tools. This might include:

    • Have managers, who have successfully used the learning and tools to produce positive business results, share their experience in meetings of other managers, either when meeting virtually or in person.
    • Share success case stories in an internal company newsletter.
    • Provide a coaching tips tool kit for managers to use in coaching follow up with those who have completed the learning and tool use training
    • Think about offering a refresher resources/course that includes case studies, best practices and more skill practice.
    • Develop a short job aid/refresher that gets sent to every hiring manager as soon as corporate is notified of a staff member change that requires a hire.

  2. Continue using the impact maps and manager/trainee discussion process with any learning initiative in the company.
    3. There is overwhelming evidence that this process contributed significantly to the positive outcomes Alamosa has and will receive from these initiatives. Given that this was the first time such a process was used and, therefore, new to both manager and trainee, the impact of this effort will only increase as both manager and trainee come to expect it as the way training works at Alamosa

  3. Clarify the role of the Chally tool in the candidate selection process.
    4. The investigators believe that it is appropriate for Alamosa to take a very strong position regarding the permission to hire a “not recommended” candidate from the Chally assessment. However, until managers are convinced that it is the right tool to use, it might be wise for Alamosa to create a way, a very cumbersome way for sure, for managers to appeal the Chally findings if they fully believe that the “not recommended” candidate is still the best fit.

  4. Long-term retention data should be carefully monitored.
    5. Given that the data in this report reflects only a three to nine month corporate experience of hiring while using the Audition, Conductor, and Chally resources, we strongly recommend that retention data be tracked very carefully over the next twelve to twenty-four months.

  5. Consider offering similar initiatives in a more “just in time” fashion if possible.
    6. Alamosa trained some managers in a process that they have not used some four to six months after the training and hope to not use the tools any time soon. The simple reason – they have no turnover in their operation.

    The investigators believe this current situation has simple remedies and therefore the overall potential impact from this investment can still be fully realized.

^ top



 Click to Download PDF Datasheet
[requires Adobe Acrobat Reader]