Session 4: Learning Analytics Dashboards: Design, Implement and Maintain Dashboards

Jeffery Berk, Vice President of Products and Strategy

www.knowledgeadvisors.com

This session is about implementing a “dashboard for learning and development. I attended this session to learn ways to measure and analyze data about BHU.

Like you, I came in wondering – what is a dashboard? Well, a dashboard is a convenient place for vital information organized into a graphical representation that is both easy to use and easy to understand.

Note: In my notes below L&D stands for Learning and Development. The equivalent of our Education and Training department.

Steps to Successful Dashboard Design:

1. Research learning metrics
   Metrics aren’t new. Go to other sources to find more information. Some suggestions:
   
   We can join the Learning Development Roundable. They’ve researched what other companies have added to their dashboards.
   
   ROI Institute – helps organizations helps people understand their companies’ ROI (return on investment)
   
   ASTD – ASTD has some great resources and chatrooms.
   
   
2. Identify macro learning contructs
   
   What is a macro construct? a small well-balanced set of broad learning metric classifications that summarize that results of the entire L&D organizations. A mile wide and an inch deep.
   
   (a) Operational: Activity contructs (how much we train)
   e.g. number of courses, number of instructors, instructor utilization, fill rates (more in micro)
   
   (b) Performance: optimization constructs (how well we train.
   e.g. evaluation scores, instructor performance, satisfaction levels (more in micro)
   
   (c) Financial: budge/fiscual constructs (how much it costs)
   e.g. L&D costs as a percent of payroll, productivity, revenue growth (more in micro)
   
   (d) Cultural: supportive environment (how conducive is it)
   e.g. how supportive is the environment of learning? (more in micro)

3. Build micro learning indicators
   What is a mico construct? A set of quantifiable measures linked to the macro learning construct that are tracked over time. An inch wide, a mile deep.
   
   (a) Operational
   # of students trained
   instructor utilization rate
   e-learning utilization rate
   average class size
   speed to market (speed to need)
   staff to management ration (training:staff ratio)
   delivery mix
   survey response rates
   class completion or cancellation rates
   
   MY COMMENT: I don’t think we have much feedback for our training practices aside from the leadership conference/RM conference sessions. Can we require surveys for our online offerings?
   
   (b) Financial
   cost per student day
   L&D costs as a % of payroll
   L&D budeget to actual
   L&D investment mix
   Revenue growth
   Human Capital Contribution Margin
   L&D ROI
   Productivity per employee (also can be a performance measure)
   
   (c) Performance
   Level 1 satisfaction socres, instructor performance, courseware quality
   Leverage effectiveness (test scores, passing scores, etc)
   Time to Job Impact (how quickly can employees apply learning? How soon does it make a difference in their work?)
   Business Result Linkage (sales, quality, cycle time, productivity, customer satisfaction, cost, savings, etc. – this must be estimated, isolated to training and adjusted for confidence)
   ROI (if not in financial construct)
   
   MY COMMENT: Do we look at financial and performance metrics in relation to our training?
   
   (d) Cultural
   Training eligibility
   Average hours of training per employee
   Available Tuition Reimbursement
   Management Support (have to build collaborative relationship with management)
   Internal validation – write up success stories and share them. Quotes from highly people in the organization
   External validation – apply to organizations like ASTD, CLO magazine, etc.
   
   MY COMMENT: We’re building job aides to get management support and buy in. Are we measuring that? How do we know that it’s happening? How do we follow up?

4. Build a process to collect and report
   
   Start with available data
   Verify that the data are credible
   Create templates to store and track (must be available!!)
   Conduct routine analysis (monthly)
   Report key findings (quarterly)
   
   Remember, the data does not have to be perfect. Just adjust for it.
   
   Collect data at every opportunity that you have – because you’re never sure when you’re going to get the chance again.
   
   
5. Design technology and templates for support
   Must have a scalable process!
   
   Need technology to manage all of this data. Metrics can be added manually or via feed from other systems (HRIS, LMS, financial)
   
   This can be done in Excel or Web tools offered from Microsoft.
   
   Technology considerations: (Last thing to think about – not the first!)
   - can we link it to feeder systems (LMS, HRIS, Access, etc)
   - understand frequency of use v. cost of integrations
   - check for system ownership and security issues (sometimes other departments aren’t willing to relinquish information)
   
   MY COMMENTS: Berk offered to send a sample template/tool in excel that we can use to enter in this information.

I liked this statement Berk made: METRICS AREN’T THE ANSWER. THEY’RE THE THING THAT DISTURBS YOU TO ASK MORE QUESTIONS TO GET TO THE ANSWER.

MY COMMENT: Obviously, we can’t do all of this (unless we hire a a couple more people…). But we certainly can take in more data to make more informed decisions. Berk offered to send tempates and samples of surveys to anyone who emails him. Once I get the materials back from him, I’ll pass it on to whoever is interested.