Demonstrating Impact From Training Highly Qualified People In Research

Daniel Banks, President, TVB Associates Inc.

How can we show impact from training Highly Qualified People (HQP) in research programs beyond
counting numbers of graduates or recounting anecdotes of successful alumni?


Demonstrating socioeconomic impact from the training HQP in research has always been difficult,
because of challenges associated with tracking graduates and following their subsequent educational
and professional careers over time.


The emergence of career-oriented social networking, however, has provided valuable tools that can be
used for this purpose. The value of any social network depends greatly on its number of users. The
biggest career-oriented network, LinkedIn, has seen a surge in usage since Microsoft took it over in
2016, and its user base is now over 600 million — about 20% of the estimated 3 billion people working
in 70 million companies around the world.1


Many HQP provide their career information on LinkedIn
The beauty of LinkedIn for tracking HQP is that individuals openly volunteer career information that
would otherwise be confidential and very difficult to get. A high proportion of HQP in North America
have LinkedIn profiles or can be otherwise identified online, and this includes professionals of all ages.
For example, in 2018, I performed internet searches for former students and post-docs who used the
now-closed Canadian Neutron Beam Centre (CNBC) for research as part of their graduate or
undergraduate programs at Canadian universities, going back as far 1984. I found 75% of these alumni
online, and nearly 60% on LinkedIn. Furthermore, 44% of LinkedIn users are women, which is similar to
the proportion of women in the workforce overall, suggesting there is little gender-bias in the data, at
least at a very high level (however, there can be difficulty in identifying individuals, often women, who
have changed their surname).


LinkedIn data reveals where alumni are working now
Some of the simplest results to obtain are the institutions where alumni are working now. For the study
for the CNBC, for example, showed that almost 80% of the alumni were working in the sectors that
contribute most directly to Canadian innovation: manufacturing, higher education, and professional and
technical services. Furthermore, a higher proportion of CNBC alumni with PhDs were working in industry
(65%) over academia, as compared to the average for natural sciences PhDs in Canada (51% half stay in
academia, according to StatsCan data).


LinkedIn data reveals individual educational and professional paths
LinkedIn data is especially useful for observing alumni’s educational and professional paths over time,
because most users treat their profiles like an online resume, listing their record of degrees and
professional positions. Such longitudinal data was essential to obtaining valuable insights in the study
for the CNBC, such as:

1. Participation in research at the CNBC as an undergraduate student was a strong predictor of
earning a graduate degree: Of the undergraduate students who came to the CNBC for a
research project, 60% went on to achieve a graduate degree. In fact, most of these alumni went
beyond a single Master’s degree: 40% of the undergraduate students later achieved a PhD, and
another 14% earned two Master’s degrees. These rates of academic achievement are far higher
than is typical for Canada as a whole: According to StatsCan data, only 44% of all undergraduate
students in Canada who are surveyed upon graduation stated intention to pursue further
education of any kind. The percentage of students who attain higher degrees is, of course, much
lower than those who intended to do so.

2. Participants in research at the CNBC have enjoyed subsequent career progression: The
LinkedIn data showed that alumni with greater years of experience tended to fill more senior
positions, while more recently graduated alumni have a greater share of non-supervisory
positions.

In the case of the CNBC study, a sample of alumni were contacted via LinkedIn and interviewed. Alumni
interviewed attributed their experience of doing research at the CNBC with motivating them to pursue
research and development or related technical careers in industry and with helping them develop skills
that have helped them in their careers. While not scientifically conclusive, the interview results provide
evidence for interpreting some causation in the above observations.


Why aren’t more institutions using LinkedIn data to demonstrate impact from training HQP?
Despite the potential that LinkedIn data holds, I have seen few studies that seek to use the data to full
advantage. A notable exception is the 10,000 PhDs project,2 in which the University of Toronto made a
significant investment of effort to identify 88% of its PhD graduates from 2000-2015 online. Many of
these alumni were found on LinkedIn. The U of T study analyzed their first and current employment
statuses. That study provided valuable insights into employment prospects after earning a PhD, and how
that employment differs across fields of study.


Perhaps LinkedIn data has not been used to its full potential because several years ago, its utility for
such studies was not as great due to lower usage levels, and one could have reasonably questioned its
long-term viability as a platform. But the activity on LinkedIn has greatly increased in recent years and it
must now be taken seriously.


Other issues could relate to interpreting the data, data privacy, or the labor required to gather and
analyze the data. These issues are discussed next.

Benchmarking to aid data interpretation
A typical challenge in demonstrating impact from training HQP is a lack of reference points to know if
the results are excellent or below average. If one believes the results will be ambiguous, then there is
less motivation to pursue the analysis.


The key to resolving the ambiguity is to determine appropriate benchmarks and build them into the
study. Sometimes the data can be compared with insights from other sources, such as StatsCan as I have
done in some of the above examples, to assist with the interpretation. Another option is to conduct the
same analyses on random samples of comparator groups (e.g. students who were not involved in
research, or were from other institutions distributed across Canada). A comparator group would be
useful to interpret the above data on CNBC alumni career progression, for example.


Data privacy
Although LinkedIn users volunteer their career information online, there are still data privacy issues to
be considered in collecting and storing the information. LinkedIn users retain the right to remove their
data from the site. Systematic duplication of their data by third parties increases the possibility of
leakage, which in turn undermines their control over their data.


The U of T study reported that student researchers who conducted the online searches were trained on
confidentiality. They entered the data they found into secure servers, at which point they no longer had
access to the data. None of the data was stored on personal computers at any time.


With reasonable precautions such as these, data privacy issues need not be a barrier to using the
LinkedIn data.


Labor to gather and analyse the data
The labor to gather and analyze the data is perhaps the biggest barrier to using LinkedIn data to its full
potential. Few professionals at institutions have time to find large samples of alumni and manually input
data from websites into a spreadsheet or database. Longitudinal analyses and benchmarking multiply
the amount of data to be found and processed.


The U of T study overcame the labor barrier by using inexpensive part-time student researchers. It
reported a $50,000 budget for a team to do the searches and data entry over an 8-month period. The
value of staff time to perform subsequent analysis on the data and publish the results can be assumed
to be in addition to this budget.


There are also smart ways of automating much of the searching, data entry and data analysis. For the
CNBC study, I was fortunate to partner with a consulting firm that had a knack for writing scripts for
these purposes. These scripts were key to obtaining results at a reasonable cost.
Furthermore, narrowing the scope of the study to the HQP trained by one or more strategic research
facilities at a university can be useful to reduce costs compared to examining the HQP trained by an
entire university.

Conclusion and questions for further discussion
The usefulness of LinkedIn as a source of data for demonstrating impact from training HQP has greatly
increased in recent years. Research institutions are just beginning realize its potential.


Are you thinking about how to show the value of training HQP in research? What kinds of messages
would you like to be able to communicate to governments and research granting agencies, but don’t yet
have the evidence to support them? Are you gathering evidence of impact from a major research
facility to support its upcoming funding renewal?


Have you been involved in studies using LinkedIn or using general online searches to find and your
research alumni? What lessons have you learned? What are your current practices to benchmark the
data, respect data privacy, or manage costs?


For further discussion, you can reach me at: http://tvbassociates.ca/#contact

References:

1 https://foundationinc.co/lab/b2b-marketing-linkedin-stats/

2 https://doi.org/10.1371/journal.pone.0209898

Blog Archives

December 16, 2019
September 30, 2019
August 06, 2019
July 15, 2019
July 02, 2019
February 11, 2019
February 05, 2019
August 31, 2018
May 15, 2018
October 23, 2017
August 22, 2017
August 14, 2017
July 31, 2017
July 17, 2017
February 20, 2017
February 06, 2017
January 23, 2017