Public sector across levels of governance is under intense pressure to provide better services with the least resources possible. It is essential to optimize the usefulness of available resources in order to achieve the best possible results. Non-profits likewise need to utilize the available tools and human resources to the maximum, since they often have to depend on external unreliable sources of funding and hence must guard against the possibility of shortage of capital and other resources.
Policy analysts and public managers, therefore, are also under the scanner for their ability to harness the resources available to the maximum. They ought to be able to decide as to which programs should be implemented and given priority over other programs with respect to the allocation of funds, time, effort and other resources. Hence, they need to be able to prioritize what services will be offered and evaluate whether the programs designed to provide such services have been effective and efficient.
The most commonly used interdisciplinary approaches and methods for evaluating policy impacts and program outcomes are cost-benefit analysis, randomized field experiments, quasi-experimental assessment, and participatory assessment. Thus we have a host of designs to choose from for the study.
The quasi-experimental design is of many types which includes the following:
1. One-Group Posttest-Only Design:
It is extremely simple and rarely found being used in social science research.
2. One-Group Posttest-Only Design with Multiple Substantive Posttests:
It allows pattern matching, as well as multiple, unique, and substantive posttests
3. Pretest-Posttest One-Group Design:
It can be implemented either the same units or different units receiving both pretest and posttest. It however suffers from single group threat.
4. Nonequivalent-Groups Design (NEGD):
Since the distribution is not random, there is a risk of selection bias. It is a method that can handle the discrepancy of internal group threats well. It incorporates the concept of a control group to eliminate internal validity problems.
A non-equivalent groups design (NEGD) was followed, based on a pre-test and post-test of both an experimental and a control group, as suggested by Trochim (2002). Only the experimental group was exposed to the intervention. The control group was included in the study to control the possible history-effect as a threat to the internal validity of the study. The non-equivalent groups design is a quasi-experimental design and is widely used in social research. It differs from a pure experimental design in that the groups are not randomly assigned (Trochim 2002).
In this study thirty-six people were included in the experimental group. They were chosen because they had already enrolled for the six-month training intervention that was to be evaluated. The participants were all between the ages of 18 and 25. Fourteen participants were female, and twenty-two were male. The experimental group members were not engaged in any formal secondary or tertiary studies at the time, although all of them had completed secondary school and some of them already had tertiary qualifications.
In selecting the control group, care had to be taken to select a group that would provide an adequate control. In order to ensure that the two groups could be validly compared in terms of previous exposure as well as motivation levels, the main criterion was that the control group, like the experimental group, should be in the same intermediary phase between school and work, and should also be actively involved in some way or another in preparing themselves for the workplace. Another criterion was that the control group should not have been exposed to the training intervention or elements thereof. For the reasons mentioned above, the control group comprised twenty second-year commerce students from the University of Pretoria, ranging in age from 20 to 22. Ten members were male and ten were female. They were all in the intermediary phase between school and work and they were all actively involved in preparing themselves for the workplace. The fact that they had already successfully completed their first year of studies, coupled with the fact that they (out of a class of more than fifty) decided to take part in the study, is a further positive indicator pertaining to motivation levels. During the six-month period that the experimental group participated in the training programme under review, the control group continued with their daily class routine. In terms of the control group not being exposed to the social constructivist training programme or elements thereof, it is significant that second-year students were chosen: the undergraduate commerce programme at the University of Pretoria is not at all structured in a social constructivist manner. Classes are very big (fifty plus); students do not form learning communities in small groups but attend their respective programmes individually; the programme is divided into separate subjects with a strong focus on conveying content and assessment is done according to traditional examination methods. Thus, although one can never create a true experimental situation where members of a control group are not exposed to any form of social constructivist learning whatsoever in their academic or personal lives, the research team felt confident that the control group experience was significantly removed from true social constructivism to provide an adequate control for the purposes of the study.
The Training Intervention
The training intervention was designed and presented by a private educational initiative consisting of team members from various fields such as psychology, education, theology and business. The intervention was a six-month full-time (8:00 -16:00) programme and was funded by private investors, as well as by the registration fees from the participants themselves. The programme aimed to develop individuals to be successful in the workplace of the future. The primary goal of the programme was to develop certain characteristics that could form the foundation for developing the participants’ ability to function in the workplace of the future. Although the characteristics they would require were seldom explicitly focused on, the programme design and training methodology aimed to develop these characteristics as an indirect consequence of the day-to-day training interventions. Some of the themes covered by the training interventions were life skills and personal mastery, entrepre-neurship and business, making sense out of history, news and politics, economics and statistical reasoning and environmental awareness.
The training methodology that was followed could be broadly divided into two main categories, namely facilitated group interventions and facilitated one-on-one interventions. While the group interventions allowed for the participants to explore and master the desired learning objectives collaboratively, the one-on-one sessions took place between a learner and a learning partner (coach) and allowed the learner to reflect on her/his progress in relation to her/his personal future goals and objectives. The learning partners and group facilitators were all experienced HRD practitioners qualified in the fields of psychology and education.
The group sessions made extensive use of activities that engaged the learners in a process of discovery and group reflection. Some of these activities were simulated, while others were real life experiences. Many activities were also designed by the learners themselves, which created considerable buy-in from learners into the learning process.
One example of a simulated set of activities that was used is a high ropes adventure experience. These activities took the learners out of their comfort zones and stimulated discussions around group dynamics, leadership, interpersonal relationships, uncertainty, change, personal goal-setting and risk. One example of a real life activity that was used was the ‘R50 business’. Each student received R50 on a Thursday morning and on the following Monday had to report on the profit she/he had made. Principles of entrepreneurship and business opportunities were derived from this activity. The subject matter was thus experienced by the learners as an integrated whole, situated within the context in which it would be used again in future.
Facilitators steered the process and created an environment in which the learners could arrive at their own conclusions. The learners also came from diverse cultural and academic backgrounds, which extensively influenced discussions and conclusions.
Assessment was an ongoing dialogical process between peers, the learning-partner (coach) and the learner, the facilitator and the learner and self-assessment. Except for the psychometric assessment that was conducted as part of this study, assessment of progress never took place by means of a test or exam and was continuously used to set new development targets and to celebrate targets that had been reached. Table 1 compares the practical implementation of the training intervention with the principles of social constructivism.
The Research Process
Based on the literature review, a number of specific characteristics were identified as necessary for success in the workplace of the future. A battery of psychometric instruments was compiled to measure these characteristics. The pre-test for the experimental group was done on the first day of the training programme and the post-test on the last day. There was thus a six-month period between the pre-test and the post-test for the experimental group. The control group also had a period of six months between the pre- and post-tests, but were not exposed to the training intervention. The control group’s tests were administered during class time, as they all continued with their day-to-day university classes over the six-month period. The testing conditions for both groups were very similar, as both groups were tested under controlled classroom conditions.
The psychometric battery that was used as the assessment instrument consisted of sixteen tests from the Situation Specific Evaluation Expert (SpEEx) and Potential Index Batteries (PIB) series. The individual tests that were included in the psychometric battery have satisfactory, well-researched reliability (ranging between 0.58 and 0.92) and validity (ranging between 0.70 and 0.94) statistical records (Schaap 2004). The tests measured the following constructs: creativity, stress tolerance, type A/B behaviour, frustration tolerance, self-acceptance, adaptability, internal/external actualisation, conformity/non-conformity and the demonstrative, Samaritan (behaviour), persevering and evaluative social styles. Table 2 indicates how the constructs that were measured are linked to the characteristics that the training programme focused on.
The Mann-Whitney U-test was used to determine the significance of the differences between the variation that took place in the experimental group and in the control group over the six-month period between the pre- and post-tests. In this test, each treated entity is compared to each control entity. This test is used when any treated entity can be validly compared to any control entity. The interpretation of the results of the Mann-Whitney U-test is similar to that of the normal except that the U-test uses the sum of rank orders rather than the statistical mean. The statistical decision criterion was set on the level of Where results were statistically significant, the practical significance (meaningfulness) and effect sizes were also calculated. The reporting of effect sizes is encouraged by the American Psychological Association in their Publication Manual (APA, 1994). For differences between averages the meaningfulness level was set at d = 0.5 (medium effect size) (Cohen, 1988). The meaningfulness (d) pertaining to the comparison of the experimental and the control groups is represented as follows (Steyn 1999):
d = Meaningfulness
= The mean of the post-measurement of the experimental group
= The mean of the pre-measurement of the experimental group
= The maximum standard deviation between the pre- and post-measurement of the experimental group.
For results with medium and large effect sizes, the Bonferroni adjustment to alpha levels was made (Hsu, 1996).
The direction of the change in the experimental group was positive in the cases of creativity, adaptability and self-acceptance and negative in the case of the evaluative social style. The differences on the adaptability (large effect size), self-acceptance (medium effect size) and creativity (medium effect size) scales are also practically meaningful according to Cohen’s test for meaningfulness (practical significance). If the Bonferroni adjustment to alpha levels is made to rule out coincidental differences, only the difference on the adaptability scale remains significant. However, when doing the Bonferroni adjustment, care should be taken not to discard results that could in fact be valid (Hsu, 1996).
If the Bonferroni adjustment is applied the alternative hypothesis can be fully accepted only for the adaptability of the experimental group in comparison to that of the control group. Still, given the fact that the creativity and self-acceptance differences are also meaningful (practically significant) according to Cohen’s formula, and given the danger of discarding valid results through the use of Bonferroni’s adjustment, the alternative hypothesis for the dimensions of creativity and self-acceptance is (at least) preliminarily accepted. Participants in the programme were thus significantly more capable of adapting to a continuously changing environment than they were before participating in the training programme. Participants were also able to display relatively higher levels of creativity and creative approaches to problem-solving than they had been prior to the intervention. The levels of confidence and acceptance of self were also relatively increased. The attributes of creativity (Grulke, 2001; Brownstein, 2001; Cetron, 1999), adaptability (Boyatzis, 1999; Herman and Gioia, 1998) and self-acceptance (Branden, 1997) are all regarded as essential for succeeding in the workplace of the future.
The demonstrative and evaluative social styles of the participants also underwent a statistically significant (though not a practically meaningful) change. This could mean that participants were relatively more willing to express themselves assertively and to take part in group discussions than they were before the intervention. They were also relatively less evaluative in their social approach, meaning that they would take decisions faster and with less worrying about having every bit of information before taking action. Being willing and able to express oneself, take risks, get involved and create action are also regarded as essential attributes for succeeding in the workplace of the future (Ridderstrale and Nordstrom, 2004; Boyatzis, 1999; Epstein, 1998; Branden, 1997).
However, according to the results, the training programme did not have a statistically significant impact on constructs such as stress and frustration tolerance, type A/B behaviour, internal/external locus of control, conformity/non-conformity and the social styles called ‘perseverance’ and ‘Samaritan’. For these dimensions, the zero-hypothesis is accepted. The training intervention was thus partially successful in developing the characteristics that were measured.
The purpose of this research was to determine whether or not a training programme which was based on the principles of social constructivism was effective in developing individuals for the future world of work. In order to determine this, a psychometric test battery was used to measure a number of constructs that are related to what certain scholars regard as being characteristics needed for success in the future workplace.
First, we need to acknowledge that the idea of a ‘future workplace’ will remain (at best) a hypothetical one that will keep on changing as we move towards it. Second, we have to admit that many questions regarding developing people for this workplace still remain unanswered as this study indicated only the development of adaptability, creativity and self-acceptance while a score of other characteristics and competencies will very probably also be required in future workplaces. However, this study does add value in that it extends the insights we already have in terms of human learning to the area of developing young people (in the phase between school and work) for the workplace(s) they are about to enter. As all organizations are confronted with the challenges and uncertainties of the future, the way in which we develop young people to enter these organizations should be re-evaluated on a continuous basis. This training programme used a social constructivist approach that acknowledges the fact that people learn best when doing, discovering and sharing. In this way the divide between learning and real life decreases and are we able to also develop ‘softer’ characteristics such as adaptability, self-acceptance and creativity.
This study had three main limitations. First, due to logistical constraints, the groups were not assigned randomly. Second, due to the cost of the intervention which limited the number of participants to the programme, a Solomon four-group design could not be followed. The Solomon four-group design would require an additional two control groups that did not take part in the pre-test evaluation (Trochim, 2002). This would control the effect that the pre-test might have had on the participants. The limited sample size also makes it difficult to generalize these results to the wider population.
A third possible limitation of the study is situated at a more philosophical level. The training programme that was evaluated was based on the educational principles of social constructivism – a markedly post-modern philosophy and approach. In addition to this, the context which forms the backdrop for this study is that of the workplace of the future – a context that certainly tends increasingly to the ‘post-‘ side of modernism. The study thus evaluates a post-modem training programme, based on a post-modem educational philosophy, in order to determine whether or not it effectively develops people for a post-modem workplace. However, the methodology which is applied in executing the study is very modernist and positivist in its approach. It is a methodology where human characteristics are broken down into psychometric constructs which are (almost mechanistically) measured and statistically analysed. This becomes especially problematic when one tries to link each construct measured seamlessly with the exact nuances that the literature associates with the successful individual of the future. The question is whether the world of metaphor, story and dialogical understanding can be subjected to the measurement instruments and research methods of the world of mechanistic and systematic understanding in any valid way. However, this study wishes to communicate its message effectively to the modernist remnants in our present-day institutions responsible for preparing young people for the workplace. It therefore has to be able to express itself in the language and idiom best understood by these institutions. Still, it would be interesting to see how a more qualitative or even narrative research model would have evaluated this training programme.
Further research that could add further valuable insights might be a post-post assessment to indicate the sustainability of the changes that took place. This could most effectively be joined with a qualitative assessment in order to determine the real level of success the experimental group achieved in the future workplace over a longer period of time. From the researcher’s subjective experience of the experimental group’s development, the question of age may also be of interest. Does age have an impact on the efficiency of a social constructivist learning experience? The age-differential between the two groups was too small to make any conclusions in this regard, although there were stages in the programme where the older participants seemed to be getting more value from the process than their younger classmates.
Trochim, William M.K. (2001), The Research Methods Knowledge Base, Cincinnati, OH: Atomic Dog Publishing, pp.191-254.
Trochim, W. M. K. (2002), Nonequivalent groups analysis, The Research Methods Knowledge Base. Available at: <http://www.socialresearchmethods.net/kb/statnegd.htm>
Shadish, William, Cook, Thomas and Donald Campbell (2002). “Ch. 4: Quasi-Experimental Designs that either lack a control group or lack pretest observations on outcome,” Quasi-Experimentation, Dallas, TX: Houghton Mifflin.
Hsu, J. C. (1996). Multiple Comparisons Theory and Methods, (London: Chapman & Hall).
Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences, revised ed (Orlando, Florida: Academic Press).
Grulke, W. (2001) Radical Innovation (South Africa: Thorolds Africana Books).
Brownstein, B. (2001) Collaboration: the foundation of learning in the future, Education 122(2), p. 240.
Cetron, M. (1999) Fair or foul? Forecasts for education, School Administrator, 56, p. 6.
Boyatzis, R. E. (1999) Self-directed change and learning as a necessary meta-competency for success and effectiveness in the 21st century, in: R. Sims and J. G. Veres (Eds) Keys to Employee Success in the Coming Decades (Westport, CN: Greenwood).
Branden, N. (1997) Self-esteem in the Information Age, in: The Organization of the Future (San Francisco, CA: Jossey-Bass).
Ridderstrale, J. and Nordstrom, K. A. (2004) Karaoke Capitalism (Stockholm: Bookhouse Publishing).