7.2 Findings from experimental study
7.2.3.4 Descriptive analyses of workshop control variables
The following paragraph reveals control analyses designed to uncover potential confounding effects of the experimental methodology and to show the systematic control variable effects that have to be controlled for in the regression analyses in order to limit the impact of biases.
Influence of different workshop locations and dates
In order to guarantee the internal validity of results in lab experiments, several measures are taken to standardise the external setting of the experiment, e.g. by testing all participants at the same time and at the same laboratory. This study is classified as an experimental field study aiming to imitate real-life ideation sessions in realistic facilities.
Thus, the workshops were held at three different premises (the creativity/ideation facilities of industrial partners as well as the Design Lab at Twente University). Thus, this section tests whether the workshop location and date had an influence on idea quality and/or quantity. To control for effects related to the time of day, each workshop was of equal length (approximately four and a half hours) and commenced at the same time of day (2pm). Table 8 shows idea quality according to workshop number. It shows that the mean idea quality is centred around 4, which is slightly above the scale centre of 3.5 (on the Likert scale ranging from 1 to 7). Workshop two stands out with an average quality of 4.330 but has also the highest level of standard deviation. Skewness is inconsistent with positive as well as negative values.
Table 8: Descriptives of idea quality of different workshops, source: own analysis Descriptives of idea quality
Workshop no. N Mean SDa Skewness Kurtosis
1 54 3,869 1,077 0,277 -0,717
2 43 4,330 1,316 -0,084 -0,892
3 20 3,766 1,266 -0,587 -0,892
4 50 3,745 0,969 0,157 -0,579
Note. N=167 a SD=Standard deviation
In order to assess whether deviations of mean idea quality levels differ systematically or just by chance, inferential statistical models are applied. As the means of more than two groups are being compared, i.e. four workshops, analysis of variance statistics (ANOVA) seems appropriate. ANOVA is a parametric statistical test. Thus, it is based on certain assumptions. One assumption is that group distributions are normally distributed.
Shapiro-Wilk tests for workshops two (S-W2=0.948, df=43, p=0.050) and three (S- W3=0.897, df=20, p=0.036) indicate that the normal distribution assumption is violated for both group distributions (see also Figure 45) and residuals. Thus, a non-parametric Kruskal-Wallis statistic for group mean comparison is utilised. It compares the ranks of each group and requires ordinal scaled data, which is present in this sample.
Histogram of idea quality per workshop
N=167 Mean
idea quality
0 1 2 3 4 5
Workshop 1
WS 2
WS 3
WS 4
Workshop 1
Workshop 2
Workshop 3
Workshop 4 Idea quality distribution
Figure 45: Idea quality per workshop, source: own depiction
The null hypothesis of Kruskal-Wallis one-way analysis of variance by ranks is that the medians of idea quality in each of the workshops is the same. There was no statistical significance between the workshop number and the median idea quality (H(3)=5.426, p=0.143). Thus, I can assume that the workshop in which the participant took part did not have an influence on how the participants performed. Subsequently, the workshop number control variable will not be included in regression analysis.
Histogram of idea quantity per workshop
N=63 Mean
idea quantity
0 1 2 3 4 5 6
Workshop 1
WS 2
WS 3
WS 4 Figure 46: Idea quantity per workshop, source: own depiction
Similarly, idea quantity is tested for differences between the workshops. Graphically, the means of idea quantity by workshop appear to be similar (see Figure 46).
After applying the Kruskal-Wallis test of rank differences, no statistically significant differences between the workshops could be found, as was the case for idea quality (H(3)=7.407, p=0.060). Again, it can be assumed that the location of the workshop in which the participant took part had no influence on idea quantity, and this will therefore not be included in further analysis.
In conclusion, I can state that workshop locations, settings and time do not have a direct effect on the outcome variables. Thus, there is no evidence that the measures adopted to keep workshop settings comparable were unsuccessful.
Influence of status of participants
The experimental workshops included both university associates, i.e. students, as well as employees from the partner companies. This section aims to uncover group differences in the outcome variables by association status of the participants. In line with the analysis in the previous section, a Kruskal-Wallis test is applied for both idea quality and quantity.
Figure 47 shows the mean differences of the two groups of participants. It shows that the industry partner associates’ ideas (mean of 4.152) were rated slightly higher than the ideas of university associates (mean of 3.810). Nevertheless, group differences are not significant following Kruskal-Wallis statistics at a 95% confidence level (H(1)=3.654, p=0.056). Thus, it can be assumed that whether the ideas originated from students or aviation industry associates does not have a direct effect on idea quality.
Histogram of idea quality by participant status Idea quality distribution
0 1
Industry partner associates University
associates 2
3 4 Mean 5 idea quality
N=167 University associates
Industry partner associates
Figure 47: Idea quality by participant status, source: own depiction
Similarly to idea quality, an analysis should be made as to whether participant status has an impact on idea quantity, which is a measure of productivity. Consistent with the findings on idea quality, industry partner associates (mean of 4.150) created more ideas than university associates (mean of 3.581, see Figure 48). But again, the differences are too low to be significant (Kruskal-Wallis statistics, H(1)=2.655, p=0.105).
Histogram of idea quantity by participant status
N=63 0
1
Industry partner associates
University associates Idea quantity distribution
University associates 2
3 4 Mean 5 idea quantity
Industry partner associates
Figure 48: Idea quantity by participant status, source: own depiction Influence of idea length on idea quality
Degree of idea elaboration, measured as number of characters, varies strongly in the ideation sessions, resulting in four outliers with ideas above 750 characters (see Figure 49). As discussed, more elaborated ideas might be perceived as being of higher value (Onarheim & Christensen, 2012). This effect seems to be evident in this sample as well (see right-hand side of Figure 49). Regressing idea length on idea quality ratings, an upward facing slope is extracted. The impact of this effect proves to be positive and significant: B=0.00132, R²=0.052, t(166)=3.002, p=0.003. This means that if the length of an idea increases by 100 characters, the idea quality is increased by 0.132 points (on a seven-point Likert scale). Thus, idea length has to be controlled for in the main multiple regression analyses.
Distribution of idea length Scatterplot with regression line of idea quality
Idea length (no. of characters) N=167 0
250 500 750 1000 1250 Idea length No. of characters
1 2 3 4 5 6
7 Idea quality = 3.467 +
0.00132 * idea length
0 250 500 750 1000 1250 Idea
quality
Figure 49: Idea quality by idea length, source: own depiction Influence of order of idea creation on idea quality
The order in which ideas are created can have a significant impact on idea quality for several reasons. First of all, participants were asked to prioritise idea generation based on the identified top needs. Thus, they should address the most pressing needs first, which in turn should result in high idea quality. Secondly, creativity and/or attention level might decrease in the course of the ideation session, which should result in lower idea quality. Box plots of idea quality distributions are shown in Figure 50. It can be seen here that the median idea quality decreases depending on the idea order, i.e. the first idea generated has, on average, the highest idea quality, followed by the second and third idea.
In order to test whether this effect is statistically significant, a mono-causal regression analysis was conducted, regressing idea order on idea quality (also see Figure 50, right- hand side). It shows a downward facing slope. Regression calculation revealed that the effect of the idea order predicts idea quality significantly: B=-0.277, R²=0.036, t(166)=- 2.475, p=0.014. Interpreting these results, this means that idea quality, on average, decreases by 0.277 rating scores when comparing first ideas with second ideas or second ideas with third ideas. Therefore, idea order will be included in the main regression model.
Distribution of idea quality by idea order Scatterplot with regression line of idea quality
Idea quality
1 2 3
1 2 3 4 5 6 Idea quality
7
1stidea 2nd 3rd 1stidea 2nd 3rd
N=167 Idea quality = 4.459 – 0.277 * idea order