Through this database we systematically explore the constructs used in survey studies in information security. We are interested in any publication that
Creates new constructs,
Applies constructs in the form of user studies,
Discusses results derived from constructs (i.e. meta analysis).
The exploration was approached through three methods:
As a first step an google scholar search for combinations of the terms ‘information security’, ‘security’, ‘survey’, ‘questionnaire’ and ‘construct’ was carried out. Unfortunately these terms are incredibly broad, and over 3 million relevant articles were returned by the search engine. The first 30 pages of search results (i.e. 3000 articles) were analysed against the three criteria above and 124 relevant publications were identified.
For every article analysed, the articles that were cited for constructs were added to the analysis queue (going backwards in time),
For every article analysed, we used Google Scholar to identify citing publications (i.e. going forward in time). Here we limited us to the 30 most cited publications (some psychology publications had tens of thousands of citations) and added these to the analysis queue if they conformed to the selection criteria above.
Initially we found that steps 2 and 3 above increased the size of the analysis queue exponentially - for every paper analysed we would add 10 papers to the queue. However after analysing 400 publications the queue started to become of fixed length, i.e. for every paper analysed we added one more paper to the queue. We have not finished processing the queue of papers at this stage, however at over 1000 constructs identified (before merging) we are reasonably confident that a comprehensive view of constructs in security has been achieved.
For each publication, we collected the following data:
A short description of the research (usually a snippet of the abstract)
research type and sample size (for example user study with 180 students / meta review / construct validation study)
the source PDF file
The constructs used or discussed, where for each construct we collected:
The exact sources referenced for the construct, or any comment by the authors if they created the construct themselves
The type of the construct (usually the theory on which the construct is based)
whether the article lists the exact questions used
whether the article gives the answer options to the questions (and if so, what type they are)
two measures of validation, as described below
The data was collected through Zotero and multiple excel spreadsheets, with a number of custom scripts. The website is statically build using Flask and Jinja2 for templating.
There are also a number of existing construct databases (although these are not security specific): Muhlenberg College
While the collection of the constructs is the primary task of this project, we also have to consider whether they have been used in a valid manner. Construct validity is the extent to which an operationalisation measures the concepts that it purports to measure (Straub, 1989). Convergent, discriminant, and nomological validation are all considered to be components of construct validity, as well as criterio-related validity and its sub-types, predictive and concurrent validity (Cronbach, 1949 and Rogers, 1995).
In the context of the Management Information Systems (MIS) literature, Boundreau et al. reviews MIS positivist quanitatitive methodologies and their validity and reliability (2001). The authors limit their analysis to high-level validation techniques:
Pretest
Pilot
Previous Instrument Utilized
Content Validity
Construct validity
They find that only between 25% and 60% of articles studied in their field perform these analysis techniques. Their research forms the basis of the two columns that we have coded for in this research: ‘Content validity’ and ‘Pretest’.
Under ‘Content Validity’ we code for the various techniques that researchers can perform to establish the degree to which items in an instrument reflect the content universe to which the instrument will be generalised (Cronbach, 1949 and Rogers, 1995). This aspect is particularly important in the case of surveys in information security, where often constructs are borrowed from other disciplines, and the content validation is never repeated. This validity is generally established through literature reviews and expert judges or panels. Lawshe (1975) describes a statistical approach to measuring content validity - although we have not seen it employed in this survey.
Under ‘Pretest’ we denote the measures the authors have taken to ensure that there are no unanticipated difficulties when executing the survey. Every survey should be pre-tested no matter how skilled the researcher (Fowler, 2009). While a study should contain both pre-tests and pilot (the pilot being the dress-rehearsal), many authors use these terms interchangeably, and hence we code both parts here under the heading ‘pretest’.
There are many constructs that have near identical questions, but have been renamed to capture a specific context, for example Ifinedo (2014) uses the construct Attitude toward ISSP compliance (Information System Security Policy), while Sohrabi Safa et al. (2016) uses Attitude towards compliance with ISOP (Information Security Organizational Policies).
Sommestad et al. (2014) conduct a systematic review of empirical studies to identify variables that influence compliance with information security policies of organizations and to identify how important these variables are. 29 studies fit their criteria and they find that none of the 60 factors identified strongly influences compliance. However, their report provides detailed disambiguation for the names of constructs. Their table forms the initial basis for us to group constructs together. In our case, we have initially identified 984 constructs that the grouping reduced to 789 constructs.