Measuring Intranet Effectiveness

In Cape Town, South Africa, The Porter Motor Group, an automotive dealer network, recently set out to evaluate user perceptions of the intranet, and the results have provided some valuable insights for future development.

 

As IT becomes increasingly recognized as a service rather than application provider, there is a greater need to develop service and customer orientation. User surveys are valuable tools for gathering qualitative information in a way that can be quantified.

 

Porter Group used a survey to provide a means of gaining insights with a minimum of data capture or complex analysis, with the goal of better understanding its effectiveness for directing actions towards service improvement. Possibly more important is that the research is a broadly applicable method of measurement. It is not specific to the Motor Industry and could find applicability across a wide range of different companies where assessment of intranet service quality needs to be evaluated.

 

Measuring user perceptions to determine intranet effectiveness is the same as measuring customer satisfaction in any other service delivery environment. While customer surveys are common in many industries, the IT organization needs to adopt a similar, albeit internal, approach to understanding user requirements and expectations.

 

Understanding the business value of technology has been a concern within the Porter Motor Group for some time and this survey is the beginning of a process that will drive IT projects for closer alignment to business requirements and improved service quality.

 

The primary aim of the survey was to measure intranet effectiveness and to establish an overall service quality index for service delivery. Analysis explored to what extent user expectations and perceptions differed and whether the instrument provided a valid measurement.

 

A service quality survey is used to determine a ‘service quality index’ based on the assumption that Service quality (Q) is the difference between user expectations (E) and user perceptions (P). The questionnaire asked for responses to the variables’ importance and performance across different service dimensions to arrive at the index. According to the literature, service quality should at least meet expectations, and therefore, Q should be equal to or greater than zero as a negative position, means service expectations are not being met.

 

This research was conducted at a time when the Porter Motor Group (PMG) was attempting to introduce Knowledge Management concepts and the redevelopment of the Intranet offered an excellent opportunity for measuring effectiveness.

 

The original corporate intranet PMGNET was introduced in 1999 and users had become familiar with the concepts. Its introduction represented a milestone in the company’s history and facilitated structural changes along with greater emphasis on efficiency and communication. PMGNET content was provided by a community of information providers publishing corporate material and general information that contributed significantly to the collaborative culture.

 

The questionnaire was based on the well-established SERVQUAL instrument developed by Parasuraman et al (1985, 1988) using a set of items (dimensions) that reflect a model of perceived Service Quality. Statements are used to measure expectation and perception (experience) for different dimensions as shown below:

 

Reliability: Provide accurate information and consistency. Intranet is dependable with minimum downtime, content and information is up-to-date and accurate, Intranet look and feel is user friendly.

 

Responsiveness: Fast Website, easy access and willingness to provide help. E-mail address for support and suggestions, Intranet response times are fast and reliable, Intranet content publishers always open to suggestions.

 

Assurance: Credible information, secure and allows users to inspire trust and confidence. Intranet content is appropriate and relative to the PMG, marketing and sales performance statistics promotes user confidence, the company Internet and E-mail use policy is accessible.

 

Empathy: Caring communication provides individual attention to users. Intranet content is easy to understand, knowledge centre is assisting users to collaborate, share and exchange ideas, community agent is encouraging team building.

 

Tangibles: Appearance of the Intranet site. Information is found with minimum key-clicks, the Intranet home page is visually appealing, navigation is consistent.

Decision makers and those with access to the internal network and Intranet were the main target of this study. Each question provided a 7-point scale on which the respondents could assess importance and performance of different aspects of the service as well as some demographic items to allow for analysis. The response rate by organizational role is shown in Table 1.

 

 

 

Overall user responses are shown in Table 2, and while measurement of individual service quality dimensions varied, all rated above the mid-point of the scale, but negative GAP scores revealed a general lack of satisfaction in the service. Analysis on the five service dimensions shows clearly that in all instances expectation (importance) exceeds perception (satisfaction).

 

 

 

The negative service quality index results indicate that the Intranet is under-performing across all service quality dimensions, and suggests that the Intranet fails to meet expectations. While users generally indicated that the Intranet is important they are disappointed with its performance. Based on these results it is clear that organisational user expectations and perceptions are significant and consequently affect success and acceptability of the PMG Intranet.

 

The results highlight a number of deficiencies that are negatively affecting overall Intranet effectiveness. Furthermore, the low Intranet performance index suggests that the executive management team needs to become actively engaged in determining content strategy and in formulating policy guidelines. As well, equal attention should be given to the current and future role of IT as facilitator and content provider. What is evident however is that an overwhelming majority of users consider the intranet to be an important business tool and an effective medium for sharing and publishing information.

 

Although corporate intranets are often cited in the literature, there aren’t many studies of intranet performance and few recommendations on how to manage and measure Intranet service quality. This study represents real world insight into the implementation of a reliable and valid survey instrument that may be beneficial in assessing and measuring the service quality and performance of an intranet.

 

The questionnaire proved sufficiently usable since the number of questions is relatively small and more likely to elicit response. The long-term intention is to repeat this process to create comparisons as a means of assessing the effectiveness of proposed developments. Who better to ask than the user?

 

Ricky Miller is the IT Manager for Porter Motor Group and is soon to graduate with Honours from the University of the Western Cape. His research into Intranet Effectiveness provides a simple and effective evaluation that can be applied in any organisation. For more information email him at: rmiller@porter.co.za

 

2004 will be the 78th consecutive year that the Porter Motor Group (PMG) (through F. Robb & Co.) has been in existence. In 1987 the JSE listed transport and mobility group, Imperial Holdings acquired 40% of the business, one year later in the summer of 1988 Imperial had full control of the company. In December 2003 PMG was consolidated into Premier Motor Holdings (PMH) under the leadership of group MD Trevor McKenzie. PMH through its dealer network has dealerships located in the Northern Cape, Western Cape, Northern Province, Free State and Gauteng.

 

For corporate information visit www.porter.co.za or www.imperial.co.za