The CAUL Performance Indicators have sold well at the price of $120 each or $300 for the set of three. All have been sold and it is time to consider republication. Orders can be satisfied on a publication on demand basis but the costs of production and handling are so much higher as to make this uneconomic. Consequently, all Australian university libraries (which received copies as members of CAUL) were surveyed to determine the extent to which the indicators have been used to date and the need for revision. Other purchasers of the CAUL Indicators are also being surveyed but the results will not be available for some time.
Twenty six Australian university libraries responded to a CAUL Questionnaire (XNTU 17.5.96) including two from the University of Western Sydney (Hawkesbury and Nepean). The experience of each respondent is presented in the attached table.
Of the 26 respondents, four had used none of the indicators and had no current plans to use them, while another three (Melbourne, Swinburne and UWA) have been using other indicators. To date, experience with the indicators has been limited:
Feedback from those who have used the indicators has generally been positive with some suggestions for improvement, which are summarised below, and some comments on their application, which are summarised in the table.
Client Satisfaction Indicator
I think that Q4 could be omitted. Several of our customers commented that the Survey was too long. Also at Macquarie the answers to this questions were of dubious validity, particularly where the response should have been small. The results were distorted by some respondents indicating they had used every facility. In some cases there could also have been a misunderstanding of the category title used by Macquarie eg "used the Special Equipment Room" which is a facility for students with disabilities to which about 20 keys have been issued attracted 90 responses. (Macquarie, Lyn McCullagh)
We had one problem with categories in the Off Site version - many academics both teach AND undertake research - we added a "teaching and research staff" category. (NTU, Anne Wilson)
We would like the question concerning remote use of services made clearer and also suggest a 'not aware of service' answer and a 'never used service' answer instead of the one 'NA'. (VUT, Doreen Parker)
We had the benefit of seeing a report of Macquarie's use of the same survey earlier in 1995 and discovered that there seemed to have been a lot of confusion between the "NA" response (not applicable) and the "0" response (very dissatisfied). People who had not used a particular service etc. often marked "0" but this was recorded as having used it and found it very
unsatisfactory. So we changed the form to put the "NA" in a quite separate column divided from the numbers by a thick vertical line, and we headed it "NOT ATTEMPTED" "DID NOT USE" etc. as appropriate. We also added to the instructions an explicit statement to mark "NA" if they had not used the service, etc. I think this was largely successful in confining the not attempteds from the very dissatisfieds.
The form is quite complicated, with lists of services and activities, and some of them might not be understood by all respondents. We know what a reference collection is, but does the typical first year undergraduate identify it as readily? We had many undergraduates rating the ILL service, but they're not eligible for ILLs. What were they thinking of? Normal loans perhaps? It could be too technical a term for unsophisticated users and for those whose first language is not English. And so on.
It is clear from the responses that many (most?) people judged the Library and its staff, services, etc. on the basis of their experience with the Library generally, not specifically on their experience on that particular day's visit. For example, there were no reader education programs under way anywhere during the survey week, but dozens of people reported having
attended them that day, and ranked their satisfaction. Also, one of the branch libraries does not provide an ILL service yet people reported having requested ILLs there that day and ranked their satisfaction. With hindsight we should have adapted the questionnaires more thoroughly to omit such activities. However, overall, we have the strong impression that many users gave general satisfaction ratings based on their past experience with the library, rather than on what they did that day. This makes the data somewhat of an apples and oranges affair if some respondents are speaking generally and others are speaking of that particular visit. I don't know how one could get around that problem.
Not all of the services, facilities, etc are really worth ranking in the context of satisfaction. In Fisher and one branch the most satisfactory activity was "returned material". This involves putting the book in a slot in the desk, so it is hardly worth rating one's satisfaction with that, and it is not helpful to find that people think this is a satisfactory way to return books. In the other branch the most highly rated activity was "other" which was mostly meeting friends. Not very useful.
The survey instructions recommend pre-printing return envelopes so people can mail their forms back if they're too busy to fill them in at the time. We dutifully did all this and now have hundreds of unused envelopes sitting here. Only a handful were returned by post and we wouldn't bother with this option next time.
The free comments at the end were very useful, and probably more helpful than the survey results themselves. We received more than a thousand, and when grouped and analysed they show quite clearly what users' concerns are (generally, not specific to that particular visit).
We intend to repeat the survey perhaps annually or biennially as part of our Continuous Improvement program, and we hope to do more branch libraries next time, but we'll pay much closer attention to the form and to cutting out unnecessary activities, etc from the lists. (Sydney, Neil Radford)
Document Delivery Indicator
( QUT, Margaret Robertson, developer of the indicator)
I would like to see use of the document delivery indicator extended to all forms of delivery in which we respond to a specific client request including acquisitions. (NTU, Alex Byrne)
Materials Availability Indicator
The separation of TAFE and higher education categories would be useful. (VUT, Doreen Parker)
The CAUL Indicators would be better presented in a more durable plastic folder. A standard folder with cover and spine pockets could be purchased for use with all the indicators. The specific title and other details for each could then be printed on card for insertion in the pockets. (NTU, Alex Byrne)
Staff members in at least 19 of the 26 respondent libraries have subscribed to the Quality listserver hosted by NTU. It would thus be a useful way of facilitating discussion on performance and related matters. However, since traffic on it has been limited to date it would be helpful to 'prime the pump'. Perhaps this report could be released on to that list to promote discussion.
Given the limited number of university libraries which have used each of the indicators to date, it would be premature to undertake a major revision of any except the Document Delivery Indicator. Since nine more plan to use the Client Satisfaction Indicator later this year so it would be useful to survey their experience and then consider the need for a revision of publication of an addendum. In the meantime, the comments from Sydney and the others who have used it will assist in its application. The Materials Availability Indicator appears to have been satisfactory but there has been to little experience with it to contemplate a revision at this time.
However, the Document Delivery Indicator is in need of revision to correct programming deficiencies and to optimise processing time. A revision would also provide the opportunity to extend its coverage to include document supply processes and other applications such as acquisitions.
I recommend that the following actions be costed by the CAUL Executive Officer for consideration by the CAUL Executive:
14 June 1996
CAUL PERFORMANCE INDICATORS
Summary of Responses to CAUL Questionnaire (XNTU 17.5.96)
Library Client Satisfaction Indicator Document Delivery Indicator Materials Availability Quality Indicator List ACU Plan to use 1996 Not used Not used Y Adelaide Not used Plan to this year Plan to this year Y ANU Not used Not used Not used Bond Plan to use in 1996 Not used Not used N Canberra No current plans but will be No current plans but will be No current plans but will be looking at performance looking at performance looking at performance measures measures measures Central Plan to use in 1996, possibly Not used Plan to use in 1996 Y Queensland Curtin Plan to use in 1996 Not used Not used Y James Cook Not used Once. All 1995 completed Plan to use second semester Y records were used. The 1996 software is slow and contains some bugs and problems. A final report of our study will be available soon. Allow lots of time for the work involved and use the fastest PC you can get your hands on. La Trobe Not used Not used Used in Abbotsford and Y Carlton campus libraries in 1994 and in all five libraries in 1995. The questionnaire required modification for the two regional campus libraries as they have their own OPACs. This also required us to modify the spreadsheets for data entry and analysis. Evaluation from all campuses still being collated. Macquarie Once in 1995. A copy of the Not used Not used Y report on our implementation of the survey was distributed at a CAUL meeting.
Library Client Satisfaction Indicator Document Delivery Indicator Materials Availability Quality Indicator List Melbourne No, since we used Van House Not used but intend to do so No since we used Van House Y methodology last year and 3 after we have implemented methodology last year and 3 years previously and would recommendations of group years previously and would have lost the opportunity to reviewing document delivery. have lost the opportunity to do trend analysis. We intend do trend analysis. We intend to look carefully at the CAUL to look carefully at the CAUL format and see how we can format and see how we can preserve trends and also preserve trends and also directly compare our directly compare our performance against the performance against the results of others who use the results of others who use the CAUL format. CAUL format. Northern We have run both the Exit and Run on a monthly basis. It Once in 1994 as part of a Y Territory Off Site questionnaires takes one staff member about quality improvement team twice, in October 1994 and half a day a month to key in project. March 96 (Offsite) and May 96 the data. We collect (Exit). The surveys were additional statistics on type straight forward to run, of client (academic, produced good response rates postgraduate, faculty, etc). and the data analysis was We don't use the last 4 reasonably straightforward. columns ie if a transaction can't be filled we consider it dead at that point whereas the Indicator has additional columns for when sent back to client, etc. Only advice to new users is to make sure you have the filed filled out correctly at the start ie define clearly what data you want to enter. NSW Plan to use 1996 Plan to use 1996 Plan to use 1996 Y Queensland No current plans but will be No current plans but will be No current plans but will be Y looking at performance looking at performance looking at performance measures measures measures QUT Not used Our document delivery staff The survey took place during Y have used the indicator week 9, slightly later in several times for both semester than suggested in intercampus and external the documentation, however document delivery. They have there is no 'best time' to been able to follow it conduct such a survey. It is without any great trouble and recommended that follow up have also been able to use surveys in subsequent years the program not strictly in be undertaken at the same accordance with the time of the year. specification. We have found it very useful.
Library Client Satisfaction Indicator Document Delivery Indicator Materials Availability Quality Indicator List QUT (cont) The most labour intensive The response rate was 34% part of using it is compiling which we found disappointing. the data as we have to select Publicity and promotion play the records from the paper an important role. Location files and then get the and personality of the staff turnaround time dates from members distributing forms is the computer system and then important. A friendly, key it all in. Our samples outgoing personality is have been small - only about required. During the survey 100 records at a time. an emphasis was placed on users entering the building. This was not entirely successful, except when numbers entering the library were high. Distributing forms to users at the catalogue terminals was found to be more effective. It is vitally important that Library staff checking the returned forms are very familiar with the catalogue, searching techniques and shelving sequence. Meeting with staff to be involved with the survey is important, especially if they have not been involved with the planning of the project. It is important to explain what is hoped to be achieved by the survey and that those checking the forms are familiar with the numeric values attached to each answer, and how these values will be used in the Excel database. Staff estimates: 38 hours to distribute 1500 forms. 38 hours to check the 512 returned. RMIT Plan to use 1996. Curtin and Not used Not used Y RMIT will benchmark the results. South Australia Once. It did not accommodate Not used Plan to use September 1996 Y a multi-campus library very well for analysis of individual campus data. We would have analysed more responses if possible. People responded well to the survey and did not appear to find it ambiguous.
Library Client Satisfaction Indicator Document Delivery Indicator Materials Availability Quality Indicator List Southern Cross Plan to use 1996 Not used Plan to use 1996 N Southern Plan to use second semester Plan to use second semester Plan to use second semester Y Queensland 1996 1996 1996 Swinburne Not used. User Service areas Not used. User Service areas Not used. User Service areas Y of the Library undertook of the Library undertook of the Library undertook significant significant significant Priority/Performance surveys Priority/Performance surveys Priority/Performance surveys and a number of focus and a number of focus and a number of focus activities over 1994/5, prior activities over 1994/5, prior activities over 1994/5, prior to the release of the CAUL to the release of the CAUL to the release of the CAUL Performance Indicator Kits. Performance Indicator Kits. Performance Indicator Kits. In 1996, the University is In 1996, the University is In 1996, the University is putting in place a quality putting in place a quality putting in place a quality management process. This management process. This management process. This includes the use of includes the use of includes the use of appropriate performance appropriate performance appropriate performance indicators. The Information indicators. The Information indicators. The Information Services Group of which the Services Group of which the Services Group of which the library is a part, will be library is a part, will be library is a part, will be relooking at the CAUL relooking at the CAUL relooking at the CAUL indicators with a view to indicators with a view to indicators with a view to incorporation and possible incorporation and possible incorporation and possible use in 1997/8. use in 1997/8. use in 1997/8. Sydney Used in October 1995 in Not used Not used Fisher and two branch libraries. We intend to use it more widely this year but it will need some adjustment following our experience last year. We used it last October in Fisher Library and 2 branch libraries. We aimed for (and achieved) 500 responses in Fisher and 300 each in the branches, total 1,100. We tailored the questionnaire to our particular situation by adding facilities and services, using local names for collections, etc. This took a bit of fiddling with the package, but was eventually successful.
Library Client Satisfaction Indicator Document Delivery Indicator Materials Availability Quality Indicator List VUT Plan to use in second Our Client Services Branch is We used in once in 1995 but Y semester 1996 using it for the whole of had difficulties in getting a 1996 and has had no satisfactory response because difficulty in collecting the of poor timing. We have also information. had difficulties in collating the data which appear to arise from a need to upgrade the hardware on the PC being used. Western Replicates a survey in us at A survey of document delivery Replicates a survey in us at Australia UWA which is run every 3 was carried out two years ago UWA which is run every 3 years and provides and no plans have been made years and provides comparative data over time. to undertake another study at comparative data over time. To change would prevent the this time. To change would prevent the comparisons being made. comparisons being made. Western Sydney No current plans No current plans No current plans Hawkesbury Western Sydney Plan to use 1996 Not used Not used Y Nepean
Alex Byrne 24 June 1996