CAUL Council of Australian University Librarians.


CAUL PERFORMANCE INDICATORS

CAUL Questionnaire (XNTU 17.5.96)

The CAUL Performance Indicators have sold well at the price of $120 each or $300 for the set of three. All have been sold and it is time to consider republication. Orders can be satisfied on a publication on demand basis but the costs of production and handling are so much higher as to make this uneconomic. Consequently, all Australian university libraries (which received copies as members of CAUL) were surveyed to determine the extent to which the indicators have been used to date and the need for revision. Other purchasers of the CAUL Indicators are also being surveyed but the results will not be available for some time.

The Survey

Twenty six Australian university libraries responded to a CAUL Questionnaire (XNTU 17.5.96) including two from the University of Western Sydney (Hawkesbury and Nepean). The experience of each respondent is presented in the attached table.

Of the 26 respondents, four had used none of the indicators and had no current plans to use them, while another three (Melbourne, Swinburne and UWA) have been using other indicators. To date, experience with the indicators has been limited:

Suggested Improvements

Feedback from those who have used the indicators has generally been positive with some suggestions for improvement, which are summarised below, and some comments on their application, which are summarised in the table.

Client Satisfaction Indicator

I think that Q4 could be omitted. Several of our customers commented that the Survey was too long. Also at Macquarie the answers to this questions were of dubious validity, particularly where the response should have been small. The results were distorted by some respondents indicating they had used every facility. In some cases there could also have been a misunderstanding of the category title used by Macquarie eg "used the Special Equipment Room" which is a facility for students with disabilities to which about 20 keys have been issued attracted 90 responses. (Macquarie, Lyn McCullagh)

We had one problem with categories in the Off Site version - many academics both teach AND undertake research - we added a "teaching and research staff" category. (NTU, Anne Wilson)

We would like the question concerning remote use of services made clearer and also suggest a 'not aware of service' answer and a 'never used service' answer instead of the one 'NA'. (VUT, Doreen Parker)

We had the benefit of seeing a report of Macquarie's use of the same survey earlier in 1995 and discovered that there seemed to have been a lot of confusion between the "NA" response (not applicable) and the "0" response (very dissatisfied). People who had not used a particular service etc. often marked "0" but this was recorded as having used it and found it very

unsatisfactory. So we changed the form to put the "NA" in a quite separate column divided from the numbers by a thick vertical line, and we headed it "NOT ATTEMPTED" "DID NOT USE" etc. as appropriate. We also added to the instructions an explicit statement to mark "NA" if they had not used the service, etc. I think this was largely successful in confining the not attempteds from the very dissatisfieds.

The form is quite complicated, with lists of services and activities, and some of them might not be understood by all respondents. We know what a reference collection is, but does the typical first year undergraduate identify it as readily? We had many undergraduates rating the ILL service, but they're not eligible for ILLs. What were they thinking of? Normal loans perhaps? It could be too technical a term for unsophisticated users and for those whose first language is not English. And so on.

It is clear from the responses that many (most?) people judged the Library and its staff, services, etc. on the basis of their experience with the Library generally, not specifically on their experience on that particular day's visit. For example, there were no reader education programs under way anywhere during the survey week, but dozens of people reported having

attended them that day, and ranked their satisfaction. Also, one of the branch libraries does not provide an ILL service yet people reported having requested ILLs there that day and ranked their satisfaction. With hindsight we should have adapted the questionnaires more thoroughly to omit such activities. However, overall, we have the strong impression that many users gave general satisfaction ratings based on their past experience with the library, rather than on what they did that day. This makes the data somewhat of an apples and oranges affair if some respondents are speaking generally and others are speaking of that particular visit. I don't know how one could get around that problem.

Not all of the services, facilities, etc are really worth ranking in the context of satisfaction. In Fisher and one branch the most satisfactory activity was "returned material". This involves putting the book in a slot in the desk, so it is hardly worth rating one's satisfaction with that, and it is not helpful to find that people think this is a satisfactory way to return books. In the other branch the most highly rated activity was "other" which was mostly meeting friends. Not very useful.

The survey instructions recommend pre-printing return envelopes so people can mail their forms back if they're too busy to fill them in at the time. We dutifully did all this and now have hundreds of unused envelopes sitting here. Only a handful were returned by post and we wouldn't bother with this option next time.

The free comments at the end were very useful, and probably more helpful than the survey results themselves. We received more than a thousand, and when grouped and analysed they show quite clearly what users' concerns are (generally, not specific to that particular visit).

We intend to repeat the survey perhaps annually or biennially as part of our Continuous Improvement program, and we hope to do more branch libraries next time, but we'll pay much closer attention to the form and to cutting out unnecessary activities, etc from the lists. (Sydney, Neil Radford)

Document Delivery Indicator

  1. Scope of Indicator - I think an adaptation of the specification to cover the supply or lending side as well as the obtaining side of document delivery might be useful.
  2. Specification -
  3. Program -

( QUT, Margaret Robertson, developer of the indicator)

I would like to see use of the document delivery indicator extended to all forms of delivery in which we respond to a specific client request including acquisitions. (NTU, Alex Byrne)

Materials Availability Indicator

The separation of TAFE and higher education categories would be useful. (VUT, Doreen Parker)

Presentation

The CAUL Indicators would be better presented in a more durable plastic folder. A standard folder with cover and spine pockets could be purchased for use with all the indicators. The specific title and other details for each could then be printed on card for insertion in the pockets. (NTU, Alex Byrne)

Quality List

Staff members in at least 19 of the 26 respondent libraries have subscribed to the Quality listserver hosted by NTU. It would thus be a useful way of facilitating discussion on performance and related matters. However, since traffic on it has been limited to date it would be helpful to 'prime the pump'. Perhaps this report could be released on to that list to promote discussion.

Conclusion

Given the limited number of university libraries which have used each of the indicators to date, it would be premature to undertake a major revision of any except the Document Delivery Indicator. Since nine more plan to use the Client Satisfaction Indicator later this year so it would be useful to survey their experience and then consider the need for a revision of publication of an addendum. In the meantime, the comments from Sydney and the others who have used it will assist in its application. The Materials Availability Indicator appears to have been satisfactory but there has been to little experience with it to contemplate a revision at this time.

However, the Document Delivery Indicator is in need of revision to correct programming deficiencies and to optimise processing time. A revision would also provide the opportunity to extend its coverage to include document supply processes and other applications such as acquisitions.

Recommendations

I recommend that the following actions be costed by the CAUL Executive Officer for consideration by the CAUL Executive:

  1. Production of another 100 copies of the Client Satisfaction Indicator for sale by CAVAL at the same price with the addition of a note on the experience of those which have used it to date.
  1. Letting a contract for the revision and extension of the Document Delivery Indicator by the end of the year. Orders should meanwhile be satisfied by CAVAL on a publication on demand basis. CAUL members will be supplied with a copy of the revised indicator on publication. All other purchasers should be offered a copy at a 50% discount.
  1. Production of another 100 copies of the Materials Availability Indicator for sale by CAVAL at the same price with the addition of a note on the experience of those which have used it to date.
  1. Introduction of a new more durable plastic folder for each at the time of publication of a revised version.
  1. Use of the Quality listserver for discussion on performance matters, 'primed' by release of this report.

Alex Byrne

14 June 1996


CAUL PERFORMANCE INDICATORS

Summary of Responses to CAUL Questionnaire (XNTU 17.5.96)


Library          Client Satisfaction Indicator  Document Delivery Indicator    Materials Availability         Quality   
                                                                               Indicator                      List      

ACU              Plan to use 1996               Not used                       Not used                       Y         

Adelaide         Not used                       Plan to this year              Plan to this year              Y         

ANU              Not used                       Not used                       Not used                                 

Bond             Plan to use in 1996            Not used                       Not used                       N         

Canberra         No current plans but will be   No current plans but will be   No current plans but will be             
                 looking at performance         looking at performance         looking at performance                   
                 measures                       measures                       measures                                 

Central          Plan to use in 1996, possibly  Not used                       Plan to use in 1996            Y         
Queensland                                                                                                              

Curtin           Plan to use in 1996            Not used                       Not used                       Y         

James Cook       Not used                       Once.  All 1995 completed      Plan to use second semester    Y         
                                                records were used.  The        1996                                     
                                                software is slow and contains                                           
                                                some bugs and problems.  A                                              
                                                final report of our study                                               
                                                will be available soon.                                                 
                                                Allow lots of time for the                                              
                                                work involved and use the                                               
                                                fastest PC you can get your                                             
                                                hands on.                                                               

La Trobe         Not used                       Not used                       Used in Abbotsford and         Y         
                                                                               Carlton campus libraries in              
                                                                               1994 and in all five                     
                                                                               libraries in 1995.  The                  
                                                                               questionnaire required                   
                                                                               modification for the two                 
                                                                               regional campus libraries as             
                                                                               they have their own OPACs.               
                                                                               This also required us to                 
                                                                               modify the spreadsheets for              
                                                                               data entry and analysis.                 
                                                                               Evaluation from all campuses             
                                                                               still being collated.                    

Macquarie        Once in 1995.  A copy of the   Not used                       Not used                       Y         
                 report on our implementation                                                                           
                 of the survey was distributed                                                                          
                 at a CAUL meeting.                                                                                     




Library          Client Satisfaction Indicator  Document Delivery Indicator    Materials Availability         Quality   
                                                                               Indicator                      List      

Melbourne        No, since we used Van House    Not used but intend to do so   No since we used Van House     Y         
                 methodology last year and 3    after we have implemented      methodology last year and 3              
                 years previously and would     recommendations of group       years previously and would               
                 have lost the opportunity to   reviewing document delivery.   have lost the opportunity to             
                 do trend analysis.  We intend                                 do trend analysis.  We intend            
                 to look carefully at the CAUL                                 to look carefully at the CAUL            
                 format and see how we can                                     format and see how we can                
                 preserve trends and also                                      preserve trends and also                 
                 directly compare our                                          directly compare our                     
                 performance against the                                       performance against the                  
                 results of others who use the                                 results of others who use the            
                 CAUL format.                                                  CAUL format.                             

Northern         We have run both the Exit and  Run on a monthly basis.  It    Once in 1994 as part of a      Y         
Territory        Off Site questionnaires        takes one staff member about   quality improvement team                 
                 twice, in October 1994 and     half a day a month to key in   project.                                 
                 March 96 (Offsite) and May 96  the data.  We collect                                                   
                 (Exit).  The surveys were      additional statistics on type                                           
                 straight forward to run,       of client (academic,                                                    
                 produced good response rates   postgraduate, faculty, etc).                                            
                 and the data analysis was      We don't use the last 4                                                 
                 reasonably straightforward.    columns ie if a transaction                                             
                                                can't be filled we consider                                             
                                                it dead at that point whereas                                           
                                                the Indicator has additional                                            
                                                columns for when sent back to                                           
                                                client, etc.                                                            
                                                Only advice to new users is                                             
                                                to make sure you have the                                               
                                                filed filled out correctly at                                           
                                                the start ie define clearly                                             
                                                what data you want to enter.                                            

NSW              Plan to use 1996               Plan to use 1996               Plan to use 1996               Y         

Queensland       No current plans but will be   No current plans but will be   No current plans but will be   Y         
                 looking at performance         looking at performance         looking at performance                   
                 measures                       measures                       measures                                 

QUT              Not used                       Our document delivery staff    The survey took place during   Y         
                                                have used the indicator        week 9, slightly later in                
                                                several times for both         semester than suggested in               
                                                intercampus and external       the documentation, however               
                                                document delivery.  They have  there is no 'best time' to               
                                                been able to follow it         conduct such a survey.  It is            
                                                without any great trouble and  recommended that follow up               
                                                have also been able to use     surveys in subsequent years              
                                                the program not strictly in    be undertaken at the same                
                                                accordance with the            time of the year.                        
                                                specification.  We have found                                           
                                                it very useful.                                                         




Library          Client Satisfaction Indicator  Document Delivery Indicator    Materials Availability         Quality   
                                                                               Indicator                      List      

QUT (cont)                                      The most labour intensive      The response rate was 34%                
                                                part of using it is compiling  which we found disappointing.            
                                                the data as we have to select   Publicity and promotion play            
                                                the records from the paper     an important role.  Location             
                                                files and then get the         and personality of the staff             
                                                turnaround time dates from     members distributing forms is            
                                                the computer system and then   important.  A friendly,                  
                                                key it all in.  Our samples    outgoing personality is                  
                                                have been small - only about   required.  During the survey             
                                                100 records at a time.         an emphasis was placed on                
                                                                               users entering the building.             
                                                                               This was not entirely                    
                                                                               successful, except when                  
                                                                               numbers entering the library             
                                                                               were high.  Distributing                 
                                                                               forms to users at the                    
                                                                               catalogue terminals was found            
                                                                               to be more effective.                    
                                                                               It is vitally important that             
                                                                               Library staff checking the               
                                                                               returned forms are very                  
                                                                               familiar with the catalogue,             
                                                                               searching techniques and                 
                                                                               shelving sequence.  Meeting              
                                                                               with staff to be involved                
                                                                               with the survey is important,            
                                                                               especially if they have not              
                                                                               been involved with the                   
                                                                               planning of the project.  It             
                                                                               is important to explain what             
                                                                               is hoped to be achieved by               
                                                                               the survey and that those                
                                                                               checking the forms are                   
                                                                               familiar with the numeric                
                                                                               values attached to each                  
                                                                               answer, and how these values             
                                                                               will be used in the Excel                
                                                                               database.                                
                                                                               Staff estimates:  38 hours to            
                                                                               distribute 1500 forms.  38               
                                                                               hours to check the 512                   
                                                                               returned.                                

RMIT             Plan to use 1996.  Curtin and  Not used                       Not used                       Y         
                 RMIT will benchmark the                                                                                
                 results.                                                                                               

South Australia  Once.  It did not accommodate  Not used                       Plan to use September 1996     Y         
                 a multi-campus library very                                                                            
                 well for analysis of                                                                                   
                 individual campus data. We                                                                             
                 would have analysed more                                                                               
                 responses if possible.                                                                                 
                 People responded well to the                                                                           
                 survey and did not appear to                                                                           
                 find it ambiguous.                                                                                     




Library          Client Satisfaction Indicator  Document Delivery Indicator    Materials Availability         Quality   
                                                                               Indicator                      List      

Southern Cross   Plan to use 1996               Not used                       Plan to use 1996               N         

Southern         Plan to use second semester    Plan to use second semester    Plan to use second semester    Y         
Queensland       1996                           1996                           1996                                     

Swinburne        Not used.  User Service areas  Not used.  User Service areas  Not used.  User Service areas  Y         
                 of the Library undertook       of the Library undertook       of the Library undertook                 
                 significant                    significant                    significant                              
                 Priority/Performance surveys   Priority/Performance surveys   Priority/Performance surveys             
                 and a number of focus          and a number of focus          and a number of focus                    
                 activities over 1994/5, prior  activities over 1994/5, prior  activities over 1994/5, prior            
                 to the release of the CAUL     to the release of the CAUL     to the release of the CAUL               
                 Performance Indicator Kits.    Performance Indicator Kits.    Performance Indicator Kits.              
                 In 1996, the University is     In 1996, the University is     In 1996, the University is               
                 putting in place a quality     putting in place a quality     putting in place a quality               
                 management process.  This      management process.  This      management process.  This                
                 includes the use of            includes the use of            includes the use of                      
                 appropriate performance        appropriate performance        appropriate performance                  
                 indicators.  The Information   indicators.  The Information   indicators.  The Information             
                 Services Group of which the    Services Group of which the    Services Group of which the              
                 library is a part, will be     library is a part, will be     library is a part, will be               
                 relooking at the CAUL          relooking at the CAUL          relooking at the CAUL                    
                 indicators with a view to      indicators with a view to      indicators with a view to                
                 incorporation and possible     incorporation and possible     incorporation and possible               
                 use in 1997/8.                 use in 1997/8.                 use in 1997/8.                           

Sydney           Used in October 1995 in        Not used                       Not used                                 
                 Fisher and two branch                                                                                  
                 libraries.  We intend to use                                                                           
                 it more widely this year but                                                                           
                 it will need some adjustment                                                                           
                 following our experience last                                                                          
                 year.                                                                                                  
                 We used it last October in                                                                             
                 Fisher Library and 2 branch                                                                            
                 libraries.  We aimed for (and                                                                          
                 achieved) 500 responses in                                                                             
                 Fisher and 300 each in the                                                                             
                 branches, total 1,100.  We                                                                             
                 tailored the questionnaire to                                                                          
                 our particular situation by                                                                            
                 adding facilities and                                                                                  
                 services, using local names                                                                            
                 for collections, etc.  This                                                                            
                 took a bit of fiddling with                                                                            
                 the package, but was                                                                                   
                 eventually successful.                                                                                 




Library          Client Satisfaction Indicator  Document Delivery Indicator    Materials Availability         Quality   
                                                                               Indicator                      List      

VUT              Plan to use in second          Our Client Services Branch is  We used in once in 1995 but    Y         
                 semester 1996                  using it for the whole of      had difficulties in getting a            
                                                1996 and has had no            satisfactory response because            
                                                difficulty in collecting the   of poor timing.  We have also            
                                                information.                   had difficulties in collating            
                                                                               the data which appear to                 
                                                                               arise from a need to upgrade             
                                                                               the hardware on the PC being             
                                                                               used.                                    

Western          Replicates a survey in us at   A survey of document delivery  Replicates a survey in us at             
Australia        UWA which is run every 3       was carried out two years ago  UWA which is run every 3                 
                 years and provides             and no plans have been made    years and provides                       
                 comparative data over time.    to undertake another study at  comparative data over time.              
                 To change would prevent the    this time.                     To change would prevent the              
                 comparisons being made.                                       comparisons being made.                  

Western Sydney   No current plans               No current plans               No current plans                         
Hawkesbury                                                                                                              

Western Sydney   Plan to use 1996               Not used                       Not used                       Y         
Nepean                                                                                                                  



Alex Byrne 24 June 1996


Updated 17 July, 1996
Return to CAUL Home Page
Comments/Suggestions