Independent Program Evaluation

The results of the 2010 independent evaluation of the performance of the Water by Design capacity building program are now available. The aims of the evaluation were to:

  • Examine whether the program has been meeting its primary objectives.
  • Assess the program’s impact and value to stakeholders.
  • Examine the continuing need for the program and its possible future scope.
  • Assess whether the program’s major products and services (e.g. types of training and guidelines) have met their objectives, satisfied stakeholders, and been delivered on time and to budget.
  • Examine governance issues that affect the programs performance (e.g. funding arrangements).
  • Identify major strengths and weaknesses of the program.
  • Assess whether institutional capacity to implement water sensitive urban design (WSUD) in SEQ has increased since it was last assessed in 2007.
  • Identify opportunities to improve the program.

Executive Summary

Executive Summary (PDF 2 pages) This is also included below.

Recommendations

Recommendations (PDF 3 pages)

Full Report

Full Report (PDF 2.6 MB, 96 pages)

Executive Summary

This report presents the findings of an independent evaluation of the performance of the Water by Design capacity building program in South East Queensland (SEQ). The aims of this evaluation were to:

  • Examine whether the program has been meeting its primary objectives.
  • Assess the program s impact and value to stakeholders.
  • Examine the continuing need for the program and its possible future scope.
  • Assess whether the program s major products and services (e.g. types of training and guidelines) have met their objectives, satisfied stakeholders, and been delivered on time and to budget.
  • Examine governance issues that affect the program s performance (e.g. funding arrangements).
  • Identify major strengths and weaknesses of the program.
  • Assess whether institutional capacity to implement water sensitive urban design (WSUD) in SEQ has increased since it was last assessed in 2007.
  • Identify opportunities to improve the program.

The methodology involved examining all data from prior evaluation activities (e.g. surveys conducted shortly after training courses) and gathering data from program staff (e.g. statistics on the use of major products). It also involved two, new, on-line surveys of stakeholders and follow-up discussions with people who were highly familiar with the program to help interpret some of the survey results.

One of the on-line surveys gathered extensive data from steering committee members , people who have been effectively engaged by the program and people who have rarely or never used the program s products or services. The other survey gathered data on the strength of different elements of institutional capacity that are needed to drive WSUD in the SEQ region.

The key findings of the evaluation are summarised in Section 5. Overall, the evaluation found the Water by Design program had been an excellent capacity building program for advancing WSUD over the past two years. Stakeholders who were surveyed typically reported that its products and services have helped to deliver WSUD on-the-ground and therefore helped to produce outcomes such as reduced consumption of potable water supplies and improved waterway health. It is highly regarded by stakeholders as a leading program, which has consistently produced high-quality products and services. Data from a variety of sources strongly indicate that the program has been meeting its goals and satisfying the majority of stakeholders who have been using its products and services.

In relation to program objectives, impact and value, the evaluation found steering committee members and Insiders were, on average, strongly of the view that the program has been successfully meeting its official mission and seven goals. In addition, steering committee members, on average, viewed the program as being very good value to the organisations they represented and to funding organisations. They also believed the program has been a more efficient model of capacity building than alternatives.

The evaluation found a strong belief amongst stakeholders who were surveyed that the program needs to continue over the next three to five years, and that the need for such a program will probably increase over the next five years. There was also strong support for expanding the scope of the program to comprehensively address total water cycle management (TWCM). In addition, there was moderate to strong support for expanding the geographic scope beyond just SEQ to include other areas where costs can be recovered (e.g. Northern Queensland).

In terms of the programs major products and services, the evaluation found that the program had demonstrated the ability to consistently deliver these so that stakeholders were, on average, highly satisfied and firmly of the view that they had met their official objectives. The evaluation also found that some the programs major projects had not met their initial budgets or timeframes. This was a consequence of a range of factors (e.g. consultants not meeting agreed timeframes, the scope of products needing to expand to meet stakeholder needs, and the initial project / work plans being too ambitious due to a lack of performance data relating to past projects). This aspect of the programs performance represents an opportunity for further improvement.

In terms of program funding, the evaluation found that it does not yet have a diverse mix of funding sources, despite attempts to secure other sources of funding. Currently the majority of funding comes from the State government. This represents a major risk to the long-term future of the program. This situation also creates a potentially problematic situation where the program effectively has two sources of influence with respect to strategic direction: the official steering committee; and the primary funding agency who, understandably, wants to strongly influence the activities that are undertaken as a result of its investment.

Surveyed steering committee members were, on average, satisfied with the performance of the program’s staff, especially in relation to their strong technical, management and leadership skills. Many strengths (e.g. their ability to engage with stakeholders) and some developmental opportunities (e.g. their ability to engage with executives and politicians) were identified in relation to the abilities of program staff. Overall, the program staff were highly regarded, and the quality and quantity of human resources were seen as major factors contributing to the success of the program.

Stakeholders identified many strengths of the program such as the quality of technical guidelines and eNewsletters, and leadership by the program. They also suggested some practical opportunities for improvement, such as placing a greater focus on the implementation of WSUD by influencing policies, assessment decisions and enforcement mechanisms. Stakeholders also emphasised the need to better engage executives, politicians, and practitioners involved with public infrastructure projects.

Thirty-one recommendations are outlined in Section 6, and seven of the most significant will be summarised here. First, the program should continue its exemplary work, whilst taking opportunities to expand its scope (e.g. to address total water cycle management), service areas outside of SEQ (where costs can be recovered), and improve its effectiveness using the feedback obtained through this evaluation. This will most likely require increased human resources within the program.

Second, the program should continue to  strengthen its project and contract management activities. Actions could include: ensuring comprehensive project plans are prepared for all major projects; using data from past projects to determine realistic deadlines and budget; negotiating reasonable deadlines with consultants; requiring contingency plans from consultants; using penalty clauses for contracts associated with excessive delays (as a last resort); and continuing to improve the host organisations financial reporting systems to help track all project costs.

Third, it is recommended that the highest priority be placed on developing and implementing a comprehensive strategy to build a more diverse and sustainable funding base for the program. A more diverse funding base would also help ensure that the role of influencing the strategic direction of the program primarily lies with the steering committee, rather than any single funding organisation.

Fourth, the program should seek to retain and develop the program’s staff as they have been a major contributor to the program s success. The program should also ensure a plan exists to manage staff succession.

Fifth, when preparing future business, work and project plans, the program should seek to address the opportunities for improvement (see Sections 4.7 and 4.8) as well as the many suggested projects / ideas (see Appendix D) highlighted in this report. Sixth, the program’s steering committee should discuss the actions needed to respond to the recommendations in this report.

Finally, it is recommended that program staff and steering committee members be highly commended for their contribution to this exemplary program over the last two years.