ANZEA Conference 2022

Aotearoa has several small research centres that focus on ethnic communities. These exist within universities as well as independent organisations. In the last decade, there has also been an increase in community and grassroots initiatives that collate ethnic community research and evaluation undertaken by ethnic communities. However, a gap exists for a dedicated ethnic community-led research and evaluation centre committed to capacity building and producing ethnic community research and evaluation.

Research conducted in 2022 to get community perspectives and responses to establishing an independent ethnic community-led research and evaluation centre in Aotearoa found that establishing a centre would be invaluable to communities, government, NGOs, and policy makers.

Benefits include:

• Enabling innovative methodologies that foster cultural knowledge and expertise 
• Providing employment opportunities for ethnic researchers and evaluators 
• Providing a space where ethnic research and evaluation are accessible. 

CERA is now in the establishment phase. This independent organisation and network is digging deep and engaging widely on reclaiming ethnic research and evaluation.

This workshop is about what it means to be ethnic community-led and what an ethnic community-led research and evaluation centre that honours Ti Tiriti o Waitangi looks like?

At its simplest form, robust evaluation is evaluation that has a clear and logical set of questions, an explicit evaluation framework that supports value judgments, and a transparent, defensible, analysis that is shared via a reporting process or report. So, what do robust questions, evaluation frameworks and reporting look like?

This workshop will explain what good looks like, and how these three components are each used to guide robust evaluation. It will also describe various common pitfalls and their solutions with some practical tips and tricks to help recognise and avoid them.

This will be useful for evaluators and those managing evaluations, with some prior experience. Good questions, frameworks and reporting look like:

1. A concise set of evaluation questions, that includes priority questions, and some evaluative (not only descriptive) questions. Each are answered in the evaluation reporting.  
2. An evaluation framework that describes evaluand and its intent, and provides agreed definitions of ‘good’ or what is valued and (where applicable) references standards.  
3. A reporting process or report that makes explicit and defensible conclusions that are acceptable and believable and are therefore useful for decisions about proving, improving, expanding, or ceasing evaluand activities.  

Good use of an evaluation framework includes using it as a lens to guide the evaluation design, to refine the evaluation throughout and to support analysis, sense-making and value judgements. Robust reporting is an explicit and defensible presentation of findings and judgements. Each finding is based on cohesively presented evidence. Judgements build on these, and make explicit use of the evaluation framework. A number of common pitfalls that will be described and solutions shared. Three examples are:

a) Asking too many or too vague evaluation questions.
b) Not defining ‘good’ or what is valued, even in draft form.
c) Making indefensible conclusions or judgments.

Scaling a successful programme or initiative is generally thought of as consolidating or deepening of the initiative, by increasing intensity or volume in the current sites or extending its reach to new communities.

Decisions about scaling of programmes are often made by funders and implementers and not the proposed recipient communities. For Māori, this is problematic because it sees scaling as transactional, divorcing the programme from its creators and origins and assumes scaling is a finite process. Te koha (gifting) is a tapu (sacred) process. Giving something of significance to another reflects mana (respect), strengthening ties and relationships and carried reciprocal obligations.

Tūārai is a whānau-centred injury prevention model of practice, embedded in the notion of holistic, whānau wellbeing. It is being developed and tested by three iwi from the Tairāwhiti. One aspect of its development is to explore scaling Tūārai. Tūārai has developed a set of ‘scaling’ principles ‘Te Koha’ that are sourced from matauranga Māori and matauranga-ā-iwi Māori; ngā kaupapa tuku iho, inherited wisdom handed down from tupuna (ancestors). Tūārai is applying and refining Te Koha as their preferred approach for gifting or sharing their programme with other hāpori (communities) and hapū.

In this panel we will share the journey to date in developing Te Koha and the implications for evaluating programmes intending to scale or in the implementation process.

• Why a different approach to scaling, how did it come about?  
• What are the initial set of scaling principles, what are the tikanga and matauranga Māori that underpin it?  
• How is Te Koha being implemented?  
• What are the considerations for evaluators, evaluation and programme funders?  

All four members of the evaluation team, three with tribal links to the Tūārai tribes, will present on Te Koha.

In 2020, Te Rūnanga o Ngāi Tahu (the office established by the Te Rūnanga o Ngāi Tahu Act 1996 to serve, protect, and advance the collective interests of the iwi) set out to develop a new approach to performance assessment and reporting. Te Rūnanga o Ngāi Tahu have for a long time been monitoring activity and performance by using output focused KPIs and targets. However, these approaches to performance monitoring, assessment and reporting were not hitting the mark. They did not support the organisation to meaningfully capture and report on the value of the investment and work being done in ways that reflect what is valued and important to Ngāi Tahu whānui.

Te Rūnanga o Ngāi Tahu want to do more than just monitor performance. They want to be able to tell an authentic and meaningful performance story steeped in what is valuable and worthwhile at whānau, rūnaka, iwi, programme, project, and service levels. They also want these performance stories to be rich and robust enough to inform future investment decisions.

Te Rūnanga o Ngāi Tahu is developing a framework and practice model that departs from standard corporate practice around the measurement of performance. It is a values-oriented approach; the premise of which is that the cultural values of Ngāi Tahu are central to the conception of what good performance looks like. The framework is explicitly evaluative; combining a mix of evidence with cultural values and a deliberative process of evaluative reasoning to reach judgements about performance.

The implementation of the Ngāi Tahu Evaluation Framework is a significant undertaking.  A developmental, learn-as-you-go approach to building and implementing infrastructure, system and processes has been taken. They are developing a system that is a good fit and grounded in the everyday realities of the work of Te Rūnanga o Ngāi Tahu.

Evaluative synthesis is the culmination of the eval reasoning process. It brings together evidence and values about whatever is being evaluated to make an evaluative judgement. Getting evaluative synthesis right is an ethical responsibility for evaluators because evaluation is used to make decisions about ongoing funding, expansion or other changes. Evaluation synthesis can be challenging but is not discussed much in the literature or in professional development opportunities. Yet it is explicitly mentioned within two domains of the Aotearoa New Zealand competencies.

This extended presentation will focus on what it takes to do competent reasoning at the synthesis stage of an evaluation, based on our experience of teaching emerging evaluators in a Masters-level course on evaluation practice. We will cover:

1. What is evaluative synthesis and why is it important? 
2. What are common pitfalls?  
3. Practical tips and resources for doing it well. 

He taonga rongonui, te aroha ki te tāngata | Kindness towards others is a precious treasure.

Receiving the call to action from Evaluation with Aroha, Louise seeks to share her journey to re-member and re-claim our ability to lead through evaluation, by actioning the capacity of aroha. Responding to wero this conference is making explicit, in particular, do evaluators still have a role as agents of social change or is this a blurring disciplinary rigour and was always going to be a challenge?

Louise will bring forward insight from her Masters journey, that is beginning to bring this provocation into focus. Fundamentally values drive approaches to leadership, just as much as our values underpin our evaluation practice and positioning. At the intersection between the theories and practice of leadership and evaluation lie the potential mobilisers or instruments for change – evaluators. Where evaluation is frequently required to “be a rhythmic alternation between attacking the causes and healing the effects” of colonialism and inequity, exploring evaluators as instruments of change is critical. However, leadership in evaluation is unspoken, implicit; so, we must lean into this ‘silence’ by resonating with the hearts and minds of those who have expressed their commitment to those they serve, and their global evaluation community.

There are opportunities to “awaken the potential of ourselves, others and situations and to then consciously manifest that potential” through evaluative leadership - a state of being that leads to action through evaluation - by actioning the capacity of aroha to contribute to positive change.  This is an invitation to wayfind; hear insight into actioning the capacity of aroha, and the continuum of evaluative leadership, and space for reflective practice.

Evaluation is often commissioned to demonstrate the value of a programme or service to a funder. An external evaluator is often brought in as an ‘independent’ expert to understand the value of an initiative.

However, this approach to evaluation reinforces a centralized or top-down power structure. The funding has been received from the top, and those receiving the funding must demonstrate their value. When the value of evaluation could be enhanced if it was more closely driven by the organisations and the communities themselves.

Building evaluation capability with organisations provides an opportunity to put evaluation practice at the heart of organisations decision-making. Shifting the power and influence of evaluation from a top-down approach to a more locally driven activity. Some initial examples of how we are working to support this will be shared during our presentation. We will also facilitate a discussion to explore some of the challenges, opportunities, and trade-offs of working in this way.  

Programme logic models have been used by evaluators to understand the theory or idea underpinning a service, programme, or initiative that they are seeking to evaluate.

Programme logic models are designed to support us in developing a shared understanding of what we are evaluating with our stakeholders, while also supporting us to focus our evaluation efforts. However, what if this focus on linear causality is already limiting our evaluation efforts? What if this approach does not help us to understand the complex realities in which these initiatives are seeking to create change?

Our session will share some systems thinking tools, such as behaviour over time graphs, to demonstrate the value they bring when building theories of change. Our practical session will give you the chance to try out these tools and consider how they might fit with your own practice.

Increasingly, projects seek multi-dimensional impacts.

For example, increased productivity and improved environmental outcomes. Speaking to the reach anticipated, the reach achieved, and the resulting impacts, scaling reflects this more holistic perspective and flows through many programmes of work regardless of their sectors or location within Aotearoa and globally. This has prompted us to take the time to explore what scaling means by reflecting on our experiences (both good and not so good) across a range of national and international programmes we have been involved in.

We asked ourselves what should or could scaling look like and what could this mean for how and why we evaluate. Do we have all the answers? The answer is emphatically ‘no’, but we are eager to grow the intellectual thought in this space by engaging this evaluative conversation.

This workshop/presentation is based on the research into Māori involvement in State Care 1950-1999 for the Royal Commission into State Care Abuse (Savage et al., 2021).

The research involved qualitative and quantitative analyses, including an integrative literature review of 482 documents (primary research, archival material and publicly available reports and paper.

Gaps in document analyses formed the basis of semi-structured interviews. The twenty-six participants were Māori and non-Māori, including former Department of Social Welfare staff, community service providers and people involved in the development and implementation of Puao-Te-Ata-Tū and the Children’s Young Persons and their Families Act.

Results presented in the report emphasise the devastating, intergenerational harms that tamariki Māori and whānau have experienced through enduring, systemic and structural racism across the state care system. These findings are not new, given a large part of analysis is drawn from published material and also highlighted in inquiries and reviews.

However, report analysis brings together in one place, a compilation of information relating to Māori
overrepresentation and Māori experiences of the state care system during the review period (1950-1999). Synthesis of this research identifies a number of social policies, Government actions and inactions, that perpetuated Māori over-representation and differential treatment in the state care and justice system.

This workshop discusses a number of implications arising from historical social policy on current day evaluation practices in the social sector. The purpose of this presentation is to look back into the past in order to understand how structural inequities continue to impact on Māori in the present day.

We explore five learning areas from a developmental evaluation (DE) commissioned in a government agency.

The government agency was developing a new approach to partnering with communities and iwi as part of a wider organisational change process. DE team members, together with the government agency team members, will share their DE experience, drawing out useful insights and lessons.

We will discuss five key areas of learning to help highlight what it takes for DE to be successful in a government context. The considerations and tensions we will explore include:

1. Commissioning for complexity 
2. Evaluation design and reporting imperatives and the value of developing high trust, valued relationships, and a shared learning agenda from early on 
3. Privileging a kaupapa Māori approach within mainstream organisation constraints 
4. National and local contexts and the ethics and practices of sharing information  
5. Authorising environments for DE practice and the importance of trust as a de-risking tool.   

The necessary conditions for effective DE contrast with many existing government practices and systems. We hope our insights about what it takes to do DE in a government context will help developmental evaluators and those in government agencies and complex social spaces to use DE more successfully.

Are you a funder, provider or evaluator working on long-standing and profoundly entrenched complex, wicked problems? Have you noticed commissioning and contracting sometimes gets in the way of the programme's intended goals or services? Have you seen or had to use workarounds to make contracting possible? You are not alone. This session is for you.  

This session is for funders, providers or evaluators looking for innovative ways to think through commissioning and contracting issues in programme and service implementation. This session may change how you design and evaluate programmes in the future. New research shows commissioning and contracting can help or hamper innovative initiatives, particularly for Māori and Pacific providers. Tensions abound in navigating traditional contracting rules and procedures associated with New Public Management.

Evaluators who understand these tensions may be better able to explain gaps in programme and service delivery and suggest ways to address them.  This session will lean into some of the critical commissioning and contracting tensions. Instead of the prevailing rigid, predetermined ways of contracting for accountability only, this session will show how a complexity framing can help focus on more equitable service provision. We will introduce you to some new and successful ways of thinking about commissioning and contracting that are complexity-informed.  These methods and approaches will quickly demonstrate alternative ways of looking at commissioning and contracting.