Loading…
ANZEA Conference 2022
10 - 12 October 2022 | Te Papa, Wellington

Log in to bookmark your favorites and sync them to your phone or calendar.

Add sessions to your personal schedule to register your attendance (Check-In).

Monday, October 10
 

1:30pm NZDT

Panel Discussion 1: LOOKING BACK. Development of evaluation as a profession
Chaired by Kate McKegg
Contributors
avatar for Kate McKegg

Kate McKegg

Director, The Kinnect Group
Kate has specialist skills in supporting evaluative thinking and practice in complex settings where people are innovating to create systems change. She has been applying these skills for over 25 years in government, non-government, philanthropic and community contexts, including many... Read More →
avatar for Fiona Cram

Fiona Cram

Katoa Ltd
Fiona Cram is a Maori (indigenous) woman from Aotearoa New Zealand. Her tribal affiliations are to Ngati Pahauwera on the east coast of Aotearoa. Fiona is the mother of one son.Fiona has a PhD in social and developmental psychology and is a widely published author of evaluation books... Read More →
avatar for Andy Rowe

Andy Rowe

Dr, Footprints Evaluation
Andy has a PhD from the London School of Economics, is a Fellow and former President of the CES and an early contributor to evaluating sustainability and mainstreaming sustainability in evaluation. He initiated the CES effort to mainstream sustainability and is a founding member of... Read More →
RP
MC
TT

Taule’ale’ausumai, Tofaga, Violet Tu'uga Stevenson

Pule Pasifika|Director Evaluation and Review Pacific, Education Review Office

Monday October 10, 2022 1:30pm - 2:30pm NZDT
Oceania
  Panel discussion, Reflecting

2:30pm NZDT

Afternoon tea break
Afternoon tea break
Monday October 10, 2022 2:30pm - 3:00pm NZDT
Oceania
  Conference Activity, Informal occasion

3:00pm NZDT

Featured Contributors 1: Ka mua, ka muri, ka mua ka ako: Reflecting on the past to inform our futures
The theme of this conference, taken from a well-known Māori whakataukī, Kia whakatōmuri te haere whakamua acknowledges the past, while at the same time encouraging us to walk into the future being genuinely aware of that past.

For those of us with some longevity in the profession, the theme compels us to think about how our early experiences, former practices, and previous achievements as evaluators and how these might inform and shape the profession of evaluation into the future.

In this presentation I will reflect on a quarter century of working in the fields of evaluation and research. Change has been a constant theme underpinning my work, however the scale of change we have witnessed in Aotearoa New Zealand in the last two years has been both significant and comprehensive. For Māori, the impacts of these changes on our community and our whānau are still being understood. What then, can we as evaluators and researchers contribute to the nations’ need to analyse, comprehend and learn from this change? What lessons can we take from the past which will allow us to stand united in an authentic partnership to meet the challenges of the future? What is our role as evaluators and researchers in that future, and what is the legacy we “oldies” leave for those new and emerging evaluators who will take up our mantle?

In this address I will provide a glimpse into what I have learned, what I might have done differently, who and what shaped my experiences as an evaluator/researcher. I conclude with the view that the skill of evaluative thinking and the theory, practice and profession of evaluation remains critical for our nation today but perhaps even more so as we look forward into an unknown future.

Contributors
avatar for Dr. Amohia Boulton

Dr. Amohia Boulton

Research Director, Whakauae Research Services
Dr Amohia Boulton (Ngāti Ranginui, Ngai te Rangi, Ngāti Pukenga, Ngāti Mutunga and Te Āti Awa o te Waka a Māui), is the Director of Whakauae Research Services Ltd, the only tribally owned and mandated health research centre in Aotearoa/New Zealand. Amohia also holds Adjunct Professor... Read More →

Boulton ANZEA 8 October 2022 pdf
Monday October 10, 2022 3:00pm - 4:00pm NZDT
Oceania
  Featured contributor, Reflecting

4:00pm NZDT

A.01 A matter of life and death: Valuing human life – history and suggestions for the future
The Treasury’s CBAx tool for helping agencies do cost-benefit analysis values a statistical life at $4.9m. Such values are used, for example, in land transport investment decisions and some COVID-19 modelling.

This session summarises the history of such values in New Zealand since the 1980s transport study that is still the underlying source for many valuations now.
The session will consider how evaluators lacking specialised knowledge of economic evaluation or the relevant data collection methods might use or even contribute to such valuations in future. In particular, it considers such valuations from several different perspectives: a philosophy of science perspective, a data collection perspective, and differences between countries.
Some resulting suggestions for the future are:

  1. Contrary to recent recommendations, New Zealand transport agencies should not soon conduct another major Value of Statistical Life survey using willingness-to-pay methods. A key reason for that is a broad philosophy of science perspective suggesting that such surveys may reflect a failed research programme (or, at a minimum, a need for greater humility about results from such surveys). 
  2. New Zealand should stop using the 2007 valuation of lives lost due to fire (related but lower than the transport valuation). Data collection for that valuation was not adequate to justify the valuation.  
  3. There are good reasons for different countries to draw different conclusions. For example, Australian transport agencies have stronger reasons to conduct a major Value of Statistical Life survey using willingness-to-pay methods.
Contributors
avatar for Charles Sullivan

Charles Sullivan

Charles Sullivan & Associates
Quant & mixed methods. Managing challenging evaluations. Administrative data,  Integrated Data Infrastructure. Wellbeing.
avatar for John Wren

John Wren

Education NZ

20221014 ANZEA Sullivan Valuing life and death v5a pptx
Monday October 10, 2022 4:00pm - 4:30pm NZDT
Oceania
  Presentation, Celebrating

4:00pm NZDT

D.01 Kaupapa Māori Evaluation: Celebrating the ordinary
Culture informs our understanding of our place in our world, but when we think about culture, we are often only thinking about the minoritised ‘other’ or about ‘other people’ in other places and how they are different from ‘us’. This view of culture leaves ‘whiteness’ unnamed and unquestioned. So when white colonisers arrive on your shore and start stealing your lands and appropriating your resources, it is perhaps unsurprising that you – the first peoples of that land – are first seen as ‘other people’ before you become minoritised and the ‘other’ in your own home place. Before this happened you considered yourself to be ordinary and so understood why these newcomers wanted to call you maori.

You treatied with the coloniser, seeking to remain ordinary and in control of the resources that sustained you. But the hope of treaty promises soon faded as theft, war and the exertion of control by colonisers pressured you to forgo your culture and accept theirs. You met this colonial insistence with decades of unrelenting resistance which, in the 1990s, included growth in the delivery of your own services and programmes.

It was then that you encountered evaluators, wanting to know if the people you had served had ‘improved’. Very quickly it became apparent that you were now in danger of insisting rather than resisting, and so dug your heels into your ordinariness to hold fast. What was required to change was not you but the accountability system that reflected back to funders how you had performed with the monies provided. The challenge of this for evaluation is how to re-present a strengthening of ordinariness that affirms culture, reconnects people to their home places, and offers the promise of people living good lives ‘being Māori’ and ‘doing Māori’.

This decolonisation of evaluation will be discussed.
Contributors
avatar for Fiona Cram

Fiona Cram

Katoa Ltd
Fiona Cram is a Maori (indigenous) woman from Aotearoa New Zealand. Her tribal affiliations are to Ngati Pahauwera on the east coast of Aotearoa. Fiona is the mother of one son.Fiona has a PhD in social and developmental psychology and is a widely published author of evaluation books... Read More →

Monday October 10, 2022 4:00pm - 4:30pm NZDT
Icon
  Presentation, Into the future

4:00pm NZDT

B.01 The CERA story: Reclaiming ethnic community evaluation in Aotearoa
Aotearoa has several small research centres that focus on ethnic communities. These exist within universities as well as independent organisations. In the last decade, there has also been an increase in community and grassroots initiatives that collate ethnic community research and evaluation undertaken by ethnic communities. However, a gap exists for a dedicated ethnic community-led research and evaluation centre committed to capacity building and producing ethnic community research and evaluation.

Research conducted in 2022 to get community perspectives and responses to establishing an independent ethnic community-led research and evaluation centre in Aotearoa found that establishing a centre would be invaluable to communities, government, NGOs, and policy makers.

Benefits include:
  • Enabling innovative methodologies that foster cultural knowledge and expertise 
  • Providing employment opportunities for ethnic researchers and evaluators 
  • Providing a space where ethnic research and evaluation are accessible. 

CERA is now in the establishment phase. This independent organisation and network is digging deep and engaging widely on reclaiming ethnic research and evaluation.

This workshop is about what it means to be ethnic community-led and what an ethnic community-led research and evaluation centre that honours Ti Tiriti o Waitangi looks like?
Contributors
avatar for Tayo Agunlejika

Tayo Agunlejika

Litmus / EkoCabs
Tayo has 15 years’ experience in the Ethnic Communities sector. He has excellent skills at engaging and relating with people of different ethnicities, races, and religions. He is a trustee of the Centre for Ethnic Research Aotearoa (CERA) and a director of Ekocabs.   Tayo was... Read More →
avatar for Sandar Duckworth

Monday October 10, 2022 4:00pm - 4:40pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Workshop/wānanga/talanoa, Into the future

4:00pm NZDT

C.01 If evaluation is a science, then robust evaluation uses a single core methodology - with plenty of contextual permutations
At its simplest form, robust evaluation is evaluation that has a clear and logical set of questions, an explicit evaluation framework that supports value judgments, and a transparent, defensible, analysis that is shared via a reporting process or report. So, what do robust questions, evaluation frameworks and reporting look like?

This workshop will explain what good looks like, and how these three components are each used to guide robust evaluation. It will also describe various common pitfalls and their solutions with some practical tips and tricks to help recognise and avoid them.

This will be useful for evaluators and those managing evaluations, with some prior experience. Good questions, frameworks and reporting look like:

  1. A concise set of evaluation questions, that includes priority questions, and some evaluative (not only descriptive) questions. Each are answered in the evaluation reporting.  
  2. An evaluation framework that describes evaluand and its intent, and provides agreed definitions of ‘good’ or what is valued and (where applicable) references standards.  
  3. A reporting process or report that makes explicit and defensible conclusions that are acceptable and believable and are therefore useful for decisions about proving, improving, expanding, or ceasing evaluand activities.  
Good use of an evaluation framework includes using it as a lens to guide the evaluation design, to refine the evaluation throughout and to support analysis, sense-making and value judgements. Robust reporting is an explicit and defensible presentation of findings and judgements. Each finding is based on cohesively presented evidence. Judgements build on these, and make explicit use of the evaluation framework. A number of common pitfalls that will be described and solutions shared. Three examples are:

a) Asking too many or too vague evaluation questions.
b) Not defining ‘good’ or what is valued, even in draft form.
c) Making indefensible conclusions or judgments.
Contributors
avatar for Anne Dowden

Anne Dowden

Anne Dowden REWA
Anne is an evaluation consultant. She runs her own consultancy providing services to Governmentagencies and other entities. Anne collaborates and partners with other evaluation experts for mostevaluation projects.Anne is part of a large Australian-New Zealand family and has a partner... Read More →

Monday October 10, 2022 4:00pm - 5:30pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Workshop/wānanga/talanoa, Developing

4:30pm NZDT

A.02 Sth Korean/NZ school students’ voices and learning outcomes on their virtual exchange
ENZ partnered with Learn English Live (LEL), Seoul and Gangwon education authorities to establish a school virtual exchange programme. The programme fosters opportunities for Korean 13–14 year olds to have English language learning, cross-cultural relationships, and NZ students in low decile schools to engage with their international peers.

This was done through Zoom sessions using a scaffolded pedagogical approach to learning (such as quizzes and other interactive activities) where students could share aspects of their life. Facilitated small, student led discussion groups for direct student engagement with peers is a key feature of the programme.

Two cohorts have completed the programme involving 785 Korean students and 334 NZ students. A combined Korean/NZ language survey was administered to all students on completion.

Key insights from the survey respective results and thematic analysis presenting student voices will be presented about learning outcomes achieved, cultural awareness, features of the programme they enjoyed and suggestions for improvements will be shown.
Contributors
avatar for John Wren

John Wren

Education NZ

Monday October 10, 2022 4:30pm - 5:00pm NZDT
Oceania
  Presentation, Celebrating

4:30pm NZDT

D.02 From Taumarunui to Whanganui + places in between: 3 challenges encountered by a Pākehā evaluator along the way
In this session, I will discuss three challenges that stand out for me about what it means to be a Pākehā evaluator in Aotearoa New Zealand.

None is ‘rocket science’ nor even particularly ‘new’; all though have been – and continue to be - critical to how I aim to ‘do’ evaluation and live in the world more generally.

The session title speaks to time and place; to a journey both through the years and across the landscape. In 1980s Taumarunui, I was part of a women’s collective that established a women’s refuge and rape crisis centre at a time when none existed between Hamilton and Whanganui. It opened my eyes to how much more there is always to learn about place, about histories and about ‘making a positive difference’ through mahi including evaluation.

Despite being from Whanganui, the only thing I knew about any connection the town had with Taumarunui was State Highway 4. Despite the refuge and rape crisis centre sharing a small office building with an esteemed Iwi leader and staunch Whanganui Awa advocate, all I saw was a stern elderly man who seldom raised a smile. Despite advocating for setting up Te Kākano o te Whānau, my appreciation of intergenerational trauma for Māori was non-existent. Racism, though a critical component of my structural analysis, had no impact on my day-to-day life as a Pākehā. So why is any of this relevant to evaluation? One of the few things I am sure about when it comes to evaluation is that as an evaluator I am the evaluation tool – no fancy technology and no precision engineering to call upon here! To be up for the evaluation challenge, the first thing I need to actively engage with is what it means for my evolving evaluation practice to be Pākehā.
Contributors
avatar for Lynley Cvitanovic

Lynley Cvitanovic

Researcher, Whakauae Research Services Ltd
Born and brought up in Whanganui (Aotearoa New Zealand), I am fifth generation Pākehā of Croatian, English and Irish descent. I joined Whakauae in 2008, as a researcher and evaluator, after spending 25 years in service delivery and middle management roles in the public health (health... Read More →

Monday October 10, 2022 4:30pm - 5:00pm NZDT
Icon
  Presentation, Into the future

4:40pm NZDT

B.02 Te Koha (the gift): An iwi (tribal) approach to scaling and its implications for evaluation
Scaling a successful programme or initiative is generally thought of as consolidating or deepening of the initiative, by increasing intensity or volume in the current sites or extending its reach to new communities.

Decisions about scaling of programmes are often made by funders and implementers and not the proposed recipient communities. For Māori, this is problematic because it sees scaling as transactional, divorcing the programme from its creators and origins and assumes scaling is a finite process. Te koha (gifting) is a tapu (sacred) process. Giving something of significance to another reflects mana (respect), strengthening ties and relationships and carried reciprocal obligations.

Tūārai is a whānau-centred injury prevention model of practice, embedded in the notion of holistic, whānau wellbeing. It is being developed and tested by three iwi from the Tairāwhiti. One aspect of its development is to explore scaling Tūārai. Tūārai has developed a set of ‘scaling’ principles ‘Te Koha’ that are sourced from matauranga Māori and matauranga-ā-iwi Māori; ngā kaupapa tuku iho, inherited wisdom handed down from tupuna (ancestors). Tūārai is applying and refining Te Koha as their preferred approach for gifting or sharing their programme with other hāpori (communities) and hapū.

In this panel we will share the journey to date in developing Te Koha and the implications for evaluating programmes intending to scale or in the implementation process.
  • Why a different approach to scaling, how did it come about?  
  • What are the initial set of scaling principles, what are the tikanga and matauranga Māori that underpin it?  
  • How is Te Koha being implemented?  
  • What are the considerations for evaluators, evaluation and programme funders?  

All four members of the evaluation team, three with tribal links to the Tūārai tribes, will present on Te Koha.
Contributors
avatar for Nan Wehipeihana

Nan Wehipeihana

Director, Weaving Insights
Nan is the director of Weaving Insights (www.weavinginsights.co.nz) and a member of the Kinnect Group (www.kinnect.co.nz). Nan is a founding member of the Aotearoa New Zealand Evaluation Association (ANZEA) and Ma Te Rae, Māori Evaluation Association – the first Indigenous Eva... Read More →
avatar for Huhana Moselen
avatar for Kahiwa Sebire

Kahiwa Sebire

Kahiwa Sebire is a puzzling pattern-spotter, an enthusiastic solution finder and a life-long learner. Exploring possibilities and meaning-making with sticky notes and whiteboards (or their digital siblings) in tow.  Kahiwa Sebire has over 10 years’ experience working in and with... Read More →
avatar for Kataraina Pipi

Monday October 10, 2022 4:40pm - 5:30pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Workshop/wānanga/talanoa, Into the future

5:00pm NZDT

A.03 Global Competency Certificate programme– how do we know it is successful in achieving its aims?
Global Citizens is one of three goals in the NZ International Education Strategy 2018-2030. ENZ entered a partnership with Massey University and AFS (formerly American Field Scholars) to design and deliver the Global Citizens Certificate (GCC). The GCC comprises 18 modules and four live facilitated dialogue sessions and has been branded for NZ. The course aims for participants to:

  • Become more self-aware
  • Increase awareness of others
  • Learn how to manage emotional responses
  • Learn how to bridge across differences. ENZ Insights was asked to undertake a small evaluation project to understand the outcomes. Objectives included understanding:
  • The level of change in knowledge about global citizenship pre and post the certificate training
  • The level of change in ability to act as a global citizen pre and post the certificate training
  • What was the most important thing they learnt
  • Satisfaction with the course
  • Areas where improvements could be made in the course. A short nine question online survey was sent to all enrollees. Respondents received an email inviting them to take part after the course, and a follow-up email four weeks later. Analysis was undertaken at the total level (all responses excluding missing responses). Comparisons were made between subgroups comprising the three different types of course. The results indicated that the initial 2021 GCC programme was delivering on its target volumes and intended course outcomes. Survey respondents regarded the course very positively and reported substantially increased levels of knowledge about global citizenship and their ability to act due to the course. Thematic analysis of free text fields shows many respondents said they learnt:
  • Increased self-awareness,
  • How cultures differ from their own
  • The importance of communication skills and techniques Improvement areas were identified, and recommendations made.
Contributors
avatar for John Wren

John Wren

Education NZ

Monday October 10, 2022 5:00pm - 5:30pm NZDT
Oceania
  Presentation, Celebrating

5:00pm NZDT

D.03 Bicultural evaluation: critical moments in our (ongoing) journey
We will share our journey as Māori and Pākehā evaluators of a bicultural education programme.

This firmly speaks to Stream 3 of the conference, “celebrating our work of today, talking about what we have learnt and challenges being faced doing evaluative work”. Te whai hua – kia ora, Sorted in Schools is a financial capability programme for secondary school students, led by Te Ara Ahunga Ora Retirement Commission. The programme is available for English-medium schools and Māori-medium kura. Te whai hua – kia ora resources are underpinned by te reo Māori, tikanga Māori, and mātauranga Māori. They are not a translation of English medium resources available in Sorted in Schools.

We are evaluating the complete programme Te whai hua – kia ora, Sorted in Schools within one evaluation. The evaluation is designed as a mixed-methods adaptive evaluation. The Māori medium strand is underpinned and guided by Kaupapa Māori methodology. Now in our third year, we have learned much along the way.

This presentation will focus on critical moments in our journey as a team, and with our clients (also Māori and non-Māori).
This includes:
  • Evaluation framework and criteria development in a bicultural evaluation and how this has shifted significantly over the three years 
  • Use of pattern spotting as an analytical tool to surface key findings within and across Te whai hua – kia ora and Sorted in Schools. 
If we have time, we will also talk about making evaluative judgements across both programmes, including the challenge of navigating differences, and ways we have woven these together in a single report. This evaluation is influencing change in financial capability education, as Te Ara Ahunga Ora are very responsive to findings reported in each phase. It is also influencing our practice as evaluators. We look forward to sharing our work with you.
Contributors
avatar for Jo MacDonald

Jo MacDonald

New Zealand Council for Educational Research (NZCER)
avatar for Nicola Bright

Nicola Bright

Kairangahau Matua, NZCER

Monday October 10, 2022 5:00pm - 5:30pm NZDT
Icon
  Presentation, Celebrating

5:30pm NZDT

Pasifika Fono
Details will be provided. 

A light supper will be provided.
Monday October 10, 2022 5:30pm - 6:30pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Conference Activity, Informal occasion

5:30pm NZDT

Tauiwi Forum: NZ’s growing ethnic diversity & its implications for evaluators – a forum for ethnically diverse evaluators
A light supper will be provided.

This forum is a unique opportunity for participants who wish to engage in a wide-ranging discussion on the growing ethnic diversity of our population and what this means for our evaluation practice. Evaluators from different cultural and ethnic backgrounds are invited to share their evaluation journey and how they are taking a bicultural treaty lens to their evaluation practice.
Contributors
avatar for Meenakshi Sankar

Meenakshi Sankar

Director, Buddhi Consultancy Ltd

Monday October 10, 2022 5:30pm - 6:30pm NZDT
Te Huinga Centre - Rangimarie Room 3
  Conference Activity, Informal occasion

5:30pm NZDT

Pākehā Forum
The Pākehā forum is open to all Pākehā attendees.

In the forum, we create a space to think together and discuss what it means to be a ‘Pākehā’ evaluator. Since our first forum in 2014, our discussions have covered who and what is included in the term ‘Pākehā’, our partnership responsibilities in terms of Te Titiri o Waitangi and decolonization, and the rights, roles and responsibilities of being Pākehā in relation to our evaluation practice. The forum provides an opportunity, in a safe space, for these discussions.

In 2022 we will re-present a draft ANZEA Pākehā position statement, first introduced at the 2018 Pākehā forum. Participants will then engage in small group discussions about current changes, nationally and internationally, with respect to Te Tiriti, indigenous rights, equity and inclusion, and explore how we make these relevant and important as Pākehā evaluation professionals and ANZEA members.

A light supper will be provided.
Contributors
RT

Rae Torrie

Evaluation Works
DB
avatar for Kate McKegg

Kate McKegg

Director, The Kinnect Group
Kate has specialist skills in supporting evaluative thinking and practice in complex settings where people are innovating to create systems change. She has been applying these skills for over 25 years in government, non-government, philanthropic and community contexts, including many... Read More →

Monday October 10, 2022 5:30pm - 7:00pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Conference Activity, Informal occasion

5:30pm NZDT

Hui Māori
Detail TBC

A light supper will be provided.
Monday October 10, 2022 5:30pm - 8:30pm NZDT
Icon
  Conference Activity, Informal occasion
 
Tuesday, October 11
 

8:00am NZDT

Registration desk open
Tuesday October 11, 2022 8:00am - 8:50am NZDT
Oceania
  Conference Activity, Informal occasion

8:50am NZDT

Welcome and introduction to day
Tuesday October 11, 2022 8:50am - 9:00am NZDT
Oceania
  Conference Activity, Informal occasion

9:00am NZDT

Featured Contributors 2: Dr. Jesse Kokaua
Contributors
avatar for Dr. Jesse Kokaua

Dr. Jesse Kokaua

Senior Research Fellow, Otago University
Dr Jesse Kokaua is a Cook Islands statistician and senior research fellow in the Va’a O Tautai – Centre for Pacific Health in the Division of Health Sciences at the University of Otago, Dunedin. Dr Kokaua has made a significant contribution to advancing the Pacific Strategic Framework... Read More →

Tuesday October 11, 2022 9:00am - 10:00am NZDT
Oceania
  Featured contributor

10:00am NZDT

Morning tea break
Tuesday October 11, 2022 10:00am - 10:30am NZDT
Oceania
  Conference Activity, Informal occasion

10:30am NZDT

C.02 Building the plane whilst we fly it! Designing the evaluation of Single Site Supportive Housing at 139 Greys Avenue
139 Greys Avenue replaces a 1960s 12-storey public housing building no longer serving customers well. Kāinga Ora recognised an opportunity to construct 276 new units to address the housing deficit in central Auckland using a single site supported housing (SSSH) model, with mixed-use commercial, mixed-tenure residential and community development to enhance customers, community and staff opportunities and wellbeing.  

Mana whenua, Ngāti Whātua Ōrakei, gifted the name Programme Raranga to the service and operating model for 139 Greys Avenue. The name means ‘weaving together’ and reflects the multi-dimensional approach to the development, including the breadth of interventions and interests, informing, serving and ultimately planning to occupy and work there. Prospective new residents include 80 former rough sleepers offered a ‘housing first’ opportunity with wraparound medical, social support and community development opportunities. The remaining public housing will be for customers with low to medium needs, identified as suitable through a place-based placement pilot. The remaining 76 apartments are allocated to market rentals, a first for Kāinga Ora.

The evaluation aims to track the rationale and assumptions underpinning the interventions through a process evaluation. Additionally, the approach will evaluate implementation and impacts on customers, community and staff over a 10-year study period.  The real-time design evaluation allows each decision and its rationale to be captured as an impact assumption. This major strength is also a challenge, as many choices remain un-finalised, and co-dependencies are unclear; furthermore, causal effects are hard to attribute with the simultaneous implementation of multiple interventions. The approach adopts a theory of change model to track anticipated impacts and outcomes, against a human-centred design model measuring success relative to the customers’ personal goals and against the design philosophy of Te Whare Tapu Whā.
Contributors
avatar for Jennifer Joynt

Jennifer Joynt

Kainga Ora
I am an environmental social scientist interested in the relationship between society and the built environment and how we adapt our social and physical settings to improve health and wellbeing outcomes, particularly for the most marginalised in society.  My applied and research... Read More →

Tuesday October 11, 2022 10:30am - 11:00am NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Celebrating

10:30am NZDT

B.03 Footprints Evaluation
Contributors
avatar for Andy Rowe

Andy Rowe

Dr, Footprints Evaluation
Andy has a PhD from the London School of Economics, is a Fellow and former President of the CES and an early contributor to evaluating sustainability and mainstreaming sustainability in evaluation. He initiated the CES effort to mainstream sustainability and is a founding member of... Read More →
avatar for Dr. E. Jane Davidson

Dr. E. Jane Davidson

Dr. E. Jane Davidson is an internationally recognized, award winning evaluation author, keynote speaker, practitioner, and coach. She was the 2019 recipient of the American Evaluation Association’s prestigious Paul F. Lazarsfeld Evaluation Theory Award.Jane is best known for pioneering... Read More →

Tuesday October 11, 2022 10:30am - 11:10am NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Into the future

10:30am NZDT

A.04 Disrupting performance and accountability with evaluative thinking and practice: An iwi initiative to shift mindsets and practices and shine a light on what it means to live by, and be accountable to Ngāi Tahu values and principles.
In 2020, Te Rūnanga o Ngāi Tahu (the office established by the Te Rūnanga o Ngāi Tahu Act 1996 to serve, protect, and advance the collective interests of the iwi) set out to develop a new approach to performance assessment and reporting. Te Rūnanga o Ngāi Tahu have for a long time been monitoring activity and performance by using output focused KPIs and targets. However, these approaches to performance monitoring, assessment and reporting were not hitting the mark. They did not support the organisation to meaningfully capture and report on the value of the investment and work being done in ways that reflect what is valued and important to Ngāi Tahu whānui.

Te Rūnanga o Ngāi Tahu want to do more than just monitor performance. They want to be able to tell an authentic and meaningful performance story steeped in what is valuable and worthwhile at whānau, rūnaka, iwi, programme, project, and service levels. They also want these performance stories to be rich and robust enough to inform future investment decisions.

Te Rūnanga o Ngāi Tahu is developing a framework and practice model that departs from standard corporate practice around the measurement of performance. It is a values-oriented approach; the premise of which is that the cultural values of Ngāi Tahu are central to the conception of what good performance looks like. The framework is explicitly evaluative; combining a mix of evidence with cultural values and a deliberative process of evaluative reasoning to reach judgements about performance.

The implementation of the Ngāi Tahu Evaluation Framework is a significant undertaking.  A developmental, learn-as-you-go approach to building and implementing infrastructure, system and processes has been taken. They are developing a system that is a good fit and grounded in the everyday realities of the work of Te Rūnanga o Ngāi Tahu.
Contributors
IP
PP
CP

Caralyn Purvis

Evaluation Specialist, Te Rūnanga o Ngāi Tahu
avatar for Kate McKegg

Kate McKegg

Director, The Kinnect Group
Kate has specialist skills in supporting evaluative thinking and practice in complex settings where people are innovating to create systems change. She has been applying these skills for over 25 years in government, non-government, philanthropic and community contexts, including many... Read More →
DB
CI

Tuesday October 11, 2022 10:30am - 11:10am NZDT
Oceania
  Workshop/wānanga/talanoa, Celebrating

10:30am NZDT

D.04 Evaluative synthesis: pitfalls and tips
Evaluative synthesis is the culmination of the eval reasoning process. It brings together evidence and values about whatever is being evaluated to make an evaluative judgement. Getting evaluative synthesis right is an ethical responsibility for evaluators because evaluation is used to make decisions about ongoing funding, expansion or other changes. Evaluation synthesis can be challenging but is not discussed much in the literature or in professional development opportunities. Yet it is explicitly mentioned within two domains of the Aotearoa New Zealand competencies.

This extended presentation will focus on what it takes to do competent reasoning at the synthesis stage of an evaluation, based on our experience of teaching emerging evaluators in a Masters-level course on evaluation practice. We will cover:

  1. What is evaluative synthesis and why is it important? 
  2. What are common pitfalls?  
  3. Practical tips and resources for doing it well. 
Contributors
avatar for Mathea Roorda

Mathea Roorda

Senior consultant, Allen + Clarke
Values inform what matters, and are at the heart of evaluation. You literally can't get to an evaluative judgement without them. I'm interested in approaches to systematically identifying what matters, and for whom. Come talk with me about the values identification matrix (VIM) I... Read More →
avatar for Christine Roseveare

Christine Roseveare

Lecturer, Massey University
avatar for Amy Gullickson

Amy Gullickson

Associate Professor, Director, Centre for Program Evaluation, The University of Melbourne
Associate Professor Amy Gullickson is the Director of the University of Melbourne Centre for Program Evaluation, which has been delivering evaluation and research services, thought leadership, and qualifications for more than 30 years. She is also a co-founder and current chair of... Read More →

2022.10.10 D.04 EvaluativeSynthesis Pitfalls&Tips Final 1example pdf
Tuesday October 11, 2022 10:30am - 12:00pm NZDT
Icon
  Workshop/wānanga/talanoa, Reflecting

11:00am NZDT

C.03 Teasing apart the impacts of 'mixed tenure': A draft monitoring and evaluation plan
The mandate of Kāinga Ora – Homes and Communities covers public housing, affordable housing and urban development.  Each involves diverse and complex social processes that interact with each other, making it difficult to measure impact.  

Recent research into the community impacts of mixed tenure (the mix of public, affordable and market housing), a key component of Kāinga Ora large-scale redevelopments, illustrates this challenge.

Understanding mixed tenure has required understanding social mix (the mix of tenure type, age, ethnicity etc), public housing concentration (the proportion of public housing versus private housing), intensification (increased density of housing) and housing typology (i.e. apartments, detached houses vs standalone houses) and a number of other interacting factors that shape communities. Not only are these factors difficult to separate from each other, but they are not always experienced in the same ways by all residents, depending on their tenure type, ethnicity, age, level of disability etc. Moreover, different parts of the organisation prioritise on different cohorts of residents, making agreement about an evaluation focus difficult.

This presentation presents a draft monitoring and evaluation plan attempting to tease out these different components of ‘mixed tenure’ at multiple levels to enable the regular reporting of digestible findings for the different parts of the business, while also telling us something about the impact of redevelopment (and all its constitutive components) as a whole.  Feedback will be required!
Contributors
avatar for Louise Humpage

Louise Humpage

Kainga Ora

Tuesday October 11, 2022 11:00am - 11:30am NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Celebrating

11:10am NZDT

A.05 He taonga rongonui, te aroha ki te tāngata | Actioning the capacity of aroha for positive change through evalautive leadership
He taonga rongonui, te aroha ki te tāngata | Kindness towards others is a precious treasure.

Receiving the call to action from Evaluation with Aroha, Louise seeks to share her journey to re-member and re-claim our ability to lead through evaluation, by actioning the capacity of aroha. Responding to wero this conference is making explicit, in particular, do evaluators still have a role as agents of social change or is this a blurring disciplinary rigour and was always going to be a challenge?

Louise will bring forward insight from her Masters journey, that is beginning to bring this provocation into focus. Fundamentally values drive approaches to leadership, just as much as our values underpin our evaluation practice and positioning. At the intersection between the theories and practice of leadership and evaluation lie the potential mobilisers or instruments for change – evaluators. Where evaluation is frequently required to “be a rhythmic alternation between attacking the causes and healing the effects” of colonialism and inequity, exploring evaluators as instruments of change is critical. However, leadership in evaluation is unspoken, implicit; so, we must lean into this ‘silence’ by resonating with the hearts and minds of those who have expressed their commitment to those they serve, and their global evaluation community.

There are opportunities to “awaken the potential of ourselves, others and situations and to then consciously manifest that potential” through evaluative leadership - a state of being that leads to action through evaluation - by actioning the capacity of aroha to contribute to positive change.  This is an invitation to wayfind; hear insight into actioning the capacity of aroha, and the continuum of evaluative leadership, and space for reflective practice.
Contributors
avatar for Louise Were

Louise Were

Hikitia | Member of the Tuakana Teina Evaluation Collective; PhD Candidate
Louise is a Māori evaluator who seeks to bring together Te Ao Māori, evaluation, design and systems thinking. Louise specialises in evaluation and policy analysis, with her professional background stretching acrossMāori, public health, disability, and community contexts, within... Read More →

Actioning the capacity of Aroha ANZEA 2022 pdf
Tuesday October 11, 2022 11:10am - 11:50am NZDT
Oceania
  Workshop/wānanga/talanoa, Celebrating

11:10am NZDT

B.04 Building capability to shift evaluation practice into organisations
Evaluation is often commissioned to demonstrate the value of a programme or service to a funder. An external evaluator is often brought in as an ‘independent’ expert to understand the value of an initiative.

However, this approach to evaluation reinforces a centralized or top-down power structure. The funding has been received from the top, and those receiving the funding must demonstrate their value. When the value of evaluation could be enhanced if it was more closely driven by the organisations and the communities themselves.

Building evaluation capability with organisations provides an opportunity to put evaluation practice at the heart of organisations decision-making. Shifting the power and influence of evaluation from a top-down approach to a more locally driven activity. Some initial examples of how we are working to support this will be shared during our presentation. We will also facilitate a discussion to explore some of the challenges, opportunities, and trade-offs of working in this way.  
Contributors
avatar for Sarah Appleton-Dyer
avatar for Nishadie Edirisuriya

Nishadie Edirisuriya

Principal Consultant, Synergia
avatar for Sarah Andrews
avatar for Dr Stella Vickers

1110 B.04 Building evaluation capability to shift evaluation into organisations 10 October 2022 FINAL pdf
Tuesday October 11, 2022 11:10am - 12:00pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Workshop/wānanga/talanoa, Celebrating

11:30am NZDT

C.04 Evaluating the redevelopment of Eastern Porirua
Eastern Porirua consists of eight neighbourhoods with approximately 18,000 residents and 2000 public houses. High concentration of poor quality housing, disadvantaged households and the long-term lack of investment since the 1970s contributed to producing poor social outcomes.

Redevelopment in Eastern Porirua will increase the social housing stock, build 1500 new homes of varing typologies and reduce the public housing concentration. This collaborative project also includes investment in infrastructure and social services. The redevelopment provides an opportunity to influence the housing, social, health and economic outcomes for its residents.

We are evaluating the outcomes of Eastern Porirua’s redevelopment, exploring how issues of construction, relocation, infrastructure enhancement affect residential wellbeing. We will present our longitudinal mixed methods approach to evaluating wellbeing outcomes of residents. We focus on the outcome areas realted to Housing, Health, Transport, Environment and Subjective Wellbeing, which will be investigated via a resident community survey, qualitative interviews with community stakeholders (e.g., kaumātua, leaders from various institutions) complimented by quantitative data obtained by our internal records and national administration data from the IDI. Collectively, all three research approaches offer a holistic picture of how the redevelopment influences the wellbeing of residents in Eastern Porirua.
Contributors
DM

David Mayeda

Kainga Ora
DH

Tuesday October 11, 2022 11:30am - 12:00pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Celebrating

12:00pm NZDT

Lunch break
Lunch Break 
Tuesday October 11, 2022 12:00pm - 12:50pm NZDT
Oceania
  Conference Activity, Informal occasion

12:20pm NZDT

Lunch discussion - Meet & Greet of ANZEA Board
Tuesday October 11, 2022 12:20pm - 12:50pm NZDT
Oceania
  Conference Activity, Informal occasion

12:50pm NZDT

Panel presentation 1: Building on the experience of Māori evaluation leaders to create a flourishing kaupapa Māori evaluation future
Our practice as kaupapa Māori evaluators has been grounded in kaupapa Māori research practices and tikanga Māori.  However, we are not a static profession, always developing and refining practice within our own contexts and with a purpose of supporting Māori development. We need to continue to develop our profession by sharing our whakaaro, experiences and practices, through collective kōrero and wānanga.

We seek to build a future for what kaupapa Māori evaluation is, what it means for us as Māori, and how we want to develop it going forward. Looking back to move forward was a key approach and highlight of my thesis research. I did this by both drawing on the kaupapa Māori evaluation literature through a systematic literature review, and interviewing kaupapa Māori evaluation leaders about the practice of kaupapa Māori evaluation and the creation of evaluation frameworks. I am honoured to share some of the findings from this research with the wider evaluation community, along with a panel of Māori evaluation leaders who will provide their whakaaro around how we, as Māori, might build on our practice.

It is through a collective approach, and whakawhiti kōrero that we will make steps towards creating a flourishing future for kaupapa Māori evaluation.
Contributors
avatar for Debbie Goodwin

Debbie Goodwin

DBZ CONSULTANCY LTD
Debbie Goodwin is the Co-Chair of EvalIndigenous from Maori tribe in New Zealand. She is a PhD fellow and member of the Ma te Rae Evaluation Association. She led the organisation of the first ever Indigenous evaluation on the theme "Indigenous Peoples' Conference on Evaluation" which... Read More →
avatar for Kataraina Pipi
avatar for Fiona Cram

Fiona Cram

Katoa Ltd
Fiona Cram is a Maori (indigenous) woman from Aotearoa New Zealand. Her tribal affiliations are to Ngati Pahauwera on the east coast of Aotearoa. Fiona is the mother of one son.Fiona has a PhD in social and developmental psychology and is a widely published author of evaluation books... Read More →
LP
avatar for Nan Wehipeihana

Nan Wehipeihana

Director, Weaving Insights
Nan is the director of Weaving Insights (www.weavinginsights.co.nz) and a member of the Kinnect Group (www.kinnect.co.nz). Nan is a founding member of the Aotearoa New Zealand Evaluation Association (ANZEA) and Ma Te Rae, Māori Evaluation Association – the first Indigenous Eva... Read More →
avatar for Dr. Amohia Boulton

Dr. Amohia Boulton

Research Director, Whakauae Research Services
Dr Amohia Boulton (Ngāti Ranginui, Ngai te Rangi, Ngāti Pukenga, Ngāti Mutunga and Te Āti Awa o te Waka a Māui), is the Director of Whakauae Research Services Ltd, the only tribally owned and mandated health research centre in Aotearoa/New Zealand. Amohia also holds Adjunct Professor... Read More →

Goodwin, D W T. ANZEA Conference presentation 11.10.22 pdf
Tuesday October 11, 2022 12:50pm - 1:50pm NZDT
Oceania
  Panel presentation, Into the future

1:50pm NZDT

B.05 Examining the practice of synthesis: robust evaluative reasoning in dynamic evaluation contexts
Evaluative reasoning is critical to any effective evaluation process. It is fundamental to establishing the value of programmes and institutions we evaluate and has been described as “the ‘core’ of our discipline, the essential piece that makes or breaks evaluation” (Davidson, 2014, p. 32).

To ensure it is rigorous and valid we need to understand how evaluative reasoning works in practice. This is especially the case in fast changing contexts, often fraught with competing interests and tensions, and where the methodology involves teams of evaluators. This presentation focuses on research undertaken within the ‘rough ground’ of evaluation practice.

The research aimed to understand the synthesis process embedded in the context of the Education Review Office’s (ERO) review practice in schools. ERO is the NZ government agency tasked with reviewing the performance and assessing the quality of pre-tertiary education providers in relation to the educational services they provide. The research examined the synthesis process in practice - how evidence was used, the iterative nature of reasoning, and the factors that impacted on sound judgements and decision-making. Methods used to get insight into this process included: a practitioner questionnaire; four case study observations of the review process in action; and follow-up interviews with the evaluators.

This presentation explores some of the strengths and challenges in undertaking a fast-paced evaluation team synthesis. It highlights the tensions between maintaining relationships and transparency of evaluation reasoning within the evaluation team and with stakeholders. Findings highlight the importance of relational trust within teams, the use of frameworks for discussion, documentation of team deliberations, stakeholder involvement and evaluator competencies in promoting evaluative thinking. The subsequent development of criteria for effective synthesis to make evaluative reasoning processes more visible in practice will be shared as a basis for discussion.

Reference   Davidson, E. J. (2014).  How “beauty” can bring truth and justice to life. In J. C. Griffith and Montrosse-Moorhead (Eds.). New Directions for Evaluation, 142, 31-43
Contributors
avatar for Ruth Pritchard

Ruth Pritchard

Senior Evaluator, Education Review Office

Tuesday October 11, 2022 1:50pm - 2:20pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Celebrating

1:50pm NZDT

C.05 Shaping future transport sector reform through evaluation learning
Rapid transformation of urban environments is needed to create safe, healthy, and liveable towns and cities and to support the reduction of transport carbon emissions (Keall et al., 2018). In Aotearoa, the Emissions Reduction Plan (Ministry for the Environment Manatū Mō Te Taiao, 2022) seeks a 20% reduction in vehicle kilometres travelled by 2035 and a 75% increase in the proportion of journeys under 5km using active transport. Enhancing the safety of walking and cycling will be essential for supporting these shifts (Waka Kotahi NZ Transport Agency, 2019).

Launched in 2018 by Waka Kotahi, Innovating Streets has contributed to the above by seeking to make it easier and quicker to undertake street space reallocation projects for active mobility. Working collaboratively with communities, the programme supported the delivery of temporary street space projects through a tactical urbanism and codesign approach (Barata & Fontes 2017; Dube 2009; Lydon & Garcia 2015). Such approach allows solutions to be demonstrated, trialled, and improved in-situ, and can bring forward benefits, build support for change, and enhance permanent solutions.

Between 2020-21, IS funding supported the delivery of 62 temporary street space reallocation projects across Aotearoa. A total of 89kms of interim street treatments was delivered through projects such as temporary cycleways, safe crossings, parklets, low traffic neighbourhoods, playstreets, and road art.
Mackie Research lead the evaluation of the IS programme and supported the evaluation of projects locally (Blewden et al, 2022). Project evaluation showed temporary treatments can accelerate safety, accessibility, and liveability benefits, while the programme evaluation derived significant programme design learning. All learning has informed Streets for People (SfP) (2022-24), the next iteration of Innovating Streets.
Important learning was also gained in respect to evaluating innovative transport reform programmes like Innovating Streets and now Streets for People. This has shaped the Streets for People evaluation framework, developed collaboratively by Mackie Research and Waka Kotahi. This presentation will discuss how the design and approach to the Streets for People evaluation has evolved and how this is expected to enhance the value and utility of evaluation at project and programme levels respectively.

References
Barata, A., & Fontes, A (2017). Tactical Urbanism and Sustainability: Tactical Experiences in the Promotion of Active Transportation. World Academy of Science, Engineering and Technology International Journal of Urban and Civil Engineering Vol:11, No:6.
Blewden, M., Raja, A., Nord, A., Hawley, G., & Gilbert, K. (2022). Innovating Streets Fund Evaluation: Final Report [Report prepared by Mackie Research and Consulting for Waka Kotahi].
Dube, P. (2009). Actions: What You Can Do with the City. C Mag C Magazine 102 (102): 47–48
Keall, M. D., Shaw, C., Chapman, R., & Howden-Chapman, P. (2018). Reductions in carbon dioxide emissions from an intervention to promote cycling and walking: A case study from New Zealand. Transportation Research Part D: Transport and Environment, 65, 687-696. https://doi.org/10.1016/j.trd.2018.10.004
Lydon, M., & Garcia, A. (2015). Tactical Urbanism: Short-term Action for Long-term Change. Island Press.
Ministry for the Environment Manatū Mō Te Taiao. (2022). Te hau mārohi ki anamata: Towards a productive, sustainable, and inclusive economy [Report]. https://environment.govt.nz/assets/publications/Aotearoa-New-Zealands-first-emissions-reduction-plan.pdf.
NZ Transport Agency (2019). Innovating Streets for People project. High level findings from baseline survey [PowerPoint slides]. Innovating Streets for People Working Group. Wellington.
Contributors
avatar for Michael Blewden

Michael Blewden

Mackie Research and Consulting, Auckland

Tuesday October 11, 2022 1:50pm - 2:20pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Celebrating

1:50pm NZDT

D.06: Supporting Te Tiriti o Waitangi responsiveness in an environmental science research programme through evaluation
Authors: V. Baker, J. Ataria, T. Chadwick, J. Forbes, H. Russell, M. Mark-Shadbolt, K. Smith, M. Walton  

In Aotearoa, the environmental policy settings of the New Zealand government are changing rapidly to respond to the role that mana whenua and their mātauranga plays in tackling complex issues like environmental contamination and plastic pollution within a Te Tiriti o Waitangi context. Conducting good science in this setting includes stronger accountability processes and increased expectations for mana whenua involvement in the co-development of research and environmental solutions. 
We share insights from the Aotearoa Impacts and Mitigation of Microplastics (AIM2) MBIE/government-funded research programme. AIM2 intends to continually improve Te Tiriti o Waitangi responsiveness. An evaluation framework has been developed, as part of wider capability development within the programme. Developing an evaluation framework, which focuses upon learning and continual improvement, has necessitated clear articulation of short, medium- and longer-term outcomes for increased Te Tiriti responsiveness within the programme. Articulation of values underpinning this work has also been beneficial, supporting a working theory of change that links how our science is conducted with aspirations for impacts of the science.
With the evaluation framework in place, capability building has greater focus building upon generic training around Te Tiriti o Waitangi and science colonization that was provided earlier in the programme. The framework also provides a mechanism for capturing learning for improving science practices, in an area with a lot of good intention, but fewer examples of success.     
We offer perspectives on the relationships and capabilities within the research programme required for Te Tiriti o Waitangi responsiveness work within the programme, steps taken in developing the evaluation framework, and intentions for ongoing use and refinement – noting the persistent challenges within current science research settings. We reflect on methods to enhance science practice and move more quickly towards Indigenous partnerships and knowledge being central, recognized and valued in research design and inquiry.
Contributors
VJ
TC

Tawhana Chadwick

Co-Director, Kūwaha Ltd
KS

Tuesday October 11, 2022 1:50pm - 2:20pm NZDT
Icon
  Presentation, Celebrating

1:50pm NZDT

A.06 Using systems thinking tools to build effective logic models
Programme logic models have been used by evaluators to understand the theory or idea underpinning a service, programme, or initiative that they are seeking to evaluate.

Programme logic models are designed to support us in developing a shared understanding of what we are evaluating with our stakeholders, while also supporting us to focus our evaluation efforts. However, what if this focus on linear causality is already limiting our evaluation efforts? What if this approach does not help us to understand the complex realities in which these initiatives are seeking to create change?

Our session will share some systems thinking tools, such as behaviour over time graphs, to demonstrate the value they bring when building theories of change. Our practical session will give you the chance to try out these tools and consider how they might fit with your own practice.
Contributors
avatar for Sarah Appleton-Dyer
avatar for Jessica Adams
avatar for Dr David Rees

Tuesday October 11, 2022 1:50pm - 2:20pm NZDT
Oceania
  Workshop/wānanga/talanoa, Reflecting

2:20pm NZDT

B.06 Te Korekoreka – a tool for evaluative practice
This presentation will discuss the indigenous approach Tokona te Raki (an indigenous social enterprise) is taking to understand evaluation as a tool for future making.

Like the whakatoki, Ka mua, ka muri we are integrating an indigenous future making approach to understanding how our past (te kore) holds important insights into our future and how our future (te po) needs to be anchored in our learning and dreaming within our current state (te ao).

The presentation will provide an overview of the approach we are taking to understand not only what value and impact we are having in the short term, programmatic space but also what difference we are seeing in our partnerships with iwi, hapū and whānau.  We will discuss the challenges, successes and early learning we are observing as a result of embedding a kaupapa Māori future making methodology at the core of our approach to evaluative practice.  In particular we will focus on sharing our methodological approach to evaluative reasoning and embedding evaluative practices into the DNA of our oragnisational culture by drawing on qualitative data and practical exemplars of key material we are developing that is fit for purpose across our organisation and wider partnerships.
Contributors
avatar for Porsha London

Porsha London

Tokona te Raki
avatar for Te Matau Flanagan
avatar for Kaya Staples

Tuesday October 11, 2022 2:20pm - 2:50pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Celebrating

2:20pm NZDT

C.06 Sustainable transport and negotiating equitable transitions: research findings and what next
Transport emissions make up a large proportion of New Zealand's carbon emissions and are a clear target for change. This means reducing reliance on private cars, especially those that use fossil fuels, and shifting to sustainable modes including public transport, cycling and walking.

Partly to address this, New Zealand’s public/affordable housing and urban development agency Kāinga Ora is building more medium and high density public housing developments, aiming to improve well-being through supporting walkability and active transport. This also means fewer carparks. However, these new developments are seeing increased parking disputes and frontline staff report public housing tenants need cars for shift work, children’s sport and transporting large families. Existing research shows low-income groups use cars less and more efficiently than the general population, and often suffer ‘forced car ownership’. Among other reasons, this is due to public transport service focusing on peak hour commuting to the CBD rather than journeys to other places at other times. Until this is rectified, public housing tenants are likely to need cars more than the general population.

This talk briefly presents findings from a tenant survey and interviews on transport experience and attitudes. It then covers challenges and first steps towards supporting well-being and rangatiratanga alongside emissions reduction, with reflection on how research and evaluation can support or lead to change in policy and practice.
Contributors
avatar for Sian Thompson

Sian Thompson

Kāinga Ora

C.06 Thompson 2022 Kainga Ora transport presentation ANZEA pdf
C.06 Thompson 2022 Kainga Ora transport presentation ANZEA pdf
Tuesday October 11, 2022 2:20pm - 2:50pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Celebrating

2:20pm NZDT

D.07 Nā tō rourou, nā taku rourou …
Come on a journey with us as we share how we, as a non-Māori and Māori, have worked together in a Te Tiriti partnership for over 20 years.        

We will explore our distinct and shared positioning through the lenses of a variety of projects we have collaborated on, both as independent consultants and as part of larger organisations.

In our exploration we will reflect on:
  • Our different pathways into evaluation 
  • The evolution of our working partnership     
  • Uncomfortable, frank, and challenging conversations with commissioners of evaluation, other evaluation stakeholders and each other    
Looking towards the future:
  • We will discuss, as evaluators in Aotearoa NZ, what we believe are our obligations as partnership role models, both domestically and internationally     
Me haere ngātahi tātou
Contributors
avatar for Teresa Taylor

Teresa Taylor

TT Consulting Tāpui Limited
avatar for Anne Dowden

Anne Dowden

Anne Dowden REWA
Anne is an evaluation consultant. She runs her own consultancy providing services to Governmentagencies and other entities. Anne collaborates and partners with other evaluation experts for mostevaluation projects.Anne is part of a large Australian-New Zealand family and has a partner... Read More →

Tuesday October 11, 2022 2:20pm - 2:50pm NZDT
Icon
  Presentation, Celebrating

2:20pm NZDT

A.07 Sounding out Scale
Increasingly, projects seek multi-dimensional impacts.

For example, increased productivity and improved environmental outcomes. Speaking to the reach anticipated, the reach achieved, and the resulting impacts, scaling reflects this more holistic perspective and flows through many programmes of work regardless of their sectors or location within Aotearoa and globally. This has prompted us to take the time to explore what scaling means by reflecting on our experiences (both good and not so good) across a range of national and international programmes we have been involved in.

We asked ourselves what should or could scaling look like and what could this mean for how and why we evaluate. Do we have all the answers? The answer is emphatically ‘no’, but we are eager to grow the intellectual thought in this space by engaging this evaluative conversation.
Contributors
avatar for Toni White

Toni White

Evaluator & Social Researcher, Plant and Food Research; ImpactAhead
•Strong background in biological sciences•Focus on bringing evaluation into the sciences in NZ•Loves working in the ECB space•Keen on learning and using new methodologies•Passionate about the social sciences•Co-facilitates the Waikato/BOP branch of ANZEA•Loves cats and... Read More →
avatar for Jordan Luttrell

Jordan Luttrell

Social Researcher, NIWA
SB
PB
AF
SF
MS

Tuesday October 11, 2022 2:20pm - 2:50pm NZDT
Oceania
  Workshop/wānanga/talanoa, Reflecting

2:50pm NZDT

Afternoon tea break
Tuesday October 11, 2022 2:50pm - 3:20pm NZDT
Oceania
  Conference Activity, Informal occasion

3:20pm NZDT

Panel Presentation 2: A conversation with Pākehā and Tauiwi evaluators reflecting on their journeys and hopes for social equity- and te Tiriti- based practice in Aotearoa.
How are Pākeha and Tauiwi evaluators reflecting on their identity, values and professional journeys so far? What are the most important lessons they have learned along the way? What are their hopes for social equity and te Tiriti-based practice and outcomes in future?

These are not new questions, rather this is a conversation that has underpinned ANZEA from its inception. It is woven through its constitution, and is reflected in the current Evaluator Competencies (2011) and Professional Standards (2015) frameworks. In 2018, a position statement was drafted as a touchstone for discussions among ANZEA members about how we understand ourselves as Pākehā and Tauiwi evaluators as well as our responsibilities for working with Māori communities and peers, and within a diverse and multicultural society. The conversation is continuing.

For this session, six practitioners who identify as Pākehā and Tauiwi evaluators have engaged in deep reflection about their ‘whys’ and what it means to them to be members of our evaluation professional community. They will share unique perspectives on their:
  •  Journeys so far and how they have felt about milestones along the way
  • Real world stories and their most important lessons learned
  • Hopes for the future of social equity- and te Tiriti-based practice in Aotearoa.
Each panelist will offer the audience an impactful question for ongoing reflection.

We warmly invite you to join this interactive opportunity for reflection, challenge and knowledge sharing that will be of interest to evaluation practitioners, commissioners, and stakeholders alike.
Contributors
avatar for Kate McKegg

Kate McKegg

Director, The Kinnect Group
Kate has specialist skills in supporting evaluative thinking and practice in complex settings where people are innovating to create systems change. She has been applying these skills for over 25 years in government, non-government, philanthropic and community contexts, including many... Read More →
avatar for jeph Mathias

jeph Mathias

Independent Development Consultant
Since arriving in Aotearoa from India as a four year old Jeph Mathias has oscillated aroundthis messy planet. He navigated school in Auckland, university in England, taught in an illegalmulti-racial school in apartheid South Africa, volunteered at Mother Teresa’s in Kolkata,returned... Read More →
avatar for Paula White

Paula White

Knowledge and Impact Manager, The Workshop
avatar for Meenakshi Sankar

Meenakshi Sankar

Director, Buddhi Consultancy Ltd
GS

Geoff Stone

Director, Ripple Collective - Research, Design and Evaluation
LS

Lauren Sweetman

Senior Adviser, Ministry of Culture and Heritage

Tuesday October 11, 2022 3:20pm - 4:50pm NZDT
Oceania
  Panel presentation, Into the future

4:50pm NZDT

Formal thank you for Taumata and Dr. Fiona Cram
TBC
Contributors
avatar for Fiona Cram

Fiona Cram

Katoa Ltd
Fiona Cram is a Maori (indigenous) woman from Aotearoa New Zealand. Her tribal affiliations are to Ngati Pahauwera on the east coast of Aotearoa. Fiona is the mother of one son.Fiona has a PhD in social and developmental psychology and is a widely published author of evaluation books... Read More →
avatar for Kataraina Pipi
RP

Tuesday October 11, 2022 4:50pm - 5:50pm NZDT
Oceania
  Conference Activity, Formal occasion
 
Wednesday, October 12
 

8:20am NZDT

Registration desk open
Wednesday October 12, 2022 8:20am - 8:50am NZDT
Oceania
  Conference Activity, Informal occasion

8:50am NZDT

Welcome and introduction to day
Wednesday October 12, 2022 8:50am - 9:00am NZDT
Oceania
  Conference Activity, Informal occasion

9:00am NZDT

Featured Contributors 3: Jeph Mathias - Presented by Allen & Clarke
TBC
Contributors
avatar for jeph Mathias

jeph Mathias

Independent Development Consultant
Since arriving in Aotearoa from India as a four year old Jeph Mathias has oscillated aroundthis messy planet. He navigated school in Auckland, university in England, taught in an illegalmulti-racial school in apartheid South Africa, volunteered at Mother Teresa’s in Kolkata,returned... Read More →

Day 3 9am JephANZEA Keynote COPY for sharing pdf
Wednesday October 12, 2022 9:00am - 10:00am NZDT
Oceania
  Featured contributor

10:00am NZDT

Morning tea break
Wednesday October 12, 2022 10:00am - 10:30am NZDT
Oceania
  Conference Activity, Informal occasion

10:30am NZDT

A.08 Having ‘Skin in The Game’ in Kaupapa Māori Evaluation
This presentation is targeted to emerging Kaupapa Māori researchers, to encourage them to ground and affirm their practice in accordance with their indigenous positioning; to look beyond the Western paradigms of research for affirmation.   We will begin by describing our journey, as three wāhine Māori Kaupapa Māori evaluators, touching on:

  • Our whakapapa whānau, and whakapapa mahi
  • How our evaluation approach and practice has been shaped and enriched by our innate connection to te ao Māori 
  • The influence our indigenous and non-indigenous positioning has on our mahi as evaluators From there, we will explore and reflect on:
  • The wellbeing and strength of whānau, personally and also in the broader sense of our mahi
  • Our obligations as Māori, wāhine Māori, wāhine Māori evaluators    “E kore au e ngaro, he kākano i ruia mai i rangiātea”
Contributors
avatar for Gill Potaka-Osborne

Gill Potaka-Osborne

Researcher, Whakauae Research Services
Ko Aotea te wakaKo Ruapehu te maungaKo Whanganui te awaKo Ātihaunui-ā-Pāpārangi te iwiKo Ngāti Tuera, Ngāti Pamoana, Ngāti Pareraukawa ngā hapū.Ko Pungarehu, ko Parikino, ko Koriniti, Ko Ngātokowaru Marae ngā marae.Ko Gill Potaka-Osborne au. In 2005, I began employment... Read More →
avatar for Kiri Parata

Kiri Parata

President, Australian Evaluation Society
Kia ora I'm Kiri, living on Kabi Kabi Country on the Sunshine Coast of Queensland. He uri ahau nō Kāpiti, kō Te Ātiawa, kō Ngāti Toarangatira, kō Ngāti Raukawa, kō Ngāti Ruanui, kō Ngāi Tahu ōku iwi. I work as a Māori health researcher and evaluator, this mahi brings... Read More →
avatar for Teresa Taylor

Teresa Taylor

TT Consulting Tāpui Limited

ANZEA 2022 Potaka Osborne Parata Taylor 091022 Copy pptx
Wednesday October 12, 2022 10:30am - 11:00am NZDT
Oceania
  Presentation, Celebrating

10:30am NZDT

B.07 Is the Pacific only for the Pasifika researcher?
This presentation adds to the discussion regarding the role of non-Pasifika researchers and evaluators working in the Pacific. The presenters propose that research can be undertaken by non-Pasifika researchers and evaluators under certain conditions, for example abiding by the research and evaluation principles of mutual respect and reciprocity, having a deep understanding of a country’s context and language, and that the work is aligned with value-enhancing outcomes for the Pacific.  

Dr Apisalome Movono will discuss the importance of creating a more accurate understanding of Pacific Island peoples to find solutions and long-term strategies for the many challenges they face. He will reflect critically on ethical fieldwork from his own experience leading research in the South Pacific and discuss the importance for both Pasifika and non-Pasifika researchers and evaluators to have an in-depth understanding of the ethics of undertaking fieldwork in the Pacific.

The presenters will also discuss a participant-led approach used in Dr Elisabeth Poppelwell’s PhD research. That research examined what innovations and lessons could be shared with New Zealand and other non-Pacific countries. This involved the participant interviewee leading the interview and the order of what was to be discussed. The researcher took a largely listening role while the participant provided the context and the linkage between core values held in their country and the participant’s decision-making in their professional roles.
Contributors
AM

Apisalome Movono

Massey University
avatar for Elisabeth Poppelwell

Elisabeth Poppelwell

MERL Consultant, Future Partners
Elisabeth has over 20 years’ experience in applied research and evaluation. She is a mixed methodologist and has undertaken a wide range of work including Pacific country programme evaluations, research on SOE governance in the Pacific, SME debt, impact of adverse events on SMEs... Read More →

Wednesday October 12, 2022 10:30am - 11:00am NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Reflecting

10:30am NZDT

C.07 Implications of historical social policy on current day evaluative practices
This workshop/presentation is based on the research into Māori involvement in State Care 1950-1999 for the Royal Commission into State Care Abuse (Savage et al., 2021).

The research involved qualitative and quantitative analyses, including an integrative literature review of 482 documents (primary research, archival material and publicly available reports and paper.

Gaps in document analyses formed the basis of semi-structured interviews. The twenty-six participants were Māori and non-Māori, including former Department of Social Welfare staff, community service providers and people involved in the development and implementation of Puao-Te-Ata-Tū and the Children’s Young Persons and their Families Act.

Results presented in the report emphasise the devastating, intergenerational harms that tamariki Māori and whānau have experienced through enduring, systemic and structural racism across the state care system. These findings are not new, given a large part of analysis is drawn from published material and also highlighted in inquiries and reviews.

However, report analysis brings together in one place, a compilation of information relating to Māori
overrepresentation and Māori experiences of the state care system during the review period (1950-1999). Synthesis of this research identifies a number of social policies, Government actions and inactions, that perpetuated Māori over-representation and differential treatment in the state care and justice system.

This workshop discusses a number of implications arising from historical social policy on current day evaluation practices in the social sector. The purpose of this presentation is to look back into the past in order to understand how structural inequities continue to impact on Māori in the present day.
Contributors
CS

Catherine Savage

Ihi Research
PC
AH

Wednesday October 12, 2022 10:30am - 12:00pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Workshop/wānanga/talanoa, Reflecting

11:00am NZDT

A.09 Achieving inclusion and safety in evaluation practice: Lessons learned from evaluating with people living with dementia
Research and evaluation practice and ethics often seeks to balance safety with inclusion. When working with communities, evaluators, ethics committees and commissioning agents sometimes prioritise safety to minimise risks. Informed consent is the standard commonly used to ensure safety.  

A safety-based approach can reflect our biases and the underlying assumptions we make about the contributions possible from some communities. A safety-based approach, while important, risks excluding communities such as people living with dementia. Historically, people living with dementia have been excluded from research because of informed consent requirements and the perception of potential harm from participation. In recent years, people living with dementia have demanded inclusion in research about dementia. These advocates created guides to enable safety and inclusion in research practice. Research shows the benefits of participating in evaluation and research include improved well-being, social inclusion, better access to care and support, and feeling valued. We think evaluation can and should achieve safety and inclusion.

Since 2018 Litmus has worked with Alzheimers New Zealand. Alzheimers New Zealand supports and advocates for people living with dementia. We conducted lived experience research in 2018 and in 2020/2021 we evaluated the services and support provided by four local Alzheimers New Zealand organisations. Across these two projects we interviewed 28 people with dementia.

This presentation shares lessons we learned about safety and inclusion working with people living with dementia. We will share resources and tools for enabling participation and the benefits of taking this approach. We will identify where and how these lessons can apply in other areas where safety and inclusion are seen to be in conflict.
Contributors
avatar for Anna Thompson

Wednesday October 12, 2022 11:00am - 11:30am NZDT
Oceania
  Presentation, Reflecting

11:00am NZDT

B.08 Papata pē ka na’e Lalanga | It may feel rough on the surface but the mat has been strongly woven together
We will share our partnership experiences, and give some examples, of designing, delivering and sharing our cross-cultural evaluation projects.

We will:
  • Describe our cross-cultural partnership. 
  • Outline the challenges faced, and overcome, to keep our cross-cultural partnership at the forefront of our evaluations. 
  • Share practices that align with our Pasifika communities: what we learned; what we recommend. 
Including: practical tips for ensuring that Pasifika voices are gathered 
  • Trusting relationships and connections provide truthful reflection 
  • Reflecting back for deeper understanding and to overcome language challenges practical tips for ensuring that our Pasifika voices are present and well understood in evaluation findings, and approaches for using evaluations findings to support change for our Pasifika communities.  
Contributors
LT

Lesieli Tongati'o

Massey University
avatar for Anne Dowden

Anne Dowden

Anne Dowden REWA
Anne is an evaluation consultant. She runs her own consultancy providing services to Governmentagencies and other entities. Anne collaborates and partners with other evaluation experts for mostevaluation projects.Anne is part of a large Australian-New Zealand family and has a partner... Read More →

B.08 Tongatio Anzea slides pdf
Wednesday October 12, 2022 11:00am - 11:30am NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Celebrating

11:00am NZDT

D.09 Advancing a system-based approach to educational evaluation by building on established participatory evaluation practices
In its 2019 review of the New Zealand education system (Supporting all schools to succeed,), the New Zealand Government confirmed ERO’s role in the review of schools. It also confirmed ERO’s role in ensuring that research and evaluation provides a strong basis for generating effective system-level information and evaluation that informs prioritisation, action, and improvement.  

ERO has since strengthened its system-level(?) approach to evaluation by establishing Te Ihuwaka |Education Evaluation Centre. The scope of this unit has expanded to include: looking at overall education system performance (including for different groups and different settings); evaluating the effectiveness of programmes and initiatives; and showcasing and sharing good practice.

When the latest cyclical review of Oranga Tamariki | Ministry for Children residential schools came around, ERO decided to frame its evaluation at both system level and institutional level. This ensured it was able to maintain its participatory reviewing approach focus (institutional level) whilst also providing a system-level view of how the residential schools as a whole were doing. ERO’s system change approach was informed by Latham’s (2014) system framework which identified enabling and inhibiting components and levers for improvement. Consequently, evaluation questions were focused on outcomes for students, the quality of the educational provision, and the enablers and barriers at a system level. The system-level questions targeted the key system elements that mattered most for educational provision in residential care settings.

The extended scope and timeframes of this combined evaluation approach enabled increased engagement of stakeholders, including across three government agencies, and culminated in collaboratively developed, cross-agency recommendations.  This presentation will explore the successes and challenges involved in integrating these two evaluation approaches (institutional and systemic). We will share some of the strategies used to manage stakeholder expectations and engagement in the evaluation, and to embed their participation in evaluation design and development of recommendations.
Contributors
avatar for Ruth Pritchard

Ruth Pritchard

Senior Evaluator, Education Review Office
avatar for Sankar Ramasamy

Sankar Ramasamy

Chief Evaluator, Education Review Office

Wednesday October 12, 2022 11:00am - 11:30am NZDT
Icon
  Presentation, Celebrating

11:30am NZDT

A.10 Ethics review and community-based practice
In this session I will share my experience of ethics reviews and approvals and what has often played out in practice (ANZEA does not have its own ethics committee).

In my experience, ethics reviews have (but not always) resulted in long participant information sheets (which few people ever read), and detailed, prescriptive research plans (that often change when fieldwork is underway). In these instances, there is a gap between ethics committees ideals on the one hand, and the reality of doing socially informed research on the other. Long participant information sheets (the last one we had approved was 6 pages) may be useful as an official accountability document, but they are usually not helpful to community members wondering what we want to do. Secondly, there is an expectation that the approved research plan will be followed through to the end of the project. This expectation may hold in the context of positivist social science, where the plan to collect data can stand independent of context. However, qualitative methodologies, which are what I often employ, are responsive to the particulars, needs and interests of the people researchers work with. The plan also needs to evolve based on what is learnt. All of this is impossible to predict in advance, certainly on longer projects.

An alternative to the rule-book approach to ethics is ANZEA’s principles-based framework. I see this as a more suitable guide for ethical evaluation practice in community settings. “Integrity” is the core value underlying the principles, which means “the practices, processes and products of evaluation are to be respectful of people, undertaken with care, methodologically responsive and appropriate, trustworthy, undertaken with competence and useful”. I’m not saying we don’t need to set out with a clear plan. We do. But the plan needs to be transparent, responsive and useful to the people we want to work with.  
Contributors
avatar for Tony O'Connor

Tony O'Connor

Point & Associates

Wednesday October 12, 2022 11:30am - 12:00pm NZDT
Oceania
  Presentation, Reflecting

11:30am NZDT

B.09 Evaluation of an expanding Tupu Aotearoa programme during COVID19 – challenges and opportunities
Tupu Aotearoa is an MPP-led initiative that has been operating and evaluated for the last 10
years to support Pacific peoples into employment.

Over the years, the programme has undergone some changes. It has evolved from the Pacific Employment Support Service (PESS) pilot established in 2010 to test multiple approaches to support Pacific youth aged
16-24 years into employment.

The original PESS programme initially ran for four years from 1 July 2010 to 30 June 2014, with the first six months dedicated to establishing the service delivery operations with the four selected providers operating primarily in South Auckland.

There was no programme in 2015-16. However, the programme was resumed from 2016-18. During the 2018-2021 period the programme was extended to a national programme with 28 provider contracts in seven regions and three metropolitan areas. Independent annual evaluations of service providers and overall programme effectiveness have been a cornerstone of the Tupu Aotearoa programme since 2012.

This presentation outlines the evolving process of evaluation over time, particularly on the challenges and opportunities for improving the evaluation of the Tupu Aotearoa programme.

The discussions will focus on the period since the recent expansion of Tupu Aotearoa to a national programme in 2019 and through the onset of COVID19 in New Zealand from March 2020.
Contributors
avatar for Dr Manuila Tausi

Dr Manuila Tausi

Ministry for Pacific Peoples
avatar for Dr Halahingano Rohorua

Dr Halahingano Rohorua

Manager, Ministry for Pacific Peoples

Wednesday October 12, 2022 11:30am - 12:00pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Celebrating

11:30am NZDT

D.10 Social License to operate. Monitoring and evaluating NZers perceptions of the value and contribution of international students and education to NZ
Social License has been defined as existing when a project has ongoing approval or broad an on-going social acceptance within the local community and other stakeholders of the activity. For Education NZ (ENZ) monitoring and reporting social license for having international students in-country and education activities is a State of Intent responsibility.   ENZ has been monitoring social approval since 2017. The most recent survey was undertaken in May 2022 using the Survey Monkey NZ population online panel.  1034 responses were collected.  Headline results, time series data and evaluative insights about the results will be presented. Evaluative questions asked include to what degree can the changes be seen as meaningful and attributable to ENZ activities? Asking the counter factual question – what if ENZ had not been active in this space – is one approach to answering the question.    The results show a gradual positive shift across 16 statements and Likert scale agreement categories about the contribution of international students to NZ.  Respondents, when asked:

  • 'overall to what extent do you agree or disagree that international students benefit NZ’, 59% of respondents ‘Agree / Strongly agree’
  • 77% ‘Agree / Strongly agree’ that students help with the economy compared to 57% in 2016
  • 66% ‘Agree / Strongly Agree’ that students provide needed skills for those that stay after study compared to 48% in 2016
  • 69% ‘Agree / Strongly agree’ that students help support the education system for kiwi students compared to 48% in 2016 However, international students remain a low priority (in bottom 4 of 16 statements) as a ‘front of mind’ issue compared to cost of living, health and well-being and affordable housing. 
The approach highlights the value of time series data for monitoring and evaluation purposes, particularly when sample sizes are small or identification and attribution of change is problematic.
Contributors
avatar for John Wren

John Wren

Education NZ

Wednesday October 12, 2022 11:30am - 12:00pm NZDT
Icon
  Presentation, Celebrating

12:00pm NZDT

Lunch break
Lunch Break 
Wednesday October 12, 2022 12:00pm - 1:00pm NZDT
Oceania
  Conference Activity, Informal occasion

12:20pm NZDT

Lunch discussion - Future of the Taumata
Since its inception, the ANZEA Taumata, comprising Robin Peace, Neville Robertson and Ian Trotman, have provided valued guidance and reflection to the ANZEA whānau collectively, as well as directly to the ANZEA Board. Our current Taumata members, Robin and Neville, are stepping aside from their roles to pursue their other passions in their retirement. This lunchtime session is an opportunity for ANZEA members to reflect on the Taumata concept and the continued role of fellows or elders to the ANZEA community.
Contributors
avatar for Adrian Field

Adrian Field

Director, Dovetail
Adrian is the director of Dovetail, an Auckland-based evaluation consultancy, and a member of the Kinnect Group. Adrian has worked in evaluation in different capacities for some 20 years and doesn't really like how old that makes him feel. Adrian's experience traverses health, social... Read More →
AC
RT

Rae Torrie

Evaluation Works

Wednesday October 12, 2022 12:20pm - 12:50pm NZDT
Oceania
  Conference Activity, Informal occasion

1:00pm NZDT

B.10 Improving online survey response rates in COVID-19 world: What should evaluators do differently?
Living in the new normal with COVID-19 in the community means most survey data collection efforts for evaluation will be done through online surveys rather than in-person. In the absence of face-to-face human interaction, it can be somewhat difficult to get people to participate in online surveys. Even more so, tāngata whaikaha (disabled people) and tāngata whaiora (people who use mental health services).

Low engagement with surveys would mean low response rates, and results not being representative of the population being studied. It is therefore important that evaluators consider what they should do differently to maintain high response rates for online surveys. Te Pou implements multiple surveys across its mental health, addition, and disability workforce development programmes. With the changing COVID-19 situation, we share some technical and culturally appropriate reflections practitioners and individual evaluators can implement to maintain high survey response rates.

Cultural sensitivity and ethics play important roles. From the Māori worldview, a survey in which the language demonstrates manaakitanga (respect, support), upholds people’s mana, acknowledges, and respects Māori tapu (sacred) and kia ngaku māhaki (being humble in your approach), would get more response than not. Other factors that determine how people engage with a survey include length of the survey, the topic, and the survey specificity. Practitioners are encouraged to take advantage of the ‘one question per page’ feature of online survey platforms. Survey questions should be simple and only make use of skip logics when unavoidable and should be targeted to enhance its engagement. These factors, if properly considered, would help potential participants feel their time is being respected, thereby encouraging online survey completion. These factors are even more important, for our Māori, tāngata whaikaha and tāngata whaiora participants.
Contributors
avatar for Charles Nnabugwu

Charles Nnabugwu

Evaluator, Te Pou
avatar for Heather Kongs-Taylor

Heather Kongs-Taylor

Manager, Evaluation and Monitoring, Te Pou
avatar for Kahurangi Fergusson-Tibble

Wednesday October 12, 2022 1:00pm - 1:30pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Developing

1:00pm NZDT

C.08 "Paddling an inclusive waka - Sport NZ's journey to develop a holistic value for money framework"
Sport NZ are on a journey to develop an approach to value for money (VFM) assessment that gets to the heart of our value proposition and defines what it really means to use resources well to create value for play, active recreation and sport in Aotearoa for tangata whenua and all New Zealanders. We believe a strong bi-cultural foundation is critical to our national identity and wellbeing.

Traditionally there have been a few competing views about value for money. There are the economists, who favour cost-benefit analysis. The impact measurement set, who like SROI. The international development sector with their 'Five Es' and key performance indicators. And then there's us, evaluators who want to provide good answers to VFM questions, bring a diverse and rich bag of valuing approaches and methods. Some of us may have flirted with CBA, SROI and the like, and can see their strengths but couldn't shake the feeling they're not quite giving us a complete picture of VFM. In Aotearoa, we also have a unique opportunity to understand VFM through the lens of Te Tiriti as well.

It's time for evaluators to own this space. Our profession has the frameworks and tools to bring together the best each method has to offer and to add our own insights to answer value for money questions.

Over the past year or so, we've been working with Julian King to develop a VFM framework that brings together rubrics, mixed methods, theory-based evaluation, participatory evaluation, economic evaluation and NZ's Living Standards Framework into a logical, intuitive, coherent evaluation framework. In this presentation, we'll show you how we've approached it.

We'll give you a peek at what we've done so far, what we're planning to do next, and what we've learned that you can apply in your work.
Contributors
avatar for Rebecca Thorby

Rebecca Thorby

Senior Evaluation Advisor, Sport New Zealand
avatar for Neil Snowling
avatar for Elaine More

Elaine More

Acting Evaluation Lead, Sport NZ
avatar for Bria Sargent
avatar for Julian King

Julian King

Director, Julian King & Associates
Julian specialises in evaluation and value for money. The Value for Investment evaluation system, developed through Julian’s doctoral research, combines evaluative and economic thinking to assess the quality, impact and value of policies and programs. Julian received the 2021 A... Read More →

ANZEA Presentation Sport NZ Julian King Oct 2022 pdf
Wednesday October 12, 2022 1:00pm - 1:30pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Reflecting

1:00pm NZDT

D.11 Evaluating contribution from the ground up
Grounded theory applies inductive reasoning and methods for generating hypotheses and theory from the ground.

Meanwhile, contribution analysis is a primarily deductive approach that tests an existing program logic to make credible causal claims about interventions and their results. What occurs when these seemingly divergent approaches are combined in a systems-focused evaluation?

In this presentation, we will reflect on our experience of using grounded theory and contribution analysis to develop warranted cases that describe the contribution of an Australian Government program that aims to strengthen comprehensive primary health care (cPHC) for Aboriginal and Torres Strait Islander people. We analysed more than 50 transcripts of interviews and workshops to surface factors that enable or inhibit contribution to cPHC.

We will provide examples of how contribution cases were developed, tested and then revised through engagement with key evaluation stakeholders. These cases were integral to supporting defensible evaluative judgements about the program’s contribution in a complex context.
Contributors
avatar for Dr Sara Lystlund Hansen

Dr Sara Lystlund Hansen

Research Consultant, Allen + Clarke
Sara Lystlund Hansen has a PhD in Cultural Anthropology and in-depth experience in ethnographic methods and analysis, including participatory research, thick-description, storytelling and narrative analysis. Sara has a longstanding interest in social justice, structures of power and... Read More →
avatar for Mathea Roorda

Mathea Roorda

Senior consultant, Allen + Clarke
Values inform what matters, and are at the heart of evaluation. You literally can't get to an evaluative judgement without them. I'm interested in approaches to systematically identifying what matters, and for whom. Come talk with me about the values identification matrix (VIM) I... Read More →
avatar for Dr Roxanne Bainbridge

Wednesday October 12, 2022 1:00pm - 1:30pm NZDT
Icon
  Presentation, Reflecting

1:00pm NZDT

A.11 What does it take to do developmental evaluation (DE) in a government context? Lessons, insights and learning.
We explore five learning areas from a developmental evaluation (DE) commissioned in a government agency.

The government agency was developing a new approach to partnering with communities and iwi as part of a wider organisational change process. DE team members, together with the government agency team members, will share their DE experience, drawing out useful insights and lessons.

We will discuss five key areas of learning to help highlight what it takes for DE to be successful in a government context. The considerations and tensions we will explore include:

  1. Commissioning for complexity 
  2. Evaluation design and reporting imperatives and the value of developing high trust, valued relationships, and a shared learning agenda from early on 
  3. Privileging a kaupapa Māori approach within mainstream organisation constraints 
  4. National and local contexts and the ethics and practices of sharing information  
  5. Authorising environments for DE practice and the importance of trust as a de-risking tool.   

The necessary conditions for effective DE contrast with many existing government practices and systems. We hope our insights about what it takes to do DE in a government context will help developmental evaluators and those in government agencies and complex social spaces to use DE more successfully.
Contributors
avatar for Alicia Crocket

Alicia Crocket

Independent consultant
avatar for Debbie Goodwin

Debbie Goodwin

DBZ CONSULTANCY LTD
Debbie Goodwin is the Co-Chair of EvalIndigenous from Maori tribe in New Zealand. She is a PhD fellow and member of the Ma te Rae Evaluation Association. She led the organisation of the first ever Indigenous evaluation on the theme "Indigenous Peoples' Conference on Evaluation" which... Read More →
JM
avatar for Kate McKegg

Kate McKegg

Director, The Kinnect Group
Kate has specialist skills in supporting evaluative thinking and practice in complex settings where people are innovating to create systems change. She has been applying these skills for over 25 years in government, non-government, philanthropic and community contexts, including many... Read More →
avatar for Rashmi Rajan
avatar for Louise Were

Louise Were

Hikitia | Member of the Tuakana Teina Evaluation Collective; PhD Candidate
Louise is a Māori evaluator who seeks to bring together Te Ao Māori, evaluation, design and systems thinking. Louise specialises in evaluation and policy analysis, with her professional background stretching acrossMāori, public health, disability, and community contexts, within... Read More →

Crocket A et al ANZEA presentation DE in government contexts pdf
Wednesday October 12, 2022 1:00pm - 1:50pm NZDT
Oceania
  Workshop/wānanga/talanoa, Celebrating

1:30pm NZDT

B.11 Recruitment resourcefulness and resilience in the context of fear and uncertainty
Evaluations focus on the important questions to inform actions and decisions and seek to determine what works for whom and why. Recruitment plays a vital role. To properly inform the actions and decisions, recruitment processes must be inclusive, equitable and effective. Failure to do so can mean an evaluation lacks rigour and is of poor quality.  

Through the COVID-19 pandemic the nature of recruitment has shifted dramatically. To maintain inclusivity, equity, and effectiveness in recruitment through COVID-19, evaluators needed to be dynamic, agile while being respectful when engaging communities. The COVID-19 pandemic forced evaluators to conduct individual and group interviews online. However, the new landscape also meant recruiters had to modify their approaches. This included greater use of texting and social media and being flexible and supporting stakeholders to ensure they have what they need to participate online, downloading apps and providing hardware.

Recruitment is also becoming more digital. The humble physical koha is being replaced with electronic vouchers, and paper-based consent forms are being replaced with e-fillable consent forms. While our practice is evolving the principles of inclusive recruitment are enduring. That is striving to include those voices and stories that need to be heard.
Contributors
avatar for Finlay Smith
avatar for Roimata Hanchard

Wednesday October 12, 2022 1:30pm - 2:00pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Developing

1:30pm NZDT

D.12 Evaluating system change: Exploring how project innovations transform business as usual
How do project innovations create changes in wider organisational systems and practice? This short paper will discuss our learning from evaluating three dynamic road safety projects working within an innovation umbrella programme.  

This session will highlight the challenges and opportunities for taking innovation to scale, reflecting on our learning from theoretical approaches outside evaluation that offer compelling new windows for evaluator’s understanding of impact and change. Grounded in real-world application of three innovative road safety projects, the paper will present the inter-weaving of socio-technical systems theory and developmental evaluation.

These approaches were used, along with their practical application through rubrics and multiple data collection methods, to explore the extent to which the projects fostered innovation that translated into sustained business operations.

This paper will provide useful ideas and reflections for participants including how collaboratively developed evaluation rubrics were used to define and assess levels and dimensions of system change that each project could reflect against, through a developmental process of engagement and reflection. Socio-technical systems theory will be provided as a lens for understanding the potential for local or niche innovations to lever changes in wider systems.

The session will conclude with an exploration of the role of evaluation in capturing and catalysing innovation.
Contributors
avatar for Adrian Field

Adrian Field

Director, Dovetail
Adrian is the director of Dovetail, an Auckland-based evaluation consultancy, and a member of the Kinnect Group. Adrian has worked in evaluation in different capacities for some 20 years and doesn't really like how old that makes him feel. Adrian's experience traverses health, social... Read More →

D.12 Field Evaluating system change 221006 pdf
Wednesday October 12, 2022 1:30pm - 2:00pm NZDT
Icon
  Presentation, Developing

1:50pm NZDT

A.12 Evaluation as public service. What does moving forward look like?
What is evaluation conceptualised as an activity of public service?  

Most reflections about the goals of evaluation and how to do it better come from the field – the evaluation practice community along with some academics. Many of us in the public sector have integrated these ideas into our design, delivery and commissioning approaches as internal evaluators or research managers.

Ideas about delivering a better public service in Aotearoa are not in short supply and reach into important territories of systems stewardship, addressing wicked problems and an effective Treaty partnership. But the positioning of evaluation is faint in these narratives and often packaged within a bunch of activities that support strong policy.

What do these progressions in public service mean for effective public service evaluation practice that delivers more public value, and where should we start?
Contributors
DS

David Stuart

Manatu Taonga Ministry for Culture and Heritage

Wednesday October 12, 2022 1:50pm - 2:20pm NZDT
Oceania
  Presentation, Celebrating

2:00pm NZDT

B.12 The changing roles and drivers for evaluation: how do evaluators help save the world, and what new skills do we need?
We are at a critical historical juncture. Inequity is increasing, climate change is well on its way to the
point of no return, and the robots are here. The time for change is now. Evaluators have a role to play in
saving the world, and we are increasingly being asked to step in. Evaluation is disrupted. Disruption
offers strategic opportunities to reset the system into a better place. We are having critical discussions
about equity, social justice, and decolonisation. We are seeing new actors, new funding streams, and
new data flows. How do we stay relevant? How do we ‘step in’? What new roles can evaluators play to
help transform the systems, policies, and practices that have created today’s challenges and help build a
more equitable and sustainable future?

In this presentation, I will share two trends I see in Australia around the changing roles of evaluators. I
ask whether the same trends are happening in Aotearoa and explore what old skills we can draw on and
what new skills we need to build.

Firstly is the rise of the impact organisation. I am seeing many not-for-profits scrambling to centre
impact into every aspect of their work. Same, same but different? A key difference is the driver; it’s not
for the funder! It’s increasingly for staff engagement and because stakeholders genuinely want to make a
difference. I share YMCA Victoria’s journey from rethinking the organisational strategy, digital
transformation, widescale staff engagement, agile approaches to learning and failing, and trying not to
overcook the whole thing.

The second trend I discuss is efforts at widescale systems transformation. I share the case of the
Early Years Catalyst inviting evaluation partners right into the messy starting phase. It seems that
evaluative thinking is crucial to this work.
Contributors
avatar for Jess Dart

Jess Dart

Chief Evaluator and Founder, Clear Horizon Consulting
Dr Jess Dart is the founder and Chief Evaluator of Clear Horizon, an Australian-based specialist evaluation company. Having received the 2018 Outstanding Contribution to Evaluation Award from the Australian Evaluation Society (AES), Jess is a recognised leader with over 25 years of... Read More →

Wednesday October 12, 2022 2:00pm - 2:30pm NZDT
Te Huinga Centre - Rangimarie Room 1
  Presentation, Developing

2:00pm NZDT

C.10 Te Matapaeroa: insights into Pakihi Māori
Te Matapaeroa is an ongoing project commissioned by Te Puni Kōkiri which examines the Māori economy, in particular Māori-owned businesses. The recently published research, Te Matapaeroa 2020, focused on the size and characteristics of Māori-owned businesses from 2010 up until 2020.

The pre-COVID view of Māori businesses forms a baseline from which future research will build upon. Updating this research yearly provides Te Puni Kōkiri and other Māori business stakeholders the evidence base to support policy and investment decisions to support Māori businesses. This research reflects the theme of the ANZEA Conference 2022, Ka mua, Ka mui: “Looking back to move forward”.

For the year ending 31 March 2020, we identified 23,000 Māori-owned businesses. This research has significantly contributed to our understanding of the size and scale of the Māori economy. For example, through this research, we identified the important role wāhine Māori play in the Māori economy. We discovered Māori-owned businesses with wāhine Māori owners employed more Māori than other Māori businesses.
In July 2022, Te Puni Kōkiri started the next development phase of the Te Matapaeroa research series, referred to as Te Matapaeroa 2021. Te Matapaeroa 2021 will provide the first insights into the effects of the COVID-19 pandemic on Māori businesses. Te Matapaeroa 2021 is a collaboration between Te Puni Kōkiri and the New Zealand Productivity Commission. Through this collaboration, Te Puni Kōkiri researchers aim to improve the methodology and approach in understanding Māori businesses using labour productivity data in Statistics New Zealand’s Integrated Data Infrastructure.
Contributors
AM

Avril Macfarlane

Te Puni Kōkiri
MB

Wednesday October 12, 2022 2:00pm - 2:30pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Reflecting

2:00pm NZDT

D.13 The unfortunate tale: great programme design let down by traditional commissioning and contracting methods - what this means for evaluators.
Are you a funder, provider or evaluator working on long-standing and profoundly entrenched complex, wicked problems? Have you noticed commissioning and contracting sometimes gets in the way of the programme's intended goals or services? Have you seen or had to use workarounds to make contracting possible? You are not alone. This session is for you.  

This session is for funders, providers or evaluators looking for innovative ways to think through commissioning and contracting issues in programme and service implementation. This session may change how you design and evaluate programmes in the future. New research shows commissioning and contracting can help or hamper innovative initiatives, particularly for Māori and Pacific providers. Tensions abound in navigating traditional contracting rules and procedures associated with New Public Management.

Evaluators who understand these tensions may be better able to explain gaps in programme and service delivery and suggest ways to address them.  This session will lean into some of the critical commissioning and contracting tensions. Instead of the prevailing rigid, predetermined ways of contracting for accountability only, this session will show how a complexity framing can help focus on more equitable service provision. We will introduce you to some new and successful ways of thinking about commissioning and contracting that are complexity-informed.  These methods and approaches will quickly demonstrate alternative ways of looking at commissioning and contracting.
Contributors
avatar for Judy Oakden

Judy Oakden

Director, Pragmatica Limited
avatar for Kellie Spee

Kellie Spee

Kelly Spee Consulting Limited

ANZEA 2022 Oakden jovxx pdf
Wednesday October 12, 2022 2:00pm - 2:50pm NZDT
Icon
  Workshop/wānanga/talanoa, Into the future

2:20pm NZDT

A.13 A Pacific MEL Framework - Lessons have we learnt from the implementation of the Kalia MEL Framework for the road ahead
Monitoring and evaluation are critical for looking at progress and for helping to learn as we go and grow. At the Ministry for Pacific Peoples, its Monitoring, Evaluation and Learning (MEL) Framework is symbolised or described within the context of a kalia (double-hulled Tongan waka). Drawing parallels to different parts of a waka, the presentation will describe how the skills, knowledge, talents, commitment and wellbeing of every person on the waka need to be harnessed and shared to keep the waka safe and on course. This relates to the roles and responsibilities of MPP leaders, managers and staff in relation to results-focused planning and management, which are underpinned by a robust MEL.

We can now look back on our progress and the lessons we have learned from one year of sailing our MPP kalia on a voyage that included challenges and opportunities presented by a crew that has more than doubled in number and the addition of many new destinations, all while navigating the ever-changing ocean and often unpredictable weather. It is a crucial time for reflection as we prepare our kalia for the journey into the 2022-23 financial year, which is proving to be a very different journey from the previous financial year. The presentation will include real-life experiences and examples, and what the lessons we have learned from this mean for the next phase of our journey. It could also help others in the same boat to set their sails for a smooth journey.
Contributors
avatar for Annalize Struwig

Annalize Struwig

Principal Analyst MEL, Ministry for Pacific Peoples
I have been humbled by many opportunities to study, learn and practice evaluation in a 20-year career spanning academia, international development and national government. What I have learnt is the importance to remain "small" in the big, diverse and dynamic world of evaluation. There... Read More →
avatar for Lealofi Kupa

Lealofi Kupa

Ministry for Pacific Peoples

MPP MEL Framework Kupa and Struwig pptx
Wednesday October 12, 2022 2:20pm - 2:50pm NZDT
Te Huinga Centre - Rangimarie Room 2
  Presentation, Celebrating

2:20pm NZDT

Using rubrics to drive systemic change for both equity and sustainability
Laudes Foundation, a family foundation based in Europe, is responding to the dual crises of
inequality and climate breakdown through synergistic and collaborative efforts in finance and
capital markets, the fashion industry, and the built environment. With an ambition this broad and
a global system so huge and complex, what kind of evaluation and learning system could most
usefully help guide and evaluate the change?

Key Performance Indicators (KPIs) had given focus and direction to the foundation staff and
trust and confidence to Board members, who were used to assessing (and viewing) progress
using KPIs (sales, revenues, costs) measured quarterly against targets. Nonetheless, these
were not systemic nor focused on quality, and they often diverted the foundation from
addressing root causes in pursuit of achieving targets.

With the help of Jane Davidson and Thomaz Chianca, Laudes Foundation understood the
limitations of KPIs and went searching for a better way – a systems change-savvy, rubric-
enhanced approach that measures what really matters.

In 2020, Laudes defined its strategy and Jane and Thomaz facilitated the development of a
systemwide theory of change (https://toc.laudesfoundation.org/). These were two essential
ingredients for the development of the rubric-based measurement and learning system. The
approach is grounded in a systemwide theory of change from which rubrics were developed to
track key outcomes and impacts. Rubrics allowed for a more nuanced and systems change-
grounded approach, one that could flex as change emerged (https://www.laudesfoundation.org/grants/rubrics).

This session will be of particular interest to advanced users of rubrics who are interested to see
them used as part of a robust and comprehensive system that drives as well as tracks systemic
change with an unapologetically ambitious and ambidextrous agenda: equity and sustainability.
Contributors
avatar for Dr. E. Jane Davidson

Dr. E. Jane Davidson

Dr. E. Jane Davidson is an internationally recognized, award winning evaluation author, keynote speaker, practitioner, and coach. She was the 2019 recipient of the American Evaluation Association’s prestigious Paul F. Lazarsfeld Evaluation Theory Award.Jane is best known for pioneering... Read More →

Wednesday October 12, 2022 2:20pm - 2:50pm NZDT
Oceania
  Presentation, Celebrating

2:50pm NZDT

Afternoon tea break
Wednesday October 12, 2022 2:50pm - 3:10pm NZDT
Oceania
  Conference Activity, Informal occasion

3:10pm NZDT

Panel Discussion 4: Future of evaluation
Chaired by David Stuart
Contributors
DS

David Stuart

Manatu Taonga Ministry for Culture and Heritage
avatar for Dr. E. Jane Davidson

Dr. E. Jane Davidson

Dr. E. Jane Davidson is an internationally recognized, award winning evaluation author, keynote speaker, practitioner, and coach. She was the 2019 recipient of the American Evaluation Association’s prestigious Paul F. Lazarsfeld Evaluation Theory Award.Jane is best known for pioneering... Read More →
avatar for Kiri Parata

Kiri Parata

President, Australian Evaluation Society
Kia ora I'm Kiri, living on Kabi Kabi Country on the Sunshine Coast of Queensland. He uri ahau nō Kāpiti, kō Te Ātiawa, kō Ngāti Toarangatira, kō Ngāti Raukawa, kō Ngāti Ruanui, kō Ngāi Tahu ōku iwi. I work as a Māori health researcher and evaluator, this mahi brings... Read More →
SF
AC
avatar for Natalie Piesse

Natalie Piesse

Evaluation and Research Analyst, Ministry of Social Development

Wednesday October 12, 2022 3:10pm - 4:10pm NZDT
Oceania
  Panel discussion, Into the future

4:10pm NZDT

Poropororoaki
Wednesday October 12, 2022 4:10pm - 4:30pm NZDT
Oceania
  Conference Activity, Formal occasion
 
Filter sessions
Apply filters to sessions.
Monday, October 10
Tuesday, October 11
Wednesday, October 12
  • Conference Activity
  • Featured contributor
  • Panel discussion
  • Panel presentation
  • Presentation
  • Workshop/wānanga/talanoa
    • Icon
    Oceania
    Te Huinga Centre - Rangimarie Room 1
    Te Huinga Centre - Rangimarie Room 2
    Te Huinga Centre - Rangimarie Room 3