Research and Evaluation

Recent Projects

Summary of IRNZ Ltd Projects 2018


Charitable Company Limited Hauora Grants Evaluation

Impact Research NZ (IRNZ) was commissioned by Charitable Trust Company Limited (CCL) to review their existing documentation and conduct an evaluation of the Hauora Grants it provides.

CCL is a wholly owned subsidiary of the Tauhara North No 2 Trust (TN2T) which manages three blocks of land in an area occupied by Ngāti Tahu. TN2T provides a range of grants and programmes to assist eligible whānau across a wide number of activities and needs. To eligible to receive a Hauroa grant, applicants must be linked to TN2T via a beneficial owner (whānau who can trace their whakapapa to those granted ownership of the land or are parties to Whānau Trusts).

Hauora Grants provide financial assistance to eligible whānau to access healthcare under a series of five separate grant categories (General Health, Vision, Dental, Dentures or Orthodontics and Hearing).

  • Data was collected via a document review to provide context and inform the evaluation.
  • Two focus groups, one in Rotorua and one in Taupo attended by 17 whānau members to gain their opinions and experiences of accessing healthcare, and the impact of receiving a Hauora grant for themselves and other whānau members.
  • An online survey was responded to by 76 whānau members who were eligible to receive a Hauora grant. Information was sought to determine their satisfaction with access to a Hauora grant, the demographic details of whānau accessing a Hauora rant, the mAn ain benefit of receiving a Hauora grant and the extent to which a Hauora grant met the costs of healthcare.

An overall report provided a high-level synthesis of key findings from participants to the focus groups and whānau responding to the online survey to inform the future direction of Hauora grants to enable CCL to continue to meet the needs of eligible whānau.

Weir, A., & Holmes, J. (2018). Charitable Company Limited Hauora grants Evaluation. Auckland, New Zealand: Impact Research NZ.


Charitable Company Limited Evaluation of Whānau Grants and Programmes

Impact Research NZ (IRNZ) was commissioned by Charitable Trust Company Limited (CCL) to review their existing documentation and conduct an evaluation of their Whānau grants and programmes.

CCL is a wholly owned subsidiary of the Tauhara North No 2 Trust (TN2T) which manages three blocks of land in an area occupied by Ngāti Tahu. TN2T provides a range of grants and programmes to assist eligible whānau across a wide number of activities and needs. To eligible to receive a Whānau grant or participate in a Whānau programme applicants must be linked to TN2T via a beneficial owner (whānau who can trace their whakapapa to those granted ownership of the land or are parties to Whānau Trusts).

Whānau grants are provided to assist applicants in sports, art and Māori cultural activities that have an educational element. The Tangihanga grant provides applicants with financial assistance to help towards tangihanga related costs. The Whānau Support programme provides support, advice and also advocates for any issues that whānau find challenging in their lives. The Te Hononga Whānau programme provides whānau with the opportunity to receive education, empower and strengthen linkeages to their whakapapa, the whenua and other whānau members through bi-annual events conducted across four locations in New Zealand (Christchurch, Tokoroa, Wellington and Auckland).

Data was collected via:

  • A document review to provide context and inform the evaluation.
  • A telephone survey delivered to 15 selected beneficial owners to explore their knowledge around Whānau grants and programmes, ease of access and main reasons for applying, potential barriers, the impact of Whānau grants and programmes and their suggestions for potential improvements.
  • A paper-based survey delivered to 124 attendees of a CCL hui to gain their knowledge around Whānau grants and programmes, the extent to which Whānau grants and programmes met their needs, potential barriers to participation and their suggestions for potential improvements.

An overall report provided a high-level synthesis of key findings from participants to the telephone and paper-based surveys to inform the future direction of Whānau grants and programmes.

Weir, A., & Holmes, J. (2018). Charitable Company Limited Evaluation of Whānau Grants and Programmes. Auckland, New Zealand: Impact Research NZ.


Support for Young People with High and Complex needs

Impact Research NZ was commissioned by the Ministry of Education (MoE) to provide an evaluation of an education facility to determine the effectiveness of the support it provides to young people and their families/whānau and/or caregivers from the perspectives of:

  • Young people engaged with the service.
  • Families/whānau and/or caregivers engaged with the service.
  • Key contacts from a range of stakeholders (e.g. Government departments, referrers to the service and support agencies.

An evaluation of current data collection and methodology was also undertaken to determine the effectiveness of the service and provide context for the evaluation.

The educational facility is contracted to provide support to young people in school years seven to 13 that have been in the long-term care of Oranga Tamariki and have high and complex needs, behavioural issues and educational needs. Young people are assessed to determine their learning and educational needs within a wider framework that evaluates all areas of young people’s lives. The service is also responsible for coordinating a team of professionals from agencies involved in supporting each young person and ensuring that the principles of the framework and model are adhered to with fidelity. The desired outcomes of the service are to provide support to enable young people to develop skills necessary for them to function in and engage effectively with their community.

Data was collected from the following groups: young people involved with the educational facility, their families/whānau and/or caregivers and representatives from key stakeholders including funding agencies and providers of allied support services.

  • Young people participated in face-to-face interviews to gather information about their expectations of the service. Their experience of the support provided by the educational facility. Perceived changes in their behaviour and attitudes. Aspects of the service that worked well for them and those that had been the most challenging.
  • Families/whānau and/or caregivers participated in face-to-face interviews to gather information about their experience of the service. The extent to which engaging with the service made a difference to themselves and their young person. What aspects of the service worked well and suggestions for potential improvements.
  • Representatives from key stakeholders participated in face-to-face interviews to gather information about their understanding of the framework within which the educational facility operates. Their perspectives of their roles in relation to others. Aspects of the service that they found helpful and those that were found to be challenging. Their opinions of the effectiveness of the service in meeting the needs of young people and suggestions for potential ways in which the service could be improved. Key stakeholders were also asked for their opinions around the potential to replicate the services provided by the educational facility elsewhere in New Zealand.

Results provided the MoE with an evaluation of how well the educational facility was meeting the needs of young people and their families/whānau and/or caregivers and other key stakeholders. An evaluation of data collection methodology to evidence the effectiveness of the service. The quality of the relationships between the service and its key stakeholders. The extent to which the service adhered to the framework of the model of service delivery and practice and the perceived potential to replicate the service in other parts of New Zealand.

An overall report provided a high-level synthesis from all stakeholder groups to inform the future direction and processes of the educational facility and further collaboration with other key stakeholder groups to ensure that the needs of young people and their families/whānau and/or caregivers are effectively met.

Weir, A., & Holmes, J. (2018). An External Evaluation of an Educational Facility. Auckland, New Zealand: Impact Research NZ.


The Far North Safer Community Council – Building Safer Communities AutoMotivate Evaluation Project: “A Passport to Life” 2018

Impact Research NZ Ltd (IRNZ) was commissioned by Far North Safer Community Council to conduct and evaluation of the AutoMotivate programme to gain an understanding of the impact that the AutoMotivate programme had on participating rangatahi and their whānau.

The Far North Safer Community Council established the AutoMotivate programme to provide education and support to rangatahi to create safer drivers and roads and reduce the likelihood of entering the criminal justice system. Utilising a whānau ora approach, the programme provides a wraparound mentoring framework to proactively address issues of rangatahi incurring excessive driving violations, driving with non-compliant vehicles, and breaches of driving licence conditions. The aim of the programme is to support rangatahi to improved access to education, training and employment and to encourage and support rangatahi to become contributing members of society, achieve their aspirations.

Data was collected via:

  • A document review to provide context and inform the evaluation.
  • Focus groups with eight rangatahi who had participated in the AutoMotivate programme to explore reasons for participating in AutoMotivate. What worked well and what did not work well, the impact of the programme on the lives of participants, potential barriers to participation and suggestions for improvements to the programme. A brief paper-based survey was also delivered to focus group members to explore driving violations and high-risk behaviours prior to participating in the AutoMotivate programme and the impact of the AutoMotivate programme on these behaviours.
  • Online and paper-based surveys delivered to 24 rangatahi who had participated in the AutoMotivate programme to gather information about driving violations and high-risk behaviours prior to participating in the AutoMotivate programme, access to the programme, goal setting, aspects of the programme that worked well and not so well and most and least important parts of the programme. Rangatahi were also asked about the impact of the AutoMotivate programme from their perspectives, involvement of other whānau members and suggestions for potential improvements.
  • Focus groups attended by nine whānau of participating rangatahi to gather information about knowledge of the AutoMotivate programme, concerns for rangatahi prior to their participation, expectations of the programme, the impact of the programme for themselves and their rangatahi. Also explored were aspects of the programme that worked well and not so well and suggestions for potential improvements.
  • Online survey delivered to 12 key stakeholders to gather information about their relationship with the Far North Safer Community Council. Their understanding of the AutoMotivate programme and its effectiveness towards its intended aims. The impact of the programme for participating rangatahi and their ongoing relationship with the Police and the potential for the programme to scale up and be replicated in other areas of high need. Questions also asked key stakeholders for their suggestions for potential improvements to the programme.
  • A face-to-face interview was conducted with member of management of AutoMotivate to provide additional context.

An overall report provided a high-level synthesis from all stakeholder groups to inform the future strategic direction of the Far North Safer Community Council and the AutoMotivate programme.

Weir, A., Lowe, G., & Holmes, J. (2018). The Far North Safer Community Council – Building Safer Communities AutoMotivate Evaluation Project: “A Passport to Life”, 2018. Auckland, New Zealand: Impact Research NZ Ltd.


The Helensville Women and Family Centre: External Evaluation of Current Services 2018

Impact Research NZ Ltd (IRNZ) was commissioned by Helensville Women and Family Centre (HWFC) to conduct an evaluation of their current services delivers to the community.

HWFC serves women and their families/whānau in the area of South Kaipara. The organisation utilises employed and contract staff and volunteers to provide services, which include a navigator service to provide advice and information about their services, counselling and also advocates on behalf of their clients to access additional support services. There are a number of family-based programmes and those targeted towards parenting and young children. Strengthening Families is a free family support service available to women, youth and families/whānau and a range of programmes, including abuse prevention, provide a safe environment for those attending. A youth group has also been established in collaboration with Kaipara college.

Data was collected via:

  • A document review to provide context and inform the evaluation.
  • A focus group attended by nine clients of the HWFC to gather information about their knowledge of HWFC services, the impact of the support provided to them by the HWFC and quality of interactions. Aspects of the services that worked well, suggestions for additional services and potential improvements to current services.
  • An online survey delivered to 53 representatives of key stakeholders gathered information about their organisation’s relationship with the HWFC, the extent to which they would recommend the HWFC to others and ease of access for clients. Further questions asked representatives for their opinions of the effectiveness of HWFC services for clients, contribution of such services to the work of their organisation, suggestions for additional services and potential service improvements.
  • A focus group attended by six HWFC staff and volunteers to gather information about training, communication between staff and the organisation and perceptions of important aspects of their roles and challenges. Staff and volunteers were also asked to comment on the services provided by the HWFC to clients and ways in which the HWFC can evidence its effectiveness for clients to key stakeholders and the wider community. Other questions asked for staff and volunteers suggestions for additional services and potential service improvements.
  • An online survey was delivered to four members of the HWFC Board of Trustees. Questions asked about their perceptions of the relevance of services offered by the HWFC to women and their families/whānau in the community. Aspects of the service that worked well, the effectiveness of collaboration between the HWFC and key stakeholders, evidence-based data to demonstrate service impact, potential additional services and future focus of services.

An overall report provided a high-level synthesis from all stakeholder groups to inform the future strategic direction of the HWFC and the services it provides to clients.

Weir, A., & Holmes, J. (2018). The Helensville Women and Family Centre: External Evaluation of Current Services 2018. Auckland, New Zealand: Impact Research NZ Ltd.


Family Violence Community Research Project 2018.

Impact Research NZ Ltd (IRNZ) was commissioned by an independent organisation that provides support to victims of family violence to conduct a high-level literature review on best practice in delivering family violence prevention and support programmes. To explore the current attitudes and beliefs of community members around issues of family violence, accessing support, and to inform the organisations’ future work in the community around education and family violence prevention.

The family violence support organisation provides residential refuge and community-based education and support to women and their children/whānau who have or who are currently experiencing issues of family violence. Through the programmes and education the organisation provides women are educated about their legal rights, ways to establish safety plans for themselves and their children/whānau and create plans to optimise their future wellbeing.

Data was collected via:

  • A brief high-level literature review to provide context and inform best practice for delivering family violence prevention and support programmes.
  • A focus group attended by six community-based members to gather their views on the incidence of family violence in New Zealand, the main causes and types of people affected by family violence and actions to help family or friends seeking help for issues of family violence. Opinions were also sought around potential barriers to help seeking, source of information and knowledge of services for perpetrators or victims of family violence and the most effective means of support from their perspective.
  • A focus group attended by seven professionals or representatives of organisations that was involved, or collaborated with family violence support services to gather their views on their views on the incidence of family violence in New Zealand, the main causes and types of people affected by family violence. Questions also sought their opinions of significant barriers to help seeking and potential solutions, knowledge of available local services, the extent to which their organisation was involved in supporting people using or receiving family violence and ways in which their organisation could contribute to reducing the incidence of family violence, A final question asked focus group participants of their views on the challenges that may be experienced by their organisation delivering support to people experiencing family violence.
  • An online survey delivered to seven professional and community-based members to gather their views of the extent to which they felt that family harm was a significant problem locally and nationally. The causes of and types of people affected by family violence. Knowledge of available support services and potential barriers to seeking help.

An overall report provided a high-level synthesis of the current bank of literature and the views of all stakeholders around the areas under investigation. This information will be used by the organisation to help better understand current attitudes and beliefs about family violence, knowledge and preferences of access to supports and to inform their future work around providing advocacy for and education around family violence and prevention.

Weir, A., Cockle, V., Lowe, G., & Holmes, J. (2018). Family Violence Community Research Project 2018. Auckland, New Zealand: Impact Research NZ Ltd.

Charitable Company Limited Rangatahi Programmes Evaluation of Kia Mau Camp

Impact Research NZ (IRNZ) was commissioned by Charitable Trust Company Limited (CCL) to review their existing documentation and conduct an evaluation of the Rangatahi programmes that it provides. With a focus on youth development and connecting rangatahi to their whᾱnau, whakapapa and whenua, one of the goals of CCL is to ensure that high numbers of rangatahi access education, develop the skills to make positive decisions about their future.

As part of a wider review, an evaluation of CCL’s Kia Mau camp was conducted. The Kia Mau camp is designed for rangatahi aged between 10 and 12 years old, is held annually during the school holidays in October. The camp aims to improve self-esteem, build confidence, support goal setting, enhance communication skills and foster the development of positive relationships. Rangatahi experience a range of activities (e.g. hiking, mystery road trips, planting trees) that are designed to enhance their ability to work in a team and communication skills.

Data was collected via:

  • A pre-camp paper-based survey delivered to rangatahi at the beginning of camp. Questions explored rangatahi’s knowledge of the Kia Mau camp and their expectations prior to attending. Using a five-point Likert scale how rangatahi felt about themselves (e.g. ‘I am happy”, “I feel confident” and how they rated their interactions with others (e.g. “ I am good at listening to others”.
  • A post-camp paper-based survey was delivered to rangatahi at the completion of camp. Questions explored rangatahi’s rating and favourite aspects of the camp and suggestions for potential improvements. For comparison purposes identical questions asked rangatahi how they felt about themselves and their interactions with others.
  • Whᾱnau/parents/caregivers of the attending rangatahi were asked about their knowledge of the Kia Mau camp and their satisfaction with the enrolment process. Further questions explored their previous experience of the Kia Mau camp and the impact of attendance for their rangatahi and suggestions for potential improvements.
  • An interim report of the findings provided a high-level synthesis of the views of rangatahi and their Whᾱnau/parents/caregivers. The results will be incorporated with evaluations of other rangatahi programmes into an overall report which will help to inform the future strategic direction of CCL’s Rangatahi programme.

Weir, A., Lowe, G., & Holmes, J. (2018). Charitable Company Limited Kia Mau Camp Evaluation. Auckland, New Zealand: Impact Research NZ Ltd.



A selection of Impact Research NZ Projects 2017


Brainwave Trust Aotearoa - 2017

Impact Research NZ was commissioned by Brainwave Trust Aotearoa to review their existing documents and evaluate the effectiveness of their programmes and seminars for those attending. The aim of the project was to develop and pilot a Results Based Accountability (RBA) framework and tools to measure their activities from the perspectives of key stakeholders and to provide baseline data. Training in RBA was provided to Brainwave staff and a RBA manual was developed to enable.

Brainwave in the future to evaluate the effectiveness of their activities. Brainwave Trust Aotearoa is a charitable trust that aims to raise public awareness around healthy brain development in babies, young children and adolescents. They deliver seminars and programmes based on robust scientific research to a variety of organisations and individuals who have an interest in ensuring the safety and wellbeing of young New Zealanders.

The project involved seven key areas for investigation:

  • The views of staff and Educators were sought via an online survey on how effective the organisation is in achieving its aims and objectives, the quality of presentation and programme development and delivery as well as areas for improvement.
  • Brainwave School’s programme asked secondary school students via an online survey and student focus group about the effectiveness of the programme. Staff members were asked via an online survey about the alignment with the school curriculum and areas for improvement.
  • Brainwave School’s programme (Health Achievement Standard) – students were asked via an online survey about how well the programme contributed to them achieving their Health Achievement Standard.
  • Canterbury Vulnerable Families Initiative programme was designed for families living in low socio-economic areas affected by the Canterbury earthquakes. Selected parents and caregivers were invited to share their experiences of attending seminars via a focus group and online survey. Professionals from referring agencies such as CYF, police and Plunket via an online survey were invited to share their views on the effectiveness of the seminar for those who they referred.
  • The views of groups and organisations who book Brainwave presentations were asked via an online survey if their needs and expectations were met by the presentations.
  • The views of Brainwave Trustees were asked via an online survey whether the organisation was on track to achieve its vision and mission, as well as adhering to the principles of its strategic direction and the effectiveness of the programmes.
  • Key stakeholders (e.g. NGOs, professionals working with vulnerable children and adolescents) were asked via an online survey about their engagement with Brainwave and how well the presentations were received by those attending and areas for improvement.

An overall synthesis report was provided to Brainwave which highlighted key findings that will be used to inform their strategic direction and utilisation of the newly created RBA framework.

Habitat For Humanity Auckland - 2017

Impact Research NZ was commissioned by Habitat for Humanity Auckland to undertake a social impact research project to inform their future direction. A Results Based Accountability (RBA) framework was developed and implemented. Two RBA workshops were held with senior management with the aim to increase the awareness of the RBA framework and how it can be utilised by Habitat for Humanity Auckland to improve its service outcomes leading to impact. A RBA manual was also provided to Habitat for Humanity Auckland to enable them to evaluate their future programmes and activities.

Habitat for Humanity Auckland offer programmes to help low-income earners in their living situations, providing them with services to maintain and repair their homes, and assistance towards home ownership.

The project involved four key areas under investigation: The status of their current programmes, perceptions of the current rental housing market, views on a potential new programme and the current funding landscape. Data was gathered via the following methods.

  • Habitat for Humanity Auckland “A Brush with Kindness” (ABWK) and “Critical Home Repairs” (CHR) programmes. The views of clients were sought via telephone interviews about their experiences of the programmes as well as areas for improvement. The view of the ABWK programme manager was also sought via a face-to-face interview about the effectiveness of
  • the programmes and potential future direction.
  • Rotary Club members were asked via an online survey and property managers via a telephone survey about their perceptions of the current rental housing market and a proposed new programme being developed by the organisation.
  • Selected Habitat for Humanity Auckland tenants shared their experiences of renting via a focus group.
  • Representatives from philanthropic trusts, local and national government were asked via face-to-face interviews about their perceptions of the current funding landscape and a proposed new programme being developed by the organisation.

An overall report provided a high level synthesis of the key findings to inform the future direction of Habitat for Humanity Auckland and its programmes. RBA workshops ensured buy-in from senior managers who are responsible for implementing RBA with their teams.

Enliven Home-Based Support service RBA refresh - 2017

Impact Research NZ was commissioned by Presbyterian Support Northern to conduct a refresh of its current RBA framework for the Enliven Home-Based Support service. The purpose of this project was to evaluate the perspectives’ of key stakeholder groups to inform the development and utilisation of a new RBA framework for future use.

Enliven provides a range of home-based support services to clients who, as a result of aging or disability, are unable to carry out day to day tasks around the home. The focus of the service is to assist clients to remain healthy, maximise their independence and engage with their community. Clients are supported to achieve their own goals, receive help with personal care and household tasks.

Data was collected from the following groups: Representatives from the Ministry of Health, Home and Community Health Association, DHBs that Enliven contracts to, Enliven clients, staff and management.

  • A document review was used to inform questions for interviews and surveys with key stakeholder groups.
  • A selected group of Enliven clients were surveyed via telephone interviews about their experiences of receiving the Enliven service and suggestions for improvement.
  • Key contacts from DHBs who contract Enliven, the Ministry of Health and Home and Community Health Association and were sought via face-to-face interviews about the Enliven service and reporting requirements.
  • The Enliven Quality Committee were surveyed via a focus group to gain their insights into current data collection practices, suggestions for additional data collection and potential changes to service delivery in response to changing client needs.
  • New client satisfaction surveys were designed and piloted in telephone interviews with a selection of Enliven Home-based support clients.

The results from all key stakeholder groups were used to inform the development and utilisation of a newly created RBA framework for ongoing use.

Enliven Bay of Plenty Home - Based Support Establishment with the Disabilities Resource Centre - 2017

Impact Research NZ was commissioned by Presbyterian Support Northern (PSN) to design and develop a Results Based Accountability framework for future evaluation in response to the partnership of Enliven Bay of Plenty (BOP) with the Disability Resource Centre (DRC).

The perspectives of all key stakeholder groups were gathered through a series of interviews and focus groups to inform the RBA framework and toolkit.

Data was collected from the following groups: Representatives from the Ministry of Health, Home and Community Health Association, DHBs that Enliven contracts to, Enliven clients, staff and management and DRC staff and management.

  • Key contacts from the Ministry of Health and the Bay of Plenty DHB were surveyed via face-to-face interviews about the Enliven service and reporting requirements.
  • The Enliven Quality Committee were surveyed via a focus group to gain their insights into current data collection practices, suggestions for additional data collection and potential changes to service delivery in response to changing client needs.
  • A focus group was conducted with DRC staff and a face-to-face interview is to be conducted with a key contact from the DRC.
  • Survey instruments were designed and delivered via online surveys with Enliven and DRC staff.

The results from all key stakeholder groups were used to inform the development and utilisation of a newly created RBA framework and toolkit for ongoing use by Enliven BOP and DRC.

Lifeline RBA Evaluation - 2017

Impact Research NZ was commissioned by Presbyterian Support Northern (PSN) to review their existing documents and processes and to pilot and establish a RBA framework for the 24/7 and Suicide Crisis (Tautoko) helplines.

The main areas of interest were: The quality factors of a helpline and the indicators of the service that were helpful to callers.

  • Data was collected via a document review.
  • A RBA framework and workshop was held with the CEO of PSN and Lifeline’s Executive Director.
  • Focus groups with Lifeline staff and volunteers in Auckland and Christchurch were conducted to gain their perspectives around current data collection practices and suggestions for future data collection.
  • A face-to-face interview was conducted with a member of Lifeline’s management for additional context.
  • Survey instruments were piloted to evaluate the effectiveness of the 24/7 and Suicide Crisis Helpline (Tautoko) from the perspectives of callers.

An overall report provided a high level synthesis of key findings from all stakeholder groups to inform the future direction of Lifeline, its processes and services for callers. The newly created RBA framework and toolkit enabled Lifeline to conduct future evaluations of the 24/7 and Suicide Crisis (Tautoko) helplines from the perspectives’ of callers.

A Women’s Refuge Trust - 2017

Impact Research NZ was commissioned by the Trust to review their existing documents and develop and trial a new RBA framework for the ongoing evaluation of the support that it provides and to inform the Trust’s future strategic direction.

The Trust provides support to women and their children who have previously or are currently experiencing the effects of family violence. Activities include the provision of residential or community-based support, advocacy and advice, education, safety plans and community-based awareness of the effects and prevention of domestic violence.

  • Data was collected via a document review.
  • RBA surveys were developed and piloted with:
    • Six current and past community outreach and refuge clients to elicit their opinions and experiences of the support received from the Trust and suggestions for potential service improvements.
    • Four members of Trust staff to gain their understanding of their role and opinions of the effectiveness of the Trust for women and children that they support.
    • Six members of the Board of Trustees for their opinions regarding the effectiveness of the support that the Trust provides.
    • Twenty-three key contacts from agencies and community organisations that have involvement with the Trust to gain an understanding of their satisfaction with their organisation’s relationship with the Trust and potential service improvements.

The results from all key stakeholder groups and workshops with Trust staff were used to inform the development and utilisation of a newly created RBA framework for ongoing use. A RBA manual was prepared and provided to KTAT to enable them to evaluate their services and programmes in the future.

SHINE in School programme - 2017

Impact Research NZ was commissioned by Shine, a division of Presbyterian Support Northern (PSN) to review their existing documents and evaluation processes as well as to develop and trial a new RBA framework and to undertake an evaluation of their SHINE in School programme.

The SHINE in School programme is currently delivered to students in years nine to 13 in a total of 14 secondary schools and youth alternative education providers in the Auckland region by a team of three educators. Presentations may be held during school assemblies and students are also able to engage in workshops which provide them with information aimed at increasing their understanding around healthy relationship and the effects of family violence. Student leaders are supported to organise community-based events aimed at raising awareness among a wider audience and act as advocates for peers experiencing family violence.

  • Data was collected via a document review.
  • Three online RBA surveys were developed and trialled with:
    • Twenty-seven students who had participated in the SHINE in School programme to gain their experiences of the programme and suggestions for potential programme improvements.
    • Three school staff who had been involved with the SHINE in School programme to gain their opinions if its effectiveness for students in their school.
    • Four key contacts of community groups that had been involved with SHINE in School programme events to gain their opinions of the relevance and effectiveness of events for raising public awareness of community-based support services
  • Face-to-face interviews were conducted with three members of Shine management and two educators to gain their opinions of the effectiveness of the SHINE in School programme and the processes currently utilised to deliver and evaluate the programme.
  • A face-to-face interview was conducted with a member of PSN management for additional context.

An overall report provided a high level synthesis of key findings from all stakeholder groups to inform the future direction and processes of the SHINE in School programme. The newly created RBA framework and RBA manual was prepared and provided to Shine to enable the SHINE in School programme to be evaluated in the future. An RBA workshop was conducted with nominated Shine staff members to further inform them of the processes involved in conducting RBA evaluations.

Kia Piki ai te Oranga a Nga Whānau me Nga Hapū
2016

Impact Research NZ worked in collaboration with Shea Pita who were commissioned by The Ministry of Health to provide a series of reports to inform the Kia Piki te Ora Māori suicide prevention post evaluation programme in Aotearoa New Zealand communities.
The first report presented a literature review to understand the context of Māori suicide by analysing available Māori suicide data, understanding the diverse causes of suicide for Māori, the current health policy framework underpinning New Zealand’s response to Māori suicide prevention and an understanding of effective interventions. The second report involved presenting a landscape review and comparative analysis of relevant strategies, programmes or initiatives that may impact upon the Māori suicide prevention landscape.
The third report involved outlining a draft model to support outcomes-focused Māori community leadership in suicide prevention.

Weir, A., McEwan B., & Shea, S. (2016) Understanding the drivers, impact and ‘what works’ for Māori suicide prevention within Aotearoa New Zealand communities. Wellington: Ministry of Health
Shea, S., & Weir, A. (2016). A landscape and comparative analysis of Māori suicide prevention within Aotearoa, New Zealand. Wellington: Ministry of Health.
Shea, S., & Weir, A. (2016). Kia Piki ai te Oranga a Nga Whānau me Nga Hapū : A draft model to support Māori communities to lead outcomes-focused suicide prevention, within the overall context of Community Wellbeing. Wellington: Ministry of Health.

The Royal New Zealand Plunket Society – Family Works Northern research on the joint Early Intervention Service
2016

The Royal New Zealand Plunket Society and Family Works Northern collaborated together to provide a short-term early intervention service. Impact Research NZ was commissioned to provide an evaluation of the effectiveness of a newly established Early Intervention Service in the Waikato area. Clients of the service participated in interviews to establish how well they thought the service met their needs, the positive outcomes experienced as a result of their engagement and suggestions for service improvements. Staff of Plunket and Family Works Northern participated in an online survey which asked them to indicate opinions relating to the success and strengths of the service and the importance of aspects of service delivery that contributed to its success. Staff were also asked for their suggestions for service improvements. An evaluation report presented the findings from the client interviews and staff online surveys to provide an insight into the ways in which the service was appropriately provided to clients and the extent to which the service provided the desired and intended outcomes for clients.

Weir, A., Holmes, J. (2015). Evaluation of the Royal New Zealand Plunket Society-Family Works Northern Huntly Early Intervention Service 2015. Impact Research NZ, Auckland.

START Taranaki Results Based Accountability (RBA) Evaluation
2016

START Taranaki commissioned Impact Research NZ to work with them to undertake and organisation review of their evaluation and reporting practices as well as to develop and implement a Results Based Accountability (RBA) framework. The project also involved providing START Taranaki staff with training in RBA, linking the strategic and operational plans to an organisational evaluation plan.
Three groups of stakeholders were surveyed utilising an RBA surveys. The perspectives of youth were surveyed via a focus group and a paper-based survey. Youth were asked about their expectations and experiences of the programme, their relationship with their key worker, and suggestions for potential programme improvements. Youth were also asked whether they had changed as a result of being on the programme, if they had developed new skills and coping strategies and if they would change their behaviour in the future.
Individual interviews and focus groups with staff and management of START Taranaki asked about their perceptions of the START Taranaki organisation, and the programme fit with other available programmes. Key stakeholders were asked for their perceptions of the underlying philosophy of START Taranaki and the benefits of their organisations’ relationship with the programme.
An overall evaluation report outlined stakeholders’ perceptions of the effectiveness of the START Taranaki programme, the extent to which it met its intended outcomes and suggestions for ways in which the programme could be improved. A Results Based Accountability manual was also created to enable START Taranaki to continue to evaluate the programme and monitor its success for themselves.

Weir, A., McEwan, B., & Holmes, J. (2016). START Taranaki Results Based Accountability (RBA) Survey Report. Impact Research NZ, Auckland.

Family Works Resolution Service RBA Evaluation
2016

Impact Research NZ was commissioned by Family Works Resolution Service to undertake and evaluation of its service by asking clients to participate in a paper-based or online survey which asked them to indicate how well they believed the service met their expectations, and how effective it was in assisting them to reach a lasting arrangement for their care of their children. A total of 77 clients responded and a report of the survey. Findings will provide details of client demographics and level of their involvement with the service. In addition clients’ experiences of the service, the benefits to clients and their children of their involvement and suggestions for improvements to the service will also be presented.

Weir, A., & Holmes, J. (2016). Family Works Resolution Service: Results Based Accountability Survey Report 2016. Impact Research NZ, Auckland.

Enliven Home Based Support Results Based Accountability (RBA) Evaluation
2016

Presbyterian Support Northern (PSN) commissioned Impact Research NZ to undertake an evaluation of the Enliven Home Based Support service. In 2016, 370 clients responded to a paper-based survey. Questions asked clients to indicate how effectively the service met their needs and situations, the positive benefits experienced as a result of their involvement with the service and suggestions for service improvements. Six Enliven sites each received a report outlining their regional results and one overall evaluation report which provided information on the effectiveness of the Enliven Home Based Support service, the extent to which the service met its intended outcomes and the positive benefits that clients experienced as a result of their involvement as well as suggestions for ways in which the service could be improved.

Weir, A., & Holmes J. (2016). Enliven: Results Based Accountability Survey 10 Service Report 2016. Impact Research NZ, Auckland.

Social Workers in Schools (SWiS) RBA Evaluation
2016

Presbyterian Support Northern (PSN) commissioned Impact Research NZ to undertake the Social Workers in Schools (SWiS) RBA Evaluation. At the time of the survey a total of 126 schools based in the upper North Island were eligible to participate. There were795 children involved with the service which was delivered by 65 social workers. Children and their parents and caregivers were provided with a paper-based questionnaire to complete and members of school staff responded to an online survey. Questions asked each group to indicate how effective they thought the service was, what the benefits of being involved with the service were for children, parents and caregivers and school staff and suggestions for ways in which the service could be improved. In 2016 responses were provided by more than 440 people surveyed. Eight PSN sites each received a report outlining their regional results to present to the schools in their area. One overall service evaluation report provided information on the effectiveness of the SWiS service, the extent to which it met its intended outcomes and clients’ suggestions for improvements to the service.

Weir, A., Holmes, J., & Schonberger, S. (2016).

Family Works Resolution Service RBA Evaluation
2016

Family Works Northern and Anglican Trust for Women and Children Youth Workers in Secondary Schools (YWiSS) Check and Connect Service, Results Based Accountability (RBA) Pilot Survey (2015)

The Youth Workers in Secondary Schools (YWiSS) Check and Connect service is jointly provided by the Anglican Trust for Women and Children (ATWC) and Presbyterian Support Northern/Family Works Northern (PSN/FWN). Impact Research NZ was commissioned to develop an RBA framework to evaluate the effectiveness of the YWiSS Check and Connect Service and its intended programme outcomes. Four groups of stakeholders were invited to participate in an online survey. Youth involved with the service, and their parents and caregivers were surveyed for their perspectives on how well the service was received, the impact of engagement for youth and suggestions for improvement to the service. Staff from seven schools in the Auckland area were surveyed for their perceptions of the YWiSS Check and Connect service and its impact on their roles within the schools.

YWiSS Check and Connect staff and members of management were also sought and questions included perceptions of the service, their roles and relationships with youth, parents and caregivers and school staff. A total of 73 responses were received and an evaluation report was provided which presented the results of all surveys. The development of the RBA framework and associated questions, agreed in consultation with ATWC and FWN were demonstrated to provide measurable data which assisted in the evaluation of the ability of the YWiSS Check and Connect service to achieve its goals and meet the needs and provide positive outcomes for clients. This approach also enabled an evaluation of clients suggestions to inform future service improvements.

Weir, A., Holmes, J. (2015). Family Works Northern and Anglican Trust for Women and Children Youth Workers in Secondary Schools (YWiSS) Check and Connect Service, Results Based Accountability (RBA) Pilot Survey Overall Service Report 2015. Impact Research NZ, Auckland.

Family Works Northern Results Based Accountability (RBA) Casework Evaluation Client Survey Report
2016

Impact Research NZ was commissioned by Family Works Northern to undertake an evaluation of their Family Works Northern Casework service, which provides support to individual clients and families/whānau through the provision of social work and family counselling. In 2016 a mailing house was trialled to invite the three stakeholders groups to participate in a client survey. Children aged 7-12 years were surveyed regarding their happiness, experience of school, problems, safety, ability to manage challenges, knowledge of where to obtain help and opinion of their Family Worker. Adult and youth clients were surveyed regarding their experiences with the service, the degree to which the service had helped them or their family/whānau and suggestions for improvement to the service. A total of 178 client responses were received and the results of all surveys were evaluated and compiled into a report. The RBA evaluation provided measurable data on the effectiveness of the Family Works Northern Casework service at meeting the needs of clients and producing positive outcomes. The evaluation also incorporated client suggestions for improvements to the service. The trial of the mailing house resulting in fewer responses than previous methods of survey delivery.

Weir, A., & Holmes, J. (2016). Family Works Northern Results Based Accountability (RBA) Casework Evaluation Client Survey Report 2016. Impact Research NZ, Auckland.

Other Projects

Family Works Resolution Service: Results Based Accountability Pilot
2015

Impact Research NZ was contracted by Family Works Resolution Service to develop, trial and evaluate a survey mechanism to gain evaluative feedback about the provision of Family Works Resolution Service. A combination of survey and interview data is being collected from Family Works Resolution Service clients, utilising a Results Based Accountability (RBA) approach.
In addition, survey and interview data is also being collected from key stakeholders, including call centre staff, contracted mediators and preparatory counsellors, and managerial personnel. The client survey mechanism is aligned with the Ministry of Justice and Family Works Resolution Service’s Key Performance Indicator (KPI) requirements, as part of monitoring and reporting on the provision of services, including the production of management and funder reports. KPI and RBA reporting, when combined, will form a comprehensive evaluative framework.

Developing NGO Capacity to Undertake Evaluation
2015 – Current

Impact Research NZ has been working with a large NGO (that offers a broad range of social services for families in South Auckland) to undertake an organisational review of their tools used to evaluate their programme and services. Workshops have been conducted on developing an organisation and programme evaluation plans linked to their Annual Plan and Strategic Plan. Other workshops focused on developing and implementing RBA and innovative models of evaluations. The NGO was also interested in its organisational responsiveness to Maori and Pasifika and both hui and fono were held facilitated by Impact Research NZ and reports provided to the NGO for follow-up.

Community Initiative – Process Evaluation
2015

Impact Research NZ, in partnership with David Turner Research Ltd, undertook a process evaluation of a community initiative pilot project funded by MSD currently being trailed in two Auckland districts: Manukau and Waitakere. The pilot project is targeted to assist low income individuals and families to improve their circumstances. A process evaluation assesses whether a program suits the context for which it was designed, if it has been appropriately implemented, and if it has reached those persons for whom it would provide the most benefit.

The evaluation methodology utilises a combination of interviews and document analysis to achieve the evaluation objective. The evaluation focuses on the quality and quantity of services delivered through the programme, both from a programme perspective and from the client perspective. The process evaluation was based on the programme’s intervention logic, which describes the problem to be addressed as well as identifying programme inputs, outputs, and intended outcomes. The evaluation l was used to inform future ongoing monitoring of the programme.

Turner, D., Weir, A., McEwan, B., & Holmes, J. (2015). Community Finance Pilot: Process Evaluation 2015. Ministry of Social Development: Wellington.

Social Workers in Schools RBA Evaluation
2015

Presbyterian Support Northern commissioned Impact Research NZ to undertake the Social Workers in Schools (SWiS) RBA Evaluation. 126 schools in the upper North Island participated in the survey. The number of SWiS in schools = 71. The number of children in SWiS service = 1653. In 2015 the service evaluation included responses from more than 650 responses. Children and their parents/caregivers participated in a paper-based survey and the school staff in an online survey designed to find out how well the service is doing, suggestions for improvements and what are the outcomes of the service for children, their parents/caregivers and the school staff. The ten regional sites each received a report to share with their schools and an overall report provides information on how well the services is doing in reaching its intended outcomes and what services improvements are to be made.

Family Works New Zealand RBA Pilot Survey
2014/15

The Ministry of Social Development funded the establishment of a common outcome measurement framework, including conducting a baseline RBA pilot survey of clients, across seven Presbyterian Support regions throughout New Zealand. Impact Research NZ was commissioned to undertake the development and implementation of the RBA pilot surveys of children, youth and their parents/ caregivers. The development of the survey instruments was done in collaboration with the seven regions. Each region received a report on their survey findings including client demographics, quantitative and qualitative data, as well as an overall report looking at the Family Works NZ service RBA survey results as a whole.

An External Review of the Response Made by The Salvation Army to the two Canterbury Earthquakes of September 2010 and February 2011
2014

Impact Research NZ was commissioned by The Salvation Army to undertake an external review of its response to the two Christchurch earthquakes of September 2010 and February 2011; with the aim of discovering what could be done to ensure that The Salvation Army’s response to future disasters was appropriate, effective and consistent with its Mission Statement. The researchers worked in collaboration with senior Salvation Army staff to formulate the overall research design, and to identify the necessary research documentation and interview participants. A literature review was undertaken to identify key research findings in relationship to best-practice disaster management.

The research project undertook two stages of data collection. In stage one, a document review and analysis was undertaken assessing a range of earthquake-response documents including: Salvation Army-based daily field reports, email communications between staff, quarterly and annual Board reports, financial reports related to earthquake fundraising and expenditure, website press releases, and two low-level review reports completed by the Emergency Services. In stage two, a mixture of face-to-face and telephone interviews were undertaken with senior Salvation Army staff; and key informants selected from government, local government and NGO personnel involved in the Canterbury earthquakes.

The research findings, in conjunction with the ‘best practice’ disaster-response research literature, identified a number of organisational opportunities for The Salvation Army to consider when reviewing its Disaster Response Policies, Protocols and Practices.


Previous

A selection of Impact Research NZ Research and Evaluation Project Summaries
2014 - Current

Impact Research NZ has been contracted by Family Works Resolution Service to develop, trial and evaluate a survey mechanism to gain evaluative feedback about the provision of Family Works Resolution Service. A combination of survey and interview data is being collected from Family Works Resolution Service clients, utilising a Results Based Accountability
Read more

Effective Client-Focused Dementia Day Programmes in the Community: Improving the quality of life for people living with dementia and their families
2014 - Current

This evaluation of day programmes for people living with dementia, aims to identify the elements of an effective client-focussed programme, identify the methods employed by organisations to measure the quality of outcomes of programmes, and lastly to determine the efficacy of the methods adopted.
Read more

The Royal New Zealand Plunket Society - Family Works Northern Results Based Accountability Pilot Evaluation
2014 - Current

In Collaboration with the Royal New Zealand Plunket Society and Family Works Northern, Impact Research NZ have designed and implemented an evaluation plan, ultising a Results Based Accountability (RBA) framework, to access the outcomes of an early intervention partnership pilot. This evaluation aligns with finding and recommendation from an earlier evaluation on the pilot service conducted by Impact Research NZ, which identified that the ongoing collection of robust outcomes-focused data would support the success of the service.
Read more

Ⓒ 'Dancing with Data' Programme
2014-2013

This practical evaluation programme was offered in partnership with Auckland Communities Foundation and funded by SKYCITY Auckland Community Trust Board. Three cohorts of participants from social service NGOs completed the programme, which comprised three workshops of two hours duration, with an expectation that development activities would be completed between the workshops; an overall commitment of approximately 12 hours over the duration of programme.
Read more

Family Works Northern Results Based Accountability (RBA) Qualitative Analysis of Client Feedback
April 2014 - May 2013

Impact Research NZ have were commission in 2013 and 2014 to undertake qualitative thematic analysis on data collected from multiple sites from clients on their experiences and outcomes of engaging with Family Works Northern’s social support services.
Read more

Enliven Stakeholder Survey
March 2014 - March 2013

An online survey of stakeholders, including District Health Boards and Needs Assessment Agencies, was designed and conducted on behalf of Enliven in 2013 and 2014. The survey gathered both quantitative and qualitative data in order to gain an understanding of the Enliven service’s performance from the perspective of its stakeholders.
Read more

ACC Home and Community Support Services: Initial scoping exercise towards a standardised approach to data collection and client outcomes
December 2013

Impact Research NZ was commissioned by ACC Home and Community Services to undertake a scoping exercise in order to determine the how the six lead providers of services currently collect and use client data to support the quality of their service provision. Multiple data collection methods were used to gather data from a range of stakeholders.
Read more

Enliven ACC, Results Based Accountability (RBA) Pilot Evaluation
November 2013

This evaluation of Enliven’s home support services for clients of ACC Home and Community Support Services designed an piloted a Results Based Accountability (RBA) survey, in order to measure outcomes of the service.
Read more

Evaluation of the Royal New Zealand Plunket Society–Presbyterian Support Northern Family Works Early Intervention Partnership Pilot
June 2012 - November 2013

The Royal New Zealand Plunket Society and Family Works Northern collaborated to pilot an innovative model of early intervention social service provision in two areas of high socio-economic over eighteen months. Impact Research NZ was commissioned to conduct an evaluation of the early intervention pilot utilising a participatory approach that engaged key stakeholders and used a range of data sources. The evaluation addressed the effectiveness of the service in terms of meeting its goals and objectives, identified factors that supported effective service delivery, and made recommendations for service delivery going forward.Read more

Presbyterian Support Northern and the West Fono Health Trust, Waitakere Family Start.Results Based Accountability Pilot Evaluation
October 2013

Waitakere Family Start is a collaborative service delivered by West Fono Health Trust and Presbyterian Support Northern. The service supports families/whanau through the early years of a child’s life, with the child at the centre of service delivery. Impact Research NZ conducted a pilot evaluation of the service, utilising two different approaches to the collection of feedback from clients, in order to assess outcomes for clients of the service, and whether there were advantages to either data collection method, in terms of supporting service outcomes and quality.
Read more

Presbyterian Support Northern, Whakamanhia Wahine Tauranga Women’s Programme Evaluation
June 2013 – November 2013

Impact Research NZ was commissioned by Presbyterian Support Northern to conduct an evaluation of their pilot programme for women offenders, funded by Department of Corrections as part of their Innovation fund aimed at reducing reoffending. The evaluation aimed to explore how effective the programme was in achieving its intended aims and included reviewing the programme implementation and how well it was received by participants and key stakeholders.
Read more

Family Works Northern Social Workers in Schools (SWiS) Results Based Accountability (RBA) Evaluation
March 2012
Read more

Previous

Incredible Years
The goal of Child Youth and Family (CYF) is for children in care to be safe, happy and part of a family. If a child cannot return to their own family, permanent care is required. Care does not always address emotional and behavioural difficulties; this can lead to a cycle of failure and aggression. Read more

Inner City Interconnectedness
Connectedness refers to the relationships people have with people around them and their surroundings. It includes relationships with family, friends, neighbours and people they have met at clubs or through voluntary work for the community. Connectedness also encompasses the way people feel about the built environment: if there are good services available, if they can walk around their neighbourhood, if the public transport is easy to use and if they feel safe.Read more

Kidpower Healthy Relationships
In 2010 Impact Research evaluated the Kidpower, Teenpower, Fullpower, Healthy Relationships programme. The programme provided training in violence prevention, personal safety and self defence. It was designed for all ages and abilities, for individuals, staff and professionals. It included risk reduction and successful communication techniques. Read more

Kiribati – Environmental Migrants
Kiribati (pronounced “Keer-ah-bhass”) comprises three groups of islands in the South West Pacific. The people of Kiribati are not Kiribatese, but I-Kiribati (pronounced “Eee-Keer-ah-bhass”). Kiribati is expected to be the first country in which all land territory disappears due to global climate change. New Zealand has an immigration agreement with the government of Kiribati to relocate the first few environmental refugees.Read more

Midlands - Results Based Assessment
In 2010 Family and Community Services (FACS) commissioned Presbyterian Support Northern to conduct an outcomes-focused evaluation of Midland family and budgeting services. Impact Research (of PSN) conducted a Results Based Assessment (RBA) pilot and recruited 26 providers to complete three RBA surveys over a two year period. Read more

National Social Workers in Schools (SWiS)
This one year pilot was commissioned by Child Youth and Family (CYF) to evaluate the effectiveness of a Results Based Assessment (RBA) framework for measuring changes in SWiS services. Read more

Plunket

This pilot-programme is trialling an early intervention that involves Plunket and Family Works Northern staff working together to support clients. It is expected that the two organisations will achieve more for clients through working together, than they could working independently. Read more

Results Based Assessment
Results Based Assessment (RBA) is a type of evaluation that measures outcomes. Government departments, not-for-profit organisations and educational services are increasingly relying on RBA to provide evidence for the success and quality of their practices. PSN has been at the forefront of RBA development since 2009. Using specially designed survey tools, PSN collects information from Family Works, Enliven, Coactive and Budgeting Services. Read more

Refugee Catch-Up Class
The refugee student catch-up class began in 2005 after Somali community members voiced concern about the under-achievement of their college-age, school children. The catch-up class is a collaborative effort between families and community. The majority of students are Somali and Eritrean. Read more

Safer Homes in New Zealand Everyday (Shine)
Shine delivers training on domestic abuse to a wide range of professionals who work in the social services or criminal justice system. A small number of these training programmes are open to the public. Read more

Social Workers in Schools
The Social Workers in Schools (SWiS) service is funded by Child, Youth and Family (CYF) under the Ministry of Social Development (MSD). The service was established in response to the impact social and family problems were having on students. Read more

The Theatre in Education Trust (THETA) Sexwise Programme
The New Zealand Health Curriculum states that the purpose of sexuality education is to provide students with the knowledge, understanding and skills to develop positive attitudes towards sexuality, to take care of their sexual health and to enhance interpersonal relationships. Read more

Waitakere Process Evaluation
In 2010 Presbyterian Support Northern (PSN) began a results based assessment (RBA) pilot with Social Workers in Schools (SWiS) based in Waitakere. Read more

Research Findings

Jun 2013 - Effective Client-Focussed Dementia Day Programmes in the Community: Improving the Quality of Life for People Living with Dementia and their Families

Nov 2012 - Evaluation of Safer Dance Parties and Large Events

Nov 2012 - Safer Parties Together: One Voice

Jan 2011 - Evaluation of the Kidpower Teenpower Fullpower Healthy Relationships Programme

Dec 2010 - Review of North Shore Family Violence Crisis and Advocacy Service Requirements


Our People
At Impact Research NZ we believe that our people are our most valuable assets. With researchers and Research Associates who are leaders in their fields, our combined experience and expertise encompasses a broad range of social sector interests.
Read more


Our Research and Evaluation
Research and evaluation are now seen as essential elements in the provision of the social services, and provide the means of measuring outcomes for clients, and informing service improvement.

Read more


Contact Us
Impact Research NZ
PO Box 9591 Newmarket
Auckland 1149
Phone: +64 027 612 4335
Email: This email address is being protected from spambots. You need JavaScript enabled to view it.
Read more