APC Mid-term Evaluation: 2013-2016 Strategic Plan (Executive Summary)

The Association for Progressive Communications (APC), established in 1990, is an international network of civil society organisations dedicated to empowering and supporting groups and individuals working for peace, human rights, development and protection of the environment through the strategic use of information and communication technologies (ICTs), including the internet. Supported by Sida, APC hired Foot in the Door Consulting to conduct a comprehensive mid-term evaluation of the outcomes to date of APC’s implementation of its 2013-2016 strategic plan, with a view to informing the development of its next strategic planning process in May 2015.

Data for this evaluation was collected using a mixed-method methodology that combined the use of quantitative and qualitative data collection tools including: 1) a literature review of past reports and research products; 2) pre-interviews with 13 members of APC’s Board of Directors and staff to formulate the evaluation questions; 3) online surveys [1] in English and Spanish; 4) an open call to participate in the evaluation process on four APC listserves; and 5) in-person and virtual interviews with key stakeholders.

A total of 268 stakeholders were initially targeted for participation in the survey and interviews, plus an additional nine people participated in the survey via the open invitation on four APC listserves. Of this total of targeted and open invitations, there was an overall participation rate of all stakeholder groups of 60% (161 respondents) including APC staff (100%), APC’s Board (100%), organisational members (81%), individual members (35%), APC-identified implementing partners (79%), donors (71%) in addition to 58 other respondents who identified as APC allies or “other”.

Findings on APC’s impacts to date

Using examples drawn from the data gathered through the key stakeholder survey and interviews, it is clear that those who participated in the evaluation believe APC is in a healthy position to deliver on the majority of its current strategic plan. Major findings include:

APC’s greatest strengths and weaknesses

All respondents (161) were asked in an open-ended question what they felt was APC’s greatest strength and weakness.

• Over 50% of respondents for this question, representing all stakeholder groups, commented on its ability to function as an active, truly global network valuing inclusiveness, diversity and communication.

• The majority of feedback on APC’s greatest weaknesses focused on how the organisation functions operationally – both internally and externally. Feedback on the highest-ranked area of weakness (e.g. challenges of effective, responsive communication and decision making within a global network) came exclusively from organisational members, project partners and allies. Feedback related to weaknesses in internal capacity, structures and systems came exclusively from staff, allies and donors – this was also the area where the most donor feedback was received. Concerns related to insufficient resources for regional, national or community-led issues were offered by representatives of all stakeholder groups, with the exception of donors.

Key result area 1: Securing and defending internet rights

The goals under this key result area are concerned with: 1) advocating for universal, affordable access to the internet and 2) the recognition that internet rights are human rights that need to be protected and defended. Eighty-nine percent (89%) of respondents (143) indicated an interest in, and experience with, APC’s work on securing and defending internet rights in this evaluation.

• 83% of respondents indicated a score of 5 or higher [2], agreeing that APC is a global leader in advocating for universal, affordable access to the internet.

• 87% of respondents indicated a score of 5 or higher, agreeing that APC is an effective advocate in promoting and defending human rights on the internet.

• 77% of respondents indicated a score of 5 or higher, agreeing that the internet freedom movement has become more inclusive of the voices and needs of activists from the global South as a result of APC’s leadership.

• 70% of respondents indicated that APC’s research on internet rights, as part of human rights and universal access to the internet, substantively informs their/their organisation’s work in this area.

• 51% of respondents indicated a score of 5 or higher, agreeing that advocating for the use of radio frequencies, TV white spaces (TVWS) and opportunities provided by digital migration strategies is still an effective approach to ensuring universal, affordable access to the internet. However, when organisational and individual members were asked if they engaged in the development of innovative uses for radio spectrum or TV white space (TVWS), only 11% said yes.

• 70% of respondents indicated a score of 5 or higher, agreeing that APC is a key player in monitoring threats and violations of internet rights.

• 66% of respondents indicated a score of 5 or higher, agreeing that APC is a key player in responding to threats and violations of internet rights. Those who rated APC low in this area commented that there are many other organisations better positioned to respond, but that APC could do more than they are doing – particularly to threats towards network members – such as an early-warning monitoring system role via information from the network.

• There was agreement among all respondents for this section that APC’s key activities [3] in this area are relevant and useful.

• There was also agreement that KRA 1 signature research products [4] have been useful, or were useful at the time, but some are no longer as useful (e.g. APC La Rue Framework, APC Internet Rights Charter). It was also noted that some products could be more useful if they were updated or revised to be more user-friendly (e.g. Internet Rights, Human Rights online curriculum).

• Almost all respondents noted the superior quality of APC’s research. However, when asked what APC could do to deepen the reach and impact of its research products, respondents overwhelmingly suggested having clearer, simpler communications and a multilingual, multi-media dissemination strategy linked to how the research can be used to further APC’s advocacy goals. This was a consistent theme across all stakeholder groups, but especially for those stakeholders trying to use the research in advocacy and policy work such as staff, members, project partners and allies. One donor noted that achieving these improvements would be a direct result of APC investing more in its internal capacity.

• Members (organisations and individuals), project partners, allies, staff and others gave very similar examples of how they use the research, which included: as references for their own research initiatives; to inform and support advocacy positions; to develop training materials; and, as a means to mobilise and engage people within their own networks. Several members and project partners also noted that they were much more likely to use the research if they had been involved in the research process. Most donors indicated that they used APC’s research to assist them in making funding and granting decisions, in addition to helping inform their internal policy dialogue on these issues.

• When asked, in an open-ended question, what the most significant change has been in this area as a result of APC’s involvement, over 50% of respondents referenced not only APC’s leadership in the promotion, establishment and ongoing support for multi-stakeholder fora, but also numerous examples of APC’s effectiveness in directly influencing these and other key fora, spaces and processes. This feedback was consistent across all stakeholder groups, and examples given by respondents were reinforced at least once by other stakeholders, indicating strong evidence that these examples can be accurately attributed to APC. For those respondents who stated that they “don’t know”, some indicated that they either truly did not know (and that this could be part of the issue of a lack of communications strategy linked to APC’s advocacy work identified in the greatest strengths and weaknesses finding), or could only agree that APC was one contributor of many to the policy gains that had been made over the past two years in this area.

Key result area 2: Foster good internet governance

The goals under this key result area are concerned with: 1) ensuring all processes related to internet governance are transparent, diverse, inclusive of civil society and accountable; 2) civil society is active in shaping the conversation and outcomes related to internet governance; and 3) the global internet governance agenda includes civil society and prioritises human rights and the public’s interest. Seventy-one percent (71%) of respondents (115) indicated an interest in, and experience with, APC’s work on fostering good internet governance.

• 93% of respondents indicated a score of 5 or higher, agreeing that APC plays an important role in ensuring a diverse set of civil society voices – especially those of the global South – are effectively brought into internet governance fora and policy processes.

• 93% of respondents indicated a score of 5 or higher, agreeing that APC is an effective advocate in the global internet governance agenda ensuring human rights concerns are seen as a priority.

• 82% of respondents indicated a score of 5 or higher, agreeing that APC is an effective advocate for the creation of legitimate global, regional and national multi-stakeholder fora, platforms and processes.

• There was agreement among all respondents for this section that APC’s key activities in this area are relevant and useful [5].

• When asked, in an open-ended question, what the most significant change has been in this area as a result of APC’s involvement, the majority of examples related to APC’s direct demonstrated influence in creating tangible advocacy outcomes at key fora, spaces and processes, and APC’s role in creating and supporting multi-stakeholder internet governance fora. There were multiple examples provided across all stakeholder groups in both of these areas, including the establishment of a Special Rapporteur on the Right to Privacy, language in multiple UN resolutions, and the establishment of the Netmundial process, among others. It was also noted by several respondents that the current friction between civil society actors participating in multi-stakeholder fora is a sign that these platforms have become a truly open space where dissent and dialogue is encouraged.

Key result area 3: Strengthening the use and development of transformative technology

The goals under this key result area are concerned with: 1) technology development that emphasises openness and open standards; 2) the promotion and adoption of sustainable technology choices by the network; and 3) the promotion and adoption of free/libre/open source software (FLOSS) by the network. Sixty-six percent (66%) of respondents (106) indicated an interest in, and experience with, APC’s work on strengthening the use and development of transformative technology.

• With the exception of some gains made in the promotion and adoption of FLOSS by the network, this is the KRA in which the least amount of progress has been made to date due to constraints in funding, internal expertise and human resources. As such, the evaluation questions for this section focused on getting a baseline of stakeholders’ current levels of knowledge on these issues, the kinds of technology they use, and their ideas of where APC could add the most value with a view to inform future decisions on this area of work.

• 68% of respondents indicated a score of 5 or higher, indicating at least the majority have some foundational base of knowledge of FLOSS issues to build upon.

• The majority of respondents (78%) indicated that they already use at least some open source software. The most common types of open source software cited were web browsing and web hosting software.

• Just as many respondents stated they would be likely to switch over to open source as those who said they likely would not. This indicates a real need to discover more, and engage in dialogue about the barriers respondents feel they are faced with, to assess which ones can be circumvented and those that cannot in planning the next phase of this work.

• Only 44% of respondents indicated a score of 5 or higher, suggesting that the majority has little or no knowledge of green and/or ethically sourced technology and indicates that there is a significant amount of awareness- raising that needs to be done before any behavioural changes can be expected. Accordingly, the overwhelming majority of respondents (71%) indicated that they do not currently use green and/or ethically sourced technology.

• Only 41% of respondents indicated a score of 5 or higher, indicating that they/their organisations would be likely to switch over to green and/or ethically sourced technology, even with support provided by APC.

Key result area 4: Ending technology-based violence against women and girls

The goals under this key result area are concerned with: 1) society recognising and rejecting VAW online and 2) ensuring that technology is used effectively to combat VAW. Sixty-seven percent (67%) of respondents (108) indicated an interest in, and experience with, APC’s work on ending technology-based violence against women and girls.

• 96% of respondents, representing all stakeholder groups, indicated a score of 5 or higher, agreeing that APC is a key player in the women’s rights movement advocating for the end of technology-based violence against women and girls.

• 85% of respondents, across all stakeholder groups, indicated a score of 5 or higher, agreeing that APC's advocacy has been instrumental in the inclusion of language on technology-based violence against women in policy documents and processes.

• 74% of respondents, across all stakeholder groups, indicated a score of 5 or higher, agreeing that APC's support to partners at the national level has strengthened their capacity to respond to threats and support survivors of technology-based VAW.

• 95% of respondents, across all stakeholder groups, indicated a score of 5 or higher, agreeing that APC's research has contributed to awareness and advocacy about technology-based VAW within and outside of the APC community.

• 88% of respondents, across all stakeholder groups, indicated a score of 5 or higher, agreeing that APC has generated awareness about technology-based VAW about and among social media platforms.

• The most valued activities [6] to respondents’ work on this issue were research and analysis linking women’s rights with internet rights, followed by APC’s training and capacity-building activities, and its role as a convener in this area. Some comments were submitted by organisational members, project partners and allies, stating that they viewed the work in this area as highly collaborative, so they did not see APC’s advocacy efforts as separate from their own.

• When asked, in an open-ended question, what the most significant change has been in this area as a result of APC’s involvement, two kinds of examples were referenced almost equally by respondents, and represented two-thirds of all examples noted. The first example, cited by representatives from all stakeholder groups except donors, was the global recognition of technology-based violence against women as a serious issue and an inherent part of not just women’s rights - but human rights and internet rights. The second most-referenced example of significant change in this area, across all stakeholder groups, related to APC and the network’s demonstrated influence in key policy fora, spaces and processes.

Key result area 5: Strengthening APC community networks

The goals under this key result area are concerned with: 1) ensuring APC members lead in mobilising communities around using ICTs for social change and 2) APC has an empowered and engaged membership.

• 93% of all respondents (161) indicated a score of 5 or higher, agreeing that APC is a trusted and valued partner.

• 71% of all respondents (161) indicated a score of 5 or higher, agreeing that APC is democratic, accountable and transparent in its decision making and programming. However, in the section on APC’s greatest weaknesses, the speed and process of decision making in collaboration with the network was cited most often as the organisation’s greatest weakness, requiring even more clarity and improvement.

• APC’s project partners, and organisational and individual members, were asked to identify all the APC trainings they had participated in, checking all that apply. Of the 65 possible respondents, only 15% had never participated in a training, and the majority of respondents had attended more than one training.

• APC’s approach to, and types of, training are relevant and effective, according to members and project partners. Twenty-seven respondents provided specific examples of the impacts from their participation in APC trainings. Changes to personal and/or organisational practices were the most often cited impacts by members and project partners. Examples of changes included improved online security, the ability to work effectively on internet rights, and using ICTs effectively for advocacy.

• Some members and project partners provided feedback related to improving the trainings themselves, including requests for more communication about trainings offered, incorporating more examples relevant to the participants in attendance, and consistency and/or more clarity on how panelists and trainers are remunerated.

• APC organisational and individual members (38) were asked a series of scale questions [7] relating to their levels of satisfaction with their membership to APC. The lowest satisfaction score was 76% (on engaging as a network in substantive debates), indicating a fairly high degree of satisfaction overall by members.

• 95% of organisational and individual members (38) selected a score of 5 or higher, indicating that they are very satisfied with their membership in APC.

• 84% of organisational and individual members (38) selected a score of 5 or higher, indicating a high degree of satisfaction that their/their organisations’ positions and feedback are reflected in APC positions and statements.

• 89% of organisational and individual members (38) indicated a score of 5 or higher, agreeing that they get support and feedback from APC staff when they need it.

• The opportunity to collaborate and engage with APC and other community network members was the highest-rated benefit, followed by the credibility that is derived from being part of the APC network.

• 84% of respondents indicated a score of 5 or higher, agreeing that the new streamlined membership application process was easy to understand and navigate – albeit still a long process. One respondent (an organisational member) who selected “strongly disagree” indicated it was because of the cost and length of the membership process.

• 54% of respondents indicated that the new member orientation was the right length and useful – but 38% of respondents were not sure. Several respondents (organisational and individual) stated that they were not sure whether they had received an orientation or not, indicating a need for APC to take more of systematic approach to new member orientation and/or be clearer with new members on what they can expect.

Cross-cutting goals

APC has three cross-cutting goals that extend across all its work, including: 1) building the information commons; 2) fostering linguistic diversity; and 3) gender equality and women’s empowerment.

• Progress made on building the information commons was the most difficult to measure during this evaluation, as the only specific indicator related to this cross-cutting goal aims to measure the migration of staff and participants at FLOSS parties over to open source technology – something which has not actually been measured to date, except for the questions in this evaluation. However, it is also clear, through the document review and respondent examples, that APC has included this goal in its advocacy messaging in KRA 1 and KRA 2 activities.

• 67% of all respondents who participated in this evaluation (108) indicated a score of 5 or higher, agreeing that APC is committed to fostering linguistic diversity. However, comments throughout the evaluation have highlighted that these efforts need to expand if APC wants to continue to be successful in its advocacy, networking and capacity-building efforts.

• 91% of respondents who participated in this evaluation (161) indicated a score of 5 or higher, agreeing that APC promotes gender equality and women’s empowerment in all of their activities and policy interventions.

APC’s theory of change

APC theory of change involves five change strategies, including: 1) its dual-track, ‘insider-outsider’ approach [8] to advocacy; 2) networking; 3) capacity development; 4) communications; and 5) research and analysis.

• When asked about APC’s approach to strengthening and influencing policy spaces, 85% of all respondents participating in this evaluation (161), indicated a score of 5 or higher, agreeing that APC’s approach to influencing regional and global policy spaces related to internet access, rights and governance is effective.

• It is clear from the numerous examples provided by respondents throughout this evaluation that APC’s ‘insider’ approach to advocacy is working. Feedback suggests that the ‘insider’ element of the strategy could be even more effective if advocacy in the different KRAs was connected through an overarching advocacy strategy, to reduce duplication of efforts and amplify impacts across the different areas. The ‘outsider’ element of the strategy could be strengthened if it was more connected to the ‘insider’ element, and strategically expanded at key “tipping-point” moments.

• Examples and findings in this evaluation reinforce that APC’s approach to networking and relationship building is effective. Feedback from respondents suggests a need to expand the APC network further – formally or informally – but in clear and defined ways (e.g. so that people understand their relationship with APC and what they can expect).

• The numerous examples of personal and organisational impacts resulting from participation in APC’s various capacity-building programmes demonstrate that APC is strengthening the movement’s and organisations’ ability to work for a fair and open internet. Respondents expressed the need for continued and expanded capacity-building opportunities – but more strongly linked to the overall advocacy strategy of APC.

• The need for a comprehensive organisational communications strategy, linking APC’s research and analysis more purposefully to its advocacy activities, was stressed by many respondents throughout the evaluation. Additionally, it was strongly suggested that APC work on the accessibility (e.g. length, simplicity, languages, format, medium) of its communication products, based on the target audience for each product as dictated by the advocacy strategy. Respondents reinforced that communications is a necessary change strategy for APC to reach its goals, but it is not working optimally as it is currently implemented.

• APC’s expertise in researching and analysing the technical, legal and political issues related to creating affordable access to a fair and open internet was noted as a critical contribution to the internet rights movement throughout this evaluation. The main theme of respondent feedback for this strategy centred on making the research more accessible to different target audiences.

Stakeholder priorities looking forward

All respondents (161) were asked which of the current APC strategic priorities were of most interest to them [9].

• Overall, “securing and defending internet rights” was the highest-ranked priority. The lowest-ranked priority was “strengthening the APC community networks.” This likely reflects that “strengthening APC community networks” is more of an internal goal executed by staff that network members benefit from, but is not one that is a direct priority for their own work. In future, APC will need to decide how to balance and articulate the relationship between this internal priority – an investment that to date has tangibly contributed to impacts in the other key result areas – with their other externally-focussed, advocacy priorities.

• Respondents were also asked if there were any other areas of work that APC should consider investing in, but was not currently. Overwhelmingly, the majority of the 111 comments received, representing all stakeholder groups, reinforced the relevance of APC’s existing five priority areas, and specific suggestions were provided for deepening the work in each area.

Considerations moving forward

Overarching

• There is a tension to manage between being inclusive and consultative with the network when making decisions and being able to be nimble and responsive in reacting to emerging issues. We suggest a mandate of authority be created with the network delineating what decisions APC can take on its own, what decisions can be taken on behalf of the network by the Board or Council, and what absolutely needs to be taken to the wider network (e.g. when input or collaboration is required beyond the members such as implementing project partners, APC allies and others). Regardless, the membership and the wider network needs more clarity about the decision making process and how their input will or will not be used.

• There is a need for a more comprehensive, articulated organisational advocacy strategy for “connecting the dots”, coordinating and leveraging the staff work among the KRAs with members, partners and allies. At least some elements of this strategy need to be created collaboratively with the network, perhaps on a regional basis. All research and communications outputs should directly support this strategy – as should all capacity building initiatives.

• Maintain an “insider-outsider” approach to advocacy. The “outsider” element of the advocacy strategy could be strengthened if it was more connected to the “insider” element, and strategically expanded at key “tipping-point” moments.

• Overhaul and update APC’s approach to communications, focusing on accessibility (e.g. length, simplicity, languages, format, medium, etc.) and a more strategic dissemination strategy of its communication products to get the products used by the people who need them, as dictated by the advocacy strategy.

• Continue and expand capacity-building opportunities – but with a stronger, more articulated link to the overall goals of APC.

• Consider involving more of the network in research – particularly in KRA 1 and 4 – where gains made in recent years on the international level are ready to be further leveraged at national level.

• Finding creative ways, perhaps with the help of network members, to increase linguistic diversity – especially as related to translation of key APC communication products linked to advocacy asks – is essential to take advocacy work on these issues at the grassroots to the next level.

• Consider incorporating a target of what you want to achieve in “building the information commons” in this strategic plan or the next one.

• Expand the APC network further – formally or informally – as dictated by the needs in each KRA, but in clear and defined ways so that people understand what they can expect in the relationship with APC.

• Don’t be afraid to communicate with the network and others your successes – both of APC and the network! It helps people understand who APC and the APC network is better.

Operational

• Invest in additional staff capacity in communications and KRA 2.

• Re-examine the systems and processes of how work is organised, managed and reported on by staff to reduce the administrative burden, duplication of work, reduce siloed work to ensure that staff’s time and expertise is being used optimally towards APC’s advocacy goals.

• Consider investing in data and information management systems that would make virtual project management and reporting more seamless.

• Consider either investing in increasing the project management skills of staff or developing a structure where project management is not done (or exclusively done) by staff responsible for front line advocacy work in KRAs.

• Create a process to regularly check-in with staff to balance needing to be ambitious with targets with being realistic about where meaningful impact can be created without increasing the risk of staff burn out and attrition.

• Consider programming quarterly staff-wide “innovation” days in the work calendar for staff to step away from their daily tasks (and possibly offices!) to regenerate, reflect, read or engage in “big picture” thinking that their daily work does not allow.

• Communicate APC's leadership succession strategy to the donors and the wider network, as appropriate.

Key result area 1: Securing and defending internet rights

• Consider resurrecting the rapid response initiative to respond to internet threats and violations – at minimum towards network members – and communicate to members what they can / cannot expect in the event of a threat.

• Prioritise updating (and keep updating) key resources in this area such as the Internet Rights Charter and the APC La Rue Framework, potentially using the network to update them as a collaborative learning opportunity.

• For resources where much has been invested, such as the Internet Rights are Human Rights curriculum, consider making it more user friendly with a clearer dissemination strategy.

• Consider investing to increase the digital security capacity of all APC staff.

Key result area 2: Fostering good internet governance

• Critically evaluate which fora to invest in over the next two years (e.g. the value of ICANN vs. IGF) – with a view to making a deeper, more strategic investment in fewer spaces. Communicate your rationale for these choices to the network.

• Keep pursuing the multi-stakeholder approach, trying to find the common ground among civil society with a view to coordinated, unified action again.

Key result area 3: Strengthening the use and development of transformative technology

• Walk the talk – if you are going to ask people to switch – be a model that people can learn from in terms of using FLOSS operationally.

• Consider revising KRA 3 indicators on the premise “first things first” – get your own house in order first by getting staff migrated to open source, then work with the network. The same would able for using sustainable technology.

• Create real learning spaces and/or case studies for people to openly share their experience in making the switch, and work together as a community to find solutions – perhaps honestly sharing APC’s own experience in making the switch and challenges faced.

• Provide simple and demystified information through communications products and trainings on what options are out there and the cost/benefit/ consequences and values-based arguments for making the switch.

• One-off trainings are not enough – iterative training and support is needed to change the habits of users.

• Offer financial and other resources to help people make the switch.

• Expand use of the METF (Member Exchange and Travel Fund) for experts from within the network to provide others with technical audits and a customized plan of action for switching to FLOSS.

Key result area 4: Ending technology-based violence against women and girls

• Continue working on the integration of gender equality and women’s empowering throughout all the organisation’s work.

• Now that violence against women online is broadly recognized as an issue, it is an opportune time to use this platform to shift the focus towards women’s rights, not the violence committed against them – particularly around sexual rights.

• Continue engagement with the private sector and social media platforms, perhaps creating a biennial edition of the report card project.

• Consider investing more deeply in regional and national work to leverage recent gains at the international level.

Key result area 5: Strengthening APC community networks

• Create more time at meetings for the network to share information and learn from what each other is doing.

• Find ways to further streamline the membership process or communicate more clearly what to expect in terms of timeline and what will be required of applicants.

• Take a more of systematic approach to new member orientation and/or be clearer with new members on what they can expect.

Conclusion

APC is in a healthy position to deliver on the majority of its current strategic plan.

Out of the 81 results indicators listed in the 2013-2016 strategic plan, APC has either exceeded or is in the process of meeting 67% (54) of these targets already [10]. The one area where APC should consider refining its targets to its current capacity and expertise, in line with stakeholder feedback in this report, is KRA 3.

The results achieved to date, documented in this report, demonstrate that the underlying assumptions in APC’s theory of change are accurate, and the strategies APC employs to affect these changes are relevant, and for the most part, achieving the desired impact.

Where APC can strengthen itself, and the impact of its work, is in taking a critical look at the mechanics of how the different strategies are currently being implemented across the key result areas with a view to connecting and leveraging the work amongst the different strategies more cohesively for maximum impact, as outlined in the considerations above.

Notes: