Policy dialogue has been an important element of international development efforts for some years now. However, its importance seems only set to grow, as the development arena becomes more contested, as commitment to multilateralism weakens and individual governments align international assistance with foreign policy.
But assessing policy dialogue is hard. Multiple factors condition the progress and outcome of a dialogue initiative, complicating the monitoring and evaluation challenge. But that said, there seems plenty of scope for more rigorous, systematic and (crucially) informative strategic assessments of dialogue investments. To date, application of such assessments has at best been patchy.
With this in mind, I wanted to share some details about an interesting effort I was involved in with Stuart Astill and Enrique Wedgewood-Young – to assess dialogue quality.
Why was it important to look at quality (as opposed to ‘satisfaction’ or ‘outcomes’)? In this instance, dialogue – specifically deliberative dialogue – was a central element in the theory of change for a long-term and complex programme. Deliberative dialogue was held to be a necessary (if not sufficient) means of shifting stakeholders’ attitudes. Understanding whether efforts to achieve this type of dialogue were being successful was therefore quite important for the strategic management of the programme.
Deliberative dialogue can be (loosely) characterised as evidence-based (drawn from trusted sources), involving a plurality of interests, conducted in a manner that builds shared understanding of different groups’ positions and the trade-offs, and focused on benefits and fairness. Deliberative dialogue is seen as a means of unblocking and reframing adversarial debates and, through the process of social learning, capable of achieving ‘enlightened’ solutions i.e. agreements that are in some sense collectively optimal and not simply the result of power relations or horse-trading (zero-sum game).
Measuring the quality of deliberative dialogue, however, is not straightforward. To do so, we examined a number of programme-sponsored dialogue events, using a mix of:
- pre- and post- contextual interviews with programme staff: to ascertain the specific objectives for the event, factual information about preparatory activities and views on success;
- a post-event questionnaire survey of participants, to obtain their views on a number of aspects of the dialogue; and
- non-participant, direct observation of the dialogue events and discourse analysis.
It is this last element that I want to highlight because this was the part that was reasonably innovative. However, important to note that the interviews and surveys, in addition to providing useful information themselves, were also key in triangulating the results of direct observation.
To operationalise the observation exercise, we developed an assessment framework informed by deliberative theory and discourse analysis, and which drew heavily on Bächtiger et al (2011) and their adaptation of the Discourse Quality Index (developed Steenbergen et al). The framework comprised nine ‘qualities’:
|Records the (time) participation by different groups attending the dialogue
|Clarity of understanding
|Records instances when participants demonstrate a lack of understanding/ clarity regarding the objectives of the dialogue
|Records instances when participants refer to other participants or their arguments
|whether the issue presented for discussion is framed from singular or multiple perspectives and whether trade-offs actively considered
|The extent and quality of justification offered for positions or arguments presented.
|Instances where participants rely on personal narratives or experiences in supporting their positions
|Respect and agreement
|The nature (tone or language) of references by participants to others’ contributions
|Common good orientation
|The basis for how arguments are cast –ranging from narrow constituency interests to principles of the common good.
|The ‘politics’ of solutions/conclusions offered, ranging from positional politics, to mediating proposals that explicitly acknowledge and seek to address different concerns
To conduct the assessment, a team of three observers were trained in the tool and given responsibility for tracking three qualities each (with tests run initially to check observer reliability). Each quality was scored either as a simple frequency count (number of instances where the quality was observed) or against a predefined scale measuring the degree to which the quality was observed.
But of course it is unrealistic to expect any dialogue to score continuously “high” in deliberative terms throughout the process. Recognising this meant our approach needed (a) to capture the variation in the dialogue over the course of the event; and (b) a basis for interpreting that variation.
In order to capture the ebb and flow and changing intensity of the discussion, we assessed each dialogue in five minute blocks – assessing the quality of dialogue throughout each session. When breakout groups exceeded the number of our observers, we had to sample pragmatically. Because we felt the nature of these smaller-group dialogues would also differ over their course, we divided each breakout session into three equal time periods – ‘beginning’, ‘middle’ and ‘end’ – and sampled across these units. The aim was to ensure the observers had a representative picture of the breakout discussions.
To interpret the results, we drew on Bächtiger et al. again, (who in turn drew on the work of Shawn Rosenberg). Depending on the combination/profile of qualities observed, we could characterise the dialogue for any period according to five types of discourse:
- Proto-discourse: communication to provide/share information and build social assurance; little or no focus on disagreements over validity claims between different interest groups;
- Competitive discourse: lacks the aim of reaching a shared understanding and any noticeable cooperative spirit; actors are not prepared to be persuaded by the “better argument” but only seek to justify their own standpoint; may involve elements of high quality dialogue but instances of respect and agreement are infrequent
- Conventional discourse: geared towards problem definition and problem-solving but not building a shared understanding; typically comprises a succession of concrete contributions intended to describe, to explain or to evaluate an aspect of the topic at hand.
- Cooperative discourse: geared towards reaching a common understanding and problem-solving. The goal is agreement among participants. To achieve agreement, diverse standpoints are evaluated; entails key elements of what many would consider high quality deliberation;
- Collaborative (rational) discourse: the most demanding form of exchange – involves the free and equal expression of personal views and a respectful consideration of others’ perspectives, fairness and the common good; in finding solutions, the goal is preference transformation both personal and collective, and disagreement is actively managed in productive, and creative ways.
There isn’t scope here to elaborate the detailed findings but suffice to say that, although some sessions in all of the dialogues demonstrated deliberative characteristics, none of the sampled dialogues overall achieved the level of deliberative discourse envisaged in the programme theory of change.
That said, the granularity of the analysis enabled by detailed direct observation and follow-up survey did point to potential actions to strengthen the deliberative quality of dialogue in the future:
- engage stakeholders earlier in the dialogue preparations by sharing more relevant, timely evidence in advance; deliberative theory places significant importance on the role of preparatory evidence, as a basis for dialogue and ensuring participants come with the appropriate mindset
- allow sufficient time and space for discussion of issues: by a) narrowing the scope of dialogue events; and b) using breakout groups more systematically. On the latter, our data indicated group discussions were typically more deliberative than plenary sessions (and involved greater participation by women); but, in practice, much of the value of group discussions was lost by reliance on rushed, abridged and (at times) incomplete feedback to plenary as the main means of capturing findings.
- finally, high quality deliberation involves tackling and resolving differences of view. This is a challenge for external agencies when the issues are highly contentious but ways have to be found if deliberative approaches are to effectively deployed. This may require greater engagement politically by supportive governments, in addition and alongside to the usual provision of funding and technical expertise.
If any of the above interests you, do get in touch!
 Bächtiger, A., S. Shikano, S. Pedrini, and M. Ryser. “Measuring Deliberation 2.0: Standards, Discourse Types, and Sequentialisation. University of Konstanz and University of Bern. 2011(?)