Systems Masterclass: Session 3 - Revealing Core Values: The Role of Critical Systems Heuristics

Yeah. We just don't have Paul, but that's the idea. Should we not worry about that? On to team two.

Thanks, Shira. Yesterday, we had Team One, where Michael Patton, Emily Gates, and Charmaine Patton provided us with a big picture overview of the systems. If anyone missed those two sessions due to time zone differences or other reasons, my colleague Shira can share the links in the chat. They are all available on the website, so you should be able to access them. And remember, if you want to earn the certificate, you can still watch the recording and complete the survey to obtain the certificate.

So today, we're moving on to P2. We have two sessions today focused on seeing and understanding complex systems. This series has been curated with NORC, specifically by Margaret Hargreaves at NORC and Ekdan. This is part of our broader Systems Initiative, which I have shared previously. In the past, we have worked extensively on systems research with Sharon Goldfield and MCRI, covering areas such as systems investing, systems measurement, and systems design and implementation. Today, we are focusing on the systems measurement component of the broader Systems Initiative, which is supported by Porticus. Thank you very much, Porticus. Just a reminder: we do offer digital certificates that you can earn throughout this series. To earn them, we need you to register for each session you would like to participate in and complete the homework.

The homework is really important because it provides a foundational background to this entirely new field that we're all collectively exploring. Please participate in at least seven out of the fourteen sessions, whether live or recorded, but they need to be completed before May 17 so that we can issue all the certifications by then. After each session, please complete the short survey. Shreya, could you please share the link to the survey so that participants have easy access right from the start? You will have access to everything; everything will be available on the website. We often get asked this, but you will have access to all the recordings, all the presentations, and all the resources, as well as ECD Connect. A lot of you have already signed into that, which is great. Let's have discussions there after the masterclass series, and we will have experts available over the next two weeks to answer any questions you may have. The format of the masterclass is pretty basic. We'll have a welcome introduction, the masterclass session, and a dialogue with participant Q&A. Just as a reminder, please ensure to put your Q&A in the designated chat. Click on the Q&A chat and submit your questions there, or we may miss them.

For the Q&A, you should have the ability to vote and comment as well as read answered questions. If you want to chat, please ensure that you’re responding to everyone and not just the hosts and panelists. We can read your lovely comments, but it’s important to include everyone. We also have Spanish translation available, thanks to UNICEF Blackborough. We are really grateful for that support. If you need Spanish translation, you can go to Interpretation and click on Spanish. A big thank you to Mitzi, our translator, who is with us. Today, we have Session Three, which is happening right now with Bob Williams, and I will introduce him along with Sharon Goffeld. Then, in a few hours at 8 AM Eastern Time, we will have the fourth session with Aaron Watson and Rebecca. So, without further ado, let me introduce both Bob and Sharon to you. Just one quick second.

Bob Williams has been working at the nexus of evaluation and systemic practice for over 30 years. We’re lucky to have him here today. He is widely recognized for his efforts in promoting systems approaches and concepts in the evaluation field. He has conducted several workshops, trainings, consultations, and applied systems thinking in program evaluation, performance measurement, organizational development, and program logic across the globe. He has also published several articles.

You can go to his website; it's a great resource with a lot of free materials available. He is the author of six books on integrating the fields of systems practice and evaluation. He is also the recipient of the American Evaluation Association's Paul Lazarsfeld Evaluation Theory Award for his contributions to the field. Joining us today are our early childhood experts, including Sharon Goldfeld.

She's the Director of the Center for Community Child Health at the Royal Children's Hospital. She's a professor, a pediatrician, a public health researcher, and a policy advisor focusing on child development and health equity. Additionally, she's the Team Director for Population Health at the Murdoch Children's Research Institute. With over a decade of experience in state government as a senior public health and education official, Sharon has an outstanding track record. She has successfully straddled the fields of research and policy.

Her expertise is highly sought after, with appointments to government, expert advisory groups, and health education and social services departments, including her recent appointment to the National Early Years Strategy Advisory Group. So we have the whole early childhood system in Sharon with us. Without further ado, I'm going to hand it over to Bob. Oh, Bob, would you mind renaming yourself? I could just do it for you, because right now you're also Sharon. I make a very good Sharon.

I have no idea why I've suddenly become Sharon. There we go. This is what happens in ACD, Bob. We just take over the world. All right, Bob and Sharon, over to you.

Yeah, there's another slight glitch that we've had, just as a warning to people. We had some sound troubles that we hope we've sorted out. But Sharon, do you have the backup? Um, if Tray has the backup... Tray has the backup, Bob, so we're good to go if yours fails. We're good to go. Anyway, we'll go with Plan A first, and if Plan A works, then we're fine. Oh, that looks promising! Let's do the usual. No, let's just check if everything is going. You can see what's happening, and we'll keep our fingers crossed that when I push the button, the sound will come on. What I'm going to do is really build on what Michael, Charmaine, and Emily have said about the systems so far for those of you who've seen it before.

Now they are focused on general principles, and initially, so will I. In fact, I'll be using some versions of the illustrations that you've already seen from those presentations. Someone has their... I think Shrey will have the answer. There we go. Thank you.

What this does is provide a deeper dive into this topic and offer important context regarding a particular systems method called Critical Systems Heuristics. It's one of the many options that Emily mentioned earlier and explores the implications of some of Michael's comments about the boundaries we establish in any systemic inquiry being social as well as physical constructs. These boundaries relate closely to our worldviews and values, as well as the boundary operating principle that Charmaine also mentioned. Consequently, this particular method, Critical Systems Heuristics, is very relevant to issues of and the values that do or should underpin ECD, as well as evaluations of ECD. Now, fingers crossed, if this goes well, it will be a pre-recorded presentation that lasts just a bit over half an hour. About halfway through, I will pause and ask you to complete a task.

So, prepare for that; it applies the four main elements of critical systems to something that you've already been involved in. It doesn't have to be big or complex. It could be getting the kids off to school in the morning. Once that presentation is finished, then Sharon and I will use a case example to explore some of the details of critical systems. After that, we'll step back and discuss any broader issues that the approach raises.

We'll also keep an eye on the comments, of course, that you make in the chat, and we'll have a go at addressing them, so keep them rolling in. Fingers crossed! Here we go. In this presentation, I'm going to be describing a particular systems approach called Critical Systems Heuristics. Of all the systems approaches I know, the ideas that underpin Critical Systems Heuristics have probably influenced my practice the most. I'm going to focus on its use in equitable practice because Critical Systems Heuristics is fundamentally about surfacing the values we aspire to and the values we practice, and equity is at the core of much early childhood development practice. I'll also talk a lot about evaluation, and I don't only mean evaluation with a big "E," but also the evaluative practice we undertake as part of our day-to-day work. Now, as the presentation proceeds, I encourage you to make a link with a particular practice and pose and consider these three questions: What do I do consciously?

What do I do unconsciously? How do I deliberate about those actions? In about 15 minutes, I will pause and give you some time to write down and explore these questions, so be prepared. But before we dive into critical systems heuristics, let's take a step back. The systems field is both old and large. It is informed by ideas that date back literally thousands of years from many parts of the world.

The modern systems field emerged from Russian and Austrian thinkers, as well as other practitioners, in the early part of the 20th century. Over the past hundred years or so, the field has drawn on many other disciplines, which have inevitably contributed to a large diversity of ideas, interpretations, and cultural orientations. In my experience, the U.S. field of systems is different from that in the U.K., which, in turn, is different from that in Germany, China, and certainly differs from the various systemic approaches that have emerged in South America. This particular diagram, produced by one of the systems field's leading figures, Ray Isen, gives you an idea of that scale and scope. On the left-hand side, he lists a range of fields that have informed modern systems practice, and as you move more towards the right, toward the center of the diagram, you go through what I would call the emergence of specific disciplines within the overall field.

And moving further to the right, there are methodologies and methods that have emerged from those disciplines and sub-disciplines. I ordered these along a continuum. At the top are methods and methodologies that primarily describe the world as it is, the objective reality. In other words, there are methods and methodologies that are primarily ontological. They describe the world as it is. At the bottom are methods and methodologies that explore the world as people see and understand it.

In other words, subjective realities. Once again, they are primarily methods and methodologies that are epistemological in nature, focused on learning. Now, as you can see, Critical Systems Heuristics falls right at the most extreme epistemological end. Indeed, in my view, it enters the third great area of exploration: that of axiology, the study of values and valuing. Perhaps for no other reason than to suggest its possible relevance to evaluators who ought to be, although often are not, interested in the nature of valuing.

To gain a somewhat less philosophical understanding of the role and potential of Critical Systems Heuristics, let's start at the very beginning: what comprises a system. There are many individual definitions, but they generally align with one developed by Dick Morris at the Open University many years ago. Here’s my version of Dick's definition: A system is a collection of entities that are perceived by someone as interacting together to achieve something.

Let me illustrate that in a slightly different way. The definition can be split into three elements. The first one is interrelationships, a collection of entities interacting together. The second is about perspectives, which are perceived by someone. And finally, achieving something can be regarded as setting boundaries.

The reason why achieving something is about boundaries is that to achieve a goal, you must focus on it. While focusing on one thing, certain aspects will inevitably fall outside that focus. In other words, you establish a boundary between what is included and what is excluded. By concentrating on your objective, you create a distinction between what is relevant and what is not within your system. All systems have a purpose, which may be explicitly defined or based on people's perceptions of the purpose.

And it's for that reason that systems are always human constructs. People can identify different purposes of the system even when the interrelationships are the same. However, in our evaluation craft, we don't just want to describe systems; we want to use systems ideas to influence our evaluation practice and the interventions we evaluate. We want to take action and engage in systemic practice. Systemic practice is more than merely describing systems.

Systems practice, or systemic practice, involves stepping into interacting entities and influencing, or at least altering the course of, those changes when possible. I want to stress that this could be within the intervention we are evaluating or indeed in our evaluation practice. To transform this description of a system into systemic practice, you need to add some verbs. Here they are: 1. Understand interrelationships. In other words, address the question: What is the reality that we are actually dealing with? 2. Engage with perspectives.

How do people interpret that reality? Finally, we must decide on boundaries. In other words, we need to determine what is truly desirable and feasible among the vast array of possibilities we have. Critical Systems Heuristics is a systematic approach to establishing boundaries, assisting us in determining what is desirable and feasible to pursue. A common misconception is that systemic approaches involve considering everything, reflecting the old idea of holism.

In fact, by now you may be realizing that the reverse is true. Every systemic practice, every intervention, every evaluation, and every approach is bounded in some way. It's impossible to view and understand every single interrelationship at every available scale. It's impossible to accommodate every single perspective. In order to be able to do anything, we end up having to draw boundaries.

Without boundary decisions, we just can't do anything. We will just stand in the lights, scared like a rabbit in the headlights. And when I say "doing," I don't just mean actions. These boundaries can be related to ideas, ideologies, or thoughts. So, for instance, in evaluation, we don't just make decisions about time, budget, or method.

We make those boundary choices. However, we also make boundary judgments and boundary choices about criteria: equity, efficiency, and effectiveness. These criteria enable us; they provide us with the boundaries to make judgments about whether something is significant or not, valuable or not, or meritorious or not in terms of those criteria. These are all vital boundaries that allow us to determine whether something is worthwhile or not, or significant or not.

So essentially, without boundaries, there is no evaluation system. What are we doing when we make boundary decisions? This issue concerned the systems thinker C. West Churchman, who developed the field of critical systems. He concluded that there will be winners and losers in every boundary decision. In terms of losers, this can be as simple as falling below the set standards or criteria in an evaluation. In other words, being ineffective, inefficient, or uncaring—whatever the criteria happen to be. Alternatively, it can be as complex as feeling that their interests have been marginalized by the evaluation or the intervention.

And since people or things will be profoundly affected by these boundary decisions, churchmen consider that boundary decisions are primarily ethical rather than primarily technical. Because they are primarily ethical, they must be justified in ethical, not just technical terms. We have to be able to claim that whatever boundaries we draw are the right boundaries to draw, both ethically and technically. Some people will challenge these claims, and they may have the power and agency to undermine that claim. We need to deliberate on whether we seek merely to bolster our argument for the rightness of our claim or, from an ethical point of view, negotiate an alternative set of boundaries that accommodates this particular challenge in some way. The current position is that these in-and-out decisions we make, these boundary decisions, are not only technical but also ethical; therefore, they require ethical deliberation on the consequences, given that people and ideas will be at least marginalized. The problem is that we make literally hundreds and hundreds of boundary decisions in any particular evaluation.

They're decisions of space, time, purpose, money, control, knowledge, and perspectives. Which ones should we focus on? We have to, therefore, make some boundary decisions about our boundary decisions. Now, every evaluation situation is different, so is it possible to identify some generic decisions that are vital in every circumstance? One of Churchman's students, interested in evaluating and social planning, Werner Ulrich, identified four key boundary decision areas and developed a series of questions or heuristics around them, and Critical Systems Heuristics was born.

The first of these four areas is purpose. In other words, this is a decision about who benefits from an intervention, including the evaluation, and by implication, who does not. Who gains? Who loses? This decision involves determining who controls the necessary resources to achieve those benefits. It represents a boundary decision regarding who has autonomy over those resources and who has the ultimate control over whether those resources are available.

In order to use resources, you need information about what kinds of resources are available and so on. Therefore, there is an issue of knowledge and the necessary decisions around whose or what knowledge and expertise are used in managing these resources. What knowledge systems are used, for instance, and what knowledge systems are ignored? That's a very pressing challenge for evaluation right now. Finally, there is a question of what Ulrich called legitimacy.

This is a complex area, but fundamentally, it poses questions about whose or what interests have to be satisfied in the decision in order to ensure that the benefits actually occur. It is a boundary choice between ideal desirability and the actual feasibility of seeing what you have implemented work on the ground and be sustained. Now, these might seem like simple, generic questions, but actually answering them exposes significant issues around the use of power, equity, and control of both the intervention and the evaluation. It raises issues around knowledge systems and the reality of the compromises and trade-offs necessary, including the compromises and trade-offs required for our evaluations to achieve what we wish them to achieve. The issue of legitimacy surrounding that is a really important aspect of critical systems heuristics, and I would argue that it is something evaluators really do need to take a closer look at.

We're all very interested in the consequences of our evaluation. Legitimacy is a major contributor to whether things are considered legitimate or not. However, here's how Critical Systems Heuristics actually handles this. The first three aspects—purpose, control, and knowledge—are essentially about who benefits and how they benefit. But the legitimacy aspect brings you down to earth with a bump.

Critical systems theorists argue that the legitimacy, and hence the feasibility and sustainability of your system, cannot be determined by those who are inside it. Within your system are people who stand to benefit from it in a direct way; they are likely to support its legitimacy. However, critical systems theorists contend that legitimacy is primarily gained by addressing the needs and desires of those who are not part of the system but whose interests are negatively affected by it. A classic example of this is a car in a cycle lane.

In other words, town planners, cycling groups, and cyclists cannot legitimize cycle lanes. The only people who can ultimately legitimize cycle lanes are power drivers and perhaps local shopkeepers. These individuals often perceive themselves as negatively affected by a system in which they do not feel included, regardless of whether that perception is accurate. The deep ethical question then arises: how many of those interests should you consider and include in your planning before you risk undermining the very benefits you desire? If you include too many interests, you may end up creating another system with different issues regarding its purpose, its controls, and its legitimacy, and this time, someone else could be negatively affected.

Let's pause here for a bit and allow you to gather your breath. Earlier, I suggested that you keep some project in mind. Well, here's a chance for you to play with it. Think about that particular project or any project, big or small, and consider what boundary choices you made, both conscious and unconscious, in terms of purpose, control, knowledge, and legitimacy, and then write them down. Cue music, and I'll be coming back in a couple of minutes. Music. I'm going to go to the next one. Music. I'm going to go to the next step. Music. I'm going to go to the next one. Music. Welcome back. Did you find the task challenging or easy?

If you found it challenging, then the next part may actually make it easier. On the other hand, if you found it easy, you might find the next part a bit more challenging. There are a total of 12 questions in Critical Systems Heuristics, three for each of the four major areas. One question tends to explore the 'who' boundary decision, the 'what' boundary decision, and the third can be considered a kind of 'so what.' It explores some key issues that are raised by the 'what' and 'who' boundary decisions, which often imply a further boundary decision.

And since they are heuristics, a major task is not just bolting the questions on, but adapting the ideas behind the questions to the particular circumstance. Let me step through them progressively. Note that I refer to the intervention and the evaluation, and that's because, as you have heard many times in this talk, evaluation itself is an intervention. Therefore, we cannot only apply these issues to the evaluand, but also, most critically, to our own evaluation practice. As I go through these questions, I suggest you reflect on how consciously you address these questions in your own evaluation designs. Reflect on, more systematically—sorry, more systemically—how these questions are consciously addressed in the evaluation system that you and others are part of. Okay, so first, purpose.

The three questions—what, who, and so what—relate to purpose: the "what" identifies the purpose itself, the "who" refers to those who will benefit from that purpose, and the third question, "so what," poses the inquiry of how you would know. In evaluation language, this is a measurement issue. For me, this is one of the most intriguing parts of CSH because it helps you understand that what you measure and how you measure it are all value-based decisions. They are influenced by our beliefs and values regarding how we observe the world. Indeed, for each question, there is an underlying inquiry regarding the beliefs and values that inform your choices about purpose, the beneficiaries, and the criteria you wish to use for measurement. This includes what you deem as the legitimate basis for assessing whether those beliefs or values are adequately reflected in your evaluation. It is important to note that these questions are posed in a neutral or observational mode.

Is, will. But in most situations, and this is where the evaluative aspect really comes in, they also need to be asked in a normative, value-explicit mode: the could, the should, and the ought. This then gives you the opportunity, should you wish, to compare the should—what is desirable—with what actually is. This comparison can also be subjected to further assessment and associated criteria, or to the tension between autonomy and control. In this case, the "what" question relates to the necessary resources in order to bring those benefits and achieve the purpose.

The "who" relates to those who have the autonomy to manage the resources. The "so what" explores where the locus of control is or ought to be, and in each of those three aspects, it identifies the value issues that arise from the boundaries between autonomy and control. It is presented in two modes, offering an opportunity to reflect on the differences between the two. As expected, knowledge questions lead to an assessment of how you would monitor and evaluate the boundaries of what knowledge is used, by whom, and what knowledge is ignored, as well as how that difference is assessed continuously. This continues in both modes and is subject to some form of comparison. Finally, as you can probably guess, the legitimacy aspect raises a number of questions about how to ensure the desirability explored in the first three groups of questions can be made feasible.

This requires understanding the value of those who may consider themselves or their interests ignored or marginalized. So we look at what is negatively affected, who is negatively affected, what are their beliefs and values that make them feel negatively affected, and then what are actually the trade-offs and compromises that you need to gain to develop sufficient legitimacy so that your endeavor is going to be feasible and sustainable in spite of these marginalized issues and marginalized people. What value base underpins your trade-offs in that particular case? And again, you can put these into the "should" mode, which is a bit more difficult to answer, and then begin to make some kinds of comparisons. So what does this all mean for equity and equitable evaluations? Well, there are a lot of hard questions with answers that will be uncomfortable and difficult for some.

Let's go through this item by item. Now, obviously, we can use critical systems and heuristics to identify purposes and those who benefit from an equitable framing, and to determine what that means in equitable terms. But what about measurement? Not only how are we going to observe whether there have been benefits, but how do we decide on these measures that are themselves based on values of equity?

Now, I honestly don't have an answer to that, but we do know that disadvantaged people literally see the world differently from we do. They will measure things differently than we do, and that has deep equitable dimensions. In terms of control, the issues are primarily over who actually has control over the intervention or the evaluation. What would be an equitable balance, for instance, between involvement in resource decisions and control of the intervention's purpose and benefits? Control over those resource decisions, issues related to autonomy regarding those decisions, how involved should different stakeholders be, and to what extent should those who are the recipients of the benefits of an intervention— including the evaluation— be involved in either the resource decisions themselves or the control of the intervention's purposes and benefits, as well as the evaluation's purposes and benefits?

There's not a lot to say about the knowledge issue. The questions are often discussed in evaluations regarding whose knowledge systems are used and respected, and which are not. This adds a few more questions to the important debates currently taking place in evaluation. The legitimacy aspect is something that I think we need to focus on a bit more, as the questions imply that, as an evaluator, you have an ethical responsibility to consider the impact of your evaluation or intervention on those who may be disadvantaged by a program based on equitable principles, or an evaluation grounded in equitable principles. Now, that's not to say you have to take any specific action regarding that, but it does raise the issue of your responsibility to those you wish to benefit from the program. How do you address this responsibility in a way that does not promote a backlash, which may ultimately result in the benefits not being achieved or sustained?

What values will inform, or ought to inform, decisions to trade off particular benefits for longer-term sustainability? And how should we evaluate those particular value judgments? That is critical systems for you. Now, apart from all the questions we have gone through in some detail, if we look beneath the role of critical systems heuristics, it really serves to enable you to assert that, either in your practice or in your evaluation, you have a basis upon which you can claim that whatever you have done or intend to do was the right thing to do.

Was it the right intervention? And was it the right evaluation? Now, obviously, there are many different ideas surrounding that, and it's not necessary to arrive at a point where you just stand up and say it's the right thing to do. After all, you probably wouldn't have done it in the first place if you didn't believe that. But now you have some way of constructing an argument, and if you can't formulate an argument that enables you to do what you want to do in an unfettered way, then it begins to prompt an examination of the trade-offs that you might need to undertake in order to sustain both what is desirable and what is feasible and sustainable.

Now, in doing this, I focused on areas related to equity. This is because, within the early childhood development field, equity is clearly an important aspect. You could probably consider any specific criterion that underpins the basis on which you wish to claim that what you are doing is the right thing to do. Okay, so that's critical systems heuristics. What Sharon and I are now going to do is have a conversation with the people who will be discussing critical systems in relation to a particular example from early childhood development. Sharon, do you want to quickly recap, and we will basically go back through the heuristic and focus on some of the more difficult and interesting areas?

But first off, Sharon, do you want to give us an overview of the project? Yeah. Yeah. First of all, can I also just say fantastic questions coming through, and I want to thank you for that really challenging overview. This idea of heuristics—these shortcuts to our thinking—might challenge us in the way we're considering early childhood development, which, of course, we'll discuss a bit later.

But, you know, we deal in microsystems in early childhood development as well, of course, because there's the child in the family. There's a whole conversation about who benefits and who doesn't benefit at the child level, the family level, as well as the community level. Exactly, so there's all this complexity, which is, I think sometimes to your point, that "rabbit in the headlights" kind of moment, where you go, "How do we deal with that complexity?" We have to take into account the child and the parent that might be at the micro level. My clinical work is at the micro level; the research, though, is at the broader level. So thank you for that. I think these are really challenging and interesting hooks for us. I think that's why you're getting the questions that are coming through, which I find really, really interesting. The other reason I want to mention is specific to where we are, the longitude that we're on, but also the fact that I'm coming to you from Australia and Bob is in New Zealand. I would like to acknowledge the traditional owners of the land that we're on here in Australia.

So I feel it's important to stop for a moment and do that. I'm on the land of the Boon Wurrung people, and I would like to pay my respects to elders past, present, and emerging. I would also like to acknowledge any First Nations people in the audience, in any country, anywhere in the world that is joining us today, and in particular here in Australia, with the knowledge that this land has never been ceded. Now, regarding this particular topic, I was a bit challenged by this. I said, "Well, I don't know which one to put up."

I don't know. I gave him a few options; I provided him with a menu, and he chose this one. This is really interesting, and I was reflecting on these heuristics as we were going through them. I'll give you a very quick summary. I'm hoping what you're seeing on the slide now gives you a sense of this, and equity was absolutely at the heart of it.

There are two issues regarding early childhood development. I think, Bob, you've hit the nail on the head with the idea that equity is very near and dear to the hearts of those of us working in early childhood. The other issue is a developmental perspective—what do we actually do for children at these very young ages? We know that what we do now is incredibly important, both in the present and for how these children will develop over time. This developmental perspective is crucial because the support we provide for infants is quite different from what we might offer to two- to three-year-olds, which in turn differs from what we would do for a five-year-old. So, "Right at Home" is based on this concept. I should also mention that it has been co-designed with end users.

And we can have a whole debate about who the end users are here in terms of the staff who deliver it and the families who receive it. However, it was designed to essentially be based on a universal opportunity for public health nursing to deliver support to families who might need something additional in order for them to have the kind of parenting opportunities they want for their children. It’s based in the home, and there is a substantial international evidence base about it. However, no one had really conducted this type of work in Australia, and this is a randomized controlled trial. What’s been particularly interesting is what happens when power is given to families; when the practitioner empowers families to determine what they want to achieve from a visit. How do you measure that?

I guess we could have a long debate about that. The second point is that we measure the impacts over time on issues that are meaningful, both from a development perspective and for the end users. Third, I noticed some of the comments that came through; one from Australia particularly stood out. Luke Williams talked about the difference between what is desirable versus what is feasible, and the challenges associated with that. I find the issue of feasibility really interesting. How do you take something that has had an impact from a research point of view, affecting both the children and the parents?

But how do you actually ensure that this can be effectively implemented so that sufficient families can benefit from it? Who benefits, and who gets in the way? I find that really interesting. We haven't been able to scale this to a level that would benefit a significant number of families. We can have a conversation, then, Bob, about what we think has impeded progress, because I find the dissonance between what is desirable and what is feasible to be quite intriguing, depending on who finds it desirable and who finds it feasible. This is influenced by a combination of reality and values, I believe.

I thought this was an interesting topic because it is fundamental, and from a public health point of view, one of the most effective interventions that we've tested. It's really important—an important part of the stack of things. There is no silver bullet when it comes to early childhood challenges, Bob. I don't think any of us would say, "Just do my program, and I'll put my feet up."

Once it's all done, all the early childhood elements will change. So, we sit in this complexity, which is why we have chosen a systems approach in early childhood. We are trying to understand this complexity and figure out how we can make a difference for children and their families, at least in the next decade, if we can. So, Bob, maybe I'll turn to you now. Just for everyone online to know, Bob and I will have a discussion and then we will reflect back a bit more. After that, we'll have time to answer questions. There are currently 15 questions in the Q&A. We might ask our colleagues at ECDUN to bundle some of those for us so that we can address them effectively.

So, Bob, over to you. Yes, and certainly, I think some of the questions will actually come up regarding power and what we mean by marginalized. These topics will emerge within some of the questions. Let me just change the slide—oh, too many slides; I'll go back. So, this is the first suite of questions, and we obviously don't have time in this conversation to go through all of them. Therefore, we've decided to focus on the ones that are particularly challenging. I believe you, Sharon, have already raised one of them, which is: How will you know, and what are the values that underpin that way of finding out?

You've mentioned, for instance, that it was a randomized controlled trial. There are certain values that underpin that—certain worldviews that inform that approach. I wonder if you would like to elaborate on the factors and consequences of this. If we were evaluating it, for example, what kind of criteria would we use to assess your methodology based on the values that underpin that particular approach? How would we determine who benefits and who does not? How would you know?

And I think this: I can see that there are some questions coming through about the setting of boundaries, and maybe that's related to this. Bob, you can sort of respond to that. Because I think it's important to always start with the end in mind, which is kind of what you're talking about here in terms of boundaries—who's in and who's out, and what's in and what's out. That's absolutely necessary. Additionally, consider the beliefs and values you bring to that. One of the beliefs and values regarding randomized control trials is to ensure that an intervention—whatever you want to call it—actually is impactful. In other words, I can tell you that the research world is littered with many things that people loved that have actually made very little difference to human outcomes.

And I think human outcomes are really—I mean, I guess I'm reflecting my own values. I think human outcomes are a really important part of what we're trying to achieve here. The other aspect is how being successful, I guess, involves ensuring that we consider the sorts of families that were included in this study and whether or not they truly reflected the equity lens. In this study, they did, which was a really important driver for us. There are many interventions that get rolled out but do not actually reach the families for whom they were intended. That's a real challenge in research.

And so I think that these are both the purpose and a bit about the boundaries we set for ourselves. Yes, I've got two comments on that, and from a critical systems point of view, one of them is: Is that how the beneficiaries, or some of the other people involved, would define what constitutes a benefit and what signifies success? How do they measure it? I mean, this is a major issue here in Aotearoa around Māori worldviews versus Pākehā worldviews. The two cultures have different ways of measuring. A systems person would say that the notion of a counterfactual in a complex system is an illusion.

It's actually based on a worldview that has no basis in reality. Now, I mean, I'm role-playing here. So those are the sorts of things that a discussion—or, again, it has to be a participatory one, which I think I didn't emphasize—ideally falls under critical systems, an area known as emancipatory systems. It's a terrible word at one level but a brilliant one at another. It's actually about allowing people to raise issues, particularly those that may not have been discussable before, and there are issues of power here. Anyway, let's move on because we're going to run out of time. I just think it's worth adding the importance of co-design. You're right in terms of the beneficiaries; I think co-design is essential. What do families actually want from these services to meet their needs? Also, as I mentioned, it's about seeding power in the interactions.

There’s a whole lot of issues surrounding the power dynamics between a health professional and a parent. A significant amount of power needs to be ceded by the health professional to allow the voice of the parent to come to the forefront. I believe that part of this program was focused on that. However, in the field of health, these are truly massive issues regarding how to cede power to ensure the voice of the beneficiary is acknowledged, enabling them to address matters that are significant to their worldview, rather than the health professional's worldview. This naturally leads into the area of control, specifically who ultimately has the authority to determine how the resources used in a particular intervention are allocated. It’s not just about managing those resources, but about who possesses, in systems language, the power to stop the system.

Who, in this particular case, ought to have that power, and who does? It raises issues of accountability, which is a concept in complex systems that, in my opinion, does not work particularly well due to the rather conservative way in which people understand accountability. It is very much about the questions of who has ultimate control over decisions regarding purpose and benefits. You've started to explain some of the strategies you had to address those issues. Yes, I think this is really interesting, and I don't know if you have more questions you'd like to share. Perhaps we can stop sharing for a moment and have a broader discussion. I do have questions, but this is the problem with critical systems: people often hesitate to engage with difficult questions and instead prefer to take a step back. No, I want to delve into the difficult questions, but I also want to have a broader discussion. No, I truly want to engage with the difficult questions.

I just don't. I want to make sure the audience can see us and not just the slides. I'm happy to talk about the resource issues because it's been a very interesting question in community and place-based work that is happening around the world. Who makes decisions about where resources should be allocated? For example, how do communities think about what they're doing for their families?

So it's a really live issue in that space in terms of where the governance should be, where the resourcing should be, and how that works. And again, who holds the power and how do you ensure equity in that space? In this particular instance, I just want to highlight some specific tensions related to the issues of resourcing and who holds the resources. Because if I’m going to falsely set boundaries, I will do it for the sake of this conversation. If you can imagine, there is a kind of policy perspective on this, which involves who holds the actual resources in terms of investment.

There's the delivery of these services, so who holds the resources in deciding what the providers should be? And there's the family themselves, who have their own internal resources as well as the ability to think about how they might, if you like, access the resources that are available to them. I think, certainly in our work, and I'm sure you do as well, they rarely align. There is often no agreement on the allocation and the consideration of those resources. While people have been discussing who benefits and who doesn't, it's really interesting to examine where the blockages or obstructions lie in terms of considering some of those resource allocations. For example, sometimes the blockage occurs at the provider level. The policy may be in place, but the providers may not want to do the work or may not find it something they desire to pursue.

Sometimes providers really want something to do, but the policy environment means that they cannot due to resource allocations and the power dynamics they cannot bear. However, I believe that from an evaluation point of view, these are really helpful ways of thinking through some of these issues, both in evaluating and, I'm going to use the word "outing," which is kind of what an evaluation is. Outing and being clear about some of these tensions involves confronting them and articulating them. The issue of control, which incidentally someone asked a question about, is significant. There are some things that are outside of our control, exactly. So, of those aspects where you have some degree of influence and control, who fundamentally ought to be able to decide the allocation of resources—wherever they are—to fulfill this purpose and achieve these benefits? Anyway, let's move on to knowledge. We could have a whole day's discussion on whose knowledge is acknowledged and whose knowledge is marginalized.

Clearly, in the description you provided of this particular project, it is something that you have thought about quite deeply. Can you discuss the relevance of the kinds of questions that critical systems surround? I think it's really interesting and very relevant in both our countries, especially concerning First Nations thinking, where both of our countries have a long history of devaluing that knowledge. However, that’s probably a longer conversation. In this particular example, it was fascinating how we brought different forms of knowledge to the table. We were very focused on positioning parents as experts, with their knowledge being the primary source. The health provider would then come in to support that parent, but fundamentally, the parent is the expert regarding their child. This aligns with what we call parent-centered practice, which is not necessarily implemented everywhere, but in this approach, that’s exactly what we aimed to do. We were trying to highlight parental knowledge and expertise, allowing the parent to guide the questions and seek out the information they were interested in. That's what we utilized for this particular project.

So here's what we used for this particular project. Here’s an interesting question: was there any system of knowledge that you accidentally or deliberately excluded? What was the reasoning behind that particular decision? You’re right.

So, there were some drivers that we also utilized. For example, while the parents' expertise came to the fore, there is also a vast knowledge of neurodevelopment that we wanted to include as well. For the ECD experts who are all online today, it won't be a surprise to them, but there are two fundamental pieces of science that were really important drivers for us. One is the fact that as a child's brain is developing, the environments that children are growing up in are crucial. In particular, the ability to bond with your parent and the use of language create a back-and-forth interaction that actually develops the temporal lobe of the brain, which is vital for children's oral language development. So, the first bit of science is that interacting with the child is a really important aspect of development. The second piece of science is regarding children's ability to self-regulate. A real understanding of this ability is a strong predictor of how children can operate in the world. Self-regulation is closely related to what is happening at home.

So, those two pieces of science helped us think about the types of conversations we might have with families regarding the activities parents would like to engage in that make sense for them. This centers around, first, how they want to bond with and communicate with their child, and second, the kinds of things they want to do at home regarding basic matters like food and sleep. Our goal was not to dictate what was right or wrong but rather to understand the science behind what was happening for those families. I acknowledge that this is quite detailed, and I’m not sure everyone wants to delve into these complexities. However, let’s explore them for a moment. Now, let’s move on to the topic of legitimacy, where people often encounter challenges. Especially in these areas, the notion of marginalizing individuals or issues can feel like an insult, as we are frequently working with people we perceive to be marginalized. The key point is that you must focus on someone or something, and in doing so, you inadvertently marginalize something else. If those marginalized individuals or groups possess sufficient power, they may actually disrupt your project or significantly hinder it. This can be seen in the debates surrounding COVID-19 and vaccination, where perceptions of marginalization have proven to be very powerful and potentially destructive.

Um, so an awareness of the fact that, even though you think you're doing very well, there are inevitably aspects that fall outside your focus is important. So, straight up, who is marginalized by your program? This is a really interesting question because I think when people hear the word "marginalized," they automatically think of the very people we are trying to include in this program. Just to briefly address the language—I'm using the terminology from critical systems heuristics. Someone mentioned the whole issue around the word "beneficiary." I know there are concerns, especially in the U.S., regarding the term "beneficiary." However, that's the language used in critical systems heuristics. We could play around with that language and adapt it to something else, but I just wanted to emphasize that you shouldn't be completely wedded to the terminology I'm using. I'm trying to reflect the formal language that has been part of critical systems for the last 30-odd years. Anyway, Sharon, I jumped in when you were discussing issues around benefits, beneficiaries, and marginalized groups. I think this ties into some of the questions that have been coming through as well, which we should probably address fairly soon. This relates to the kind of policy translation thinking here: by focusing a program on families who likely have the least power in the system, we actually created challenges in considering the ongoing feasibility of this approach. So, in your words, we ended up marginalizing the majority.

And by marginalizing the majority, we have actually had less cachet to sustain and scale this program because it was less about the majority. So it’s kind of like I’m just getting my head around it; it almost feels like a double negative. But what ended up happening was that the trade-off was that not everyone received this because it wasn’t needed as a universal program. However, by not everyone getting it, there were trade-offs between universal versus targeted offerings. In the end, the trade-off was twofold. First, it affected the majority.

And number two, we're actually asking the healthcare workforce to work in a very different way. That was the other trade-off; we are asking people to engage in a deeper way.

It took more work. It needed more support, so both of those. Oh my God, we could talk about this more—it's definitely more difficult! Yeah, we could talk about this all day or all morning, depending on what time zone you guys are in. We probably should turn to some of the questions that people have. Well, I just want to point out one thing in terms of marginalization. You had to select an area of operation, which meant, in effect, you marginalized people outside of that area. It's not just about professional boundaries, which I found interesting, but also the trade-offs surrounding where that program was available. I suppose my final question on this, although it may not be the easiest to articulate, is regarding the values that drove your decision-making when you were making those trade-offs. In other words, what were your bottom lines, and what were those values based on? If you can answer that in three minutes, we can then spend a few minutes addressing some of the questions that are coming through, because I think the values are probably at the core of this project and likely at the heart of what drives a lot of early childhood development research.

The values at the very heart of our project were equity and prevention. While I’m not sure if prevention counts as a value, equity certainly does. These two concepts were foundational to the entire project and program. Equity played a crucial role in terms of delivery, power, and outcomes. They were truly at the core of the values driving our efforts.

And I think we're really clear about that from the get-go. However, in terms of the split between the desirable and the feasible, we had to make trade-offs. What were the values behind those trade-offs? Again, the values centered around equity. In an ideal world, we could say that everyone should receive 20 visits from a maternal child health program. Therefore, we had to trade off the universal approach, where everybody gets something, to what we call proportionate universalism, where people receive services according to their needs.

So that was the other value that we had to consider as a trade-off, keeping feasibility in mind. So who decided what the need was? What was the value of that need based on? You see, we could go on forever. Don't answer that.

Sure! Here’s the reformatted text with correct capitalization and spelling: --- Yeah, let's stop, Bob. Let's step back now because we've only got, what, not the Bob and Show? Yeah, we've only got a few minutes left. So, let's step back. You can kind of get the idea, I mean, with the people who are the observers of this little corridor. You can see how even bending around the questions themselves—right? That's why it's heuristic. It's about raising the space for debates that often are not explicit. One of the reasons why this is generally known as the emancipatory end of the system stuff is that it's perceived these discourses are important from a democratic perspective, which is one of the ethical value systems that drove people like C. West Churchman and Norwick to develop critical systems heuristics. It is based on a set of values and assumptions that the practitioners critique all the time. But let's have a look at some of the—uh, and I don't know if Shakufa or Shreya want to bundle any of these. I’m just trying to look through them, and oh, they’re so good! One from Shikufa—yeah, I know they're really good. Did you have a look through them to ask, Bob? Did you have any more content to go through? It seemed like you wanted to... No, no more. No more content except a photograph of some books screen-sharing so that we can actually see you guys. But people have enjoyed the conversation. I think they're really riveted by the dialogue, so that's really great. Thank God it's at night; otherwise, Sharon and I would go on all day, as it were. Bob did warn me ahead of time. He said, “I can go on for a long time; make sure you stop me.” So, what do you think? There’s a good question by Anita. She asked, “In the case you discussed, going back to if the idea is to focus on parents and health workers and ultimately think about positive outcomes for young children, what are the evaluations that could have been used that would have been primarily important for parents, health practitioners, etc.?” Actually, there are so many good questions. There's another one by Luke Williams: “Is the overarching perspective that if you take a decision within certain boundaries, members outside of those boundaries will always be marginalized?” I mean, that's really... exactly! The notion of critical systems is that you have an ethical obligation to consider that. The problem is, as I pointed out earlier on, we make millions of boundary choices in any intervention. So what Ulrich did was say these three are at the core; they raise those particular four question areas. But you're absolutely right.

We make boundary choices every single moment of the day. We make boundary choices when crossing the road. I think for early childhood development, this kind of purposeful, reflective thinking is so important, isn't it? Many of us probably engage in it, but we have not codified it in the way that you have in critical heuristics. There is always value in codifying the way many people think.

But for ECD, it's like for everyone. Everything we're thinking about is how to create a layered approach. How do we actually bring this together not just for one program but for multiple programs targeting multiple people? I think Chris Cuthbert had the question about whether, at a broader level, these critical heuristics work, or if they are really more designed for the program level or the micro level. Do you have a view on that, Bob? Because I think there's a lot of thought regarding ECD systems, and Bob pushed back on me when I mentioned big systems thinking and systems thinking at the program level. Do you have a thought about the macro version versus the meso and micro versions? No, I think it depends really on what the focus is. In the end, you've got to concentrate on something.

The decision on which you primarily focus is itself a boundary choice. You just need to go through a process by saying, to be honest, we need to be aware of, shall we say, for want of a better word, the bigger picture. But actually, in terms of what we have some authority and control over—going back to that—what about the things we have absolutely no control over? Or the things we have the ability to manage? Maybe that's a sort of set of criteria by which you could perhaps decide. That would be my own comment. But yes, absolutely, there are so many layers to this particular onion.

At some stage, you need to work out which is the best one. I'd be interested to hear; I don't know if there are any policymakers online, but it's interesting. I actually think policymakers have a lot of control over many themes, but they don't necessarily make these decisions in the way you might be thinking. They have various opportunities to consider these issues all at once. They don't have to think on a program-by-program basis. If they're really great critical thinkers, they can analyze multiple aspects simultaneously.

And I don't know what other people think online, but I mean that the opportunity in ECD is multi-sector, multi-layered thinking. So I think that's very, very interesting. Shaquipu, where do we go from here? Should we answer more questions online? There's just one more question from me as well.

There are some people who have their hands up. Yeah, and I have one question I’d like to answer as a rapper. So, can I just ask this? Kelly says it’s a great discussion about power, but the end result is still that the RCT is what is held up as a gold standard. Who benefits from this, and who doesn't? I know this comes up quite often, so I just wanted to put that out there as well. Yeah.

Yeah, I'm going to go. I'm going to put that out there as well. Yeah. Yeah. I'm going to have a quick answer, and then Bob, you can have the final word, and then we have to go. Let's give it to Shakufa because we're going to run out of time.

And then, Bob, you can do your wrap-up question as well. Yes, it's a great wrap-up question. I think RCT, depending on the sector, might be the gold standard, but I just don't believe it's the only methodology. In my view, it's a case of "forces for courses." It depends, again, on your values and how important it is to know whether something is effective.

But more importantly than anything else, it does harm. I can tell you that there are lots of well-intended interventions that have been delivered in ways that have actually created harm in the end. Sometimes, I think people forget that without conducting these trials, we don't know whether things have caused harm, and that harm can emerge much later. However, that's a much longer conversation. This is one approach to evaluation, but it is not the only approach to evaluation.

Over to you, Bob. Yeah, I'm not going to comment on RCTs. I just want to do the wrap-up thing, which ties into the final slide. I'll just throw it up.

Hopefully, you can see it. How do you actually do this stuff? Farhan asked, "Bob, could you explain how you would go through this heuristics process with a group of people willing to start a project? Any recommendations on how to introduce them?" Farhan asked, "Bob, could you explain how you would go through this heuristics process with a group of people willing to start a project?"

Any recommendations on how to introduce them and so on and so forth? This is a common question and is really, really important. There are some people I know who approach the questions in a fairly systematic way. I, however, do not. What I do, when I'm involved in the design of a particular project, is that I have these issues buzzing around in my mind, and they inform the questions that I ask, both directly and indirectly. Remember, it's a heuristic, right?

It's not a set of given questions. You can bend and modify the questions to some extent. This final slide is one that I show in all my talks about every single methodology. I've given talks on just about, well, not every single methodology—there are thousands of them—but at least the ones that I know.

Here’s a corrected version of your text: "Are these kinds of strategic guidelines: be careful, be safe, be useful, and be creative? That's a good place to start. So, figure out whether you're truly interested in this process. Are you really interested, or is it just somewhat interesting but you have other priorities? Just be careful."

I mean, you can see how conversations between Sharon and me could have easily gotten way out of control. Um, you just have to be a little bit careful. Um, be safe. In other words, you know, I used to be a snowboard instructor, right? You know, um, you would never take people up to the top of the mountain on the first day.

Right. Even if they were really experienced and wanted to learn something new, you took them down to the bottom of the mountain. It was often very hard to persuade them to do so, but they usually thanked you in the end. Don't play games with people in the sense of just using this when you think it will be useful. Be creative and don't take these questions as if they were written on a stone tablet. These are ways of generating ideas and discussion. And then the final illustration, which I’m not sure whether it's showing or not—let's just see. Oh no, there we go. Have some fun!

This should be, this should be, this should be, it's never a walk in the park. Um, but it shouldn't be a nightmare for all concerned. Some of these questions can generate conversations that, um, can stop being fun. So just, you know, I hope that helps. I mean, it's a bit vague and a bit imprecise, but that's what I've found works for me.

I think it's been really helpful, Bob. And can I just say, I believe the ethical overlay for everybody on this call has been significant; I'm sure everyone will sit back and reflect on this for sure. I do think you might have taken me to the top of the slope and not the bottom, but I'll go with that anyway. I'll take one for the team, and I might just hand over to Toto Kufay now to wrap up.

Sharon, it's been great. Thanks, Bob. Amazing. What an amazing session, everyone. Really, really amazing.

So, just to wrap up, thank you again, Sharon and Bob. I think the comments speak for themselves; that was a riveting session. Thank you to both of you. Again, I feel like a broken record, but please remember to complete the survey. If you want the certificate, the link has been shared several times and is also available on the website.

This was one of the two themes for today, and the next session is in a couple of hours on mapping causal loops and leverage points with Aaron Watson. So, I hope to see you there! If you haven't signed up, I can share the link. Tomorrow is Day Three of the masterclass series, and we have some really exciting sessions lined up. Each day has been remarkable! Tomorrow, we will move into exploring application, which is quite thrilling. We will have Nancy Pelix from the Pelix Foundation, who will share a philanthropic perspective on how they view systems thinking and apply it. She will also discuss the framework they've developed to evaluate resilience. We will get to explore how a single philanthropy and foundation has implemented some of this work across their initiatives. Additionally, we have Session Six with Efren Aguilar from the University of California, Los Angeles, on unleashing the power of equity measurement and collective sense-making to uncover the roots of population health, resilience, and flourishing.

We actually have the Mayor of Pomona, California, Tim Sandoval, who will be online with us, along with the Deputy Cabinet Secretary for the New Mexico Early Childhood Education and Care Department. Additionally, Hero will be our expert early childhood specialist, so that should be exciting! We will also have policymakers on the call. Finally, if you are looking for resources, please visit the EKTAN website. Under the thematic pages, we have a section on systems thinking where you can find more resources. Let's continue this conversation over on connect.ektan.org. Each of the sessions has a thread available now, so we can keep the discussion going there, and the experts will be checking in over the next two weeks to respond to questions. A big thanks, as always, to the EKTAN team, to Shreya, who has been online and the backbone of this work, as well as Hannah, Catherine, Carol, and Hamid. Without them, none of this would be possible. Wishing everyone a good morning, good afternoon, or good evening, wherever you are. Yes, please spread the word! We want to benefit as many people as possible, and the masterclass is entirely free. So feel free to share it across your networks and with your friends. I even got my husband to watch some of these, and he shared it with his colleagues.

So spread the word loud and wide. And again, a big thank you to Bob and Sharon for having a riveting session with us today. Thank you to both. Thanks, everyone. Thanks, everybody.

Certainly! Please provide the text you would like me to reformat, and I'll be happy to assist you.