Thank you very much. Some years ago I was a young analyst with a military think tank and my job was to analyze an information flow during a military crisis. I went to a meeting and I was very disappointed because no one seemed to listen -- I wanted to tell them about my new model. My supervisor was amused by this young woman who thought she had the answer to military command and control. I said, "I don't know what to do, I'm a female and I'm too young." He said, "I wouldn't worry. The first problem you can't do anything about, and the second problem will go away."
So I'm here to tell you about my new model. I'm not going to discuss this application in a military command and control situation. Rather, I'll apply to health care, and in particular to community health information networks.
When the Community Network Society did its survey about a year ago, major hospitals and key executives were asked what they thought was the most important objective for a managed care system. They said improved communication among physicians for clinical and referral purposes. Now most of us here build systems with the consumer at the center, and most of us believe the consumer is best served if information is captured at grass roots with the community as the provider. Community networks facilitate this process. There have been several attempts to evaluate the role they play in increasing collaboration. The ODPHP sponsored a study that would help organizations decide whether to adopt community networks.
Community networks are interorganizational networks. If this is going to advance collaboration, it must enhance the benefits of working together. Interorganizational networks will become the new institution of the future due to efficiency and adaptivity. Smaller organizations will make decisions collaboratively to adjust to market conditions. The key to this success will be developing new and creative products and services in a shorter time. We tested this in a health care setting, and it did apply.
I'm going to show how these costs and benefits were applied to community networks, when we interviewed front line workers and administrative personnel. Front line workers tended to focus on benefits. Emphasized adaptive efficiency, by reducing time of development of treatment plan. Administrators asked, who owns the system? Will I have to share? How will the organization be awarded by sharing? Physicians wanted community nets for referrals. We decided to use as an outcome measure the time it took to develop a treatment plan. This was one of the primary goals of the community network. There was a conference in Washington, DC a few weeks ago in which Connie Morella, a U.S. Representative from Maryland, spoke, and I was pleased to hear her say, "We need to provide more timely treatments."
We're talking about community networks and how they can achieve true community networks. So, this is our objective, to show that after one year a community service network will reduce by 30% the time it takes to develop a treatment plan. Why 30%? A series of focus groups developed this goal as a shared vision. You can see the hypothesis that leads to those objectives, identified by the participants. There are other intermediate benefits as well.
But this is not enough. Just because we decrease the average time to develop a plan, how do we know that plan is better? More appropriate? Time is a good measure because it can be used across many organizations in a community, and across many human services. Second objective was to show that after one year the community network will reduce time to place patients in an alternative continuing care facility. This is a problem in the human services area -- patients don't carry out their treatment. Community networks, if they allow patients to visually communicate with continuing care facilities, tend to follow up the care that has been prescribed. You'll notice a broad range -- nursing home, home care, rehabilitation -- this goes across health and human services. This still isn't enough -- what question are we really asking? That the outcomes have improved.
What we do now as a 3rd objective, we have to show that a community network will reduce the time it takes to achieve the health outcome. This is almost impossible to prove on a case by case basis. It means you have to track patients and see what the outcome was. It means you have to build a medical record that follows the patient. It's even better if you can do this electronically. My model allows you to track patients through various facilities over time. That's a subject for a different talk, but I'll give you a hint of how that's achieved.
The most improved communication -- a lot of our discussion has been about the Internet and the ability to search, to take information that is stored in database -- but what physicians are telling us is that they want the technology for improved communication. I think we are in the midst of a paradigm shift in which the Internet and other interactive technologies are going to be used more for communication than for selection of information. This was hinted at several times in our discussions this morning.
The model shows who talks to whom and how effectively. A nurse talks to a nurse, or a physician, when developing a treatment plan. It's tied together through referrals. This allows you to analyze specific patient episodes in focus sessions to see how effective a managed care organization is in providing successful health outcomes. I won't go into this now because of limited time, but I'll be happy to discuss this later.
The purpose is to introduce the new model and its method to evaluate a community network. The study design comes from Campbell and Stanley, the traditional approach to health services and evaluation, in this case the intervention is the community network. It doesn't have to be the entire community, just those in organizations that refer patients to each other. Then we take a comparison group that does not have the community network, and we observe both. So we're using a standard approach that is normally used on other interventions, but we're using the community network as the intervention.
I'm going to show you that process. We conducted preliminary information flows -- who talks to whom, about what, and how effectively. This shows who should be involved in your evaluation. We selected the participants, the intervention group, and the comparison group, and collected information on the amount of time it took to develop the plan. These times were converted into costs. Each organization in the network can measure whether the network has decreased costs for them. At the same time the collective outcome can be measured to see if the benefits outweigh the costs. The participants can determine the price they are willing to pay. The beauty of tracking cases is communities can track the process they are using to provide health care.
The major advantage is that large investments are not necessary. You don't need to start with the entire community, just start with those who are collaborating. So the advantage is the bottom-up assessment of the interorganizational network. Individual participants can assess the economic benefits themselves. Patient privacy is protected, since we're only measuring times that can be converted to costs.
The model for community health outcomes that traces who talks to whom doesn't require knowledge of patients at all. Simply collect information at each institution about who referred to whom, and you've got the complete picture. It's a virtual electronic record -- you can then ask the patient if they'd be willing to work with you in a focus group on the messages that each institution contained. You can protect patient privacy in this method. The government will often stipulate in funding that there cannot be a duplication of effort. This is one of the absolutes of a community network -- interorganizational networks are more efficient if the participants have unique capabilities, as there is a natural incentive to reduce duplication and the costs of providing health care.
The conceptual framework -- economic efficiency is very important, but we also have to look at adaptive efficiency. Over time, communities will realize that there are economic efficiencies to working together. I start by showing that there are some benefits, but right now we think there is more in enterprise networks than in interorganizational benefits. Over time, they will realize that the net benefits exceed the costs of working together. At the same time, the enterprise networks benefits will be less obvious. At some point, you have an equilibrium between enterprise and community networks -- so many of each type of network.
Suppose we use the technology -- when I talk about community networks I'm just talking about people who get together and collaborate on the best treatment for their clients -- but suppose we provide this group with some technology. We're suppose to show the collaborative technology enhances collaboration. Here's what would happen. What does that do to the equilibrium point? We have in fact improved adaptive efficiency -- we have reached equilibrium in a shorter period of time. We're also reaching it at a higher point with higher net benefits to the enterprise benefits, because a community network enhances administrative activities. Everyone benefits -- better adaptive efficiencies, and both kinds of networks improve.
I have taken the two approaches that we seem to be taking right now toward networks and technology -- the vertically integrated systems that we're developing in managed care organizations, and the collaborative technologies that help us communicate in parallel, such as provider and consumer networks. I think you can play this little game -- what falls into which technology? Collaborative technologies help us work together, but integrated systems focus on information. They help us share information, and that's what our technology is leading us toward -- more communication over information. Integrated systems are top down, collaborative from the bottom up. Integrated systems are enterprise systems that extend to community. Integrated systems are more structured; it's very hard to adapt your coding system into the integrated coding system. Collaborative technologies are adaptive. One focuses on stores of information, the other on flows. One is on administrative information -- most community network activity is in fact being developed for administrative and financial capabilities and far less to help the physicians and the consumers collaborate. Where is the consumer in all this?
The whole idea is that all of this can go on. I could talk further, but I'm told that I'm out of time.
Q: You mentioned a survey of community health information networks. Is that available?
A: It definitely is. Pamela Hanlon, president of COMNET, has this information.
Q: When you draw a diagram, are you representing a particular perspective on cost and gain?
A: I'm representing the cost of benefits of building a community network -- the costs and benefits of whoever wants to invest.
Q: [unintelligible]
A: They can also be integrated. You have the data repository at top, then the various specialists. I'm talking about the vertically integrated network of contractors and affiliates.
Q: I"m having a hard time understanding your use of the term vertical and integrated.
A: A lot of the literature uses these terms, but that's an excellent point. We think of it this way because the patient is seen as a at the bottom of this. Many of these networks can be represented as a consumer who sees a primary care physician who is a gateway to the specialized services. Maybe seeing the consumer at the bottom isn't fair.
Q: You mentioned issues of visual communication in passing.
A: That came from our interviews. We found that front line workers that when the client was able to actually see the person that they were being sent to, through a video, that tended to improve the probability that the client would go there. A treatment is recommended, and sometimes they never show up. That improves that process. That was from focus groups with the patient.
Holly JimisonMy title is comprehensive evaluation. Everyone here is an expert, we all have a notion of evaluation. What I'd like to do today is provide a framework for community aspects -- one project we're in involves a community-wide intervention in Boise. I'd like to bring you into the discussion of the variety of aspects of evaluation and also get your feedback on whether this is a useful framework, and ask you to chime in on the experiences you've had too.
Evaluation goes on through in the planning stages and the culmination of deployment. Needs assessment can be thought of as evaluation. Prior to design, especially from the technical side -- say you've been trained in computer science and engineering -- you're probably not the most appropriate person to say how people want to interact with systems. Hewlett Packard was once known for excellent design for their colleagues, but not so much for the consumers. You may not be the most prototypical consumer yourself --we're educated, higher income, motivated -- are we trying to reach similar people or reach others? Linda Adler started the informed patient consumer group at Stanford and started with means assessments though ethnographic interviewing of the patients and providers. Has anyone here done a means assessment or have other techniques?
Aud: We took a community information network, had focus groups with patients, visited housing projects to see what will happen and how they'll use the equipment.
Aud: We did a participant evaluation watching clinicians who practiced.
Aud: We started a small consumer health information network and did a survey for needs assessment.
Sometimes you have to do a qualitative needs analysis to even get the questions for ht survey. The results will give you input for design specifications of the system. It's at this time you start defining your outcomes. Don't wait until you've developed -- then your specifications will reflect your design. Start to design objectives first, then ensure that the design of the system does something to move towards your goals.
Ongoing process evaluation during development. Whenever I talk to people in the commercial world, if you're in a competitive for- profit market, they rarely think about this. What's your impression?
Aud: We come up against that as a support concept. They have a hard time adapting.
Aud: [unintelligible]
Even in the commercial world, no one does evaluation until you see market share. That's a little late. It may take additional time, but it also could improve your product to the point of making it cost effective. I'd like to bring up the example of a project that we're working on and we'll attest to the fact there is definite tension. What could be done in this stage of evaluation as you're developing, you want to take your systems apart, make them fail- safe. That can be one of the most frightening things to deploy. This is part of the ongoing system development. The accuracy of what you're doing -- risk communication, response times issues, navigation is important, then how do you do that? How do you make sure it's meaningful, no mismatch between expectations and what they see? Are their needs and objectives met? This is usually done during usability testing.
Aud: What we're seeing in the Internet, there's a lot more focus group capability -- we have an Internet team that brings people in for the look and feel and the content side -- I really think that in this medium that the medium draws them in through real-time feedback.
But in that model you can't test very well. In our first stage to develop a kiosk to help consumers choose a health plan, you don't assume understanding of graphs or high reading ability. You want to communicate measures so they can make an important choice. Our first step was to mock up a series of screens in Hypercard, take them to focus groups, and find out what icons they thought were meaningful, can they read graphs, do they understand what's being presented -- a map with hospitals, enter their home and find the closest clinic. Some people won't admit they can't read maps, but they'll say their friends can't. You can't just put it out and see what happens.
One reason we advocate mockups, before you get your ego invested in your approach. In focus groups we presented a variety of ways. But your point about the incredible value of being able to get real feedback from whoever's online -- remember that those online are those already on the Internet and are facile with the technology, might not be the feedback you want. We're proposing a framework for getting the analysis outcomes you want.
Protocol analysis brings in subjects to a lab setting, starts them on the system with no coaching, and ask them to talk aloud every thought that comes into their mind. Typically it's videotaped -- you can do rough analysis on colleagues to get your first debugging. It's a very important step. Many CD-ROMs are getting flashier, but not more useful. A lot of times all the flash takes away from the functionality. We've heard a variety of speculation -- one finding, and then assumption -- these things have to be tested. The thing you can do in this process is to look at different components -- do you need interactive video? Are still images and audio all right? How do you tailor to patients? This is iterative development. Don't wait to the end -- get feedback as you go along. Does anyone have experience with this?
Aud: We've had colleagues and patients give us specific responses in interviews. We have a system built in so when people don't understand we track the number of hits there to show statistics on how understandable each segment is.
Excellent idea. And that's not hard to implement.
Aud: [unintelligible] It's very intimidating, very complicated. Going up to this computer and having to say what's on your mind while it's watching you, with no online help, no manual.
But that's the real world, that's what happens.
Aud: if you want news or feedback on your tools, always include "don't understand" as an option. Your alternatives then are to rewrite the frame, or include additional layers of help. Include this as a final option in the system.
Aud: [unintelligible]
That's an excellent point. Focus group results must be taken with a grain of salt -- it's a very small number. The focus is to inform more careful later study, but it's much better than launching off with preconceived notions.
Aud: In our own project, focus groups -- it's such an investment. The most embarrassing thing I've done is videotape my own focus group, the one I was running. But it's not easy to hire someone who knows our objectives. We work with a consultant who gets to know us and can learn what we need to know. We also try to bring groups back twice to follow up.
We had trouble with our focus group leader -- the information that we had to present was heavy on data, and he was more afraid of the statistics than our participants. You don't want someone biased by being enculturated into medicine, but you get bogus results when they know nothing about medicine.
Aud: [mostly unintelligible] In other cultures, some of these assumptions may not work. Where I'm from [New Zealand], some of these won't translate.
Studies have shown that people are more accurate predicting neighbors behavior than their own behavior. That's an interesting effect.
Aud: We do limit the options of our participants before we start. There has been work lately to try to remove this bias so we can look at these kinds of situations better. There have been a lot of developments in this research to try to get rid of all these biases.
So, to sum up, your recommendation is for participatory design.
Most of you think about evaluation is testing equipment in a controlled setting. I won't spend too much time here, usual design is you have intervention and control group, but we should spend some time focusing on access, availability, how and when and who is using it, measure process variables where the real outcomes is behavior change, health outcomes, and satisfaction. Intermediate outcomes are confidence, attitudes, and self-efficacy. Is there anything I've forgotten here? The big issue, are these benefits worth the cost? Will this be a success? There are a variety of methods -- those all have different costs. Are the benefits worth the cost?
Aud: Something about operational impacts. What does this do to the nurse in the implementation?
These are things we wouldn't intuitively think of. This all happens in a real setting. This must be tailored to the actual application. What we want here are clear placeholders for outcomes measurements.
We're measuring the same kinds of things, but I wanted to discuss the difficulty once your deploy this in the community, you have to describe and monitor the process of the intervention. The example I wanted to bring up is evaluating a large community intervention from Boise, Idaho. I imagine most of you are familiar with the HealthWise handbook. They're mailing this to every household in four counties in Boise. That's phase one, phase two is a kiosk to increase shared decision making, over the phone, in libraries and community centers, employee sites, scattered throughout the community. This includes urban and rural Boise. We did a huge baseline survey and compared it to other communities, but we thought you couldn't do it all by survey. In monitoring the background trends, about 20% of them already had the handbook. With regard to the computer, they have a couple of computer companies, many are already using CD-ROMs for health information and will search the web at work, how can you have an impact when this is already happening in the background. This will all ramp up while we're trying to measure the impact.
It's not just a book and computer, it's publicity, workshops, community organizing, and we're testing the added effect of these components. HealthWise is great for working people up to a religious fervor over intervention. THe purpose of an evaluation is to give people an idea of could this generalize to another community. Should I do this somewhere else as well? if you think it's just mailing a book, you'll totally miss the results.
The community we've seen so far it's been difficult to show a community wide effect, and if your sample is one intervention and two controls, it's very difficult with a sample of three. When there's so much happening culturally, you have to very carefully describe that and then put your intervention on top of that. Factors in Boise -- rapid growth, few physicians, employers make it difficult to refer to specialists even though there is not much managed care. We are tracking this over a three year period. In the background is all of the other interventions that're being mentioned in this conference. We have to understand that impact.
There's a broad spectrum now of ways to think of evaluation. They're all difficult. Think throughout the development to have an optimal system when done with your investment. Where will all this busyness get us? it's well worth thinking about evaluation.
Aud: What are some words of wisdom to generalize this?
Jimison: When I think of the HealthWise example, we're being asked by other groups to do similar experiments, but they want to tack on changes. It's difficult -- you can break things into components, then it's more costly to evaluate. We already know from an HMO viewpoint we should mail out these books, it improves patient satisfaction and loyalty. if you want to see how behaviors change, that's very difficult. I wouldn't assume it's going to generalize. I would have more caution than to say I have a smart way to do it. There's so much going on in society already. If you can look at subgroups, the intervention might be very important in rural areas, but it's very difficult to sample those groups. In a community-wide intervention, you're looking at the whole. The other way is to buy into appropriate utilization, get a handle on all the different methods.
Aud: are these books going out all at once?
Jimison: HealthWise mailed them all out in one wave with a huge publicity campaign. What's being phased is the computer comes second. Six months after the book is mailed, we evaluate utilization. Then we roll out the knowledge base and evaluate the effect of that. Ideally for them it's all one intervention. It's only for evaluation purposes that you have the phases.