Child welfare systems have adapted to the national health emergency with ingenuity, including finding new ways to provide services and recruit, develop, retain, and support families. These adaptations provide a special opportunity to evaluate how you’ve done your work in the past and plan for the future. Collecting and analyzing data about your service changes now and how they affect families’ experiences can equip you with valuable information for making program decisions in the future.
One theme we have heard from many systems is that prospective and current foster and adoptive families appreciate being able to access orientation, preservice training, and ongoing support online. Some agencies have shared that they are seeing increases in the number of families participating in orientation and also seeing more families continue on through the preservice training and approval process.
Though the reasons more families may be reaching approval are likely multifaceted, it’s certainly exciting to see that even during this challenging time, some agencies are seeing increases in the number of families becoming resources for children in foster care.
Below we outline ways to explore if your services are accomplishing all you want when they are offered only or mainly online, and to compare satisfaction or outcomes with in-person efforts. Even if comparisons to pre-pandemic services aren’t possible, it’s always worthwhile to begin or continue collecting data to better understand your services.
Questions to consider—and data to collect
How effective are your online recruitment efforts?
Compare the effectiveness of all online recruitment efforts—including through social media—to previous efforts, especially in your ability to reach targeted audiences. Keep track of which activities you are continuing, which previous activities you have suspended or cancelled (such as booths at in-person events), and what new or significantly revised recruitment activities you are using (e.g., new online efforts, presenting to groups virtually). Note how many families each effort attracts and how far they move forward through the process, if at all.
Also keep track of information from prospective parents who contact you about which recruitment efforts and messages they heard or saw. You can use this data to help you assess which efforts are bringing in families.
How does the availability of online orientation affect the number of families attending orientation and moving ahead?
Keep track of which families have begun the fostering, adoption, or kinship process through various versions of initial orientation (online, one-on-one, over the phone) and compare it to information on the experiences of families who attended in-person orientation. You can use this data later to determine whether there are differences in rates of families continuing through the process, differences in the pacing that families move on to subsequent steps, or other differences. Also, if you evaluate sessions for satisfaction or knowledge gained, you can compare that data to see if one method is seen as more effective than the other.
What differences do you see in the effectiveness of preservice vs. in-person training?
For years, child welfare systems have asked whether there is data available on the effectiveness of online training as compared to in-person training. Now, as many systems move to online delivery of preservice training, you have the opportunity to collect data and compare the satisfaction, preparedness, and ongoing experience of parents whose preservice training was delivered online compared to those who attended in-person training.
Consider asking prospective parents questions after they’ve completed preservice training that assess their knowledge attainment. Whenever possible with post-training evaluations, use the same questions you had been using before so you can make comparisons.
What are the benefits and challenges for families in accessing support online or in other remote formats?
Collect feedback from families about their experiences accessing support in various online and remote formats (including phone calls and text messages) and what works best for them. If they’ve used both in-person and virtual support services, ask them which they prefer and why.
Explore whether families have different preferences depending on their needs and situations (such as differences for ongoing support and connections vs. support for crises or differences based on where families live—in more urban or more rural areas).
Tips for successful evaluation and comparison
Use existing tracking systems whenever possible
Only create new tracking tools when the data you seek cannot be captured elsewhere. Instead, add data points to existing tracking systems. For example, when tracking which families have completed preservice training, you can add if that training was completed in-person, online, or a combination of the two.
To determine what data to track, think about the questions you want answered. If you want to know if online services generate more numbers or higher percentages of approved caregivers, you might ask questions such as:
- Are people who have access to online orientation more likely to submit an application than those who do not have access to online orientation?
- Do more prospective parents complete preservice training when it’s online?
If you want to know how services compare in terms of effectiveness, you might ask:
- Are virtual support services effective in meeting families’ support needs?
- Are families who receive online training as prepared to parent as those who received in-person training?
Use surveys to capture family input over time
Families’ opinions about the benefits and drawbacks of virtual support are subject to change. Survey families periodically to better understand how their feelings change or what may affect a family’s ability to benefit from virtual support.
Be proactive in turning evaluation into action
As you begin to routinely seek feedback from families, they will take notice if the needs and challenges they report go unaddressed over time. It’s important to plan for how you will use the information you collect, be prepared to change your practices based on your evaluation results, and to provide updates to families about how you are using the evaluation data. System improvement can only take place when you are ready to take action based on your data.
It’s never too late to start evaluating. If you weren’t gathering data on some of these questions before, you can start now. If comparisons between your in-person and virtual events aren’t possible for drawing meaningful conclusions, you can still learn about how effective your services are in the current environment. Focus on what you can learn now, and continue gathering information when some in-person events return.
Recognizing limitations
While we encourage you to assess the effect of new approaches to conducting your recruitment, development, and support efforts, we also caution you that it is difficult to isolate the impact of remote services from the many other changes that families are facing now. Families may be experiencing increased stress related to job loss, public health threats, and managing online school for their children. These demands may affect their experiences and satisfaction. It may be harder for them to retain new knowledge from preservice training classes, or they may have higher support needs. These other changes certainly affect your ability to draw firm conclusions about these new approaches to your work.
Consider partnering with experienced evaluators, such as a program assessment team, a continuous quality improvement team, or an external evaluator like a university, to better understand both the potential and the limitations of your evaluation efforts.
As you assess the impact of changes in how you deliver services and feedback from families, keep in mind that there is likely no one-size-fits-all solution for how best to recruit, respond to, prepare, and support foster, adoptive, and kinship families. However, the data and insights you gather while you are working in these new ways can be incredibly valuable to help your system determine what variety of formats will work best when you are able to conduct services both in-person and remotely. It may well be that an ideal model may include a mix of virtual and in-person activities.
