I co-founded a group that provides secondary school scholarships to children in Southern Tanzania and we ran into a very similar problem (jiamini.org). Like you

I co-founded a group that provides secondary school scholarships to children in Southern Tanzania and we ran into a very similar problem (jiamini.org). Like your scholarship, ours was based on a combination of need and merit where the ideal candidates are those that have the capacity to excel in school, but don't have the resources to fulfill their potential. The first step in our solution to the selection problem was to have children nominated for the scholarship by teachers, headmasters and other community leaders. Nominees could not be direct relatives. They had to be in the top 10% of their class or had to be vouched for by community leaders that they were capable of this type of ranking given changes in their home life. They also had to have a difficult home life - i.e. living with relatives living with AIDS, orphans, living in a mud hut, etc. We also required any set of nominees offered up by a teacher or headmaster to be half female. Given a large enough sample size - typically we select about 15 kids per year from 2 primary schools - we are able to get about 30 nominees. School ranking, sex and kinship are relatively easy to verify. We can't directly verify home life or income so we provide a simple questionnaire for all nominees to fill out. Answers provided on the questionnaires are then confirmed by other teachers and community leaders we trust. After all of this, it is relatively easy to identify a group of 15 kids who demonstrate ability and have few available resources. In other words, we've also found there is a strong correlation between income and scores, but there always seem to be kids who score high from the low income groups. People at school and in the community often know who these kids are. You're right, you can't trust a single source of anecdotal information, so the key seems to be to build in redundant sources of qualitative data. This approach won't pass an audit with the same ease as a regression analysis and it may miss some ideal candidates, but it does help find some kids that could use the help. What group are you working for?