Suppose there's a test. They made the test extremely difficult. Then, it's natural that you would expect that this example of a screening mechanism would result in high quality passers. I would expect so too. Then again, we could be wrong.
In their latest Economic E-Journal paper entitled "Tougher Educational Exam Leading to Worse Selection," Eduardo Andrade and Luciano de Castro shows us instead the possibility of a counterintuitive result: an increase in the exam difficulty may reduce the average quality of selected individuals. Well, their study seems more focused to the labor market but the authors also applied their analysis for teachers and students as well:
"This apparently counterintuitive fact arises because tests do not emphasize all abilities that are important for job performance. A large number of papers show that noncognitive skills not tested in exams are important determinants of the performance in the labor market. When the standard rises, at the margin candidates with relatively low cognitive skills but high noncognitive skills decide not to make the effort to meet the new standard. Candidates who succeed display more cognitive skills but the average level of noncognitive skills falls. As all skills contribute the workers' productivity in the market, the net effect may be a reduction on the average quality (productivity) of those individuals who pass the standard."
Bottomline is, I think Andrade and de Castro's main message is that these tests have to take into consideration non-cognitive aspects, which are likewise important--not just in the labor market but in a school setting as well. I tend to agree more to the point that tests have to be customized to what it is intended for in the first place--and this seems to be one of their recommendations:
"Our results also offer a testable implication: the test is more effective in enhancing productivity when the mix of skills tested is closer to the set of skills needed in the job... it is more important to design the exam in order to test skills directly relevant to the jobs than to raise the standard."