There has been a lot of discussion recently about defeating CAPTCHAs (Completely Automated Public Turing Test to Tell Computers and Humans Appart). I won't reiterate all of the hacks and arguments here - for many links, see here. What I did want to bring attention to is a set of criteria for evaluating CAPTCHA systems posted yesterday to the web security list by Jeremiah Grossman (link to full message).

He calls it the CAPTCHA Effectiveness Test:

1. The test must be able to be administered where the human and the server are remote to each other over the network.
2. The test must be easy for humans to pass. Less than 0.01% of humans should fail the test on the first attempt.
3. The test must be hard for computer to pass - Computers should have less than a 1 in 10,000,000 chance of guessing the correct answer. (Even after a pre-determined amount of analysis time)
4. The test must be able to be completed by a human in less than a several seconds.
5. Knowledge a test question, answer, or result (or combination thereof) must not impact the predictability of following tests.
6. The test should not discriminate against the blind or the deaf. Or provide a solution to address the issue.
7. The test should not possess a geographic, cultural, or language bias.