He picks up on all the clues present that Eliza's perfect English is the product of coaching, but when he assumes that her manners are innate, he comes to the conclusion that she's a foreign aristocrat instead of that she's a lower-class Englishwoman.