If they cannot give you a handful of legitimate clients, run away. Having an intermediary is not always a bad thing, but let us assume you want to go the factory directly for whatever reason, then here are some red flags that you may be dealing with a middleman:
Avoid those that refuse to list the name or location of the production facility.
Focus on those factories that can clearly show manufacturing experience with your particular model or production method. They should have samples and quality documents readily available if they are a real factory.
If you are able to arrange a factory visit, do your contact's business cards match the factory staff's information? If the cards do not match in name, color and address, then your contact is probably a middleman.
Do the people at the factory clearly know your contact or does he give out business cards to factory staff when giving you the tour of "his" factory? At worst case this may be his first time working with the factory and you may as well build your own relationship without him.
Look for clear information about operation size, equipment and staffing on the website. Most intermediaries do not offer the same level of detail that real factories present.
Be wary if they supply a very large range of products. If they say they make toothbrushes and TV sets, most likely one or both are outsourced.
Be aware that polished English skills do not reflect production skills. Often the most polished websites are set up by trading companies.
Ask for ownership papers of the factory. By law, the business license should be hanging from a wall in an easy to see location. Granted, it is often in Chinese, but get a copy (and make sure the copy you are given matches the one on the wall) and later you can translate the Chinese to show valuable information like ownership, years in business, scope of business and such.
Be explicit that the production location may be audited by you or a third party, and that this location cannot be changed without approval from the buyer. You will be surprised at the number of middlemen who will take the buyer on a visit of a factory only to change the location to a less expensive and poor quality option after the buyer leaves