Basically, who should eat and who should not depends on the occupation of a person. If you ask a modern doctor, he will simply tell you that if you have need for more calories and you do a lot of physical labor, then only you should eat non vegetarian food. If your occupation is of sedentary type and you eat a lot of high-calorie food, you are bound to gain more weight and invite many sorts of diseases related to obesity.
The medieval caste system of India was based upon the occupation of a person. Therefore the people from a particular caste doing more physical labor were allowed to eat meat. For example, job of a Kshatriya was to fight with enemy and protect the people. So, they required a lot of energy. Hence, Kshatriyas were allowed to eat meat. In the same way, a farmer who requires a lot of physical work was allowed to eat meat. On the other hand, other castes like Brahmins who do Prayer and intelligent work requiring less hard labor were restricted from eating meat. In the same way, weavers, businessmen were not allowed to eat meat.
Aside from that, some non vegetarian products like meat and chicken are hard to digest. If you do not have enough body movement, it will be hard for you to digest that food and you will be unnecessarily inviting the ailments.