If I understand what you are saying, this cat never was quite "right"? Lacked energy the others in the litter had, lacked appetite always, etc?
Get off all the wormers, flea stuff, etc, unless he actually HAS fleas. I use the stuff sparingly only when necessary- it's all somewhat toxic, and more for some individual animals than others.
IMHO, it's way past time to get a vet's opinion here. My suspiscion is some type of chronic- probably congenital- problem. Very possibly a heart defect, although kidney or liver problems are also possible.
A home made "food" I've used on ill, convalescent and weak cats for years is a combination of milk, hard boiled egg, tuna fish and oatmeal. Usually with a dose of cat vitamins mixed in, and often with a teaspoon of cod liver oil added as well. While proportions may vary depending on the animal (kittens get more milk, for example) I figure 1/2 cup milk, 1 egg, 1 can tuna (in oil, if thats at all possible) and 1/2 cup cooked oatmeal. I usually mash it up with an egg beater, but for a small kitten or very weak cat will put it in a blender or food processor and make it into a thick liquid mush.
Canned cat food is something I avoid, always. Although there may be some good brands out there, most of it is made to appeal to *people*, and if you read the labels, you'll see that the nutrition is really lousy. Our cats thrive on the basic Sam's Club brand dry cat food, but they also get raw milk and various meats several times a week. When I butcher chickens, I cook the hearts and livers and chop them in the food processor, then freeze them and give them a small amount every couple of days. Right now they are working on a bunch of chicken I pulled off the bones when I made soup with the backs and necks.
Still, although cats can be picky eaters (if people allow them to!), this sounds like he's got a fairly severe health problem that needs addressing.
Summerthyme