I found the below in a discussion on obesity and its relation to wealth here. It's really interesting and I suggest you give it a read. As that blog states, though obesity in Western countries today largely correlates with poverty, it was once a strong indicator of wealth. The more money you had, the more Willy Wonka-style feasts you could afford. It wasn't a perfect correlation–there was such a thing as being too fat in the 19th century (looking at you, George IV), and slender, athletic, muscly men certainly were considered attractive. However, the Victorians treated the concept of "fat" quite differently and with far less stigma than we do today.
In 1866, a "Fat Man's Club" was opened in Connecticut. Each member had to weigh at least 200 lbs to qualify for membership. Apparently the club was so popular, it spread. The below image was taken of members of a Fat Man's Club in 1894 in Van Zandt County, Texas. According to the County Genealogical Society, each man was weighed in a cotton gin to verify his qualifications.
In many ways, the Fat Man's Club was a Victorian country club, only instead of directly measuring the power of its members through their wallets, it was thought that their power and status could be read directly in their physique. Not sure what the activities were–probably not swimming and golfing like a normal country club. I'm guessing "Sitting around and stuffing yourself with pies" would be high up on the activities list.
Then, around the turn of the century, food started to be made available and affordable to everyone. The railways had a lot to do with this, since food could be shipped in bulk quickly from the country and coast and arrive still fresh in the city. You didn't have to be a millionaire to eat well anymore. So the lower classes started filling out a bit. And when that happened, the upper classes started losing weight. If they couldn't be the fattest, then they'd be the thinnest, goddamn it! (Check out any socialite today and tell me I'm wrong.) Suffice it to say, the Fat Man's Club closed in only 1903, leaving us with our current views on weight and wealth.
Though I have been unable to find and reproduce it here (help, anyone, if you can), I once read an amazing non-fiction novel called Devil in the White City, which had in it a menu that was typical of high-powered business dinners in the 1890s. If you don't believe me that food was associated to power, you should see this thing. It must have had ten courses, all of them decadent and covered with cream and filled with cheese and artery-clogging goodness, with several cigarette and booze breaks in the middle. I would buy that book just to have the menu on hand.
So the moral of the story is, if someone says you're fat, just tell them it's because you're from old money and prefer to keep the traditions alive.
A friend of mine managed to find the menu from Devil in the White City. It is as follows, with the course underlined and the accompanying alcohol in italics:
Blue Points a l'Alaska.
Consomme printanier. Creme de Celeri.
Rissoles Chateaubriand. Amandes salees. Olives, etc.
Bass rayee, sauce hollandaise. Pommes parisiennes.
Miersfeiner. Moet et Chandon. Perrier Jouet, Extra Dry Special.
Filet de Boeuf aux champignons. Haricots verts. Pommes duchesse.
Ris de Veau en cotelette. Petit Pois.
Romaine fantaisie. Cigarettes.
Canard de Tete Rouge. Salade de Laitue.
Petits Moules fantaisies. Gateaux assortis. Bonbons. Petits-fours. Fruit assortis.
Roquefort et Camembert.
Cognac. Cordials. Cigars.