The "Canoe Rock" House of Big Stone Gap was built about 1840, probably by William Richmond, and is the oldest structure still standing in Big Stone Gap. He was one of the leading citizens of early Wise County, being one of the commissioners who located the county court and also the first presiding justice of the Wise County Court. The Richmond estate extended from Roaring Branch along Stone Mountain to Cadet and back up Aviation Road. (Note: since the publication of The Big Bear Grass, the Richmond House has burned.) The house was situated by the trail crossing of the Middle Fork of the Powell River on the north bank. The road on the north side of the river forked with one branch going beside the river to present Lee County and the other going up stream to present Appalachia and Stonega. The original house was a four room log cabin with a puncheon floor. It was named after a rock on the opposite bank where people landed their canoes after ferrying the river. The house was later enlarged to a two wing frame affair. At some point after the Civil War the farm passed into the Flanary family, and they maintained a store beside the road. Indians (?Melungeons) used to come out of the mountains and after finding out where the barrels of coffee, sugar, and so forth were located would return at night and bore a hole through the floor into the bottom of the barrel and drain off what they wanted and then plug the barrel back up with a stob. There is a persistent tradition that there was a post office in this store with the postmark of "Three Forks". If this is so, it is not on the list of Wise County post offices listed by the U.S. Postal Service as given in Johnson's Wise County, Virginia, and it would have existed at the same time as the Big Stone Gap Post Office at Elkanah Gilley's on Country Boy Island.

One of the stories William's family left us with is when Mrs. Mary Richmond learned that bushwhackers had found out she had some meat and wheat left and that they were coming after it. She had been saving the wheat for seed but had not sewn it as it was not time yet. Nevertheless, upon learning that it was to be stolen from her she sewed it and suspended the meat high up in the chimney and built a fire in the fire place under it. When the foragers came and demanded the food she told them the meat was used up and that the wheat was sewn across the ground and that they could have it if they wanted to pick it up. The bushwhackers left and returned on another occasion intent on hanging the entire family. William, Jr. was in his twenties, but was at home because he had a short deformed leg. His nickname was Flitter Bill. The bushwhackers were drunk and got to laughing how they were going to hang Flitter Bill first and make his legs the same length. They got so drunk that they were unable to hang anybody, and the Richmond's were saved.