Scarth started his career as a foreman in the East Indies. His skill led him to the HBC. In 1845 he made a model of a ship to replace the aging Cadboro, which turned into the schooner Prince of Wales. He began losing his sight and, once too blind to work in 1853, was given a house near the beach store at Fort Nisqually where he lived with his small family. He became known as a great storyteller, treating visitors to tales of his time with the East India Company. His wife Nancy, whom he had married in 1850, left him in 1856, so he lived within the fort from then until 1863 or later. As Nancy died in 1859, Scarth died a widower sometime in the 1870s, leaving a daughter behind.
James Scarth is possibly the son of the historically famous Isobel Gunn, who is an example of a woman who fooled her way into the HBC by pretending she was a man. This James was born on 29 December 1807 in Pembina, North Dakota. She was sent back to the Orkneys (Scotland) on the Prince of Wales on 20 September 1809, the same ship she had traveled on to the New World, and possibly the namesake of the ship James would later contribute to.