Gorman met his wife, Johanna O'Keefe, also from Ireland, when buying supplies in Los Angeles and they were to have three sons and a daughter. The Gormans built, or moved into, a large log house that sat between the road and along the mountain stream running through the area. The house had rooms for travelers upstairs and a large dining room downstairs where meals were served. In 1876, only eight years after their marriage, James Gorman died when he was run over by his wagon as he was returning from Los Angeles with a load of supplies. The next year James and Johanna's little girl was to die as well.

The first post office was established at "Gorman's Station" in December 1877 with Henry Gorman, probably James' brother, as the first postmaster. With the help of her brother-in-law and her sons, Johanna continued to run the family farm and the roadside "way station" until her death in 1889.

With the closing of Fort Tejon the economy of the area shifted to mining and the community of Gorman became the main source of supplies, services and entertainment for the many miners of the Hungry Valley and Frazier Mountain areas. By this time the settlement consisted of a store, post office, saloon, blacksmith shop, stable, hotel and numerous homes and ranch buildings. In the 1890's, writer Mary Austin wrote of the town: "We took dinner at 'Hell', a supply station and hostelry, known sometimes as 'Gorman.'"

The Gorman brothers sold their property to Oscar Ralphs in 1898. Mr. Ralphs and his brothers had established the Ralphs Markets in Los Angeles in the early 1890s. Another property owner in the area by this time was James McKenzie. His oldest daughter, Mary, married Mr. Ralphs in 1901. They lived in the Gorman family home with their nine children and established a dynasty that has carried on for over 100 years.