Armour was descended from colonial settlers of Scottish and English origin, with his surname originating in Scotland. He was educated at Cazenovia Academy until the school expelled him for taking a ride in a buggy with a girl. Among his first jobs was that of driver on the Chenango Canal.
Mrs. Fannie J. Bailey who celebrated her 103rd birthday January 27, 1935, was a student at Cazenovia Seminary in 1847-48. Wholesome good times were enjoyed by male and female students, despite the seminary’s strict rules. Buggy rides and walks were popular. Her first boyfriend was Armour. But she gave up her friendship with him because “I couldn’t stand the grease he used on his hair.”