He was born in 1829/30 (4 (31 in 1861; muster-out roll has '41')).
He enlisted and was mustered into service, at Philadelphia, Pennsylvania, on 21 or 26 October 1861 (1, 3 [as corp], 4 [as pvt], 5 [21 Oct]). He was mustered in as a private, in company H (1, 4, 7-10).
He was transferred on 4 December 1861 to company D (1, 4, 5).
He was promoted to corporal (4).
While in Philadelphia, he cut Samuel Wilson in the head with a saber because Wilson refused to run after a soldier who ran away from him (6). (I assume this is the right Patterson; the other candidate, Thomas Patterson, was a corporal in company B.)
He was killed on 13 December 1862 at Fredericksburg, Virginia (1, 2, 3, 4 [missing, supposed killed]). He was a corporal, in company D (1, 7-10).
1 Bates, Samuel Penniman. History of Pennsylvania volunteers, 1861-5. Harrisburg: B. Singerly, state printer, 1869-71. 5 volumes. 'Ninety-first regiment', volume 3, pages 186-233. (In the roster) (James Patterson [D]; Samuel Patterson [H])
2 company D, register of deaths (James Patterson)
3 company D, list of non-commissioned officers (James Patterson)
4 Civil War Veterans' Card File, available at the Pennsylvania State Archives, searched 8 August 2004 (James Patterson; roll has 'Samuel Patterson')
5 company H, register of men transferred (Samuel Patterson)
6 Senate report 1163 [re pension for Samuel Wilson], 27 May 1890, Serial Set 2709, 51st Congress 1st session, session volume 7 (Corporal Patterson)
7 index to compiled service records of volunteer Union soldiers who served in organizations from the state of Pennsylvania (Samuel Patterson)
8 index to compiled service records of volunteer Union soldiers who served in organizations from the state of Pennsylvania (James Patterson)
9 index to compiled service records of volunteer Union soldiers who served in organizations from the state of Pennsylvania (Samuel Paterson)
10 index to compiled service records of volunteer Union soldiers who served in organizations from the state of Pennsylvania (James Paterson)