Parker resided in what was by then the ancestral home, a famous structure today known as the Old House (see picture), which is currently part of a museum complex on the village green of Cutchogue. Open to the public during the summer, it is considered one of the finest American examples of domestic 17th century English architecture, and was a mansion in its day when it was built in 1649 (see historic marker). One of the oldest houses in New York State, it had been constructed by prosperous shipping merchant John Budd, great grandson of the 16th Earl of Warwick, and given to his daughter Anna and son-in-law Benjamin Horton as a wedding present. The house was originally erected in the village of Southold, but in June of 1660 it was moved six miles to Cutchogue, then a wilderness area with no European settlers nearer than two days' journey to the west. The Hortons had no children, so Parker's grandfather purchased the house in 1699 and in 1749 it was inherited by Parker. As detailed below, the house was confiscated at the end of the American Revolution, and auctioned off in 1784 to Parker's nemesis Jared Landon, who because of Parker's efforts, had previously been imprisoned for guiding Patriot raids against Long Islanders. However, the Wickham family regained possession of the property in 1867, when it was purchased from Landon's grandnephew by William Harrison Case and his wife Hannah "Nancy" Wickham, Parker's great-granddaughter. They were the last family to live in the house, and by the early 1900's, the house was showing its age and became a storage facility for farm machinery. Eventually, its great historical value was recognized, and it was acquired from the Case family by the Independent Congregational Society of Cutchogue in 1939. With the help of many groups, it was restored and dedicated as part of the Tercentenary Celebration of Southold Town in 1940, became a National Historic Landmark in 1962 and was listed on the National Register of Historic Places in October, 1966. In 1999, the 350th anniversary of the construction of the Old House was celebrated, with descendants from the Budd, Horton, Wickham, Landon and Case families attending. The rectangular house, which features clapboard siding and a spectacular English-style chimney, measures 20 x 40 feet, and has two floors and an attic, but no cellar. The first floor contains a kitchen and parlor, each with large fireplaces, while the second floor contains two bedrooms. Of utmost significance are the three-part casement window frames on the north wall of the second floor, considered the most outstanding examples of this architecture in the country (see picture).

Parker's father died in only his 47th year, so at age 22, Parker inherited a sizable amount of property including 590 acres of farmland in Cutchogue, 900 acres of additional property in nearby Riverhead, and the storied 445 acre Robins Island, situated in the center of Long Island's Peconic Bay. Parker's grandfather Joseph Wickham had purchased the Cutchogue property in 1699 and Robins Island in 1715. The Riverhead property had been given to Parker's parents in 1727 by his maternal grandfather John Parker. As the eldest son, Parker received all of the ancestral landholdings because his family believed in the English aristocratic concept of primogeniture. On December 14, 1752, Parker married Mary Goldsmith and they eventually had six children: Joseph Parker, John, Thomas, Parnel, Hellen and Sarah. Parker had already begun a slow but relentless climb in local area politics, first being elected to the position of Fence Viewer and Prisor of Damage in 1751, responsible for establishing property boundaries. In the years 1754 and 1755, he was elected Overseer of the Poor, charged with providing for the area's paupers. He was appointed justice for the County of Suffolk in 1763. After being elected town assessor in 1765 and 1766, he was elected town supervisor of Southold Town for the first time in 1767, the highest post of the local government, and held the position for an impressive nine one-year terms. At the time, the Town of Southold included what is today the Town of Riverhead, so Parker oversaw an area stretching approximately 35 miles along the north-east coast of Long Island, a district of about 121 square miles of land. In his capacity as a public servant, Parker was active in working to improve the living conditions of Long Island's few remaining Native Americans. Working with a legal representative of New York's colonial governor, he took statements from local citizens about Indian properties and the documentation bearing his signature still survives. Parker was also commissioned a major in the local militia, after he and seven other townsmen petitioned the lieutenant governor of New York to form themselves into "a compleat organized company of Troopers" for the Town of Southold.

Just 19 days before the start of the American Revolution, Parker left elected office for two crucial years, from April 1775 to March 1777, although it was clear he at least tacitly supported the Loyalist cause, as did many other family members, including his nephew John Wickham, the prominent Federalist, who enlisted in a Loyalist regiment. In August of 1776, Washington was routed by the British in the Battle of Long Island, so the Town of Southold came under firm British control. Many die-hard Patriots moved to Connecticut, which was used as a staging ground to launch raids into Long Island. While some of these raids were directed against British military posts, often times the raiders would pillage the farms of suspected Loyalists and even kidnap the owners. This caused economic hardship for the affected families and created an atmosphere of public fear. In April of 1777, Parker came out of political retirement and was elected to his ninth term as town supervisor, during a time when British military fortunes were at their peak, shortly before the crushing British defeat in upstate New York at Saratoga in October of that year, an event which proved to be the turning point of the war. With a clear mandate to improve public security and proclaiming that "I will not raise my hand against my king," Parker, in his role as an elected government official, denounced those he suspected of guiding the raids against Long Islanders, causing some to be jailed and earning him considerable enmity from certain neighbors, especially the Landons, a politically active family committed to the Patriot cause who had publicly quarreled with Parker before. However, Southold at the time had a majority of Loyalist citizens and Parker enjoyed wide, though hardly universal, support in his efforts to protect citizens from what he saw to be a fringe minority breaking the law. But on December 13, 1777, a Patriot force from Connecticut raided Southold and kidnapped Parker, who was still serving out his term as town supervisor. He was taken back to Connecticut, where he has held captive in a house under armed guard. When Washington learned of the raid, which included additional objectives besides Parker's abduction, he was quick to express his concerns that these operations did more harm than good, writing to a subordinate that he has "ever been averse to these small excursions, especially when they divert our attention from more material Objects."

Parker was paroled later in the war, so he returned to Long Island, which remained under British control, but on October 22, 1779, New York's legislature passed a bill of attainder which named Parker as one of 59 English sympathizers (and one of three in Suffolk County), stripping him of his property and banishing him from the state. Bills of attainder were outlawed by Article I of the Constitution of the United States in 1789, but previous to that time they were frequently used by lawmakers to destroy the careers of political rivals by declaring them guilty of some crime, but without any burden of proof. With Long Island still under British control, Parker had little to fear of being attainted, but after the British defeat at Yorktown in 1781, it was clear what the future held. Meanwhile, Parker's wife Mary died on April 4, 1783 at age 49 and was buried in Cutchogue. After the British evacuation of eastern Long Island on November 22, 1783, New York State was finally able to enforce its bill and Parker's property was appraised and sold at auction in the late summer of 1784 by the commissioners of forfeiture. Political opponent Jared Landon moved into Parker's home, while Robins Island, after being purchased at auction by two of Washington's top spies, went to Ezra L'Hommedieu, a Southold native who had prepared the confiscation bill while in the state assembly. Parker's estate in Riverhead was turned into the downtown district of that growing village. An ailing Parker had already been forced to move to New London County, Connecticut, due to the banishment provision of the act of attainder, which reads: "Persons herein before particularly named, shall be, and hereby are declared to be forever banished from this State; and each and every of them, who shall at any time Time hereafter be found in any Part of this State, shall be, and are hereby adjudged and declared guilty of Felony, and shall suffer Death as in Cases of Felony, without Benefit of Clergy." Broken by the experience, Parker soon died in Connecticut on May 22, 1785 and was buried there in Waterford (see gravestone). Just before his death, he wrote a family member that "I have acted consistently and consciously throughout my whole conduct, with a firm belief there is a future existence, and defy the state to produce one instance wherein I have acted rigidly, defrauded, or abused one member of it, although it was in my power." His son Joseph, who had already sailed to Antigua to engage in trade, was clearly embittered by these events, writing from Dublin in October of 1790 that "I never more expect to see my Native Country. In short the suffering of our family have been so Great in it that I don't want to see it. I am a Single Man and the World is before me." But a year later he returned to Cutchogue and eventually purchased a farm there.

Ironically, the timing of Parker's death just after the close of the American Revolution helped enable the Wickham family to remain on Long Island, rather than moving to Nova Scotia or England, since only Parker had been named as a Loyalist. Furthermore, the British government agreed in December of 1789 to partially compensate Parker's son Joseph for the loss of property, paying him 2,800 pounds sterling, a modest fortune that was about half of his total losses. Before his death in 1785, Parker considered seeking additional relief from the State of New York, but decided it was unethical, writing: "As application is already made for redress of grievances to another quarter, I think it scandalous and hypocritical to apply for redress of grievances to two opposite parties for the same thing." Financially aided by the payment from the British government, members of the Wickham family, such as Parker's great-grandson William Wickham, a four-term district attorney, continued to be prominent in the Cutchogue area, somewhat to the surprise of Parker's political enemies. (Other descendants such as healthcare pioneer William Hull Wickham, farmer James Parker Wickham and banker spouse Matilda Briggs Wickham would also prosper in New York State.) Yet the confiscation of Parker's property has remained controversial over the centuries for a number of reasons, including that it occurred in violation of the terms of the Treaty of Paris, which ended the American Revolution. In addition, Parker's guilt was determined by a legislature rather than a court and there was no chance to appeal. No evidence or witnesses were ever presented and Parker's accusers were never identified. While at least one historian has chosen to label Parker as "notorious," this is an inappropriate characterization of a man who spent almost his entire career helping to improve the lot of fellow citizens and who, more than anything else, was simply opposed to violence against government officials. While many of those targeted by the attainder bill were royal appointees who were later shunned by society, Parker continued to command considerable local respect because he was a fourth-generation American who had achieved his positions by winning open elections. In recent years, Parker's iconic stature has grown, with a book about Parker's career and legacy, The Wickham Claim by legal scholar Dwight Holbrook, being published in 1984 which questions Parker's "guilt" and postulates that Parker was the victim of a machiavellian scheme hatched by covetous political rivals. Parker will always remain a complicated figure: a proper, ethical and democratic man advocating change through peaceful means who believed in a fundamentally undemocratic system, who was then persecuted for those beliefs and deprived of his civil liberties by those professing to be bringing liberty to all.

Not all of Parker's family shared his political views. His daughter Parnel was regarded as a staunch Patriot and married James Reeve, who served in the Third Regiment of Minute Men in Suffolk County. Perhaps more seriously, Parker's brother Thomas, a Yale graduate and formidable political figure in his own right, took an active role in the conflict on behalf of the Patriots. Prior to the American defeat at the Battle of Long Island, he was in charge of nearby Gardiners Island and the stock there, and served in the 1st, 2nd and 3rd Provincial Congresses. Relocating to the safety of Stonington, Connecticut in September 1776, he was appointed Auditor of Refugees� Claims, and became the captain of a privateer sloop called the Hampton Packet, which carried eight guns and a crew of thirty. During one raid, he seized Parker's cattle that were grazing on Robins Island. On another occasion, he captured the Peggy, a 50-ton schooner carrying flax-seed and oysters, off Fire Island Inlet. In his most audacious operation, he helped to lead a surprise boat attack on the Long Island port of Sag Harbor, which was guarded by Hessian troops. The other vessels were driven ashore, leading to the capture of their crews, but Thomas bravely succeeded in recapturing the lost vessels and rescuing all but one of the prisoners. Thomas was also appointed to a committee that advised General Washington on management of military supplies on Long Island and Staten Island. As a trophy, he kept a weight at his front gate made from grape shot fired by the British during their attack on Stonington. After the American Revolution, he served as a member of the New York Assembly in 1791.

In 1989, several of Parker's descendants filed a widely publicized claim in the United States District Court for the Eastern District of New York, arguing that Robins Island, (see picture), then under the threat of development, was unlawfully seized from Parker and that it should be returned to his heirs, who remain numerous on Long Island. Considered one of the most valuable undeveloped islands in the United States, Robins Island occupies a unique place in eastern Long Island folklore, with those competing to own it over the years including Prince Moulay Abdullah of Morocco and John Jacob Astor, whose plans were cut short when he perished aboard the Titanic. Prior to 1850, Robins Island was mainly used for farming, cattle grazing and lumbering, as well as brick making, but in later years it was turned into a hunting preserve for wealthy sportsmen. But by the 1970's, rising property taxes made the shoots uneconomical and plans were made to build a few hundred houses there. To combat this, a non-profit corporation called Robins Island Preservation Fund, dedicated to the preservation of Robins Island in its natural state, was established as successor-in-interest to Parker. The group claimed that New York's Act of Attainder, under which the island was confiscated and sold, was invalid because New York had no sovereignty over Robins Island at the time, which was then under British control. They further argued that by the time New York did achieve sovereignty over Robins Island, the Treaty of Peace of 1783 between Great Britain and the United States had already been enacted, which prohibited future confiscations of British and Loyalist property. The group alternatively argued that the seizure was illegal because Parker had inherited the property as an estate tail, a form of ownership no longer allowed in the United States which was used by wealthy British settlers to keep ancestral lands in one family by having property without exception pass to the heirs. This would mean that even if Parker's property loss was lawful, his son Joseph would continue to maintain a future interest. Although he married twice, Joseph had no children and made no mention of Robins Island in his will, so presumably descendants of his siblings inherited any ownership interest. Because of the enormous lapse of time and the different legal standards of the day, the claim was found to be without merit, even though such a confiscation would be illegal today. However, the owner of Robins Island at the time of the trial, Southold Development Corporation, was battered by the protracted legal struggle and abandoned its plans to develop the island after collapsing into bankruptcy. Under new ownership, most of the island has since been reserved for open space conservation in perpetuity under an easement to The Nature Conservancy.