Episcopal services began on Orr's Island in July of 1894. They were held in the dining room of a boarding house called Bellevue Cottage. Among the guests was the Rev. Elwin K. Smith of Lambertville, New Jersey, who conducted a series of five services during that summer. An offering made at the last service of that summer constituted the beginning of a fund for the building of a Chapel on the Island. The consent to forward this project was cordially given by the Right Rev. Henry A. Neeley, Bishop of Maine, who wrote, “I heartily and gratefully approve of the movement for the erection of a simple church on Orr's Island, believing that such a provision would be for the spiritual advantage, not only of summer visitors, but also of permanent residents”.
The next summer the Rev. Mr. Smith returned to the Island as the guest of Miss Harriet S. Baquet of Burlington, New Jersey, and Miss May S. Decker of Manchester Green, Connecticut, both of whom were vitally interested in the building of a Chapel. In 1897 two services were held in Moulton's Ice Cream Parlor, and in 1898 services were held in Seaside Hall. During the years between 1894 and 1900, the plan of building the Chapel was carried out by summer residents.
Land for the Chapel was purchased for $80 from a local resident who hauled, free of charge, all the lumber and materials needed and used in the building of the Chapel. A total of $1,000 had been raised by the summer residents-just shy of the amount needed to pay for the total cost of building the Chapel. In the summer of 1900, another summer resident, Emma DeCoursey, put on a musical on the grounds of her home, Kinsale (now the home of Harry and Betsy Allyn), on Orr's Island. The $100 proceeds provided the additional funds needed to finish constructing the Chapel.
On August 6, 1900, the Feast of the Transfiguration, all debts having been met, the Chapel was consecrated by the Right Rev. Robert Codman, Bishop of Maine and successor to Bishop Neely who had died in 1899.
In 1901 the services were in charge of the Rev. Samuel Upjohn of Germantown, Pennsylvania, and the Rev. John B. Falkner of Germantown, the latter continuing the work during the summers of 1902 to 1905. In 1907, the Rev. Charles Henry Arndt of Germantown conducted the services. The Rev. Arndt continued his ministry at the Chapel for many years, later marrying the daughter of the Rev. Falkner.
Dormer windows and extensions for side aisles were added during the Rev. Arndt's tenure. In the nearly one hundred years since its consecration, many of the furnishings have been given as memorials. Descendants of the Rev. Arndt, as well as the grandchildren and great-grandchildren of several of the founders, continue to worship at All Saints'. The Chapel is also the site of many weddings, baptisms, and memorial services.
In 1994, the Chapel Committee purchased a triangular parcel of land across from the Chapel. Five years later, under the design leadership of Bob Brownell, plans were developed for a new parking area and the creation of a Memorial Garden. The All Saints' Memorial Garden was dedicated by the Right Rev. Chilton B. Knudsen, Bishop of Maine, on August 6, 2000, the 100th anniversary of the consecration of the Chapel.