German explorer Pierre Yax (b.1763) in Grosse Pointe, New France (now Michigan) was the first recorded non-Native American in the New Baltimore area.  Later, French settlers came to this area and took residence along the waterfront and later developed farms inland. The earliest evidence of a settled community dates to 1845 when Alfred Ashley platted 60 acres, which became the Village of Ashley. This land was on both sides of Washington Street.
Mr. Ashley opened businesses in shipping, real estate and lumber. The original Village of Ashley occupied what is now the center of downtown New Baltimore, extending northwest along Clay, Base and Maria Streets from Anchor Bay. This settlement bore Ashley’s name until New Baltimore was incorporated in 1867.  

In 1868, Peter Schaars constructed the Baker Building out of which he sold dry goods,
clothing, medicine and groceries. In 1870, he sold the building to William Baker Sr. and from that day forward, it was known as the Baker Building. Mr. Baker and his heirs used the
building for over 100 years selling everything from petticoats to potatoes. In 1920, the
building was partitioned and has since been home to many other businesses; American (Hensel’s) Bakery, Ashley’s Sweet Shop, and most recently Stahl’s Famous Original Bakery.