Line the bottom of a 9x13 inch baking pan with crackers.
In a small saucepan, melt 2 sticks of butter. Stir in brown sugar, granulated sugar, graham cracker crumbs, and milk. Mix together until all the ingredients are thoroughly incorporated.
Spread half of the filling on top of the base layer of crackers, then top with another layer of crackers. Spread the remaining half of the filling on top, then add a final layer of crackers.
In a separate small saucepan, melt together 1/2 cup (1 stick) butter, 1/2 cup chocolate chips, and 2/3 cup peanut butter. Pour over the final layer of crackers and spread evenly over the top.
Refrigerate for at least 4 hours, until firm.