Preheat oven to 350 degrees.
Coat a 11/2-quart baking dish with olive oil
Combine eggs, 3 Tbsp honey, milk, and bread in a large bowl.
Beat with a mixer until smooth.
Add sweet potato, vanilla, cinnamon, and allspice; beat until smooth.
Pour into prepared baking dish. Scatter pecans and ginger over top.
Bake at 350 degrees for 25 minutes until pudding is set and slightly puffy
While pudding bakes, combine yogurt and remaining 1 Tbsp honey; stir until smooth.
Divide pudding among 4 bowls, and top evenly with yogurt mixture.