Butter the bottom and sides of 9-inch cake pan. Cut a circle of parchment to fit the bottom and press into the pan.
Whisk together the almond flour, polenta, flour, baking powder, salt and lemon zest in a bowl.
Put 1 cup of the sugar in a small saucepan with 1/3 cup water. Stir to dissolve, then place the pan over medium heat. Bring to a simmer and cook until the sugar begins to form large bubbles and turns a light amber color (be patient — this can take 15 minutes or so). Remove from the heat and drop 2 tablespoons of the butter into the syrup, swirling to melt. Immediately pour into the cake pan.
Arrange the plums in a single layer around the bottom of the pan (careful - the syrup is very hot).
Beat the remaining 2 sticks butter and 1 cup sugar on medium-high speed in a heavy duty mixer fitted with the paddle attachment until light and fluffy. Add the eggs one at a time until each one is emulsified, then add the almond extract. Stir in the flour mixture in 2 additions until blended.
Scrape the batter over the plums and even out the top. Bake for 1 hour and 20 minutes or until the top is golden, crackled and firm to the touch. Set the cake on a rack and cool completely.
When the cake is cool, run a dull knife around the edge of the pan to loosen the cake. Put a large plate over the top and invert the pan. Peel off the parchment paper.