Gage tree not setting fruit

[pedlar]

I have a six-year old Oullins Golden Gage tree pruned as an upright cordon which has never fruited. Last year it flowered properly for the first time but only set 3 fruit, the rest remained as small green fruitlets which eventually dropped. This year it looks to be doing the same thing. My understanding is the Oullins Golden Gage is self fertile. My garden is on a slight rise overlooking an open field and I live in one of the windiest inland areas in England. Could this be the problem? Do I wait and hope or I was thinking of replacing the tree with an apple cordon. All I've read about Oullins Golden Gage is what good variety it is, but mine isn't! Please advise.