The contracts have limited duration. And there are always people holding contracts that they have to get rid of before they come due.
If a single entity, with sufficiently deep pockets, is prepared to take delivery on the oil, then they could affect the end price. But consider that the world oil market is over $250 billion dollars per month. Specifically, what organization are you claiming is doing this? It doesn't work unless you have one entity, or a very small number of entities (3 would be pushing it) working in tandem. (Like the hunt brothers and the silver market in the 70s. But there, the total dollar value of the silver market was only a few billion dollars).
It would be very easy to pinpoint this if it were happening.
And in any case, this entity would not be a "speculator". It would be an entity trying, illegally, to gain control of the market. And it would have to be willing and able to shell out a substantial fraction of $3 trillion a year to do it.
Again, who do you think is doing this? Please be specific.
-Steve