Imagine wearing a pair of glasses that change all of the colours, for instance, into their compliments like a photographic negative. This type of experiment is well known: your brain will re-interpret all the colours so that you see green as red and vice-versa.
Things become more interesting, however, if there is a constant shifting of the colours. Now your brain must come up with a "meta-concept" to label the colours by identifying objects that more permanently keep their colours. For example, there are two objects in the room (at the time of first writing) that would allow me to identify the colour red with almost 100% certainty: a Coke bottle and a bouguet of fake roses, unless the roses are yellow and Coca-Cola has decided to rebrand their bottles in the same colour, both highly unlikely eventualities.
It gives one pause for thought: how much do we rely on reference points to calibrate our colour-sense, which is deceptively constant, despite widely varying lighting conditions? With the magic glasses on, the mind is called upon to create this constant colour-sense, even though the colours of objects are constantly shifting at the same time as the environment. It would be an interesting way to mess with someone's mind: yellow Coke bottle labels, orange leaves, black roses, etc.
The magic glasses would be easy to build: a pair of web-cams, virtual reality glasses and a palm-top computer would be all you need.