Well, a lot probably. Most islands aren't idyllic paradises. Many are too small, too isolated or lack a water source. The climate may be arid, too hot or too cold. It could be too windswept to grow anything. Perhaps it's no more than a rocky outcrop, a patch of low lying infertile sand or overshadowed and threatened by an active volcano. And how do you survive if you're the only person on an island or even if you're part of a tiny community? What happens if you are injured or get sick? Do you want some of the trappings of modern civilisation like electricity? Would you prefer to buy clothing or, if you have the time and inclination to make your own clothing, would you prefer to have someone else make the cloth? After all it takes many hours to spin and weave enough to make even one garment. Living a subsistence life is all very well but how do you get extras without something to trade or the money to pay for it? How do you get work to earn money for things you might consider essential. On a small, isolated island with a tiny community there's not going to be many options. What happens if you fall out with your neighbours? More, what future is there for your children? How do they get an education? So many questions.
Still there are islands that do fit our fantasy. These are islands large enough to support a fair-sized population, where there are job opportunities from a variety of sources ranging from agriculture and fishing to tourism, but are small enough to feel special. I'm thinking of islands like the medium sized ones in the Caribbean or the Mediterranean. Even in those places difficulties remain with a drift of the younger folk to the mainland where there are more opportunities in education and employment.
I'm still drawn to an island life, though, and given the chance - like winning a major Lotto prize which would obviate most of the problems listed above (money after all is a great facilitator) - I would happily move to one, just as long as it wasn't too far from the mainland.