But Panem is not a world that stands up on close examination, so giving it more depth ruins the verisimilitude. The specialized districts work fine as representative of different parts of society and illustrating across the board oppression. As an actual setting, it raises a lot of skepticism, making the specialized nationalities of David Eddings' Belgariad seem grounded in historical fact by comparison. There is really only one district where they mine and one where they farm, and one where they fish? Ooookay.
If you think about the way that major corporations/companies have been going to third world countries since the age of imperialism and forcing them to produce cash crops, the specialized district thing checks out. It's a way to force the districts to be dependent on the Capitol. They each produce a piece of the economy, but can't support themselves without imports from other districts. As a means of suppression it seems pretty effective, and it has been show to work in basically any place that has been "colonized" by the first world.