They were Imperialist from day one and took over a large chunk of the world enslaving people and supplementing their culture religion and language. The Caliph was a hereditary emperor and before the Mongrels attacked them they were actively trying to conquer all the Christian nations. They had a larger population and more advanced culture and they should have succeeded. One of the favorite whipping boys of the white guilt mindset, the Crusades, was a the direct result of Muslims attacking the Christian city of Constantinople and all the land fought over had previously belonged to Christians for hundreds of years before they were conquered by imperialist Muslims. Why so many people feel guilty over that is beyond.
Both cultures practiced empire and the Muslims lost in no small part because of the ant-intellectual nature repressive nature of their culture. They are not backasswards because they were colonized, they were colonized because being backasswards allowed them to be out paced in technology by a group they at one time were in conquering. The Muslims were right there next to Europe during the Renaissance and they got left behind because they shunned new ideas and turned inwards. The rest of the world was taken by surprise when the Europeans came pouring out with their canons and ships that could navigate the world but the Muslims have no such excuse they saw it happening and chose to stagnate.