I go to at least one design meeting each day at work, and I know the kind of optimizations you're talking about. The company I work for produces the major software behind most of the world's biggest cell phone networks, too, so it's not trivial stuff being designed. There IS math, but there's not a lot of it, nor is it complicated. It's mostly the recognition and application of algorithms. Algorithms that were developed and proven by people with lots of math. The theory guys. The REAL "computer scientists." The use of their algorithms, though, doesn't require a lot of knowledge.

Note of course that I'm talking relatively advanced math. You definitely need to know some discrete math, statistics, a bit of linear algebra, some geometry and newtonian physics if you want to work in graphics, etc. Calculus, though, beyond maybe knowing how to get a first derivative for estimation purposes? Or differential equations? You could work a thousand different programming jobs at the highest level, and never see them. I don't argue that there may be some very specific tasks that require higher math concepts, but they're few and far between.