You say you're a math person. That makes sense if I look at what you write.

Let me try to put it this way.
Lets make NS the function f(. Imput are traits, output is probability of survival (or probability of survival increase/decrease).

Now I go into the field and collect data, given the fact that I can.
f(1) -> 17
f(2) -> 3
f(3) -> 3
f(4) -> 99
f(5) -> 0
...

In case you start calculating: I'm giving random numbers.
The point I'm trying to make is this: if you "put in" random traits, the output (probability of survival) will be random.

Now since the output is random, there is no way you can predict what any given new trait will give you. Only afterwards you can say that f( -> something, but never before.

When I say that trait 3 will give a survival benefit of 3 I am right. When I say that f(3) = 3 I'm also right. But all I did was repeating myself.
Now I know that even random functions have their use in math, but the point is that once you have established that certain traits have certain survival benefits, f( does not give the explanation for why trait so-and-so gives so much survival benefit, it only repeats it.

In the same way NS does not explain why the gene pool changes, it only 'puts it in a function'.

Now, saying that there are many factors involved, including a non-stable fitness landscape does not essentially change this this problem. All that does would be:
f(g(h(i(j(k(l(m(n())))))))
And so basically clouding the fact that all these functions combined do give random results.
Hence, it has to be true. It is true. By definition, but therefor it has no explanatory potential.