Halliday & Matthiessen (2014: 421):
Note that in the case of polarity, the negative adverb not has the adverbial group as its domain, not the clause in which the adverbial group serves; this can be checked by adding a Moodtag: with the clause not infrequently the supply of communication services in fact precedes the growth of incomes the unmarked tag would be negative (doesn’t it), following the general principle of reversed polarity and thus showing that the clause itself is positive.
The different types may, of course, combine, as with polarity + intensification and intensification + comparison; for example:Not altogether surprisingly, my wife had fainted.