Halliday & Matthiessen (2014: 166):
What is the analysis of let’s? Given its place in the paradigm, it is best interpreted as a wayward form of the Subject ‘you and I’ (note that the marked person is realised by Ictus
on let’s, parallel to that on you). The only anomalous form then is the response Yes, let’s!, No, let’s not!, which on this analysis has Subject and no Finite; but in each case there is an alternative form with the Finite element in it, Yes, do let’s!, No, don’t let’s!, which also suggests that let’s is felt to be a Subject. (The order do let’s corresponds to the earlier second person ordering as in Do you look!.)