Avoids potentially inlining a lot of repeated destruction logic. This
also allows the class to play nicer with forward declarations.
While we're at it, we can also explicitly delete the move
assignment/constructor (previously they were implicitly deleted, given
the class contains a const reference).