core: don't create a repository needless for loading its refs
This is a regression from 26cde2c0e6bb, in !201 (merged), which made us use the function Dulwich provides for loading remote references, which caused us to always create a Git repository.