This was due to a backup that was created for a file that existed before (e.g. backup for Page0.xaml is created, Page0.xaml is deleted manually in file explorer, backup is still there unfortunately, now when creating a new Page file with name Page0.xaml it will try to load the backup for old file).
Fixed by removing backup for old file when creating a new file.