You’d expect to have the AWS credentials loaded up by your application via %userprofile%\.aws but no, it doesn’t.
I think I’ve spent about 2 hours searching for the solution. Thanks to some random person, I figured out that you have to place the files here instead:
C:\Windows\System32\config\systemprofile\.aws\credentials
Now I finally got my ASP .NET Core 6.0 backend API running on my windows server.