Hello, I've checked at https://www.xwikiplayground.org/xwiki/bin/view/Sandbox/#Attachments and I can see that all links are HTTPS.
I believe you have a configuration issue somewhere. I'm tempted to close this issue as cannot reproduce.
The code is at https://github.com/xwiki/xwiki-platform/blob/a46b2b47e3e1eae1849b417a62110790c43f6598/xwiki-platform-core/xwiki-platform-web/xwiki-platform-web-templates/src/main/resources/templates/attachment_macros.vm#L146-L152 and I see no reason that $xwiki.getURL($attachment.getReference()) would not return a URL with HTTPS (this api is used in lots of other places in the XWiki UI).
Closing for now, we'll reopen if more info is provided.