I have a lab with Windows Server 2022 domain controller and a Windows 11 domain-joined workstation, both enterprise editions. There is nothing else in this lab, no ADCS, no nothing. I have out-of-the-box settings for the domain controller, my only modification is allowing incoming RDP connections on both machines.
When I connect from one to the other in RDP using a domain admin account, I am immediately connected, no questions asked. However, when I check a network capture between the two machines, I can see that TLSv1.2 is involved for the RDP connections and that both machines use self-signed certificates.
From what I remember, if you connect using RDP from a non-domain-joined machine, a linux with remmina for instance or a WORKGROUP windows, you will have a warning on the client side regarding the certificate of the machine you connect to because it's self-signed.
Why am I not prompted to trust the certificate?
Is there a previous secret shared between the workstation and the server during the authentication phase that they rely on during the TLS part that make the certificates useless?
Are they blindly trusting each other because the authentication phase succeeded?
I'm having a hard time finding out so any explanation or resources like blogs etc... would be welcome.
Is there some previous secret shared between the workstation and the server during the authentication phase that they rely on during the TLS part that make the certificates useless ?
Yes, the session key within the Kerberos ticket.
In an AD environment (and with NLA/CredSSP enabled), RDP uses Kerberos – which provides user authentication, server authentication, and optionally transport encryption independent from TLS.
(Overall flow)
To start with, every Kerberos principal (every user and every service) shares a secret with the KDC. You and the RDP server have two long-term shared secrets (user↔KDC and server↔KDC) which are used to distribute a third temporary shared secret (user↔server).2
Your Kerberos secret is based on your password (fed through PBKDF2 to produce an AES key); the AD-joined server likewise has a "computer account password" from which its Kerberos key is derived. Outside of AD, it can also be a random (non-password) key held in a .keytab file.
When you obtain a Kerberos ticket for a service, the KDC includes a randomly-generated session key together with the ticket – one copy is separate and visible to you (encrypted using your key), while a second copy is stored inside the ticket (encrypted using the service's long-term key).
So when you present that ticket to the RDP server as part of Kerberos authentication, you're also sending it an encrypted copy of this session key, which only the correct service is able to decrypt.
(This handshake also requires you to send a one-time "authenticator" produced from the session key, meaning that Kerberos doesn't rely on TLS to protect the ticket from being stolen at this point – indeed it was originally designed to be done in the clear.)
Now that both sides have decrypted their copies of the session key, it
can be used to establish a secure channel – either encrypted ("sealed") or only integrity-protected ("signed"). The SSPI interface on Windows and the GSSAPI on Linux provide functions to GSS_Wrap()/GSS_Unwrap() arbitrary messages using the established Kerberos context, while old pre-GSSAPI programs use krb_mk_priv()/krb_rd_priv().
Specifically in RDP, the client and server use these Kerberos-protected messages to confirm that the TLS certificate seen by the client is the correct one ([MS-CSSP]).
(More general aside)
In other words, Kerberos was designed so that it alone can serve as a full replacement for TLS (mainly because it predates SSL/TLS by nearly a decade), and Windows indeed uses it that way for securing LDAP, MS-RPC, and WinRM/PS-Remoting traffic. All initial "domain join" traffic is protected by Kerberos sealing, for example, which is why it doesn't require setting up LDAPS.
SMBv3 encryption also relies on the same feature to negotiate its own session keys, although SMB and RDP don't rely on Kerberos for bulk encryption since it seems to use a cipher mode that's much harder to optimize for performance. For comparison, NFSv4 does use Kerberos encryption and it is much slower than e.g. TLS or IPsec.
Many other protocols don't use Kerberos sealing but still rely on the same session key purely to authenticate the server to the client (i.e. Kerberos provides mutual authentication) without needing a separate server certificate – the server's ability to decrypt the ticket and reply with an encrypted message authenticates it by proving that it knows its long-term shared key (keytab or machine password or whatever). Usually programs only need to pass the "require mutual auth" flag to SSPI or GSSAPI to achieve this.
(Back to RDP)
Finally, the way it integrates into RDP NLA (as part of the "CredSSP" mechanism) is documented in [MS-CSSP]; in short, it seems that after the Kerberos ticket auth is done, is used to exchange an encrypted/"sealed" hash of the server's TLS public key that was seen, and the client expects to receive a sealed confirmation from the server that it's the right one. Again, the server's ability to decrypt your message and encrypt its reply (with Kerberos's built-in anti-replay protection) is what authenticates the server to you.
(Something similar also happens when you use LDAP with TLS, at least in Windows/AD environments, where it is called "channel binding" and doesn't fully replace the TLS cert verification, and is documented in RFC 4121/6542 although the integration into LDAP is a non-standard Microsoft variant – standard RFC 4752 can't do channel bindings.)
(From non-AD-joined clients)
From what I remember, if you connect using RDP from a non-domain-joined machine, a linux with remmina for instance or a WORKGROUP windows, you will have a warning on the client side regarding the certificate of the machine you connect to because it's self-signed.
You can have this work even from non-domain-joined machines, so long as they're able to find your domain through DNS (i.e. you didn't use some nonsense domain name). When prompted for RDP or SMB credentials you must specify your username either in the AD UPN format or in the Kerberos principal format (i.e. either user@domain or user@REALM) – not in the legacy NT DOMAIN\user format – and then the standalone client will just do Kerberos without needing any configuration.
(That is to say, the whole thing doesn't rely on the client having machine credentials – at least not until you configure your DCs to make FAST armoring mandatory. Otherwise the user's password is enough to get an initial ticket.)
Remmina on Linux (or rather its FreeRDP backend) also supports NLA with Kerberos starting with FreeRDP 3.1 or so; AFAIK you must specify the username in the raw Kerberos principal format user@REALM, e.g. [email protected], although it might accept the lowercase UPNs as well. I don't think it fully implements the TLS+Kerberos combination though – it still asks to verify the certificate before it gets to speaking Kerberos.
* In theory, NTLM can also provide a secure channel (as far as the hardcoded RC4 can be called "secure"), but it cannot authenticate the server (only the client), so in a workgroup environment it's not really good enough. Future Windows versions are apparently going to switch to Kerberos directly between standalone machines.
1 Each Kerberos enc-type (see table) separately defines the KDF to use. For example, in addition to modern AES keys with PBKDF2, most AD DCs still store "RC4-HMAC" keys which use MD4 for key derivation (which is completely identical to what NTLM uses; meaning that it used to be possible to upgrade a legacy NT domain to a Kerberos-based AD domain without requiring a mass password reset).
2 This is a simplified version that skips the TGT acquisition, but is still plausible (there are in fact situations where a TGT is not used), and either way it doesn't change the point being made. The full flow involving a TGT merely adds a second near-identical layer of the same thing.
