The TLS secure digital communications protocol includes a sub-protocol called TLS Handshake. This enables the server and client to authenticate each other in turn then select an encryption algorithm and cryptographic key before the application sends the data. Here’s how it works.
TLS authentication: a closer look at the Handshake protocol
Like TCP/IP, the TLS protocol is a client-server protocol suite made up of four sub-protocols. The Handshake sub-protocol has the following aims:
- Negotiate the encryption algorithms (symmetric and asymmetric)
- Negotiate symmetric key lengths
- Negotiate signature algorithms (HMAC)
- Server authentication by the client
- And, as an option, client authentication by the server.
The client and server select the most powerful shared algorithms. If no algorithm is found, the communication is terminated (Alert protocol). Keys are negotiated in the same way
Authentication is performed with X.509 certificates.
Details of Handshake protocol exchanges for TLS v1.2
Below is a diagram showing all the preliminary exchanges between the client and the server to ensure the security, confidentiality and integrity of the exchanged messages.
1 – Establishing the connection
The client initiates communication with the server by sending a SYN (synchronize) message. The server accepts the communication by sending a SYN-ACK (synchronize-acknowledgment) message. Finally, the client informs the server that the message has been received by sending an ACK (acknowledgment).
2 – Negotiation
The client sends the unencrypted HELLO_CLIENT message to the server. The message includes:
- the TLS version (the most recent one the client supports)
- A random message for signing the data
- A session identifier and the list of cipher suites supported by the client in descending order of preference.
The server replies to the client with a HELLO_SERVER message containing the TLS version supported by the client, the random message sent by the client, the session identifier and the strongest cipher suite supported by both client and server.
If the TLS version or cipher suite proposed by the client is not available on the server, the communication is immediately interrupted.
3 – Server authentication
Once the algorithms have been negotiated, the server authenticates itself to the client by sending its X.509 certificate (Certificate message).
As an option, the server can also send its public key (which is different from the server certificate public key) to the client to encrypt the session key (Server Key Exchange). Depending on the server configuration, it sends the Client Certificate Request message to ask the client for its certificate.
The client checks the certificate format, expiration date, status (revoked?) and whether the certificate is trusted (the server’s certificate authority chain must be contained in the client trust store). If one of these verifications fails, the transaction is abandoned.
Finally, the server sends the Server Hello Done message to indicate that it has finished.
4 – Client authentication and session key generation
The client sends its own certificate to the server (Client Certificate) if requested. The server performs the same checks as those performed by the client (see point 3) on the server certificate.
The client produces a pre-master key which it encrypts with the server certificate’s public key. If the server has sent Server Key Exchange, this encrypted key is encrypted again with the server’s public key. The result is sent to the server via the Client Key Exchange message. This secret then enables the client and server to generate session keys which will not be exchanged.
If the client has sent its certificate, it also sends the Certificate Verify message containing the fingerprint of all the previous messages signed with the client’s private key. This message proves that the client has the private key associated with the certificate.
5 – End of TLS handshake
The client sends Change Cipher Spec and Finished messages which are encrypted and signed with the previously mentioned keys to indicate that the TLS tunnel is now established. The server does the same thing and the Handshake ends.
The client and server now communicate according to the Record protocol, guaranteeing the confidentiality and integrity of all the exchanged messages.
