What layer is TLS?
TLS stands for "transport layer security". And the list of IP protocol numbers includes "TLSP" as "Transport Layer Security Protocol". These two things would leave me to believe that TLS is a transport layer protocol.
However, most people seem to talk about TLS over TCP. Wikipedia lists it as an "application layer" protocol. This is further complicated by the fact that TCP doesn't have anything like a protocol number: it just packages up raw bytes, so how do you parse out that you are getting a TLS packet, vs a packet that just starts with
The OSI model, that categorizes communication protocols into successive layers, is just that: a model. It is an attempt at pushing a physical reality into neatly defined labelled boxes. Nobody ever guaranteed that it works...
Historically, that model was built and published when the ISO was pushing for adoption of its own network protocols. They lost. The World, as a whole, preferred to use the much more simple TCP/IP. The "model" survived the death of its initial ecosystem, and many people have tried to apply it to TCP/IP. It is even commonly taught that way. However, the model does not match well TCP/IP. Some things don't fit in the layers, and SSL/TLS is one of them.
If you look at the protocol details:
- SSL/TLS uses an underlying transport medium that provides a bidirectional stream of bytes. That would put it somewhere above layer 4.
- SSL/TLS organizes data as records, that may contain, in particular, handshake messages. Handshake messages look like layer 5. This would put SSL/TLS at layer 6 or 7.
- However, what SSL/TLS conveys is "application data", which is, in fact, a bidirectional stream of bytes. Applications that use SSL/TLS really use it as a transport protocol. They then use their own data representation and messages and semantics within that "application data". Therefore, SSL/TLS cannot be, in the OSI model, beyond layer 4.
Thus, in the OSI model, SSL/TLS must be in layer 6 or 7, and, at the same time, in layer 4 or below. The conclusion is unescapable: the OSI model does not work with SSL/TLS. TLS is not in any layer.
(This does not prevent some people from arbitrarily pushing TLS in a layer. Since it has no practical impact -- this is just a model -- you can conceptually declare that TLS is layer 2, 5, or 17; it won't be proven false.)
Thanks. That really helps. Do you know the answer to my last question then? Since TCP doesn't have any protocol numbers, how do I tell that I'm looking at a TLS packet?
Usually, TLS happens because client and server agree that TLS ought to happen. For instance, when connecting to a `https://' URL, a Web browser just knows that it should begin some TLS, because that's what the 's' in 'https' means. _From the outside_, you can usually infer that TLS is occurring because you recognize the TLS record headers. Each record begins with a five-byte header: one byte for the record type (0x14 to 0x17), then protocol version (0x03 followed by 0x00, 0x01, 0x02 or 0x03), then record length (over two bytes, big-endian).
To sum up, you usually know that TLS is there because of the context (e.g. if it is on TCP port 443, then chances are that it is TLS), and you can _heuristically_ recognize TLS records with a low false-positive rate. E.g., just yesterday, I was looking at some RDP packets, and right in the middle of them I recognized an encapsulated TLS handshake. Notably, the handshake implies the sending of a certificate from the server, and the encoded certificate really stands out in hexadecimal dumps (well, for me at least).
One of the most useful concepts in the OSI model is separation of layers. TCP/IP follows this to some degree. So TCP doesn't know anything about what's encapsulated in the packets, thus no protocol number. So you have to imply the packet through what's generally called "deep packet inspection".
@AndrewSpott TCP does indeed not have a protocol number but it has port numbers which to some extent serve the same purpose. Usually you will have no more than one service listening on a specific port number, and that service knows which protocol to be speaking on that port. In less common cases one may decide to have a service understand different protocols on a single port and have that service deduce from the bytes sent by the client, which protocol is being used. In some cases it can be a security vulnerability if client and server speaks different protocols.
It turns out that in the upper layers OSI is too fine-grained (in the "real world", we use the Internet protocols, which compress layers 5-7 into a single layer) while in the lower layers it is too coarse-grained (again, in the "real world", we use IEEE-based protocols such as Ethernet or WiFi, which split OSI layers 1 and 2 into three layers: PHY, MAC, and LLC).
@JörgWMittag: The OSI layers model has additional limitations when communicating through links with different propagation times and reliability characteristics, and probably contributes to a lot of user frustration with WiFi. If a user clicks a button and gets no response, the local machine's WiFi handler will know if the requests associated with the click have yet reached the router, and will also know how long it is since it was last informed that the router has no data available, but there is no mechanism by which it can inform the application.