Session encrypts your messages using the Session Protocol, an end-to-end encryption protocol built on libsodium, a highly-audited and widely trusted cryptographic library. Session moved away from the Signal protocol in order to build the standard for decentralised encrypted messaging apps. Learn more here.
A technical description of the Session Protocol can be found in this blog.
You can also refer to Session's whitepaper for further details.