Fully Homomorphic Encryption Inference of Neural Networks Using CKKS-TFHE Scheme Switching and Accelerated Linear Layers

Abstrak

Fully homomorphic encryption (FHE) allows neural network inference to be performed directly on encrypted data and models, preserving end-to-end privacy. One of the most widely used FHE schemes, CKKS, supports approximate arithmetic over real numbers and is well-suited for computing polynomial operations. However, CKKS does not natively support conditional branching, making it unsuitable for non-linear functions such as ReLU, which behaves differently depending on whether the input is negative or nonnegative. To address this, ReLU is typically approximated using polynomials. Unfortunately, this approach introduces two key drawbacks: (1) the approximation is only valid within a certain input range, and (2) identifying this range typically requires prior exposure of cleartext data to perform profiling, thus partially compromising privacy. This requirement is problematic in practice, as small profiling errors can lead to substantial accuracy degradation. On the other hand, the TFHE scheme supports binary gate-level computation, enabling precise implementation of conditional operations such as ReLU. Yet TFHE lacks the SIMD capabilities of CKKS, leading to significant computational latency. This paper explores the use of scheme switching for encrypted neural network inference: leveraging CKKS for polynomial-compatible layers and switching to TFHE for layers that require conditional logic, such as ReLU. We demonstrate that this hybrid approach substantially improves the numerical fidelity of encrypted neural network inference: while polynomial approximations suffer from numerical divergence, scheme switching more closely matches non-encrypted inference.

Identifier

• DOI: https://doi.org/10.1145/3779208.3807493
• OpenAlex: https://openalex.org/W7163586139