If you want to make this scheme more safe you can add haproxy layer between keepalived and kube-apiserver.

Just install haproxy package into your system, and add the next configuration into /etc/haproxy/haproxy.cfg file