Container Networking Is Harder Than Container Running
Running a containerized application is straightforward. Connecting it to other services, enforcing network policies, and securing east-west traffic in a Kubernetes cluster — that is where most teams discover the complexity they signed up for. Container networking sits at the intersection of Linux kernel networking, distributed systems design, and cloud provider specifics, and the documentation rarely explains how these layers interact. This deep dive covers CNI plugins, service mesh architecture, and Kubernetes network policies with the level of technical detail that actually helps when things break.
How Container Networking Actually Works
Before evaluating CNI plugins, you need a clear mental model of what problem they solve. When a container starts, it gets a network namespace — a kernel-level isolation boundary with its own network interfaces, routing tables, and iptables rules. By default, this namespace has no external connectivity. The Container Network Interface (CNI) specification defines how a plugin takes that isolated namespace and connects it to the rest of the world.
Every time a Pod is created in Kubernetes, the kubelet calls the configured CNI plugin(s) in sequence. The plugin is responsible for:
- Assigning an IP address to the Pod from a pre-allocated CIDR
- Creating a virtual ethernet pair (veth pair) connecting the Pod’s namespace to the host namespace
- Ensuring that traffic destined for any other Pod’s IP can reach it, regardless of which node it is running on
That third requirement — Pod-to-Pod reachability across nodes — is where CNI plugins diverge significantly in their approach.
CNI Plugins Compared
Flannel: Simple Overlay, Limited Features
Flannel is the simplest production-grade CNI plugin. It creates an overlay network using VXLAN (or host-gw mode for better performance on trusted networks). Each node gets a subnet, and Flannel encapsulates cross-node traffic in UDP packets routed over the host network.
# Flannel ConfigMap — VXLAN backend (typical production config)
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-flannel-cfg
namespace: kube-flannel
data:
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan",
"VNI": 1,
"Port": 8472
}
}Flannel’s simplicity is also its limitation: it provides no network policy enforcement. To use Kubernetes NetworkPolicy objects with Flannel, you need a separate policy enforcement daemon like Calico in policy-only mode. For clusters where network policy is not required, Flannel remains a solid, low-overhead choice.
Calico: BGP Routing and Rich Policy
Calico uses BGP (Border Gateway Protocol) to distribute routing information between nodes instead of an overlay network. Each Pod’s IP is a real routable address, and nodes exchange routing tables via BGP. This avoids overlay encapsulation overhead, which matters at high packet rates.
# Calico IPPool — defines the address range and routing mode
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
cidr: 10.244.0.0/16
ipipMode: Never # No overlay encapsulation
vxlanMode: Never # Using BGP routing instead
natOutgoing: true
nodeSelector: all()Calico’s network policy implementation is also significantly richer than the standard Kubernetes NetworkPolicy API. Calico GlobalNetworkPolicy objects can enforce policy across namespaces, and Calico’s HostEndpoint resource extends policy enforcement to host network interfaces — a capability relevant for bare-metal or VM-based clusters where you want to secure traffic that never touches a Pod.
Cilium: eBPF-Native Networking
Cilium replaces iptables-based packet processing with eBPF programs loaded directly into the Linux kernel. This architectural choice has significant performance implications: eBPF programs run in the kernel without the context-switching overhead of iptables chains, and they can be updated atomically without reloading the entire ruleset.
# Cilium NetworkPolicy with L7 HTTP rules
# This is not possible with standard Kubernetes NetworkPolicy
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: api-service-l7-policy
namespace: production
spec:
endpointSelector:
matchLabels:
app: api-service
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "/api/v1/products"
- method: "POST"
path: "/api/v1/orders"This policy only allows the frontend service to make GET requests to /api/v1/products and POST requests to /api/v1/orders. Any other HTTP method or path is rejected at the kernel level — not after reaching the application process. This is L7-aware network policy that iptables-based CNIs cannot express.
Cilium also includes Hubble, an observability layer that provides real-time flow visibility across the cluster without sampling:
# Observe traffic flows with Hubble CLI
$ hubble observe --namespace production --follow
# Filter for dropped traffic (policy violations)
$ hubble observe --namespace production \
--verdict DROPPED \
--follow
# Example output:
Mar 29 08:23:11.492: production/frontend-6d8f9b-xyz:52341
-> production/payments-7c4d2a-abc:8080
http-request DROPPED (policy-deny)
GET /api/v1/admin/refund HTTP/1.1This visibility is invaluable when debugging why a service cannot reach another — you can see in real time whether traffic is being dropped by a policy and which policy is responsible.
Service Mesh: What It Adds and What It Costs
A CNI plugin handles IP connectivity and basic network policy. A service mesh adds a layer above that, focusing on traffic management, observability, and security for service-to-service communication. The two concerns are complementary — you typically run both.
The Sidecar Model vs. Ambient Mesh
Traditional service meshes like Istio (pre-1.21) and Linkerd inject a sidecar proxy container into every Pod. This proxy intercepts all inbound and outbound traffic for the Pod and applies mesh functionality: mTLS, traffic shaping, retry logic, and telemetry collection.
The sidecar model has a well-documented cost: every Pod carries an additional container (Envoy for Istio, linkerd-proxy for Linkerd) that consumes CPU and memory. In a cluster with 500 Pods, that is 500 additional proxy processes. Istio’s ambient mode, released as stable in Istio 1.22 (2024), addresses this by moving the mesh functions to per-node proxies (ztunnels) and a shared waypoint proxy for L7 processing.
# Enable Istio ambient mode for a namespace
kubectl label namespace production istio.io/dataplane-mode=ambient
# Verify ambient enrollment
kubectl get pods -n production -o wide
# Pods do NOT show a sidecar container injected
# But traffic is still intercepted by the node-level ztunnel
# Deploy a waypoint proxy for L7 policy in this namespace
istioctl waypoint apply --namespace production
# Now L7 AuthorizationPolicy objects work for ambient-mode pods
kubectl apply -f - <Traffic Management: The Practical Patterns
The service mesh features teams actually use in production tend to be a small subset of what the documentation covers. The highest-value patterns are:
Canary releases with traffic shifting:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-service
namespace: production
spec:
hosts:
- api-service
http:
- route:
- destination:
host: api-service
subset: stable
weight: 90
- destination:
host: api-service
subset: canary
weight: 10
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: api-service
namespace: production
spec:
host: api-service
subsets:
- name: stable
labels:
version: v2.1.0
- name: canary
labels:
version: v2.2.0-rc1Retry and timeout policies:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payment-service
namespace: production
spec:
hosts:
- payment-service
http:
- timeout: 5s
retries:
attempts: 3
perTryTimeout: 2s
retryOn: 5xx,reset,connect-failure
route:
- destination:
host: payment-serviceKubernetes NetworkPolicy: What It Can and Cannot Do
The standard Kubernetes NetworkPolicy API is enforced by the CNI plugin (not Kubernetes itself), which means it only works if your CNI supports it. Flannel does not. Calico, Cilium, and Weave all do.
# A reasonable default-deny policy for a production namespace
# Apply this BEFORE deploying services, then add allow rules as needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {} # Selects all pods in namespace
policyTypes:
- Ingress
- Egress
---
# Allow DNS resolution (required for almost everything)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCPThe standard NetworkPolicy API has significant limitations: it cannot express L7 rules (no HTTP method/path filtering), it has no ordering semantics (all matching policies are additive), and it cannot reference external IP ranges by domain name. For these use cases, you need Cilium's CiliumNetworkPolicy or Calico's GlobalNetworkPolicy.
Choosing the Right Stack
The decision matrix for container networking comes down to three questions: What is your performance budget? Do you need L7 network policy? Do you need a service mesh?
- Simple cluster, no network policy needed: Flannel VXLAN. Low operational overhead, wide documentation.
- Network policy needed, BGP-capable infrastructure: Calico with BGP mode. Native routing performance, rich policy expression.
- L7 network policy, high observability requirements: Cilium. eBPF performance, Hubble observability, L7 policy support.
- Service mesh for mTLS and traffic management, lower overhead: Istio ambient mode or Linkerd. Ambient mode eliminates per-pod sidecar cost.
Do not run a service mesh unless you have a clear use case that justifies the operational complexity. For most teams, Cilium's network policy capabilities handle the security requirements, and a service mesh becomes worthwhile only when you need sophisticated traffic management for deployments or when mTLS between every service pair is a hard compliance requirement.
Key Takeaways
- CNI plugins handle IP assignment and cross-node routing. The overlay vs. BGP routing choice affects performance and network topology requirements.
- Cilium's eBPF approach enables L7-aware network policy that iptables-based CNIs cannot express, plus real-time flow observability via Hubble.
- Service meshes add mTLS, traffic management, and telemetry above the CNI layer. Istio's ambient mode reduces the operational cost of the sidecar model significantly.
- Always apply a default-deny NetworkPolicy to production namespaces before deploying services, then add explicit allow policies.
- Standard Kubernetes NetworkPolicy cannot express L7 rules — for HTTP-aware policies, use Cilium or a service mesh AuthorizationPolicy.