eBPF Engine ​

The eBPF Engine is the core component responsible for loading, verifying, and managing eBPF bytecode programs within the ecapture system. It handles kernel compatibility detection, dynamic bytecode selection between CO-RE (Compile Once, Run Everywhere) and non-CO-RE modes, program lifecycle management, and probe attachment. This engine serves as the bridge between user-space capture modules and kernel-space instrumentation.

For information about specific capture modules that utilize this engine, see Capture Modules. For details on eBPF program development and structure, see eBPF Program Development.

Architecture Overview ​

The eBPF Engine operates as a multi-layer system that translates capture requirements into active kernel instrumentation:

Architecture Flow: Capture modules instantiate the eBPF Engine through ebpfmanager, which performs BTF detection to determine kernel compatibility. Based on BTF availability and container status, the engine selects either CO-RE or non-CO-RE bytecode from embedded assets. The cilium/ebpf loader loads the bytecode and attaches probes, with the kernel verifier ensuring safety before allowing execution.

Sources: Makefile:1-269, .github/workflows/go-c-cpp.yml:1-128, README.md:1-335

CO-RE vs Non-CO-RE Modes ​

The eBPF Engine supports two distinct compilation and runtime modes to maximize kernel compatibility:

Build-Time Compilation ​

The build system produces both bytecode variants simultaneously, embedding them into the final binary through go-bindata.

CO-RE Compilation (Makefile:118-135):

• Uses clang -target bpfel with BTF type information
• Produces position-independent bytecode with CO-RE relocations
• Single bytecode works across kernel versions (4.18+/5.5+ for arm64)
• Smaller build artifacts due to single version

Non-CO-RE Compilation (Makefile:146-183):

• Requires full kernel source headers at build time
• Uses clang → llc pipeline with -march=bpf
• Bytecode contains hardcoded kernel structure offsets
• Kernel-specific: must match build kernel version

Sources: Makefile:118-183, .github/workflows/go-c-cpp.yml:16-33, builder/init_env.sh:1-106

Runtime Detection and Selection ​

The engine performs environment detection at startup to select the appropriate bytecode:

The detection logic prefers CO-RE when available, falling back to non-CO-RE only when BTF is absent and not running in a container. This ensures maximum portability while maintaining kernel-specific optimization capability.

Sources: README.md:82-102, CHANGELOG.md:228-230, Makefile:38-63

BTF Detection ​

BTF (BPF Type Format) is a metadata format that describes the types used in BPF programs and the kernel. The eBPF Engine uses BTF to enable CO-RE functionality:

BTF Detection Process ​

The engine checks for BTF support through multiple indicators:

Detection Implementation:

1. Primary check: /sys/kernel/btf/vmlinux file existence and size
2. Fallback: Parse /boot/config-$(uname -r) for CONFIG_DEBUG_INFO_BTF=y
3. Runtime log indicates detected mode (see log output in examples)

BTF Benefits for CO-RE ​

BTF enables several critical capabilities:

The CO-RE relocations happen during eBPF program load, where the kernel's BTF information is used to resolve field offsets, type sizes, and existence checks.

Sources: README.md:89, README_CN.md:90, CHANGELOG.md:249-250

Bytecode Loading ​

The eBPF Engine uses the cilium/ebpf library to load bytecode into the kernel. The loading process involves bytecode selection, asset extraction, and kernel verification:

Bytecode Asset Management ​

All eBPF bytecode is embedded into the binary at build time using go-bindata:

The asset generation happens in the build process (Makefile:186-195):

• go-bindata scans user/bytecode/*.o files
• Generates assets/ebpf_probe.go with embedded byte arrays
• Each bytecode file is accessible by its original filename

Loading Sequence ​

The loading process includes:

1. Bytecode Selection: Based on module name, version, and detected mode
2. ELF Parsing: Extract program definitions, map definitions, and BTF info
3. Kernel Loading: Submit bytecode via bpf() syscall
4. Verification: Kernel verifier checks safety constraints
5. Relocation (CO-RE only): Resolve structure field offsets using BTF

Sources: Makefile:186-195, README.md:89-102

Probe Attachment Mechanisms ​

The eBPF Engine supports three primary probe attachment types, each serving different instrumentation needs:

Probe Types ​

Uprobe Attachment ​

Uprobes instrument user-space functions by placing breakpoints at function entry and return points:

Example: OpenSSL module attaches to SSL_read at offset determined by parsing /usr/lib/x86_64-linux-gnu/libssl.so.3 ELF symbols.

Kprobe Attachment ​

Kprobes instrument kernel functions for network connection tracking:

Purpose: Correlate user-space captured data with network connection metadata (source/dest IP, port).

TC (Traffic Control) Attachment ​

TC classifiers capture raw network packets at the data link layer:

TC Integration Flow:

1. Attach TC classifier to network interface
2. Classifier executes on every packet
3. Lookup PID/UID from connection tracking map
4. Filter packets based on BPF filter program
5. Write matching packets to ring buffer

Sources: README.md:183-184, CHANGELOG.md:618-637

Map Types and Usage ​

eBPF maps provide shared memory between kernel eBPF programs and user-space applications. The engine uses multiple map types:

Map Type Overview ​

Perf Event Arrays ​

Used for high-throughput event streaming from kernel to user-space:

Configuration (README.md:99-100):

INF perfEventReader created mapSize(MB)=4

Characteristics:

• Per-CPU buffers to avoid contention
• Configurable size via --mapsize flag (default 5120 KB = ~1024 KB per CPU on 5-CPU system)
• Used for: SSL data events, connection events, master key events
• User-space polling via perf_event_open() syscall

Ring Buffers ​

Introduced in kernel 5.8+, provides shared memory ring for efficient event delivery:

Characteristics:

• Single shared buffer across all CPUs
• More memory-efficient than per-CPU perf arrays
• Used for: Network packet data (TC classifier output)
• Automatic backpressure handling
• User-space polling via epoll() on ring buffer FD

Hash Maps ​

Store connection metadata for correlation:

Use Cases:

• Connection Tracking: Map socket FD to network tuple (IP:port)
• PID/UID Lookup: Map network tuple to process information
• TLS Session Data: Map client random to master secret
• Protocol State: Track HTTP/2 stream state

Example Map Definition (conceptual):

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 2048);
    __type(key, __u64);      // socket FD or tuple
    __type(value, struct conn_info_t);
} conn_map SEC(".maps");

Sources: README.md:99-101, CHANGELOG.md:674

Error Handling and Diagnostics ​

The eBPF Engine implements comprehensive error handling and diagnostic logging:

Common Error Scenarios ​

Diagnostic Logging ​

The engine outputs structured logs at various levels (from examples in README):

2024-09-15T11:51:31Z INF Kernel Info=5.15.152 Pid=233698
2024-09-15T11:51:31Z INF BTF bytecode mode: CORE. btfMode=0
2024-09-15T11:51:31Z WRN OpenSSL/BoringSSL version not found from shared library file, used default version OpenSSL Version=linux_default_3_0
2024-09-15T11:51:31Z INF BPF bytecode file is matched. bpfFileName=user/bytecode/openssl_3_0_0_kern_core.o
2024-09-15T11:51:32Z INF perfEventReader created mapSize(MB)=4

Log Interpretation:

• BTF bytecode mode: CORE - CO-RE mode selected
• BPF bytecode file is matched - Bytecode loaded successfully
• perfEventReader created - Maps initialized

Kernel Compatibility Checks ​

The engine validates kernel requirements at startup:

The version requirements reflect eBPF feature availability:

• 4.18+ (x86_64): BTF support, bounded loops
• 5.5+ (aarch64): ARM64 eBPF JIT improvements
• 5.8+: Ring buffer support (auto-detected, falls back to perf arrays)

Sources: README.md:14-16, README_CN.md:14-17, CHANGELOG.md:41

Integration with ebpfmanager ​

While the raw eBPF loading is handled by cilium/ebpf, the engine integrates with a higher-level manager for lifecycle control:

Manager Responsibilities ​

The ebpfmanager component (referenced in diagrams and logs) provides:

1. Program Lifecycle: Init → Load → Attach → Run → Detach → Close
2. Multi-Program Coordination: Manage multiple eBPF programs per module
3. Map Management: Create, populate, and clean up eBPF maps
4. Resource Cleanup: Ensure proper teardown on module close

Module Integration Pattern ​

The manager abstracts the low-level eBPF operations, allowing capture modules to focus on data processing rather than kernel interaction details.

Sources: README.md:91-102, CHANGELOG.md:462

Performance Considerations ​

The eBPF Engine is designed for high-performance event capture with minimal overhead:

Memory Configuration ​

Map Size Tuning (CHANGELOG.md:709):

• Default: 5120 KB total (distributed across per-CPU buffers)
• Configurable via --mapsize flag
• Calculation: mapsize * PAGE_SIZE * num_CPUs
• Impact: Larger maps reduce event loss but increase memory usage

Event Loss Prevention ​

Mitigation Strategies:

1. Increase --mapsize for high-throughput scenarios
2. Use multiple event workers for parallel processing
3. Enable filtering at eBPF level (e.g., PID filter, pcap filter)
4. Batch event processing in user-space

eBPF Program Optimization ​

Verifier-Friendly Patterns:

• Bounded loops (kernel 5.3+) or unrolled loops
• Stack size ≤ 512 bytes
• Helper function call limits
• Avoid dynamic jumps

The CO-RE mode additionally benefits from kernel-optimized relocations, avoiding manual offset calculations in the eBPF program itself.

Sources: CHANGELOG.md:709, CHANGELOG.md:674, Makefile:23-24