Engula

Engula
docs
docs

Engula NetTrace Synth

Engula provides a TCP traffic replay tool named NetTrace Synth that enables comprehensive performance testing and workload analysis of your database systems. This powerful utility captures real-world traffic patterns and replays them against your Engula instance to provide accurate benchmarking and performance evaluation.

Goal

The primary goal of this workload replay tool is to:

  • Performance Benchmarking: Accurately measure system performance under realistic traffic conditions by replaying actual production workloads
  • Capacity Planning: Help determine resource requirements and scalability limits through systematic workload testing
  • Regression Testing: Validate system changes and updates against consistent, repeatable traffic patterns
  • Stress Testing: Evaluate system behavior under various load conditions to identify bottlenecks and potential failure points
  • Compatibility Verification: Ensure that Engula maintains consistent behavior when processing identical workloads compared to reference implementations

This tool bridges the gap between synthetic benchmarks and real-world usage scenarios, providing valuable insights into how your Engula deployment performs with authentic application patterns and query distributions.

Not-Goal

The tool does not act as a data transfer tool. It is specifically designed for traffic pattern analysis and performance benchmarking rather than data migration or synchronization purposes.

Key Limitations:

  • Data Integrity: The tool does not guarantee data consistency or integrity during replay operations, as its primary focus is on workload simulation rather than data accuracy
  • State Preservation: No preservation of database state between capture and replay sessions, meaning the target system should be pre-populated with appropriate test data
  • Data Migration: Not intended for transferring data between different database instances or environments
  • Backup/Recovery: Does not provide backup or disaster recovery capabilities; it merely replays command sequences without ensuring semantic correctness of the underlying data
  • Synchronization: Cannot be used as a real-time data synchronization mechanism between source and target systems

Important

Root permissions are required to execute this tool, as it utilizes pcap (packet capture) functionality to intercept network traffic at the interface level.

Guide

On your Redis server host, execute the following commands to install the prerequisite components. The following example assumes Ubuntu 22.04 as the operating system.

Download engula server

Refer to Engula Downloads

Install OpenJDK 17.x

1sudo apt-get install openjdk-17-jdk
2export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
3export PATH=$PATH:$JAVA_HOME/bin
4java -version

Install Gradle 8.14.x

1wget https://services.gradle.org/distributions/gradle-8.14.2-bin.zip
2unzip gradle-8.14.2-bin.zip
3export GRADLE_HOME=$PWD/gradle-8.14.2
4export PATH=$PATH:$GRADLE_HOME/bin
5gradle -v

Build pcap_resp_replay

1git clone https://github.com/montplex/pcap_resp_replay.git
2cd pcap_resp_replay
3gradle jar
4cd build/libs
5java -jar pcap_resp_replay-1.0.0.jar --help

Command-line interface documentation:

Usage: java -jar pcap_resp_replay-1.0.0.jar [-d] [-b=<bufferSize>] [-B=<sendCmdBatchSize>] [-c=<maxPacketCount>] [-f=<filter>] [-h=<host>] [-H=<targetHost>] [-i=<itf>] [-m=<bigKeyTopNum>] [-p=<port>] [-P=<targetPort>] [-r=<readTimeout>] [-R=<readScale>] [-s=<runningSeconds>] [-W=<writeScale>] TCP monitor / filter and then replay / redirect to target redis server. -b, --buffer-size=<bufferSize> buffer size, default: 1048576 (1M) -B, --send-cmd-batch-size=<sendCmdBatchSize> send cmd pipeline size, default: 1, max 10, means no pipeline -c, --max-packet-count=<maxPacketCount> receive max packet count, default: -1, means not limit -d, --debug debug mode, if true, just log resp data, skip execute to target redis server -f, --filter=<filter> filter, default: tcp dst port 6379 -h, --host=<host> host, eg: localhost -H, --target-host=<targetHost> target host, eg: localhost -i, --interface=<itf> interface, eg: lo, default: lo -m, --big-key-top-num=<bigKeyTopNum> big key top num, default: 10, max 100 -p, --port=<port> port, eg: 6379 -P, --target-port=<targetPort> target port, eg: 6380 -r, --read-timeout=<readTimeout> read timeout seconds by capture from network interface, default: 10 -R, --read-scale=<readScale> read scale, default: 1, max 100 -s, --running-seconds=<runningSeconds> running seconds, default: 60, max 36000 -W, --write-scale=<writeScale> write scale, default: 1, max 100

Testing Your Workload with Engula

Launch Engula Server

engula-server --port 6380 --save "" --appendonly no --protected-mode no

Execute pcap_resp_replay for Initial Validation

1sudo java -Xmx1g -Xms1g -XX:+UseZGC -XX:+ZGenerational -jar pcap_resp_replay-1.0.0.jar -i lo -p 6379 -H 127.0.0.1 -P 6380 -s 30

Analyze Performance Output and Statistical Summary

1+---------+---------+-----------+
2| ps_recv | ps_drop | ps_ifdrop |
3+---------+---------+-----------+
4| 4490    | 42      | 0         |
5+---------+---------+-----------+
6+--------------------------+-------------------------+-----------------------+----------------+-----------------+-----------------+
7| timeout packet get count | invalid resp data count | valid resp data count | read cmd count | write cmd count | other cmd count |
8+--------------------------+-------------------------+-----------------------+----------------+-----------------+-----------------+
9| 2                        | 0                       | 2190                  | 764            | 1414            | 12              |
10+--------------------------+-------------------------+-----------------------+----------------+-----------------+-----------------+
11+--------------------+--------------------------+---------------------+---------------------------+
12| forward read count | forward read error count | forward write count | forward write error count |
13+--------------------+--------------------------+---------------------+---------------------------+
14| 764                | 0                        | 1413                | 1                         |
15+--------------------+--------------------------+---------------------+---------------------------+
16+-------------+-------+
17| cmd         | count |
18+-------------+-------+
19| append      | 51    |
20| client      | 6     |
21| config      | 2     |
22| decr        | 59    |
23| decrby      | 66    |
24| get         | 595   |
25| getdel      | 49    |
26| getex       | 53    |
27| getrange    | 68    |
28| getset      | 53    |
29| hello       | 3     |
30| incr        | 51    |
31| incrby      | 57    |
32| incrbyfloat | 68    |
33| mget        | 53    |
34| mset        | 51    |
35| msetnx      | 51    |
36| ping        | 1     |
37| set         | 617   |
38| setex       | 66    |
39| setnx       | 49    |
40| setrange    | 73    |
41| strlen      | 48    |
42+-------------+-------+
43big key top n 10:
44+---------------------------------------------+--------+
45| key                                         | length |
46+---------------------------------------------+--------+
47| string:4a65d70f-2260-4c1d-9ef9-ef36f3041c0c | 749    |
48| string:9825eed7-7529-4cfb-bb79-611281ebb2ee | 787    |
49| string:90d40f8d-cd9d-430a-85a7-e0e1d4cf7077 | 780    |
50| string:da6ee415-b4b3-4d54-925e-02aa3900f29e | 927    |
51| string:67bd18a8-ffdb-4418-a1ad-6df4dfc116cf | 892    |
52| string:1918dec2-c048-4879-b045-ac8291ab0e05 | 944    |
53| string:76096a81-36df-4671-8402-fa6cd64ba7c9 | 992    |
54| string:0e5b9f2f-938d-4005-8f62-723152e6f42e | 955    |
55| string:848cadae-e9fb-4c38-89ac-a3f024a6b1bc | 943    |
56| string:667cd229-4e33-455a-a08e-7da1280e97ab | 962    |
57+---------------------------------------------+--------+

Pre-populate Engula Server with RDB Data File

For meaningful workload replay results, it's essential to have representative datasets already loaded in the server, ensuring that real-world data access patterns can be accurately tested. We strongly recommend loading an RDB file into the Engula server prior to conducting workload replay tests. To generate an RDB file from a Redis server, execute the BGSAVE command, followed by copying the resulting RDB file to your Engula server directory.

Adjusting Read/Write Traffic Scaling Parameters

When conducting workload replay tests, you may need to adjust the read/write scale parameters to simulate different traffic patterns and load characteristics according to your testing requirements.

Key Parameters

  • --read-scale=: Controls the scaling factor for read operations during replay, default value is 1
  • --write-scale=: Controls the scaling factor for write operations during replay, default value is 1

Usage Scenarios

  • Read-heavy workloads: Increase --read-scale value while keeping --write-scale lower to simulate content management systems or read-intensive applications
  • Write-heavy workloads: Increase --write-scale value while keeping --read-scale lower to simulate logging systems or write-intensive applications
  • Balanced workloads: Keep both parameters equal to maintain the original read/write ratio from captured traffic
  • Load amplification: Use values greater than 1 to amplify traffic beyond the original captured volume
  • Traffic throttling: Use values between 0 and 1 to reduce the intensity of the replayed workload
ON THIS PAGE