Escript

Megaload supports specifying tests by means of an Erlang Escript, as an alternative to JSON files, to test XMPP servers. Escript and JSON test specifications differ in the way they define the activities performed by each worker and in the target load model.

The Escript test specification supported by Megaload is described below. The XMPP client used is the open source library escalus, developed by Erlang Solutions.

An example is available in Examples.

Model

The Escript must define:

  • The function main/1, implementing the job each Megaload worker shall perform;
  • Two callbacks related to updating the target load generated by the Megaload node hitting the system under test (SUT). The target load is modelled as target number of workers and target rate.

The escript may define:

  • The initial targets for the workers and rate.
  • The arrival rate for all the test.
  • The total number of unique ids generated by Megaload

When instructed to start the load test, Megaload begins spawning short-lived workers -- each executing the main/1 function, then dying. Megaload ends doing so only when instructed to stop the load test.

Each Megaload node spawns new workers according to the local target number of workers, and throttles the invocations of the main/1 function according to the local target rate.

Each Megaload node periodically sets the local targets (i.e. target number of workers and target rate) to the values returned by the dedicated load regulation callback, capped in order not to exceed the configured memory and CPU usage limits (configuration parameters mem_load_limit and cpu_load_limit of the loader_utils application).

Load regulation model

The load regulation sets local target load -- i.e. number of workers and rate -- for each Megaload node. On each Megaload node, local mechanisms enforce the local targets.

The load regulation is per-node, i.e. each Megaload node decides the local targets independently from the other Megaload nodes in the cluster, based on custom SUT-related metrics.

For the sake of simplicity, the workers arrival frequency experienced by the SUT in case of Megaload cluster with multiple Megaload nodes is a default value (hardcoded in Megaload) multiplied by the number of Megaload nodes.

At the start of the test, each Megaload node invokes the specified initialization callback supposed to initialize the custom SUT-related metrics.

During each phase, on each Megaload node, the load regulation periodically:

  • Reads the custom SUT-related metrics;
  • Invokes the specified regulation callback, that returns new local targets (from previous local targets and metrics values);
  • Reads memory usage and CPU utilization;
  • Computes and sets new local targets as returned by the regulation callback and capped by memory usage and CPU utilization.

Test specification

Introduction

In order to specify a valid test, the Escript has to define main/1 and other two callbacks -- there is no need to export them.

The following is a minimal template Escript:

#!/usr/bin/env escript

main(_) ->
  %% Hit SUT.
  %% Update request counters.
  %% Update custom SUT-related actual metrics.
  ok.

init_load_regulation_metrics() ->
  %% Initialize target and actual custom SUT-related metrics.
  InitializedMetrics = [...], %% Non-empty list here.
  {ok, InitializedMetrics}.

load_regulation_targets(_MetricsValues, OldTargets) ->
  %% Compute new targets.
  NewTargets = OldTargets, %% Dummy logic.
  {ok, NewTargets}.

In order to write the Escript, it may be useful relying on Erlang modules not released with plain Megaload. Refer to the Megaload installation guide for how to build a Megaload package with additional applications.

During development and prototyping:

  • The Escript can write entries in the Megaload custom log using the API in the loader_custom_log module. Such logging must be removed during the actual load testing of the SUT;
  • You may prefer validating the Escript using escript -s (warnings can be ignored) before feeding Megaload with it.

Job performed by each worker

The Escript has to define the job to be performed by each worker in the main/1 function.

Main

Hit the SUT, update request counters, update the custom SUT-related actual metrics (e.g. response time of the SUT) and return.

Type
-callback main(any()) -> ok.
Expected behaviour

In order to update the request counters, use the Megaload API in the loader_requests module.

In order to update the custom SUT-related actual metrics, use the Megaload API in the loader_metrics module.

Load regulation

The Escript has to define two load regulation-related callbacks:

  • Initialization callback init_load_regulation_metrics/0;
  • Regulation callback load_regulation_targets/2.

Initialization callback

Initialize custom SUT-related metrics.

The returned metrics datapoints are read by Megaload in the regulation loop.
It is guaranteed to be executed on the Megaload node.

Type
-callback init_load_regulation_metrics() -> {ok, InitializedMetrics}
  when InitializedMetrics :: [Metric, ...],
       Metric :: {ShortName, Target, Actual},
       ShortName :: atom(), %% E.g. time_to_deliver
       Target :: LoaderMetricsMetric,
       Actual :: LoaderMetricsMetric,
       LoaderMetricsMetric :: {Name, DataPoints :: [DataPoint, ...]}.

The prefix of the returned names should be <<"load_regulation">>, followed by <<"target">> | <<"actual">> and the short name of the metrics e.g. <<"time_to_deliver">>.

Expected behaviour

A sequence of calls to loader_metrics:new_*.

Regulation callback

Compute new proposed targets for number of workers and rate.

It is not guaranteed to be executed on the Megaload node.

Type
-callback load_regulation_targets(MetricsValues, OldTargets) -> {ok, NewTargets}
  when MetricsValues :: [MetricValues, ...],
       MetricValues :: {ShortName, TargetDataValues, ActualDataValues},
       TargetDataValues :: DataValues,
       ActualDataValues :: DataValues,
       DataValues :: [LoaderMetricsDataValue, ...],
       LoaderMetricsDataValue :: any(),
       OldTargets :: Targets,
       NewTargets :: Targets,
       Targets :: {{workers, TargetWorkers}, {rate, TargetRate}},
       TargetWorkers :: pos_integer(),
       TargetRate :: pos_integer().
Expected behaviour

Purely functional logic (i.e. deterministic and free of side effects) e.g. it shall not attempt to read metrics.

Rate, workers, arrival rate and unique ids

The initial values for rate and workers, and the value of the arrival rate for all the test may be defined using Erlang attributes. If these are missing, they default to:

  • rate = 1, the load regulation will increase this value based on available CPU
  • workers = 1, the load regulation will increase this value based on available memory
  • arrival_rate = 20

The unique ids may be used by an Erlang escript to have a unique integer identifier on each concurrent scenario. As an example, this identifier can be used in XMPP to generate unique user/password pairs. The unique id is stored in the process dictionary and can be retrieved using erlang:get(unique_id).

The total number of unique identifiers is split among nodes to avoid duplicates. By default, Megaload will assign 1,000,000 unique identifiers per node. As an example, two nodes with 15000 unique ids are split such as one node has identifier 1 to 7499 and the other identifier 7500 to 15000. Note that if a node needs more identifiers than provided, it may generate duplicates by overlapping ids with other nodes.

Example

-rate(15). %% requests per second
-workers(150). %% concurrent_scenarios in the JSON specification
-arrival_rate(20). %% new workers/scenarios per second
-unique_ids(15000).