Scaling Dynamic Worker Pool Connections

Jade configuration settings affect the operation of the dynamic worker pool used by web server applications such as REST servers and SOAP servers.

You can configure a pool of workers to be dynamically scaled in response to changes in the incoming communication load. When the incoming rate of requests exceeds the response capacity of the server, the Jade application detects this situation and adds processing capacity by spawning another worker application.

An incoming request is added to the request queue if there are no workers available to process the request. If the number of queued requests continuously remains above the queue depth threshold for the configured period, an internal JadeMultiWorkerTcpManager class, which manages TCP connections to a Jade application, adds a new worker to the pool if the number of current workers is less than the configured maximum number of workers. Each worker in the pool is a complete single‑threaded Jade application, running on the same Jade node as the original Jade application that spawned it.

If a worker process is idle for more than the configured timeout period, it is removed from the worker pool if the total number of workers is greater than the configured minimum number of workers.

The scaling of the pool of workers is controlled by the following initialization file settings.

• The [WebOptions] section of the Jade initialization file (jade.ini) can contain the following parameters for Jade applications to directly control the scaling of the dynamic worker pool.

  • MaximumInUse , which specifies the maximum number of instances of a Jade application that will be run on a single Jade node. It should be matched to the resources available (that is, the number of CPU cores with allowance for the operating system and the Jade system application) or a target resource cost.

  • MinimumInUse , which specifies the minimum number of instances of a Jade application that will not be subject to removal if they become idle.

  • QueueDepthLimit , which specifies the number of queued requests (that is, the queue depth) that must be continuously exceeded before a new worker will be spawned.

  • QueueDepthLimitTimeout , which specifies a time duration. If the queue depth limit has been continuously exceeded for the specified time, a new worker is spawned.

  • WorkerIdleTimeout , which controls how long an application instance is retained in the connection worker pool when it is idle.

  System designers need to consider their overall strategy when setting values for the dynamic worker pool scaling. If the server is to be configured for maximum responsiveness, set a queue depth limit of 1, a queue depth limit timeout of 5 seconds, and a worker idle timeout of 10 seconds. If the server response is to be optimized for resource cost, more‑detailed analysis will be required. For example, a REST request may take 5 milliseconds to service. The maximum response time that client applications may accept for that request could be half a second. A request could therefore sit in a queue that was 100 entries deep before it would exceed the acceptable response time.

  The request queue grows when the incoming rate of requests is greater than the capacity of the server to respond. Even though the load may increase beyond the capacity of the server to respond, it will take some finite time before the queue reaches an unacceptable depth. For example, a 20 percent increase in request load for 5 seconds could result in the maximum queue depth increasing from 3 to 8 requests if the incoming request rate was low (that is, 10 per second) and the response time was high (that is, 100 milliseconds). In this example, if an acceptable response time is 1 second, there would be no need to add server capacity. The overall system performance criterion could be met for a lower resource cost.

  Optimal configuration for any system will therefore be a matter of managing the server queue depth, or more precisely, the likely rates at which the queue depth can grow during the specified queue depth limit timeout period.

  The growth rate of the queue depends on several factors, including the:

  • Average time for requests when the server has spare capacity

  • Asymmetry between the incoming request rate and the capacity of the server to respond

  • Amount of time the asymmetry exists

  • Distribution of incoming request types (assuming not all requests take the same amount of time to service)

  Given the number of factors, a system designer would probably need to determine the best queue depth limit and queue depth limit timeout values experimentally. A good starting point for any such experimentation would be to start with a maximum queue depth calculation for each request type and then work in factors like expected load asymmetries (request volatility) and the likely distribution of request types.

  When the range in response times for different request types is large or distributions hard to predict, consider servicing these requests in different web server applications.

  To disable all dynamic worker pool scaling, configure all of the above parameters in the [WebOptions] section of the jade.ini file to zero (0).

• The MaxInUse parameter in the [application-name] section of the jadehttp.ini file affects the dynamic worker pool scaling. This setting does not control the scaling, but it can prevent the scaling from working if it is set too low.

  The value must be large enough to allow enough connections between IIS and Jade so the queue depth remains greater than the QueueDepthLimit parameter value for the period configured in the QueueDepthLimitTimeout parameter defined in the [WebOptions] section of the jade.ini file.

  In some respects, the MaxInUse parameter sets the upper limit for the response time, but it is better to manage response times using the parameters in the [WebOptions] section of the jade.ini file. In general, most REST server usage patterns would be well‑served by a MaxInUse parameter value that is 10 times larger than the queue depth limit, with a value not less than 50.

  The MaxInUse parameter is the only configuration setting the affects how often the system will respond with a “Server Unavailable” 503 response. In a direct HTTP configuration, all requests are queued, and as such, never return a 503 response.

• The specified number of application copies in the Application Copies text box on the Web Options sheet of the Define Application dialog provides the Jade node with the starting number of worker instances.