Liveliness updates

Eventually you may find that your spawning application has spawned multiple servers and each will have it’s own lifecycle. We refer to the lifecycle of a spawned server as it’s liveliness. We can specify the --liveliness-endpoint command-line option when launching ts3d_sc_server so that the spawned server knows what endpoint to send liveliness updates. When this option is specified, its value acts as a base URL to which messages are posted. The server’s id is used as the endpoint to the spawned server base URL and the message type is specified in the liveliness query parameter.

Let us imagine we have launched ts3d_sc_server with --id abc and --liveliness-endpoint http://localhost:55555 would consist of the following messages:

Ready: This one-time message indicates the spawned server has initialized and is ready to receive a WebSocket connection. The request would be a POST to: http://localhost:55555/abc?liveliness=ready
Ping: These messages act as a “heart beat” and let us know the spawned server is still alive. The request would be a POST to: http://localhost:55555/abc?liveliness=ping
Disconnect: This one-time message indicates the spawned server has terminated. This can happen when the user closes the tab containing a WebViewer or programmatically calls the shutdown() method from the JavaScript API. The request would be a POST to: http://localhost:55555/abc?liveliness=disconnect

To receive and process these methods we will need to make some modifications to our code. Begin by adding the following line of code to the argument list in our spawn method, just after the line containing the –-sc-port option:

let args = [
  "--id", spawnInfo.spawnId,
  "--sc-port", spawnInfo.wsPort,
  "--model-search-directories", this.modelDirectory,
  "--license-file", this.licenseFile,
  '--liveliness-endpoint', `http://localhost:${this.serverPort}/liveliness`
];

We now need a method on our Spawner class to receive these updates. Add the liveliness method to the Spawner class in spawner.js:

liveliness(req, res) {
  const spawnId = req.params.spawnId;
  const spawnInfo = this.activeSpawns.get(spawnId);

  if (spawnInfo === undefined) {
        console.log(`unknown id: ${spawnId}`);
        res.status(404).json({error: "invalid spawnId."});
        return;
  }

  const updateType = req.query.liveliness;

  switch (updateType){
        case "ready":
          console.log(`liveliness ready: ${spawnInfo.spawnId}`);
          break;

        case "disconnect":
          console.log(`liveliness disconnect: ${spawnInfo.spawnId}`);
          break;

        case "ping":
          console.log(`liveliness ping: ${spawnInfo.spawnId}`);
          break;

        default:
          res.status(400).json({error: "invalid liveliness value"});
  }
}

Examining the code, we can see that when we receive a liveliness update we first ensure that the spawnId we parsed from the url parameter is valid. The corresponding SpawnInfo object that was created in the spawnViewer method is then retrieved. We finally have a switch statement for processing the update types that were outlined above.

In order to see this in action, we will need to bind our new method to the correct route. Add the following code right after we bind the spawnViewer method inside the start method of the Spawner class:

this.expressApp.post("/liveliness/:spawnId", this.liveliness.bind(this));

Running our application now will result in terminal output that looks similar to the following:

liveliness ready: 8f187c7c-6f35-44d5-8370-26042aea66ae
liveliness ping: 8f187c7c-6f35-44d5-8370-26042aea66ae
liveliness disconnect: 8f187c7c-6f35-44d5-8370-26042aea66ae
spawn 8f187c7c-6f35-44d5-8370-26042aea66ae exited with code: 0

Now that we can receive messages about spawned servers we can create meaningful implementations for the corresponding cases of our switch statement.

The ready message

In our current implementation of the spawnViewer method, we are immediately returning a response to the caller with the WebSocket port. Since spawning and initializing the server is an asynchronous operation, we need to wait until the server is ready to receive a connection before returning a response to the caller. You will notice that we save off the response object on the SpawnInfo for that instance. The ready liveliness message handler is a fine place to now fulfill that request.

Begin by adding the implementation for the ready liveliness message inside the corresponding case of the switch statement:

case 'ready':
  console.log(`liveliness ready: ${spawnInfo.spawnId}`);

  spawnInfo.response.status(201).json({
        id: spawnInfo.spawnId,
        endpoint: `ws://localhost:${spawnInfo.wsPort}`,
        rendererType: spawnInfo.rendererType
  });

  spawnInfo.response = null;

  break;

After you have added the above code, remove the following from spawnViewer

res.status(200).json({result: "spawn viewer!"});

The above code simply returns the id, endpoint, and renderer type to the client within our liveliness method, which is why we no longer need an HTTP Response sent back to the client in the spawnViewer method. The returned endpoint will be passed directly into the Communicator.WebViewer object that is created on the client’s browser. For now, the renderer type will always be client-side rendering. We will enable server side rendering later in the tutorial.

The disconnect message

The main purpose of the disconnect message is to allow us to return the ports that the instance was using to the available list, so that they can be reused by future spawned servers. Our implementation for the disconnect case looks like this:

this.availableWsPorts.push(spawnInfo.wsPort);

this.activeSpawns.delete(spawnInfo.spawnId);
res.status(200).end();

In addition to returning the ports to the available list, the code also removes the SpawnInfo object from our activeSpawns map to ensure our recordkeeping is up-to-date. Now when a viewer shuts down, its port will be re-used when a new viewer is spawned.

The ping message

For the purposes of this tutorial, we are not doing anything special for the ping message aside from illustrating its presence. We will simply record the last time we received a ping message from the particular instance inside the ping case of our switch statement:

spawnInfo.lastPingTime = new Date();
res.status(200).end();

In a production environment, you will most likely require additional error checking and handling. If a ping or message is not received from an instance for a prolonged period of time with no disconnect message, it is possible an application error has occurred. In this case it may be prudent to examine logs or even kill the application so that its port may be reused. Additional logic may be also placed in the process exit handler to detect the unlikely event a program crash and the absence of expected liveliness messages.

Our application server is now starting to shape up. We are processing messages from instances of spawned server and are reusing ports as viewers are spawned and disconnect. In the next section we will turn our attention to the front end and get models rendering on the screen.