Fuser Auto-Scaling

PhotoMesh offers two options to scale project production up and down based on resource needs:

§  Amazon Web Services (AWS) Fusers – AWS fuser instances can be dynamically launched whenever there are unassigned pending fuser tasks, to further accelerate model creation.

§  Fusers Pool – Fusers can be shared between several PhotoMesh Build Managers in your organization, to dynamically allocate and free fusers based on each project’s resource needs.

These options can be used together or separately. When either option is used, PhotoMesh Build Manager continuously monitors the fusers’ progress and status and the processing requirements and allocates or frees fusers as required.

Using Amazon Web Services (AWS) Fusers

PhotoMesh can work with Amazon Web Services to scale project production up and down based on resource needs. AWS fuser instances can be dynamically launched whenever there are unassigned pending fuser tasks, to further accelerate model creation. PhotoMesh Build Manager continuously monitors the fusers’ progress and status and the processing requirements of the current step, adding and terminating AWS fusers as required. See "Getting Started with AWS" for more information.

To use AWS fusers:

1.     In the Build Manager, toggle on Use AWS Instances. You can also access these settings from the Tools tab (AWS Instance Settings command) and from the Start page's Automation Dashboard.

Note:        The Automatically use new available fusers check box must also be selected, so that PhotoMesh will use the AWS fuser instances after they are launched.

2.     If PhotoMesh Build Manager is running on a local computer, and this is the first time connecting to the AWS fusers, the AWS Security dialog is displayed. Enter your AWS security information, and click OK.

Note:        If PhotoMesh Build Manager is running on an AWS instance with an IAM (Identity and Access Management role), security credentials are not required. See "Creating an IAM User for Running PhotoMesh Project Using an On-Premises Master" in the "Getting started with AWS" chapter for information.

AWS Security Dialog

Access Key ID	Token that allows PhotoMesh to communicate with your AWS account. Note:    You can find your Secret Access Key on the My Security Credentials link in the AWS Management Console.
Secret Access Key	Token that allows PhotoMesh to communicate with your AWS account. Note:    You can find your Secret Access Key on the My Security Credentials link in the AWS Management Console.
Region	Select an AWS region.
Remember my credentials	Select this option to save the password. The password will be saved and will be displayed using asterisks the next time the dialog is opened.

3.     In the Auto-Scaling Fusers Settings dialog, select the Use AWS Instances check box, and then enter the following information:

Maximum Price per Spot Instance	Set the maximum price you want to pay for an EC2 fuser spot instance.
Maximum Instances	Set the maximum number of fuser instances to launch.
Launch Rate	Set the maximum number of fuser instances to launch per minute, e.g. if Maximum Instances was set to 200, and the Launch Rate to 50, the Build Manager will spread the launch of the fusers over 4 minutes. This prevents a potential bottleneck that could be caused by a large number of fusers attempting to read the build data at the same time.
Terminate AWS instances when build finished	Select the check box if you want to terminate AWS fuser instances when the build is complete.
Select fuser instance launch parameters
Region	The region the master is running in, or the region set in the AWS Security dialog (Read-only).
Configuration Name	Name of the JSON configuration file to use. The JSON file contains an array of fuser instance types that PhotoMesh should use, ordered based on preferability. During production, PhotoMesh iterates over the elements of the array, beginning with the first instance type, until it finds an available instance. PhotoMesh will always attempt first to launch instances from one of the subnets listed in the Subnet ID (below), before it moves to the next machine type listed. Select based on the steps that require processing and other relevant factors. Click Edit to edit the JSON file. You can also open and edit the file in a text editor See "Recommendations Based on PhotoMesh Step" in the "Getting Started with AWS" chapter for information. Additional JSON configuration files can be created and placed under ../photomesh/AWSConfigurations where the two example JSON files are also saved. Each instance object in the JSON file's array has four properties: ami, instanceType, demandType, and fusersCount. The ami is the ID of the Amazon Machine Image (AMI). The instanceType defines the hardware of the host computer used for your instance. Each instance type offers different compute, memory, and storage capabilities. The demandType determines the pricing arrangement: "onDemand" or "spot." The fusersCount determines the number of PhotoMesh processes to run on each instance. More powerful instances have the capability to run multiple fusers simultaneously, thereby enhancing cost efficiency. Example: {     "configurations": [         {             "ami": "ami-12345",             "instanceType": "g4ad.4xlarge",             "demandType": "spot",             "fusersCount": 4         },         {             "ami": "ami-12345",             "instanceType": "g4ad.2xlarge",             "demandType": "spot",             "fusersCount": 2         },         {             "ami": "ami-12345",             "instanceType": "g4dn.4xlarge",             "demandType": "spot",             "fusersCount": 4         },         {             "ami": "ami-12345",             "instanceType": "g4dn.2xlarge",             "demandType": "spot",             "fusersCount": 2         }     ],     "Name": "GPU" }
Build Priority	Select either Choose all listed options, to use all machine types listed in the JSON file, with priority based on the order they are listed in the file, or select a specific machine.
Security Group ID	Select the instance’s security group ID. Security groups control inbound and outbound traffic using rules.
Subnet ID	Select the subnet of the virtual private cloud (VPC) dedicated to your AWS account, into which to launch your fuser instances. If you want to use additional subnets, click Additional Subnets, and type a list of all the subnet ID's you want to use. See "Building a PhotoMesh Project Using the Master Instance" in the "Getting Started with AWS" chapter for more information.
Key Pair Name	The name of the Key Pair used to configure the PhotoMesh AWS fuser Amazon Machine Images (AMI's). Key Pairs are used to encrypt and decrypt login information.
Additional Per Project Fuser Instance Tags Add tags to help you easily identify a fuser instance in the Amazon EC2 console after it is launched. These tags are saved in the JSON configuration file.
Customer	Name of customer who commissioned the project.
Description	Free text describing the project.
Shutdown inactive fuser instance…	Select one of the options for shutting down inactive fuser instances: §  Shut down inactive fuser instances before the next billing hour §  Shut down inactive fuser instances after XX minutes

4.     If you want to use another account, click Use another AWS account, and enter the AWS security information (see step 2 for more information).

5.     If you also want to use pool fusers, toggle on the Use Fusers Pool link, and enter the required information. See "Using a Fusers Pool" in this chapter for information.

6.     Click OK.

Fusers Pools

The fusers pool is used to share fusers between several PhotoMesh master computers in your organization. When working with a pool, the PhotoMesh master computer’s Build Manager requests fusers from the pool, and returns them when they are no longer needed. A distributed management approach is used to manage the pool fusers, whereby each PhotoMesh master computer manages its own fuser needs and resources. Fusers are allocated on a first come first served basis, i.e., to the first PhotoMesh Build Managers that request fusers from the pool. If a manager submits a request for a user, but no fusers are currently available in the pool, the manager continually checks the fusers pool for a fuser that was returned to the pool, at which point it assigns the fuser to its project’s working folder.  

Fuser allocation can be manually set from the Pool Manager page. From this page, you can monitor and manage all the fusers in the pool, e.g., assign fusers to specific projects’ working folders and prevent their being returned to the pool.

Running the Fuser Service

The PMFuserService.exe should be run on all fuser computers in the pool since it is responsible for running the required fuser (PhotoMeshFuser.exe) version for the requesting manager and assigning it to the manager’s working folder. This enables different PhotoMesh versions to use the same fusers pool. The PMFuserService.exe can be run from the fuser folder under the installation folder (.\Application Files\Fuser\PhotoMeshFuser.exe) or from a file share or network drive that is accessible to all fuser computers. It is recommended to run the fuser service on startup using auto-logon. See "Running Fusers Upon Login" in this chapter for information.

 

Creating a Fusers Pool

To create a fusers pool:

1.     Designate a pool directory which will hold the pool.json file created in the next step as well as the working folders that will be created automatically for each of the projects using the fusers pool.

Note:        If you are also using local fusers for a project, the working folder must be the one automatically created in the pool directory. It is recommended to create the working folder on a fast drive that is different from the one used for the project data, most preferably on a solid-state drive (SSD) that is shared with the master machine and other fuser machines.

2.     Create a pool.json file with the name and IP of all fusers in the pool. Use the following structure and syntax in creating the file:

§  Create an array called "Pool" (enclosed in square brackets).

§  Enclose within curly braces each of the fuser objects. Each fuser object is composed of two name/value pairs separated by a comma: Name: FuserName, and IP: FuserIP.  

{

"Pool": [       

        {

            "Name": "pm-020",

            "IP": "http://192.168.204.20:8085"

        },

        {

            "Name": "pm-021",

            "IP": "http://192.168.204.21:8085"

        }

    ],

    "Name": "ProductionPool"

}

§  When using more than a single fuser on a machine (See "Launching Multiple Fusers" in this chapter), a unique /X must be added after the IP address for each of the fusers, e.g., "IP": "http://192.168.200.001:8085/1", "IP": "http://192.168.200.001:8085/2", "IP": "http://192.168.200.001:8085/3" will launch three fusers from the same machine.

Note:    The number of fusers set to run on a machine should be limited by the amount of memory on the machine. See "Launching Multiple Fusers" in this chapter for more information.

Note:    The launching of multiple fusers on a single machine from a pool.json is only supported in PhotoMesh 7.8.3 or higher.

The fusers are launched in the order that they are listed in the json file. So if you want to launch a single fuser on each machine before launching additional fusers, each of the machine IPs should be listed first in the json file with /1 after each IP. To launch additional fusers, enter the machine’s IP followed by a unique /X. The example below will launch fusers in the following order: a fuser from computer 1, a fuser from computer 2, a fuser from computer 3, a second fuser from computer 1, and finally a second fuser from computer 2.

{

    "Pool": [

        {

            "Name": "fuserX1",

            "IP": "X.X.X.X/1"

        },

        {

            "Name": "fuserY1",

            "IP": "Y.Y.Y.Y/1"

        },

        {

            "Name": "fuserZ3",

            "IP": "Z.Z.Z.Z/1"

        },

        {

            "Name": "fuserX2",

            "IP": "X.X.X.X/2"

        },

        {

            "Name": "fuserY2",

            "IP": "Y.Y.Y.Y/2"

        }

    ],

    "Name": "ProductionPool-Multiple fusers on machines"

}

Managing a Fusers Pool

Fusers are allocated on a first come first served basis, i.e., to the first PhotoMesh Build Manager that requests fusers from the pool. If a subsequent manager submits a request for a user, but no fusers are currently available in the pool, the manager continually checks the fusers pool for a fuser that was returned to the pool, at which point it assigns the fuser to its project’s working folder. If you want to manually allocate fusers, you can do so from the Pool Manager page. From this page, you can monitor and manage all the fusers in the pool, e.g., assign fusers to specific projects’ working folders and prevent their being returned to the pool.

To manage a fusers pool:

1.     In the Build Manager dialog, in the Auto Scaling section, click the Pool Manager link. The PhotoMesh – Pool Manager page is displayed, with information on all the fusers in the pool. You can also access the Pool Manager from the Tools tab and from the Start page's Automation Dashboard. You can set any of the following for either selected or all fusers:

§  Assign to a project

§  Modify their idle timeout

§  Set private or public status

§  Add a comment

2.     If you want to manually assign fusers to a project’s working folder, do the following:

a.     Select the fusers’ check boxes, and click Assign. The Assign Fusers dialog is displayed.

Assign Fusers

b.     Enter the working folder and fuser folders.

c.     If you want to prevent these fusers from being restored to the pool after completing their current tasks, select the private check box.

d.     Click OK.

3.     If you want to modify the idle timeout set for certain fusers, select the fusers’ check boxes, and click Set Idle Time. Then enter the time in seconds, and click OK.

Set Idle Time

4.     Fusers can be made available to all master computers in a pool or restricted to a specific project's working folder. If you want to change fusers’ availability in the pool, i.e., to either make them available in the pool (public) or restrict their use to a specific project (private), select the fusers’ check boxes, and click Private/Pool. Then select the private check box to make them private (or clear to make them public), and click OK.

Note:        Private fusers are not restored to the pool even after completing their current tasks. If you want to assign fusers to a specific project’s working folder, but have them returned to the pool after they complete their tasks, clear this check box, and see step 3.

5.     If you want to write a comment for certain fusers, select the fusers’ check boxes, and click Comment. Then type the comment, and click OK.

Using a Fusers Pool

The PhotoMesh master should be run from a shared network drive that is accessible to all fuser computers, to enable the PMFuserService.exe to launch a fuser that is the same version as the master.

To use a fusers pool:

1.     Copy the PhotoMesh application files from the PhotoMesh master computer to a shared network drive that is accessible to all fuser computers, to enable the PMFuserService.exe to launch a fuser that is the same version as the master. Then run PhotoMesh from that path on the master computer.  

2.     In the Build Manager, toggle on Use Fusers Pool. The Auto Scaling Fusers Settings dialog is displayed. You can also access these settings from the Tools tab (Fuser Pool Settings command).

Note:        The Automatically use new available fusers check box must also be selected, so that PhotoMesh will use the pool fusers when they become available.

3.     Select the Use Fusers Pool check box, and enter the following information:

Fusers Pool Folder	Click Change to browse to the location of the pool folder. The pool folder must contain the pool.json file. See "Creating a Fusers Pool" in this chapter for more information. Note:    If you are also using local fusers for a project, the working folder for the entire project, i.e. for both the local and pool fusers must be located in the pool directory.
Fuser Folder	Click Change to browse to the location of the fuser folder. This folder is set, by default, to the UNC path of the folder from where PhotoMesh is running so that the Fuser service launches the fuser version required by the master. Make sure that this path is accessible to all the fusers, e.g., by running PhotoMesh from a file share or network drive that is accessible to all fuser computers.
Return inactive fuser to pool after	Set the number of seconds after which inactive fusers should be returned to the pool.
Maximum Fusers	Type the maximum number of fusers to run. If fusers are available in the pool, PhotoMesh Build Manager adds pool fusers up to the maximum number set.  Note:    Make sure your pool limit does not exceed your license fuser limit, to avoid rejection of your fuser request by the pool.

4.     If you also want to use AWS fusers, select the Use AWS Instances check box, and then enter the required information. See "Using Amazon Web Services (AWS) Fusers" in this chapter for information.

5.     Click OK.

6.     If you want to monitor and manage all pool fusers, e.g., assign fusers to specific projects’ working folders and prevent their being restored to the pool, in the Build Manager dialog, click the Pool Manager link. See "Managing a Fusers Pool" in this chapter for information. You can also access the Pool Manager from the Tools tab.