Administering FSC & Troubleshooting FSC Failure-to-Start

 

During teamcenter installation, FSC is installed under local user account, service is started and configuration files (.xml) are created by TEM. Later on, FSC service can be stopped and started to add volumes or to update its configuration files with the changes. Sometimes due to different reasons, FSC service may fail to start. The post will discuss on possible reasons of such failure and recommended steps to be taken to fix any such issue. Before this, it is important to understand how FSCs are administered and managed.

Administering FSC

While initially installing Teamcenter with TEM, single FSC (that mounts on single volume) is installed under local user account. After FMS has been installed, more volumes can be added and FMS can be customized for achieving different FMS configurations for handling huge data, frequently used data, remote client data request etc.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\administeringfsc.jpg

Managing FSC with fscadmin utility

In various situations, FSC may need to be managed. When changes are made to master configuration file, FSC will need to be restarted. When troubleshooting FSC, there may be need of redefining logging level to retrieve more information from log file. There may be situations where there is need to clear or purge cache.

So, while managing FSC there may be need to check status of server (started, offline etc.), to perform a shutdown (stop), to check if file has been previously cached (cachedetail), to clear cache (clearcache), to get the configuration details (config) of FSC , to redefine or modify logging levels (warn, info, debug, error etc.) and many others. All of these can be achieved by using the utility fscadmin with various arguments.

Understanding the commonly used arguments with fscadmin utility

Before using this utility, FSC_HOME and JAVA_HOME must be set correctly. FSC_HOME is set to the fsc directory of the Teamcenter installation. Now for example, if status (running status) of local FSC needs to checked, the utility can be used on TC command prompt from FSC_HOME directory as follows:

D:\blog\images\teamcenter administration\fms\troubleshooting fms\troubleshootingFSC\fsc_command.JPG

Where:

-s

Specifies the protocol server and the port to which FSC will communicate with.

FSCID

The command described above is intended for the FSC with given ID. Instead, (.) can be used, if connecting to local (current) FSC. E.g. fscadmin –s http://pun05:4343 ./status

FUNCTION [/SUBFUNCTION/…]

This specifies the part of the command that is run with fscadmin utility to achieve the desired information or result. E.g. fscadmin –s http://pun05:4343 FSC01/status OR

fscadmin –s http://pun05:4343 FSC01/cachesummary/read

Few of the frequently used functions (and some of its sub functions) with fscadmin utility are listed below:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\fsc_function_subfunction.jpg

Managing FSC manually (when running on windows)

FSC is installed by TEM as a Windows service. If not installed by TEM, it can also be manually installed using installfsc batch script. From the TC Command Prompt, change to the FSC_HOME directory and run:

installfsc %JAVA_HOME% %FSC_HOME% fscid

Where JAVA_HOME & FSC_HOME should be set to correct paths and fscid is the name of FSC to be installed. If JAVA_HOME or FSC_HOME contains spaces, they must be quoted on the command line. When FSC has been installed through batch script, it is important to change Log On information from System account to User account in order to get proper network connectivity.

Configuring FSC manually

Changing configuration in FSC’s master file for installed FSC

After installing the FSC with batch script, fmsmaster_fscid.xml file needs to be configured correctly, so that this FSC is included with its defined volume (with its correct id and path) in the enterprise.

This configuration can be achieved manually by executing backup_xmlinfo utility. List of current volume definitions in the database can be obtained by running this utility .The generated definitions including FMS ID tags for volumes and transient volumes are stored in output file backup.xml.

The backup_xmlinfo utility is located in the TC_BIN directory .Executing this utility from TC Command prompt will create the output file backup.xml in the directory from which this utility was run.

backup_xmlinfo -u=infodba -p=password -g=dba

An example of generated backup.xml and configuring fmsmaster_fscid.xml with it is as follows:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\backup_xmlinfo.jpg

Please Note:

For four-tier client file access, transient volumes must be declared within the FMS Master Configuration file. They are declared with the FSCs that host the transient volumes.

The backup_xmlinfo utility generates transient volume information for the current server context (tcserver) in which the utility is being run. All other server pools with transient volumes are not identified.

Updating all FSCs master file with the changes configured for installed FSC

When there are multiple FSCs, FMS Master Configuration file is initially downloaded for all the other FSCs from master FSC in FMS network. These copies of the master configuration file should be in synchronization in the network. So, changes made to any of FSC’s master configuration file (fmsmaster_fscid.xml), needs to be updated to all the other FSCs master configuration file in FMS Network

To achieve this, FSC configuration can be reloaded by stopping and restarting all the FSCs .This updates their respective master configuration file. The other way, to reload the configuration is through fscadmin utility. The configuration will be reloaded across entire FMS network by running the command:

fscadmin -s http://fschost:fscport ./config/reload/all

Troubleshooting FSC Failure-to-Start

There can be several reasons, FSC service will not start. Based on the errors displayed, there can be different reasons for the failure. Few of the possible solutions are illustrated below with the cause of failure:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\fms_enterprise_error.jpg D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\volumerror.jpg D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\logfile_info.jpg D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\TROUBLESHOOTINGFMS\TROUBLESHOOTINGFSC\fsc_corrupted.jpg

Other suggestions:

  • Alternatively to start FSC or to check issues with FSC service start, following command can be directly run from TC Command Prompt (changing to fsc directory):

startfsc –Dfsc.config=%FSC_ID%.xml –Dfms.config=fmsmaster_%FSC_ID%.xml

  • There can be FSC FCC connectivity issue due to firewalls. It’s difficult to get clue for such issues. E.g. Error – error code (say, Error-2550)

Also Read:

Understanding FSC & FCC

FMS Configuration for Multiple FSCs & Cache Server

FMS Configuration for Remote Clients

Teamcenter Architecture & Communication Between Tiers

  ... 

FMS Configuration: Remote Clients & Understanding Local Volumes

 

Considering the remote location of clients, FMS configuration will be required that accesses and serves data requests without affecting performance.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\locationofclient.jpg

LAN (Local Area Network) & WAN (Wide Area Network)

Remote location computers can be connected through WAN .Before further understanding this configuration, a short description on two types of networks that provides interconnectivity between computers is as follows:

 

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\lan_wan.jpg

LAN (Local Area Network) is group of computers and network devices connected together usually in same building. Connections are high speed and relatively inexpensive.

WAN (Wide Area Network) provides widespread connectivity and allows computers to be connected in different cities and countries. A WAN connects several LANs, and may be limited to an enterprise (a corporation or an organization) or accessible to the public.

LANs are typically faster and more secure than WANs.

Why default FSC Cached Configuration is not a good idea?

Please Note: symbol indicates any client or server at remote office location.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\bestpractice.jpg

The first configuration shown in diagram above is of FSC Cached Configuration discussed in last post, that can also be used to upload/download files from clients at remote office location. But, for each data request from client at remote office location, WAN trip has to be made to the FSCCache at home (central) location. Files will be uploaded/downloaded for each WAN trip. This will decrease performance and increase cost of file transfer.

Configuration files will be same as discussed under the section of FSC Cached Configuration except that master configuration file will include WAN transport.

The second configuration shown in diagram above is the best practice that can be adopted .It introduces remote FSCCache at client’s remote office location. This remote cache server will cache files and can be shared between all clients at remote office location .Files are accessed directly from remote FSCCache in LAN network (LAN at remote location) for all the clients at this location. This will take care that WAN trip is not made every time for each data request from clients at this remote location. This increases performance and decreases cost for this configuration. The said configuration is discussed below:

FSC Cached Remote location Configuration

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\cacheserverremote.jpg

Clients Directly Connect to Remote Cache Server

Property FCC_EnableDirectFSCRouting enables clients to directly connect to any volume servers. In case there are both volume servers and cache servers in FSC group, this property needs to be disabled by setting it to false (default is true), so that clients can connect to assigned cache server and not to volume servers directly.

But, for current configuration in discussion, there is remote cache server but no volume servers at remote location, so this property enabled or disabled has no effect and need not be set. Clients will anyway connect to assigned remote cache server.

Sending Data Requests to Central Office Location

  • Incoming data requests from remote office location are sent to cache server of central office. This cache server at home location performs as entry FSC for requests from FSC groups outside the central office FSC group.
  • WAN acceleration is enabled from the remote office location to the central office location using the linkparameters fromgroup statement.

Accessing Files in Remote Location

  • Remote office location clients are assigned remote cache server (shared cache to clients) in its LAN.
  • When the new file is uploaded or downloaded to volume server (at central location) through WAN trip, a copy of this file is maintained in remote cache server. Next time, when the client needs to access this file, it does not have to make a WAN trip and can access it from its remote cache server in LAN network.

The configuration files for this configuration that contains remote cache server are as shown below:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\master_cacheserverremote.jpg

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\fsc_cacheserverremote.jpg

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\fcc_cacheserverremote.jpg

FSC Cached Remote location Configuration with Volume Servers

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\REMOTECONFIGURATION\volumeserverremote.jpg

In the above diagram, remote users (Remote Location 1) have access to requests from outside FSC Groups through its shared FSCCache (FSC Remote Cache 1). Locally served requests at this remote location are routed directly to FSCs with volumes (Volume Server). So basically at this remote location,

  • Local Requests are handled by its Volume Servers.
  • Outbound Requests & Incoming Requests are handled by its Cache Server.

Outbound request

It is a request originating mostly within the FSC group, for accessing data stored on a resource outside its FSC Group.

Roles of Cache Server at Remote Location

Remote FSCCache

Recently accessed files from other FSC Groups (central or other remote location FSC Groups) are populated in this remote FSCCache. The populated data is then served to its clients on request, in the LAN network. Serving data from its cache, prevents an unnecessary WAN trip.

This FSCCache can perform as exit FSC or entry FSC based on the way it is handling the request.

FSCExit

Outbound requests originating within the FSC group are routed to an exit FSC that routes this request on FSCs outside its FSC Group. Identifying exit FSC (here it is FSC Remote Cache 1) becomes important when there are also volume servers in its FSC Group.

FSCEntry

For any incoming requests from FSCs outside its FSC Group, the FSCCache performs as entry FSC. So for this remote location (Remote Location 1), exit FSC routes the requests to outside FSC Group’s entry FSC. Also, if an outside FSC Group is accessing data from this remote location (Remote Location 1) , the same FSCCache(FSC Remote Cache 1) would also have performed as entry FSC for the outside FSC Group. ...  Continue Reading

FMS Configuration: Multiple FSCs & FSC Cache Server Configuration

 

Different FMS configurations may be required based on volume of files to be transferred or stored, how frequently  files are accessed by clients, whether clients are at remote or home location, managing single or multiple sites/database etc. Few of the frequently used FMS configurations are discussed here in this post. The diagram below depicts the different FMS configurations that may best meet the mentioned purposes.

 

Single FSC with Multiple Volumes

In a typical small deployment, teamcenter installation provides a single FSC that mounts a single volume. Adding few additional volumes to this deployment will create the following  FMS Configurations:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\CONFIGURATION1\singlefsc_multiplevolume.jpg

 

This type of configuration may be required when volume of files to be stored is huge or there are additional integrations with teamcenter.

In this case, multiple volumes are defined for single FSC.In FMS Configuration file, each volume must have an entry under this FSC.The volume statement also specifies the root directory for each volume.The volumes can be either local disk volumes or mounted volumes on network.

File Upload & Default Volume:When a new file is uploaded by user, the default volume for the file is determined by the volume attribute of either the user or the user’s group in the Organization application.So,for a specific user or for a specific user’s group, files are uploaded to a specified default volume.

Moving Volume data from one volume to other

Volume data can be moved from one volume to another volume using the move_volume_files utility. To move volume data, an XML template file contains allocation rules with defined user criteria, , Item/Item Revision criteria,dataset criteria(including type) and volume criteria(space available) based on which files are moved. The move_volume_files utility is used to retrieve this XML file containing the volume allocation rules template, edit the rules as required and then the utility is used to move the volume files(arguments that can be used with utility : [-outrulesfile=xml_file] , [-rulesfile=xml_file –f=move] ). This utility can come useful if specific type of data needs to be maintained in each volume (e.g. putting JT files and CAD files in separate volumes).The utility can be run as schedule task or cron job.

Multiple FSC Configuration (Volume Servers)

This configuration may be required when high volume of data is requested simultaneously through different clients. Since there are multiple FSC servers, it can approximately handle ‘n’ times (number of FSC servers) the file transfer volume of single FSC deployment .The configuration contains a single FSC group with an FSC deployed for each volume.

Assigned FSC to client is just to identify the FSC group and any of FSC in this group can be used to transfer files. This configuration with direct user access via direct routing (routing not only to assigned FSC) is sometimes called FCC direct connect configuration.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\CONFIGURATION1\multiplefsc_multiplevolume.jpg

 

All FCCs and other non-master FSCs download the configuration file from fscA (master FSC ) .Data requests of FCCs can be handled by any of the FSCs.

FSC Cached Configuration

In this configuration, most accessed files are maintained in a FSC cache server and clients access files from this cache server instead of making trips to volume servers every time. It provides improved client performance due to high speed cache server. When user uploads or downloads a file, a copy of file is maintained in cache server while uploading/downloading the files to/from volumes. So, in case of short network outages for volume servers, clients continue accessing files from the cache server.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\CONFIGURATION1\fsccacheserver.jpg

As explained in last configuration, by default FCCs can still make direct connect with volume servers (bypassing cache server). This direct connect of FCC has to be disabled; else the whole purpose of introducing FSC cache server will be defeated.

FCC_EnableDirectFSCRouting = false

Also Read:

Understanding FSC & FCC

Troubleshooting FSC Failure to Start

FMS Configuration for Remote Clients

Teamcenter Architecture & Communication Between Tiers

Structure Manager in Teamcenter

Understanding FMS Server Cache(FSC) & FMS Client Cache(FCC)

 

Understanding File Management System

File management system manages uploading and downloading file data between clients and volumes in both two-tier and four-tier teamcenter architecture.

It centralizes data storage while enabling fast access of data to user.  FMS is responsible for data storage (volumes), caching (FSC & FCC), distribution (via file servers at remote locations) and access system (no direct access to files).

Generic diagram to explain downloading and uploading of files through FMS:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\generic fms.jpg

FMS servers do not permit direct access to volume data or cached data.Access is provided when a valid security ticket is presented by requestor.

FMS provides:

  • Server Cache for data access
  • Client Cache for rich clients
  • Volume servers for file management
  • Transient datastore for transporting reports,plmxml and other non-volume data

FMS Server Cache (FSC)

FMS Server Cache is responsible for file upload and downloads to other FSCs and to FCCs. It is a FMS process that runs on a server host and performs as one or more listed server:

Volume Server

When FSC runs on a host where volume is located or directly mounted, it performs as Volume Server. FSC can contain zero to multiple mounted volumes .It will read /write files from volumes.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\volume server.jpg

Cache Server

When FSC runs on a host, where either volume is not located or not directly mounted or requested data is not directly available on mounted volume, it performs as Cache Server. This FSC has no direct access to the volume containing the file, so it caches data from other FSCs.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\cache server.jpg

Configuration Server

FMS configuration information has to be passed to all FSCs and FCCs. When this information is passed by the FSC in question to other FSCs and/or to file client caches, it is performing as configuration server. This configuration server reads the fmsmaster.xml configuration file to distribute the information to FSCs and FCCs.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\configuration server.jpg

Transient Server

When FSC delivers transient files like PLMXML and generated reports (non-volume data) to clients, it is performing as transient server. FSC performs as transient server only in four-tier teamcenter architecture. Each business logic server in enterprise tier reads and writes file to a temporary disk location (transient volume).This FSC has the capability to deliver this temporary data (generated files in temporary disk location) to and from the client. The transient volume must reside on the same machine as the FSC and the Pool Manager.

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\transient server.jpg

Teamcenter server resides on the client machine in two-tier architecture, so it writes and read files from the temporary disk location on the client machine itself. There is no need to deliver the data to client by FSC, so there is no FSC performing as transient server in two-tier architecture. FCC will directly process the read files from the transient volume. That means, FCC is performing as transient server in two-tier architecture.

Note: There can be one or more than one FSCs for one teamcenter installation. FSCs can be deployed for each volume created. FSCs can be deployed to perform as one or more of the servers as mentioned above.

Also, FSC manages two caches for files (Write Cache & Read Cache).

FMS Client Cache (FCC)

FCC is a FMS process that runs on a client host .FMS Client Cache uploads files to FSC, requests files from FSC and it also caches files to client host. Each teamcenter rich client host requires a local FCC. FCC manages two caches for files. Write cache containing files uploaded to teamcenter volume and read cache containing files downloaded from teamcenter volume. FCC caches are high performance cache that accelerates file downloads over noisy WAN lines for remote sites. Required software for WAN acceleration comes along with base FMS install.

How file consistency is maintained in cache?

FMS system uses a file GUID for file contents. This identifier determines when to pull a file from its local cache, rather than retrieving the file from a far lying default volume. Every file in teamcenter storage has a single file GUID associated with every replicated copy of the file. If file is moved, copied, renamed, reassigned to a new owner, its file GUID remains the same. However, if file content is changed, a new file GUID must be created to describe the file’s new contents. So, there are no issues or conflicts with change of file and cache consistency is maintained. This holds true for both client and server cache.

Understanding FMS Configuration & Configuration Files

There can be several different FMS configurations required to transfer files and data. Few of them are listed below to understand that such scenarios can be handled by File Management System.

  • Single FSC deployed for teamcenter installation
  • Multiple FSC deployed for the teamcenter installation
  • Client is situated faraway from volume server containing the file
  • Client may need access to multiple database
  • Files need to be transferred over the WAN to remote site
  • Files need to be transferred between multiple PLM systems

To achieve the above configurations, File Management System (FMS) uses three files:

  • fmsmaster.xml– FMS Master configuration file
  • fsc.xml– FMS Server configuration file
  • fcc.xml-FMS Client configuration file

Parameters and elements defined in these files will help us achieve different configurations ranging from simple to much more complicated ones.

FMS Installation & Initializing Configuration

When File Management system is installed, it initially installs and configures FSC server, FMS Master configuration file and FSC configuration file(s). For multiple deployed FSCs, FMS Master configuration information is retrieved from master FSC .For non-master FSC, address to master FSC is provided for FMS master configuration information. Initial FCC configuration is downloaded from FSC (bootstrap address of FSC is mentioned in FCC configuration file) .In FCC configuration file, FSC can be assigned to route data requests from FCC.

Below diagram depicts an installation with four FSCs deployed, FSC3 being master FSC.FCC configuration is downloaded from FSC( diagram assumes the shown FSC bootstrap address, mentioned in FCC configuration file).

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\FMS installation.jpg

FMS Master Configuration File

FMS Master Configuration file describes the FMS network, defining the group of deployed FSCs and their read-write cache location. Default values defined in fmsmaster.xml can be overridden with values defined in fsc.xml & fcc.xml file. An example of fmsmaster.xml file is as shown below:

D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\fmsmasterconfigfile.jpg

Key elements of this file include:

  • fmsenterprise –This element contains definition of FSC default, FCC default, FSC and its assigned volumes. It also includes an ID attribute that should match the unique identifier of database
    • fscdefaults – specifies where FSC read & write cache files are stored.
    • fccdefaults – specifies the default location of user’s cache.
    • fscGroup –contains list of all FSCs installed as part of LAN configuration .They cannot be deployed across WAN and file transfer between two FSCs are directly routed.
    • fsc – defines the fsc identifier and denotes its address.
    • volume -Each volume must have entry under fsc .Multiple volumes can be defined under one fsc. Volume statement specifies the root directory for each volume, which can be located on the disk or mounted over the network.
    • clientmap -Specifies that all clients on a particular network address (146 prefixed) are assigned a specific fsc (assignedfsc).
    • transientVolume-Specifies the temporary directory used by teamcenter business servers for writing and reading temporary files (generated report, PLMXML etc.) in four-tier configurations.

    FSC Configuration file

    FSC Configuration file configures an individual FMS Server Cache (FSC) .This file defines values such as address of master FSC. It can also define default values for FCCs and other FSCs. An example of fsc.xml file is as follows:

    D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\fscconfigfile1.jpg

    D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\fscconfigfile2.jpg

    Key elements of this file include:

    • fscmaster –Specifies that this FSC is the master configuration server .Other FSCs will download the FMS Master configuration file from this FSC (specifies address of this FSC with non-master FSCs)
    • fsc– Specifies the FSC ID of this installed FSC.

    FCC Configuration file

    This file specifies the (bootstrap) address of FSC configuration server (FSC) that allows the FCC configuration to be downloaded. Also routes FCC data requests to assigned FSC. An example of fcc.xml file is as follows:

    D:\blog\images\WATERMARKED\TEAMCENTER ADMINISTRATION\FMS\fccconfigfile.jpg

    Key elements of this file include:

    • parentfsc – This element identifies which parent FSC to use for initial configuration download for FCC. Multiple parentfsc can be specified to provide for failover.
    • FCC cache location – Windows client contains FCC cache at the location specified by this xml attribute.
    • assignment mode- FCC data requests is routed to assigned fsc. FSC can be assigned to FCC through two assignment mode – clientmap or parentfsc (preferred when client changes network).This element of FCC can override the default client mapping in FMS Master configuration file.

    In order to upload & download files with high performance in different possible configurations (remote user configuration, multiple FSC configuration, FSC cache configuration etc.), the three files (fmsmaster.xml, fsc.xml, and fcc.xml) can be configured effectively. In the next few posts, examples of configuring these files for different scenarios will be explained. Also, we will talk about administering FMS and other important details related to file management system.

    Also Read:

    FMS Configuration for Multiple FSCs & Cache Server

    FMS Configuration for Remote Clients

    Troubleshooting FSC Failure to Start

    Teamcenter Two-Tier & Four-Tier Architecture

    Structure Manager in Teamcenter