Cloud Agent
The Cloud Agent comes in handy when you have data sources on your firewall-protected network. When installed in the same network, the Cloud Agent initiates web socket connections to the CData Virtuality SaaS. After that, CData Virtuality uses the web socket connections as tunnels to connect to the actual data sources in your private network.
API Key
Before connecting to the CData Virtuality Server, the Cloud Agent must give it a valid API key. It is stored in the CLOUD_AGENT_CLIENT_API_KEY option and can be obtained using the following SQL statement:
SELECT optionValue FROM "SYSADMIN.OptionValues" WHERE NAME = 'CLOUD_AGENT_CLIENT_API_KEY'
The API key is generated automatically on the CData Virtuality Server startup if it does not already exist, and a new value can be generated using the SYSADMIN_VDB.generateCloudAgentApiKey() procedure.
Example
Here is a data source creation example:
CALL SYSADMIN.createConnection(name => 'testDatasource', jbossCliTemplateName => 'postgresql', connectionOrResourceAdapterProperties => 'cloudAgent=true,db=test_tables,user-name=postgres,password=Password1!,check-valid-connection-sql="select 1"') ;;
CALL SYSADMIN.createDatasource(name => 'testDatasource', translator => 'postgresql', modelProperties => 'importer.tableTypes="TABLE,VIEW",importer.useFullSchemaName=FALSE,importer.importIndexes=TRUE,importer.schemaPattern=public,importer.defaultSchema=public', translatorProperties => '', encryptedModelProperties => '', encryptedTranslatorProperties => '') ;;
Note host
and port
connection properties are replaced with the cloudAgent=TRUE
directive, indicating to the CData Virtuality Server that this is a data source connection accessed through the Cloud Agent.
On the server side, connection object is created with host as localhost
, and port as a random port. This means that from the Postgres driver perspective, the presence of the Cloud Agent tunneling is transparent, i.e. as if it is connecting to a Postgres instance located at localhost and listening on random port. Upon connection creation, a server socket with the mentioned random port is also created and is listening for connections.
Once a connection from the CData Virtuality Server is initiated on the localhost server socket, a socket is created that accepts the connection and the open socket is placed in the queue to be further handled by the Cloud Agent server side logic.
If there is a Cloud Agent connection for the data source, the Cloud Agent server side takes the socket from the queue and starts routing the traffic from/to it through the Cloud Agent.
If there is no Cloud Agent connection for the data source, the open socket remains in the queue for some time waiting for Cloud Agent to connect, and is closed and removed from the queue after a timeout. This is also useful in situations when the CData Virtuality Server is starting up but there is no Cloud Agent connection to it, and without closing the socket that accepted the connection, some drivers wait indefinitely for the response preventing the server to start up (because this is not the same as when the driver cannot reach the target data source at all, because we always accept the connection at localhost
in case of Cloud Agent connections, but we just do not respond with anything on the socket before we 'forward' the request to the Cloud Agent).
The CLOUD_AGENT_HOST_ADDRESS option value can be used to specify a different host other than the localhost if needed.
This is the Cloud Agent configuration for the connection above:
dvsaas.ws.url=ws://localhost:8080/cloud-agent
client.api.key=f5ba2a5c-5246-4c4d-a05b-2b44790a78d3
datasources=testDatasource_VDB_ID_1
testDatasource_VDB_ID_1.host=localhost
testDatasource_VDB_ID_1.port=15432
On the Agent side, we also append VDB ID, because multiple data sources with the same name can exist in different VDBs on the server side.
Here is also an example of the web service data source creation:
CALL SYSADMIN.createConnection(name => 'wsDatasource', jbossCliTemplateName => 'ws', connectionOrResourceAdapterProperties => 'cloudAgent=true,cloudAgentWsHost="catfact.ninja"') ;;
CALL SYSADMIN.createDatasource(name => 'wsDatasource', translator => 'ws', modelProperties => '', translatorProperties => '', encryptedModelProperties => '', encryptedTranslatorProperties => '') ;;
and the Cloud Agent configuration for it:
dvsaas.ws.url=ws://localhost:8080/cloud-agent
client.api.key=f5ba2a5c-5246-4c4d-a05b-2b44790a78d3
datasources=wsDatasource_VDB_ID_1
wsDatasource_VDB_ID_1.host=catfact.ninja
wsDatasource_VDB_ID_1.port=443
wsDatasource_VDB_ID_1.useSSL=true
Web service data sources support available since v4.7
Configuration
The configuration directory contains the information on how to connect to the CData Virtuality SaaS as well as to data sources being proxied by the Cloud Agent. All files with the .properties
extension found in the directory are loaded. If not explicitly specified via configDir
system property, properties are searched by default in the config
folder in the working directory. Any configuration changes in the directory are dynamically detected and loaded without the need to restart the Cloud Agent (scanning for changes is done every 30 seconds).
Here is an example properties file:
dvsaas.ws.url=ws://localhost:8080/cloud-agent
client.api.key=someApiKey
datasources=someDatasourceName_VDB_ID_1, anotherDatasourceName_VDB_ID_1
someDatasourceName_VDB_ID_1.host=localhost
someDatasourceName_VDB_ID_1.port=15432
anotherDatasourceName_VDB_ID_1.host=localhost
anotherDatasourceName_VDB_ID_1.port=443
anotherDatasourceName_VDB_ID_1.useSSL=true
Parameter | Description |
---|---|
| The URL of the CData Virtuality SaaS to which the Cloud Agent would initiate web socket connections. The URL format is |
| Contains a comma-separated list of names of all data sources proxied by the agent |
| The host address of the data source being proxied by the Cloud Agent |
| The port number of the data source being proxied by the Cloud Agent |
|
|
|
|
Configuration Updates
The Cloud Agent listens for changes in the configuration folder and automatically refreshes its configuration when changes are detected.
The Cloud Agent sends its configuration on startup as well as on any changes.
The CData Virtuality Server can send configuration to the Cloud Agent too.
clientId
(a random UUID) is generated on a agent instance on startup if it does not already exist and stored in theclient_id
file in the agent configuration directory. It is used to identify an instance so that it can be distinguished from other instances in case there is a cluster of agents.Also, if there is a configuration error (like invalid data source definition, etc), the Cloud Agent still connects to the CData Virtuality Server if the CData Virtuality SaaS url and API key are correct, so that the (invalid) configurations can be inspected and fixed through the CData Virtuality SaaS UI too if the user would like to do so.
Cloud Agent Web Socket Connection Types
There are three 'types' of web socket connections that Cloud Agent establishes with the CData Virtuality Server:
Master connections: Used to establish the initial connection to the CData Virtuality Server, as well as exchange the Agent configuration with the CData Virtuality Server.
Data source management connections: Used by the CData Virtuality Server to notify the Agent when actual data source connections are initiated by the CData Virtuality Server.
Data source connections: Used to exchange data between the CData Virtuality Server and actual data sources.
For example, for the data source above:
Agent establishes data source management web socket connection with the CData Virtuality Server for the
testDatasource_VDB_ID_1
data source.The CData Virtuality Server wants to establish a connection to the proxied Postgres instance. It sends the session id to the Agent through the management connection.
The Agent establishes the data source web socket connection with the CData Virtuality Server and provides the session id.
the CData Virtuality Server starts sending actual data source data through that connection.
The Agent receives the first package, opens the connection to the actual target data source and starts exchanging the packages between the actual data source and the CData Virtuality Server.
This architecture makes sure that if one web socket connection gets overloaded, broken or closed for any reason, it does not affect other connections which continue to function normally.
System Procedures
There are several special system procedures for working with the Cloud Agent:
SYSADMIN_VDB.getCloudAgentConnectionStatus
SYSADMIN_VDB.updateCloudAgentConfiguration(IN file string, IN configuration string)
cSYSADMIN_VDB.getCloudAgentConfiguration(OUT clientId string NOT NULL, OUT file string NOT NULL, OUT configuration string)
For detailed information, please refer to SYSADMIN_VDB Procedures.
Logging
By default, application logs are written to the standard output and in logs
folder in the working directory with daily rotation (max 30 log files; current log file is named cloud-agent.log
, older ones contain the date in the name e.g. cloud-agent-2023-11-21.log
). The following configuration parameters can be provided to customize logging behavior:
logDir
system property to specify logs folder (i.e.-DlogDir="<path to log directory>"
)log4j.configurationFile
system property to specify custom log4j configuration file (i.e.-Dlog4j.configurationFile="<path to log4j config file>"
) to change e.g. log level, log format, rotation strategy, etc.
See Also
Cloud Agent Configuration for a description of the Cloud Agent-related options