Oracle Cloud Infrastructure Logging Service
The Oracle Cloud Infrastructure Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. You can configure Verrazzano to send logs to Oracle Cloud Infrastructure Logging instead of OpenSearch. For general information, see Oracle Cloud Infrastructure Logging Overview.
Set up custom logs
Verrazzano can send its logs to Oracle Cloud Infrastructure custom logs. You will need to provide two Oracle Cloud Infrastructure Log identifiers in your Verrazzano installation resource - one for Verrazzano system logs and one for application logs. Follow the steps in Creating Custom Logs to create two custom logs. Do not create an agent configuration when creating a custom log, otherwise the log records will be duplicated.
The Fluentd plug-in included with Verrazzano will use Oracle Cloud Infrastructure instance principal authentication by default. Optionally, you can configure Verrazzano with a user API signing key. API signing key authentication is required to send logs to Oracle Cloud Infrastructure Logging if the cluster is running outside of Oracle Cloud Infrastructure.
Oracle Cloud Infrastructure Logging is enabled in your cluster when installing Verrazzano. The Verrazzano installation custom resource has fields for specifying two custom logs: one for system logs and one for application logs. Here is an example Verrazzano installation YAML file for each type of credential. Note that the API references Kibana, upcoming releases will use OpenSearch Dashboards in the public API.
Override the default log objects
You can override the Oracle Cloud Infrastructure Log object on an individual namespace. To specify a log identifier on a namespace, add an annotation named
verrazzano.io/oci-log-id to the namespace. The value of the annotation is the Oracle Cloud Infrastructure Log object identifier.
Here is an example namespace.
apiVersion: v1 kind: Namespace metadata: annotations: verrazzano.io/oci-log-id: ocid1.log.oc1.iad.ns.app.example creationTimestamp: "2022-01-14T15:09:19Z" labels: istio-injection: enabled verrazzano-managed: "true" name: example spec: finalizers: - kubernetes status: phase: Active
Note that if you add and subsequently remove the annotation, then the logs will revert to the default Oracle Cloud Infrastructure Log object specified in the Verrazzano custom resource.
To search Verrazzano logs, you can use the Oracle Cloud Infrastructure Console, Oracle Cloud Infrastructure CLI, or Oracle Cloud Infrastructure SDK.
For example, using the Oracle Cloud Infrastructure CLI to search the system logs for records emitted by the
$ oci logging-search search-logs --search-query=\ "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \ where \"data\".\"kubernetes.container_name\" = 'verrazzano-application-operator' | sort by datetime desc" \ --time-start 2021-12-07 --time-end 2021-12-17
Search for all application log records in the
$ oci logging-search search-logs --search-query=\ "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \ where \"data\".\"kubernetes.namespace_name\" = 'springboot' | sort by datetime desc" \ --time-start 2021-12-07 --time-end 2021-12-17
For more information on searching logs, see the Logging Query Language Specification.
If you are not able to view Verrazzano logs in Oracle Cloud Infrastructure Logging, then check the Fluentd container logs in the cluster to see if there are errors.
$ kubectl logs -n verrazzano-system -l app=fluentd --tail=-1
If you see
not authorized error messages, then there is likely a problem with the Oracle Cloud Infrastructure Dynamic Group or IAM policy that is preventing the Fluentd plug-in from communicating with the Oracle Cloud Infrastructure API.
To ensure the appropriate permissions are in place, review the Oracle Cloud Infrastructure Logging required permissions documentation.
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.