OpenTelemetry: Benefits, Architecture, and Best Practices

What Is OpenTelemetry?

OpenTelemetry is an observability framework that emerged from the merging of Google-sponsored OpenCensus and CNCF-sponsored OpenTracing projects. It provides a unified approach to collecting and analyzing telemetry data, including traces, metrics, and logs. Major companies contributing to the project include Google, Microsoft, Red Hat, and Amazon, among others. 

‍

The long-term objective of OpenTelemetry is to provide a comprehensive and standardized set of APIs, libraries, and instrumentation for monitoring and troubleshooting distributed systems, facilitating improved observability and performance for modern applications and services. By simplifying the observability landscape, OpenTelemetry aims to lower barriers to adoption and enhance collaboration across the industry.

‍

This is part of a series of articles about distributed tracing.

How Does OpenTelemetry Work? 

Observability is a measure of how well the internal states of a system can be inferred from its external outputs. In the context of modern software systems, observability encompasses the collection and analysis of the following:

• Metrics: Numerical representations of data points over a specified time interval. They can be used to monitor system performance, resource usage, and other quantitative aspects. Examples include CPU usage, memory consumption, and request latency. 
• Events: Discrete occurrences within a system that provide insight into its operation. They can be used to track specific actions or incidents, such as user login, errors, or configuration changes. 
• Logs: Textual records generated by applications, services, or infrastructure components. They contain detailed information about events, errors, or other notable occurrences. 
• Traces: Track the end-to-end flow of requests through distributed systems, capturing the path of execution and timing information for each operation. Traces help developers identify bottlenecks, latency issues, and service dependencies, which is crucial for optimizing and troubleshooting distributed applications.

‍

OpenTelemetry API provides a set of standardized interfaces for instrumenting applications to collect and export telemetry data in a vendor-neutral way. Key components of the API include:

• Tracer API: Responsible for generating and managing trace data. It allows developers to create and propagate spans, which represent individual operations in a trace. The Tracer API helps capture the flow of requests, measure latency, and identify service dependencies within distributed systems.
• Metric API: Enables the collection and aggregation of metrics data. It provides interfaces for defining and recording different types of metrics, such as counters, gauges, and histograms. The Metric API allows developers to monitor system performance, resource usage, and other quantitative aspects of applications.
• Context API: Deals with context propagation across process boundaries. It ensures that trace and other contextual information, such as authentication data or custom metadata, are consistently transmitted between services, enabling end-to-end visibility and correlation.
• Semantic conventions: OpenTelemetry defines a set of semantic conventions to standardize the naming, structure, and attributes of telemetry data. These conventions promote consistency and interoperability.

OpenTelemetry Architecture and Components 

OpenTelemetry currently consists of these primary components:

• A cross-language specification
• Utilities for gathering, processing, and exporting telemetry data
• SDKs tailored for each language
• Auto-instrumentation and supplementary packages

‍

By utilizing OpenTelemetry, developers can eliminate the reliance on vendor-specific tools and SDKs to generate and export telemetry data.

The Collector

This is a service that gathers, processes, and exports telemetry data from various sources. It serves as an intermediate component, enabling high-performance processing and allowing for vendor-agnostic instrumentation. The Collector can aggregate, enrich, or filter data before exporting it to various backends or monitoring platforms. This promotes flexibility and scalability while reducing the resource impact on the instrumented applications.

The Exporter

This  is a component responsible for sending collected telemetry data, such as traces, metrics, and logs, to various backends or monitoring platforms. Exporters enable data transmission in a standardized format, allowing for seamless integration with multiple observability solutions while maintaining vendor-neutrality.

Language SDKs

OpenTelemetry's language SDKs are language-specific libraries that provide developers with the necessary tools and APIs to instrument applications and services. These SDKs enable the collection, processing, and export of telemetry data (including metrics, logs, and traces) in a standardized and vendor-neutral way. Language SDKs are available for popular programming languages, such as Java, Python, Go, JavaScript, Ruby, and more, ensuring broad support across diverse applications.

Auto-Instrumentation

Automatic instrumentation refers to the process of injecting telemetry collection code into applications without requiring manual modification of the source code. It simplifies observability by reducing the development effort and minimizing the chances of errors. 

‍

Automatic instrumentation leverages language-specific techniques, such as bytecode manipulation in Java, monkey-patching in Python, or middleware in JavaScript, to capture traces, metrics, and logs from commonly used libraries and frameworks. 

‍

The underlying approach may differ between languages due to their unique runtime environments, but the goal remains consistent: to streamline and enhance observability with minimal developer intervention.

‍

Learn more in our detailed guide to OpenTelemetry architecture

OpenTelemetry Benefits and Challenges 

OpenTelemetry offers numerous benefits that make it a powerful and attractive choice for monitoring and observing modern applications and systems:

• Simple setup: OpenTelemetry is designed to streamline the process of instrumenting applications and services for observability. By providing language-specific SDKs, automatic instrumentation, and well-documented APIs, it significantly reduces the complexity and development effort required to gather telemetry data. Developers can spend less time configuring and integrating monitoring solutions and more time focusing on their application logic and performance. 
• Flexible data handling: OpenTelemetry offers a range of tools and components, such as the Collector, which enables the aggregation, enrichment, filtering, and processing of telemetry data before exporting it to various backends. This flexibility allows organizations to adapt their observability pipelines to their specific needs and requirements, optimizing data flow and reducing resource impact on instrumented applications. 
• Vendor-neutrality: By standardizing the format and transmission of data, it enables seamless integration with multiple monitoring and observability platforms, without locking users into a specific vendor or proprietary solution. This freedom to choose and switch between different backends allows organizations to select the best tools for their needs.

‍

Despite its numerous benefits, OpenTelemetry also has certain limitations that users should consider when adopting it for observability:

• Unsupported data types: OpenTelemetry primarily focuses on collecting and exporting traces, metrics, and logs, but it does not currently support other data types, such as application security (AppSec) events or code profiling information. Organizations that require these additional data types might need to rely on complementary tools or solutions to cover their complete observability and monitoring needs.
• Language stability: As a relatively young and evolving project, some language SDKs might not have reached full stability or maturity. This can lead to inconsistencies or incomplete support for certain features across different languages. 
• Maintenance: OpenTelemetry's rapid development pace and its reliance on community contributions might introduce potential maintenance challenges for users. Keeping up with updates, bug fixes, and new features can be time-consuming, especially when dealing with multiple languages and components. 

OpenTelemetry vs. Prometheus 

Prometheus is an open-source monitoring and alerting system, primarily designed for reliability and scalability. It provides a powerful query language called PromQL to analyze collected metrics data. Prometheus pulls time-series metrics data from instrumented targets, stores them efficiently, and enables real-time alerts based on predefined rules, making it widely popular for monitoring containerized and microservices-based environments.

‍

Here are some of the main ways that Prometheus differs from OpenTelemetry:

• OpenTelemetry enables instrumentation of code for generating telemetry data, while Prometheus serves as a metrics monitoring tool. Both offer client libraries for code instrumentation, but OpenTelemetry libraries deliver a comprehensive solution for generating logs, metrics, and traces, while Prometheus solely focuses on metrics.
• Prometheus features a basic visualization layer, whereas OpenTelemetry does not aim to include visualization. Data collected with OpenTelemetry can be sent to any backend analysis tool. 
• OpenTelemetry establishes the foundation for building observability practices, essential for microservices-based architectures. Utilizing Prometheus for observability requires additional tools for traces and logs.
• While Prometheus offers short-term storage and can be used alongside long-term storage solutions like Cortex or Thanos. OpenTelemetry, on the other hand, does not provide storage; instead, it offers exporters that can be configured to transmit data to the backend analysis tool of your preference.

OpenTelemetry Best Practices 

Here are some of the best practices to make the most of OpenTelemetry.

Use Attributes

Attributes in OpenTelemetry are key-value pairs that provide additional context and metadata for telemetry data, such as traces, metrics, and logs. They enrich the collected data by supplying extra information, aiding in the analysis and understanding of system behavior. The main types of attributes in OpenTelemetry include user, software, data, and infrastructure attributes.

Create a Shared Attribute Library

Creating a shared library for known attributes is useful because it promotes consistency and reusability across different applications and services. A shared library helps standardize attribute names, formats, and semantics, ensuring compatibility and simplifying data analysis. 

‍

Additionally, it reduces the chances of errors and duplication while improving maintainability, as updates to attributes can be made centrally and propagated across all instrumented components, streamlining the observability process.

Consider the Cardinality

Cardinality, in the context of observability and monitoring, refers to the number of unique values a particular data element, such as a metric, attribute, or tag, can take within a dataset:

• High-cardinality data allows for more fine-grained analysis, as it provides greater detail and context about individual events or operations. This granularity is beneficial when diagnosing complex issues or understanding specific user behavior patterns.‍
• Low-cardinality data is valuable for aggregation, as it simplifies data analysis and enables more efficient processing. Aggregated data is helpful for identifying broader trends, patterns, and anomalies across the entire system.

‍

Related Content: Read our guides on distributed tracing tools & datadog distributed tracing