Adobe’s Use of OpenTelemetry Collector for Observability
Adobe leverages the OpenTelemetry Collector as a central hub for managing their vast amounts of trace and metrics data, simplifying the developer experience.
Chris Featherstone and Shubhanshu Surana from Adobe showcased the OpenTelemetry Collector’s versatility and utility during their talk at the Open Source Summit North America.
The primary use case for Adobe is tracking extensive observability data collected by the company, including metrics (with 330 million unique series per day), span data (equating to 3.6 terabytes per day), and log data (exceeding 1 petabyte per day).
Featherstone, a senior manager for software development, explained that while not all data flows through his team or the OTel collector, a significant portion does. Adobe’s company structure comprises numerous acquisitions, each with its own preferences for cloud providers, tools, and technologies. This creates challenges when trying to stitch a trace across various clouds and vendors. To overcome this, Adobe adopted the OpenTelemetry Collector in 2019 while building a distributed tracing platform based on Jaeger agents.
Starting from April 2020, Adobe gradually replaced Jaeger agents with the OTel Collector. Initially focused on trace ingestion, they expanded its usage to include metrics in September 2021 and are planning to incorporate logs as well.
Adobe’s team primarily utilizes OpenTelemetry libraries, particularly auto-instrumentation in Java, to instrument their applications. They also perform application enrichment by incorporating Adobe-specific data and enhancing data pipelines as it flows into the collector. Custom extensions and processors are utilized, and configuration is managed through GitOps whenever possible.
Featherstone highlighted the dynamic nature of the collector, as it can extend to multiple destinations while handling a single set of data. This versatility makes it the “Swiss Army knife of observability” for Adobe. The team at Adobe, known as the developer productivity team, aims to facilitate better and faster code development for developers.
For Java services, Adobe provides a base container that includes OpenTelemetry Java instrumentation by default. This configuration ensures that any Java service running in Kubernetes at Adobe automatically participates in tracing. Metrics, however, require additional effort to retrieve, although Adobe has simplified the process. While Java represents approximately 75% of their workload, they aim to implement similar concepts for Node.js, Python, and other languages.
Managing the Data
Regarding data management, Adobe performs enrichment and ensures sensitive information is not included in their tracing or metrics data. They employ various processors within the OpenTelemetry Collector to eliminate specific fields, enrich data with additional attributes, and prevent service name collisions. Adobe has its own service registry, where each service has a unique ID attached to its name, allowing engineers to easily identify and locate services within the tracing backend.
Additionally, Adobe sends data to multiple export destinations, enabling engineering teams to work with their preferred backends and tools. The introduction of the OpenTelemetry Collector and the OTLP format has simplified data routing and reduced the need for significant changes in engineering teams’ backend or application code.
Adobe’s future plans involve improving data quality by eliminating unnecessary data and implementing rate limiting for spans at the edge. They are also exploring trace-first troubleshooting approaches and aiming to integrate OpenTelemetry logging libraries with core application libraries. The team seeks to leverage OTel collectors as sidecars to transmit metrics, traces, and logs, and they are exploring the use of the new connector component and developing a trace sampling extension at the edge to enhance data quality.
In conclusion, Adobe appreciates the plug-in-based architecture of the OpenTelemetry Collector, as it enables sending data to different destinations with a single binary. The flexibility provided by the collector’s extensions and processors empowers Adobe to work with data in versatile ways. Overall, Adobe views OpenTelemetry as.