What is OpenTelemetry, how to set it up on AWS, and what it costs a small team.

OpenTelemetry is used to instrument your code once and get traces, metrics and logs out in one format, OTLP, that any backend can read. OpenTelemetry (OTel) is a Cloud Native Computing Foundation project, formed from the merger of OpenTracing and OpenCensus. Its own docs put the main point bluntly: "OpenTelemetry is not an observability backend itself." It produces and moves telemetry; something else stores it and draws the graphs. The parts you will meet:

• API and SDK. The API is what your code calls to start a span or record a metric. The SDK, one per language, implements it: sampling, batching, exporting.
• Auto-instrumentation ("zero-code"). An agent or wrapper that patches common libraries, HTTP servers and clients, database drivers, AWS SDK calls, so you get traces without writing span code. For a typical Node.js or Python API this covers most of what you need.
• OTLP. The OpenTelemetry Protocol, the format every part speaks. Every backend in this guide accepts it.
• The Collector. A separate process that receives OTLP (and Prometheus, StatsD and others), processes it (batching, sampling, dropping attributes) and exports it to one or more backends.
• Semantic conventions. Standard attribute names, such as http.response.status_code, so every backend understands the same data.

There are three main signals: traces (one request's path through your services, made of spans), metrics (numbers over time) and logs. Their maturity differs by language, and that matters more than the marketing suggests:

Notes that save an afternoon:

• Use a distribution that has sigv4auth. It lives in the contrib repository, not the core Collector. The ADOT Collector and the contrib build both include it.
• Exporter name. Recent upstream releases renamed otlphttp to otlp_http and keep the old name as a deprecated alias. We used the old name above because AWS's examples still do; switch when your Collector version supports the new one.
• memory_limiter goes first, batch after it and after any sampling, as both READMEs recommend. Set GOMEMLIMIT to about 80% of the container's memory.
• Turn on Transaction Search first. AWS's docs require it before you send traces to the OTLP endpoint. The endpoint takes up to 10,000 spans or 5 MB uncompressed per request, which is why the batch size above is capped; the batch processor's default trigger is 8,192 spans with no upper limit.
• This sends traces to X-Ray and Transaction Search, not Application Signals. AWS's feature table lists Application Signals as unsupported with the upstream Collector. For Application Signals, put the CloudWatch agent in this position instead (section 08).
• The task role needs permissions, for example the CloudWatchAgentServerPolicy managed policy, and never an access key baked into the image.

Two Python traps. opentelemetry-distro defaults the OTLP protocol to gRPC, so without OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf it would talk gRPC to the collector's HTTP port 4318 (or point it at 4317 instead). And behind Gunicorn or uWSGI with several workers, the batch span processor is not fork-safe; the OpenTelemetry Python docs say to set up tracing in a post_fork hook, and AWS's Application Signals docs have a troubleshooting entry for pre-fork servers.

The SDK's default sampler is parentbased_always_on: every trace is kept unless the caller said otherwise. parentbased_traceidratio with 0.2 keeps 20% of new traces and always follows the caller's decision, so a trace is never half-recorded across services. The default propagators are W3C tracecontext and baggage; if any service still uses X-Ray headers, add xray to OTEL_PROPAGATORS on all of them during the migration.

For Application Signals, use AWS's ADOT packages instead (on ECS an init container copies them in; on EKS and Lambda they're injected for you). The variables are the same family, plus a few AWS-specific ones listed in section 08.

1. Enable Application Signals in the account once (console or the AWS::ApplicationSignals::Discovery resource), and turn on Transaction Search.
2. Store the agent config in Parameter Store: a small JSON with "application_signals": {} under traces.traces_collected and under logs.metrics_collected.
3. Add the ecs-cwagent sidecar to the task definition, from the CloudWatch agent image on ECR Public, reading that parameter.
4. Add an init container from the ADOT auto-instrumentation image for your language, which copies the agent into a shared volume, and make the app container depend on it.
5. Set the app's environment: OTEL_RESOURCE_ATTRIBUTES with service.name and deployment.environment, OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true, the exporter endpoints on localhost:4316, and the xray sampler and propagator. Python also needs OTEL_PYTHON_DISTRO=aws_distro, OTEL_PYTHON_CONFIGURATOR=aws_configurator and PYTHONPATH.
6. Give the task role the CloudWatchAgentServerPolicy managed policy, as AWS's guide does, and the execution role read access to the parameter.

ECS doesn't discover service names; what you put in service.name is what the dashboards show, so agree a naming rule first. The tasks also need a route to ECR Public to pull the images, through a NAT gateway or a public subnet.

Effort, our estimate: one to two days for the first service, including the Terraform module, then an hour or two per extra service. A vendor-neutral version with the ADOT Collector instead of the CloudWatch agent is the same shape: swap the sidecar image and pass your own config through AOT_CONFIG_CONTENT from Parameter Store.

On EKS, install the CloudWatch Observability add-on or the OpenTelemetry Operator and opt pods in with an annotation. On Lambda, add the ADOT layer or tick Application Signals. On EC2, run the CloudWatch agent or a Collector as a service next to the ADOT agent.

EKS: the CloudWatch add-on, or the OpenTelemetry Operator.

The quick path is the Amazon CloudWatch Observability EKS add-on. It installs the CloudWatch agent and turns on Container Insights and Application Signals. You then opt workloads in with one pod annotation, for example instrumentation.opentelemetry.io/inject-python: "true" (also inject-java, inject-nodejs, inject-dotnet), and the add-on injects ADOT into the pods on restart. On EKS, Application Signals discovers service and cluster names itself. One catch: from add-on version 5.0.0 Application Signals is on by default, and AWS warns it affects any OpenTelemetry APM you already run, so read its opt-out notes before upgrading a cluster that has one.

The vendor-neutral path is the OpenTelemetry Operator (or AWS's ADOT add-on, which installs the ADOT Operator). You define an OpenTelemetryCollector resource for the collectors and an Instrumentation resource for exporter, sampler and propagators, then use the same style of annotation. Most installs need cert-manager first. Give the collector's Kubernetes service account an IAM role, as AWS's example does with eksctl create iamserviceaccount.

Effort, our estimate: half a day to a day with the add-on; two to four days for the Operator with your own collectors, cert-manager and IAM. If you are still deciding between the two orchestrators, our ECS vs EKS guide covers the rest of the trade.

Lambda: a layer and one environment variable.

Tick "Application Signals" in the function's monitoring settings, or add the AWS Lambda Layer for OpenTelemetry yourself and set AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument. Layers exist for Python, Node.js, Java and .NET. The current layers don't run a separate collector inside the function; the older collector-based layers are marked "not recommended". Go, Rust and Ruby have no Application Signals layer; for Go, AWS points to instrumenting with the SDK by hand and adding the ADOT Collector layer.

Lambda specifics: sampling is parent-based by default (set OTEL_TRACES_SAMPLER to change it); the execution role needs the CloudWatchLambdaApplicationSignalsExecutionRolePolicy managed policy; and functions deployed as container images can't attach layers, so you copy the layer's contents into the image at build time. The instrumentation runs inside your function, so expect some extra init time and duration on the Lambda bill; to limit cold starts, AWS enables only the AWS SDK and HTTP instrumentations by default for Python, Node.js and Java. Measure a cold start before rolling it out to latency-sensitive functions. Effort: an hour or two including the IaC change.

EC2.

Install the CloudWatch agent with Application Signals enabled and start the app with the ADOT agent for its language, or run the ADOT or upstream Collector as a systemd service. Half a day per kind of host, on our estimate, if the AMI or user data is already automated.

How to read it:

• Free allowances distort small bills. Application Signals gives 100 GB of span data and 1 million indexed spans free for the first three months per account. New Relic's 100 GB is shared with your logs and metrics, so it fills up fast if logs go there too.
• Without Transaction Search, Application Signals bills per request, not per stored trace, as AWS's own pricing example counts it, so sampling doesn't lower that line. With Transaction Search it bills per GB, so it does. AWS's docs suggest always_on sampling with Transaction Search so you see every span; at 50 GB that's still small, but check it against your traffic.
• Grafana's APM views cost extra. The Grafana line is Tempo storage only. Grafana's Application Observability, its service-level APM pages, is billed at $0.025 per host-hour (about $18 a host a month) on Pro.
• Datadog's bill follows hosts, not spans. 50 GB is far inside the 1.5 TB that 10 APM hosts include. The full Datadog picture, and how to trim it, is in Datadog pricing, explained.
• Spans stored by Transaction Search are not charged again as log ingestion, per AWS's pricing page.
• Add the collector. On Fargate that can be $9 a month per task (section 08), which can exceed the data bill.

Metrics and logs over OTLP into CloudWatch.

OpenTelemetry metrics sent to CloudWatch cost $0.50 per GB, with 15 months of retention and no per-series charge; PromQL queries cost $0.01 per million samples, and console and dashboard queries are free. That's the way around the $0.30 per classic custom metric, which our Datadog guide warns about. Logs sent to the OTLP logs endpoint are priced like other CloudWatch Logs: $0.50 per GB in the Standard class, $0.25 in Essentials (the class formerly called Infrequent Access, which drops Live Tail, metric filters and subscription filters). On EKS, Container Insights with OpenTelemetry is $0.08 per GB. Amazon Managed Prometheus charges $0.90 per 10 million samples after the first 40 million, plus $0.04 per hour and $0.03 per 10 million samples collected if you use its managed collector instead of your own; Managed Grafana is $9 per editor and $5 per viewer a month. Where these lines sit in a whole startup bill is in how much AWS costs a startup.

Self-hosting removes the vendor invoice and adds a system to patch, size and back up; our Datadog guide prices a single-instance open-source stack and the hours it takes. For a team without an infrastructure person, a managed backend is usually cheaper in total.

SOC 2 and data residency. Traces are an easy place to leak personal data: auto-instrumentation can record URLs with query strings, and people add user emails as attributes. Decide which attributes are allowed, strip the rest in the collector, and set retention on purpose. Spans in Transaction Search live in a CloudWatch log group, so CloudWatch Logs data masking and retention settings apply to them. Each SaaS backend is one more subprocessor your auditor and your customers' DPAs will ask about; keeping telemetry in your AWS account avoids that conversation. Traces are not an audit log: access and change evidence comes from CloudTrail, covered in SOC 2 on AWS.