Signoz

SigNoz uses a columnar storage format (ClickHouse) for logs, making high-cardinality searches significantly faster than traditional text-based stores.

Is available at apm.perma.health.

The official documentation can be found here.

Credentials: developer@permaplant.net:OYD@ayu6@SV@Omd0rczMy#vrIlVn#IVt

Overview

We integrated telemetry data creation, export and collection via Opentelemetry.

Architecture

Its integration is described in the architecture proposal.

1. Metrics (Service Health & Infrastructure)

SigNoz aggregates metrics via the OTel Collector. This section is used for high-level monitoring and performance profiling.

You can query all metrics in the summary overview or in the dashboards.

  • Dashboards: Pre-built or custom JSON-defined dashboards using ClickHouse as the storage engine. You can query metrics using PromQL or the Clickhouse Query builder.

  • Key Views: * Service Overview: Automatically calculated RED metrics (Rate, Errors, Duration) derived from span data.

  • DB/External Calls: Latency and throughput for downstream dependencies (SQL queries, Redis commands, HTTP egress).

  • Usage: Metrics are powerful for more than just monitoring. They also help in setting up Alerts on specific metrics like P99 latency thresholds or error rate percentage spikes.

2. Traces (Distributed Request Context)

The Traces tab provides a granular view of request flow across distributed systems.

  • Search & Filter: Filter by tags like http.status_code, service.name,deployment.environment or custom business logic attributes injected into the span context. At the time there are no custom attributes added.
  • Flamegraphs & Gantt Charts: Analyze the execution path of a single TraceId. You can identify synchronous blocking calls or inefficient sequential processing.
  • Span Details: View raw metadata for each operation, including the service.name, operation, and any associated Events (like exceptions) attached to the span.
  • Usage: Essential for identifying tail latency and debugging failures in complex microservice orchestrations.

3. Logs (Structured Event Data)

The Logs tab shows all logs gathered and provide a view to follow logs through the whole application stack.

  • Log Management: Supports structured logging (JSON). Fields like severity_text, body, and attributes are automatically indexed.
  • Trace-Log Correlation: This is the "killer feature". If your application logs include the trace_id and span_id, SigNoz provides a one-click pivot from a specific log line to the distributed trace that generated it (and vice versa).
  • Usage: Searching for stack traces, specific user_id contexts, or debugging internal state transitions that aren't captured by spans.