Centralized log management and deep event search
Bring agent and agentless logs into one searchable, self-hosted event pipeline.
01
The operational problem
Security evidence is often split between host files, journald, cloud object storage and vendor APIs. Different formats make investigations slow, while arbitrary nested JSON and long retention periods demand storage designed for large analytical workloads. A useful platform must preserve the original record and still make important fields searchable.
02
How Tranzify Watch helps
Tranzify Watch receives events from Linux agents and scheduled S3 or API sources. Each source writes to an explicit dataset. Raw payloads remain available, while versioned parsers turn text lines or JSON into normalized fields. ClickHouse supports time-bounded analytical search, full-text candidates and filters over promoted or dynamically extracted fields. The query builder combines nested AND and OR groups with equality, comparison and pattern operators, so analysts can move from a broad message search to a precise field such as payload.process_id.
Operational guide
What this use case covers
A practical deployment model that connects technical telemetry with ownership, validation, and measurable operations.
Operational context
Centralized log management across agents, S3 buckets, APIs, text files, and JSON sources. The design begins with an explicit asset scope, operational owner, risk level, expected event volume, and retention objective instead of a collection-everything default. The first design decision is to define ownership, the systems in scope, the expected signal volume, and the response that operators must be able to perform from the collected evidence.
Data and evidence
Ingest raw messages with source metadata, dataset, parser revision, timestamps, and normalized fields while preserving arbitrary nested JSON for exact field search. Raw evidence remains available beside normalized fields so an analyst can verify the source and explain every alert or incident. Keep original evidence available alongside normalized fields so analysts can verify a finding, rebuild a parser, and explain why an alert or incident was created.
Measurable result
Analysts search one evidence trail instead of moving between hosts, cloud consoles, archives, and application-specific viewers. Useful indicators include coverage, freshness, failed collections, false-positive rate, investigation time, and the percentage of assets on an approved policy. Measure the result through coverage, collection freshness, failed checks, alert quality, investigation time, and the percentage of systems operating under an approved policy.
Implementation
A production-ready workflow
Start with a controlled scope, prove data quality, and expand only after the operating team can act on the result.
Define scope
Document the business question, included assets, exclusions, owners, escalation path, retention, and acceptance criteria for centralized log management across agents, s3 buckets, apis, text files, and json sources.. Record exclusions and success criteria before enabling production collection.
Collect safely
Use checkpoints, deterministic event identifiers, decompression limits, parser versions, and a dead-letter path so retries do not create silent gaps or duplicates. Begin with a representative pilot and verify permissions, network paths, rate limits, timeouts, retries, and rollback before wider rollout. Start with a representative group and confirm permissions, limits, and network paths.
Structure data
Preserve the original value, assign a dataset, validate types, and map only stable fields. Nested JSON and unstructured text must remain searchable even when no dedicated parser exists. Preserve the raw value whenever a transformation changes its representation.
Evaluate
Reconcile object counts and offsets, test multiline and malformed records, and confirm that raw and normalized representations remain linked. Test expected data, malformed input, missing fields, delayed delivery, duplicate records, partial outages, and a realistic peak volume. Test expected and unexpected inputs, including missing fields, delays, and partial failures.
Operate
Turn the result into a documented operating action: assign an owner, define notification and escalation, record evidence, and review the workflow after material infrastructure changes. Assign an owner, document escalation paths, and review the workflow after material infrastructure changes.
Engineering controls
Security, quality, and capacity
These controls keep the deployment explainable and stable as event volume, retention, and the number of monitored systems grow.
Security boundary
Use least privilege, signed and versioned configuration, encrypted transport, secret redaction, append-only audit records, and strict separation between monitoring and remote administration. Treat credentials, headers, attachment contents, and collected payloads as sensitive operational data.
Collection cadence
Set collection frequency per signal: seconds for lightweight health telemetry, minutes for operational state, and hours for expensive inventory, package, or historical checks. Use faster intervals only for signals whose response value justifies the additional CPU, network, and storage cost.
Quality assurance
Reconcile object counts and offsets, test multiline and malformed records, and confirm that raw and normalized representations remain linked. Surface the last successful run, policy or parser revision, rejected records, queue delay, and validation reason in the interface. Failed validation must remain visible and actionable instead of silently producing incomplete evidence.
Capacity planning
Partition ClickHouse by time and dataset, batch inserts, control dynamic-key cardinality, and use retention tiers instead of indexing every JSON path equally. Plan partitions, batching, backpressure, retention tiers, query limits, and cardinality controls before production volume arrives. Review growth by dataset and source, then tune retention and expensive searches before capacity becomes an incident.
Common questions
Planning and operating the use case
Answers for teams evaluating an on-premise rollout or replacing fragmented monitoring tools.
How should a production rollout start?
Begin with a small, representative group and an approved baseline. Document the business question, included assets, exclusions, owners, escalation path, retention, and acceptance criteria for centralized log management across agents, s3 buckets, apis, text files, and json sources.. Compare the collected result with the source system before expanding coverage, and keep a documented rollback path for policy changes.
How is data quality verified?
Reconcile object counts and offsets, test multiline and malformed records, and confirm that raw and normalized representations remain linked. Surface the last successful run, policy or parser revision, rejected records, queue delay, and validation reason in the interface. Operators should be able to see the last successful collection, the parser or rule version, and any rejected records without searching server logs.
How can resource consumption be controlled?
Set collection frequency per signal: seconds for lightweight health telemetry, minutes for operational state, and hours for expensive inventory, package, or historical checks. Partition ClickHouse by time and dataset, batch inserts, control dynamic-key cardinality, and use retention tiers instead of indexing every JSON path equally. Plan partitions, batching, backpressure, retention tiers, query limits, and cardinality controls before production volume arrives. Separate lightweight health signals from expensive inventory, package, or historical operations and give each one an appropriate schedule.
Which security principle is most important?
Use least privilege, signed and versioned configuration, encrypted transport, secret redaction, append-only audit records, and strict separation between monitoring and remote administration. Apply least privilege, retain an audit trail for configuration changes, and never expose secrets in URLs, command history, downloadable reports, or user-visible logs.