Sondar
Sandbox Install
DocsPricingSecurityAbout Sandbox Install Sondar

Migration · ELK / OpenSearch

Moving from the ELK stack.

Days, most of it in parsing and dashboards. Filebeat keeps running with one output changed; your grok patterns come with you; Kibana searches translate almost word for word. What does not come across is the Elasticsearch API itself, and anything that depends on it.

  1. 01

    Filebeat: change one output

    Minutes

    Point Filebeat's Logstash output at a Sondar agent, which listens for the same protocol. Inputs, modules and processors stay as they are. Filebeat's message becomes the event's text, the file path its origin and the host its host. We ran Filebeat 9.5.4 and 8.19.22 this way, at their default settings, and found every event by keyword. 

    # filebeat.yml
output.logstash:
  hosts: ["sondar-agent.internal:5000"]

# the Sondar agent
readers:
  - lumberjack_listener:
      address: ":5000"

    Sondar does not speak the Elasticsearch API. Its storage layer is descended from Elasticsearch, but nothing on its port answers Elasticsearch requests, so an elasticsearch output (from Beats, Logstash or a client library) cannot point at it. Logstash's own outputs have not been tested against Sondar; send through Filebeat or the Sondar agent instead.

  2. 02

    Parsing: keep your grok patterns

    Days

    The agent parses with grok at collect time and ships Logstash's standard pattern file, so %{COMBINEDAPACHELOG}, %{SYSLOGBASE} and the rest mean what they meant. Custom patterns go in a file beside them. Or move the parsing to search time: a regex or JSON extraction rule on a source type applies to data already stored, so it needs no re-ingest.

    Transformers · Indexes and field extraction

  3. 03

    Searches: KQL becomes SonQL

    Hours

    A Kibana search is mostly the same shape with = in place of :. As in KQL, AND binds tighter than OR; parenthesise when you mix them. Everything after the search is SonQL commands piped together, the part Kibana does with its visual builders.

    Kibana (KQL)SonQL
    status:500status=500
    status >= 500status>=500
    service:checkout and status:500service="checkout" status=500
    not level:debugNOT level="debug"
    level:(error or warn)level in ("error", "warn")
    "connection refused""connection refused"
    host:*host=*

    Searching logs with SonQL

  4. 04

    Dashboards and visualizations: rebuild them

    Days

    There is no importer for Kibana saved objects. Each panel is rebuilt from a SonQL search, saved as a report and placed on a dashboard. That is where most of the migration's time goes, and a good moment to retire the panels nobody opens.

    Dashboards

  5. 05

    Index lifecycle: set a retention

    Minutes

    An ILM policy becomes a retention period on each Sondar index. The index a new install writes to keeps 15 days; set each one to what its data needs.

    Indexes and field extraction