The Ingestion Controls section is where users can tailor their log tracking infrastructure. Easily toggle your Middleware Agent logs, metrics, and trace ingestion on and off. The Middleware Host Agent will search for changes in your account configuration and automatically update the configuration YAML file and reload itself. Fine-tune your log collection process by selecting your preferred hosts for tracking infrastructure file-based logs, Middleware Agent Logs, and specify custom directory paths for monitoring your log collection.

​
Infrastructure File-Based Logs

Quickly enable OR disable the Middleware Agent from monitoring file-based logs for collection. Infrastructure filed-based logs are collected by default. To disable the collection of file-based logs, navigate to Settings, select Ingestion Controls, and press the Infrastructure File-based Logs radio button.

​
Middleware Agent Logs

Quickly enable or disable Middleware Agent log collection. Middleware Agent pod-logs are generated by the Middleware Agent.

​
Middleware Agent Log Paths

This feature is not available in the Docker Agent since Docker does not allow dynamic volume bindings.
You can dynamically select the log files you want to monitor. Add directory paths in the Middleware Agent Log Paths section. Below is the accepted regex pattern: 
'^(\\/(?:[\\w_\\-^* ]+\\/)*(?:[\\w_\\-*]+(?:\\**\\/)?)?[\\w_\\-.]+[^.]$)'
The follwing are examples of available paths:
Asterisks (**) can represent any number of directories. 
/home/ubuntu/abc.log

/home/ubuntu/*.log

/home/ubuntu/**/*.log

​
Custom Attribute Controls

This feature is only available when logs are scraped from a filesystem by the MW Host Agent.
Middleware supports standard regex rules to allow extraction of attributes and their values from log messages. The attributes scraped will be available in filters, search bars, and expanded log attributes.

​
Log Extraction Example

Learn more with the following GitHub Syntax article. Refer to the Grouping table.
Using the below log message as an example, we are going to extract four key attributes: date, process_id, level, and content. [2024-02-26 09:15:41 -0500] [671097] [ERROR] Shutting down: Master We can extract the desired attributes with the below regex: \[(?P<date>.+)\] \[(?P<process_id>.+)\] \[(?P<level>.*)\] (?P<content>.+) Given the original log message and above regex, we receive the following key and value pairs: 
date = 2024-02-26 09:15:41 -0500
process_id = 671097
level = ERROR
content =Shutting down: Master
The following image shows how this looks from within the Middleware Application:
We have a special attribute named level, which accepts ERROR, WARN, INFO, DEBUG as valid values. Based on the level attribute, logs are available under specific severity levels.

​
Sensitive Data Masking for Logs

Middleware provides users the ability to mask data at time of ingestion. This means that all data will be masked before it leaves your systems, allowing full control over security regulations and company policies. Similar to the above attribute extraction, Data Masking is also based on standard regex rules. The setting will expect 2 inputs from users as shown in the image below:
  1. A Regex Pattern for the keywords that users want to hide
  2. A Replacement Value, e.g. *****, HIDDEN, etc.
If your log content looks like: Transaction ID: 675-98-3434 \\d{3}-?\\d{2}-?\\d{4} modifies the log before sending: Transaction ID: *****

​
Metrics Ingestion

By default, MW Agent metrics are automatically ingested. To disable this feature, press the radio button and click Save. Disabling metrics will limit your overall understanding of your infrastructure vitals and performance.

​
Trace Ingestion

If you disable traces, you will in turn disable all active APMs as well.
By default, MW Agent traces are automatically ingested. To disable this feature, press the radio button and click Save. Disabling traces will limit your ability to isolate issues within your infrastructure.
Need assistance or want to learn more about Middleware? Contact us at support[at]middleware.io.