Middleware supports the ingestion of APM traces and logs from the DataDog Agent. We use their Dual Shipping feature to redirect your APM traces and logs into Middleware.
This feature is supported in DataDog’s Linux and Kubernetes Agent

​
Prerequisites

1

DataDog Agent - Traces

DataDog Agent version 6.7.0 or above
2

DataDog Agent - TCP Logs

DataDog Agent version 6.6 or above
3

DataDog Agent - HTTP Logs

DataDog Agent version 6.13 or above

​
Linux Agent

​
Add Trace Config

Add the following additional_endpoints within the apm_config section in your datadog.yaml file:
If you already have the apm_config section in your datadog.yaml file, add an additional endpoint in the same section. Datadog agent does not support multiple apm_config sections.
The Linux DataDog Agent is configured by default in the following YAML file: /etc/datadog-agent/datadog.yaml
datadog.yaml
apm_config:
  additional_endpoints:
  "https://datadog.middleware.io":
  - "<Your MW API Key>"

​
Add Log Config

Add the following additional_endpoints within the logs_config section in your datadog.yaml file:
Unlike the APM Traces, you only need to provide your hostname (datadog.middleware.io) without the protocol scheme (https://)
datadog.yaml
logs_config:
    use_http: true
    additional_endpoints:
    - api_key: "<Your MW API Key>"
        Host: "datadog.stage.env.middleware.io"
        Port: 443
        is_reliable: true

​
Restart DataDog Agent

Run the following code in your terminal to restart the DataDog Agent:
Shell
sudo systemctl restart datadog-agent

​
Kubernetes Agent

The Datadog Kubernetes agent supports Helm and Operator methods of installation. Middleware can ingest traces and logs from the Datadog agent installed using any of these methods.

​
Helm chart

If you have installed the Datadog Agent using their Helm Chart, you must add the following section in your datadog-values.yaml file that you created from step 3 here:
You can remove either the logs_config or the apm_config from the below file depending on what data you would like to send to Middleware.
datadog-values.yaml
agents:
  useConfigMap: true
  customAgentConfig:
    logs_config:
      container_collect_all: true
      use_http: true
      additional_endpoints:
        - api_key: lzfywamjxpfaepytxwcavxgnxmkwebkuccyi
          Host: datadog.middleware.io
          Port: 443
          is_reliable: true
    apm_config:
      additional_endpoints:
        "https://datadog.middleware.io":
          - "<Your MW API Key>”
Below is an example of the full datadog-values.yaml file that will send logs and APM traces to the Middleware platform:
datadog-values.yaml
datadog:
  site: us5.datadoghq.com
  kubelet:
    tlsVerify: false
  apiKeyExistingSecret: datadog-secret
  logs:
    enabled: true
    containerCollectAll: true
agents:
  useConfigMap: true
  customAgentConfig:
    logs_config:
      container_collect_all: true
      use_http: true
      additional_endpoints:
        - api_key: "<Your MW API Key>"
          Host: "datadog.middleware.io"
          Port: 443
          is_reliable: true
    apm_config:
      additional_endpoints:
        "https://datadog.middleware.io":
          - "<Your MW API Key>"

​
Amazon ECS on EC2 Cluster

Middleware can ingest Logs and APM Traces from the DataDog Agent running on ECS clusters that use EC2 instances.
For more information on ingesting Log and APM Traces using Amazon ECS with DataDog, navigate here.

​
Add Trace Config

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping APM traces in the DataDog Agent.
The DD_APM_ENABLED and DD_APM_ADDITION_ENDPOINTS environment variables are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition. 
"environment": [
   ...
   {
     "name": "DD_APM_ENABLED",
     "value": "true"
   },
   {
     "name": "DD_APM_ADDITIONAL_ENDPOINTS",
     "value": "{\"https://datadog.middleware.io\": [\"<Your MW API Key>\"]}"
   },
 ]
Ensure that you have replaced <Your MW API Key> with your desired Middleware API key from your Middleware account.

​
Add Log Config

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping Logs in the DataDog Agent.
The four environment variables mentioned above are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition. 
"environment": [
  {
    "name": "DD_LOGS_CONFIG_USE_HTTP",
    "value": "true"
  },
  {
     "name": "DD_LOGS_ENABLED",
     "value": "true"
   },
   {
     "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
     "value": "true"
   },
   {
     "name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
     "value": "[{\"api_key\": \"<Your MW API Key>\", \"Host\": \"datadog.middleware.io\", \"Port\": 443, \"is_reliable\": true}]"
   }
 ]
Ensure that you have replaced <Your MW API Key> with your desired Middleware API key from your Middleware account.

​
Update Agent Service

Once you have updated the ECS task definition for the Datadog Agent, update the relevant Datadog Agent Service to redeploy the agent with your new configuration. APM traces and logs will start flowing into your Middleware account.

​
FAQ

​
How do you stop sending APM traces and logs to DataDog and only send them to Middleware?

If you want to stop sending APM traces and logs to Datadog, you can change the api_key (or environment variable DD_API_KEY) in the /etc/datadog/datadog.yaml file to something invalid.
The Datadog Agent does not work if you comment out your api_key or set it to an empty value.
Below is an example of setting your api_key and site to invalid values:
datadog-values.yaml
api_key: using middleware
site: example.com

​
I can’t see my APM traces and logs on Middleware.

  • Execute the following command to ensure the DataDog Linux Agent is in an active state:
Shell
sudo systemctl status datadog-agent
  • Verify that your Middleware API key and target are correct. The apm_config target is http://datadog.middleware.io whereas the logs_config Host field value is datadog.middleware.io (without https://).
  • Make sure you only have one apm_config and logs_config section in the following configuration file: /etc/datadog/datadog.yaml
  • Check your DataDog Kubernetes Agent. Ensure the datadog-agent and cluster-agent pods are operational with a liveness and readiness check for both pods.
  • Check your Kubernetes Helm chart installation method by checking the datadog-values.yaml and ensure that the agent section is at the same level as the datadog section. The agent section should NOT be inside the datadog section.
Need assistance or want to learn more about Middleware? Contact us at support[at]middleware.io.