Simple filebeat logging with Elastic Beanstalk

Simple filebeat logging with Elastic Beanstalk

Tom Weiss, Mon 04 September 2017

We host a lot of different services for our clients, and to ensure they all run well it's important that we can collect all of our logs together in a single instance. We've been using an Elastic stack for quite a while and have recently migrated to the excellent hosted solution from logz.io.

With logz.io we install the filebeat client on each of our servers and all of the logs from that instance go straight to logz.io where we monitor and alert on them in a single place.

So far, so good. However, one of our latest client's systems runs on Elastic Beanstalk which has given us an interesting problem. Elastic Beanstalk is a great scalable environment from Amazon Web Services, and one of its best features is a single configuration for all environments - development, live, staging, etc. - with the differences between the environment managed solely through environment variables.

However, the example code provided to configure logging using Filebeat from Elastic Beanstalk does not work with environment variables. It took us a long time to work out exactly what the problem was and wanted to share the solution here.

To understand why, we first need to take a look at how the-the configuration files for Elastic Beanstalk, the .ebextensions files work.

From the documentation on AWS, we can see that we have six different commands:

  • The packages section installs different packages as required
  • The groups and users sections creates Linux groups and users
  • The sources section will download files from a public archive
  • The files section specifies files to write onto the system
  • The commands section specifies commands that should run after the files section
  • The services section determines services will run when the instance starts
  • The container_commands section specifies commands that should run once the environment is set up and ready to use

The problem with all of the examples we've seen online is that they initiate filebeat in the commands section which is before the environment variables are set. As the environment variables are set up as part of the container, we need to switch the filebeat configuration to the container_commands sections and everything works!

You can see our setup below

files:
  "/etc/filebeat/filebeat.yml":
    mode: "000755"
    owner: root
    group: root
    content: |
        ############################# Filebeat #####################################
        filebeat:
            prospectors:
                - 
                    paths:
                        - /var/log/eb-commandprocessor.log
                    fields:
                        logzio_codec: plain
                        token: ___YOUR_TOKEN_HERE___
                        environment: evname
                    fields_under_root: true
                    ignore_older: 3h
                    document_type: eb-commands
                - 
                    paths:
                        - /var/log/eb-version-deployment.log
                    fields:
                        logzio_codec: plain
                        token: ___YOUR_TOKEN_HERE___
                        environment: evname
                    fields_under_root: true
                    ignore_older: 3h
                    document_type: eb-version-deployment
                - 
                    paths:
                        - /var/log/eb-activity.log
                    fields:
                        logzio_codec: plain
                        token: ___YOUR_TOKEN_HERE___
                        environment: evname
                    multiline:
                        pattern: '^[[:space:]]'
                        negate: false
                        match: after
                    fields_under_root: true
                    ignore_older: 3h
                    document_type: eb-activity
                -
                    paths:
                        - /var/log/httpd/error_log
                    fields:
                        logzio_code: plain
                        token: ___YOUR_TOKEN_HERE___
                        environment: evname
                    fields_under_root: true
                    ignore_older: 3h
                    document_type: apache_errors
                -
                    paths:
                        - /var/log/httpd/access_log
                    fields:
                        logzio_code: plain
                        token: ___YOUR_TOKEN_HERE___
                        environment: evname
                    fields_under_root: true
                    ignore_older: 3h
                    document_type: apache
                -
                    paths:
                        - /opt/python/log/supervisord.log
                    fields:
                        logzio_code: plain
                        token: ___YOUR_TOKEN_HERE___
                        environment: evname
                    fields_under_root: true
                    ignore_older: 3h
                    document_type: supervisord
            registry_file: /var/lib/filebeat/registry
        ############################# Output ##########################################
        output:
            logstash:
                hosts: ["listener.logz.io:5015"]
                ssl:
                  certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

container_commands:
  01_download_filebeat:
    command: "curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm"
    cwd: /home/ec2-user
  02_install_filebeat:
    command: "rpm -ivh --replacepkgs filebeat-5.0.1-x86_64.rpm"
    cwd: /home/ec2-user
  04_get_certificate:
    command: "mkdir -p /etc/pki/tls/certs"
    cwd: /etc/pki/tls/certs   
    command: "wget https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt"
  05_stop_filebeat:
    command: "/etc/init.d/filebeat stop"
  06_set_environment_name:
    command: "source /opt/python/current/env; sed -i -e 's/evname/'$ENVIRONMENT_NAME'/g' /etc/filebeat/filebeat.yml"
  07_start_filebeat:
    command: "/etc/init.d/filebeat start"

Need help? Get in touch...

Sign up below and one of our data consultants will get right back to you

Other articles about aws


Dativa is a global consulting firm providing data consulting and engineering services to companies that want to build and implement strategies to put data to work. We work with primary data generators, businesses harvesting their own internal data, data-centric service providers, data brokers, agencies, media buyers and media sellers.

145 Marina Boulevard
San Rafael
California
94901

Registered in Delaware

Thames Tower
Station Road
Reading
RG1 1LX

Registered in England & Wales, number 10202531