Complete ELK-stack example with Docker

I wanted a quick setup for an Elasticsearch Logstach and Kibana (ELK-)stack to work with. But searching on the internet gave me too many long-winded not really working examples.

That’s why I created this page. Use it to quickly get up-and-running with an ELK-stack of your own.

Create the file docker-compose.yml

# file: docker-compose.yml
version: "3"

services:
  elk:
    image: sebp/elk
    ports:
      - "5601:5601"
      - "9200:9200"
      - "5044:5044"
    environment:
      - MAX_MAP_COUNT=262145
      - ELASTICSEARCH_START=1
      - LOGSTASH_START=1
      - KIBANA_START=1
      - TZ="Europe/Amsterdam"
    volumes:
      - elk-data:/var/lib/elasticsearch

volumes:
  elk-data:

Now start up with docker-compose up -d. That’s it!

5601: endpoint for Kibana
9200: endpoint for elastic search

Add some security

Don’t leave your elastic-search open for everyone.

Add some basic security by adding a .htpasswd config to your webserver.

$ sudo sh -c "echo -n 'myelasticuser:' >> /etc/nginx/.htpasswd"
$ sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd"
Password:
Verifying - Password:

Add it to your webserver, like nginx.

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /var/www/html;
    index index.html index.htm;

    server_name localhost;

    location / {
        try_files $uri $uri/ =404;
        auth_basic "Restricted Content";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

Reload nginx.

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ sudo service nginx reload

Some notes

I chose the Docker image of sebp because he’s got great documentation. Go check it out!

Especially the part with the Frequently Encountered Issues.

There, you’ll see that you’ll:

• need 4GB of memory for the Docker container
• need to set the amount of virtual memory on linux by setting the max map count:sudo sysctl -w vm.max_map_count=262144

Free SSL certificates with LetsEncrypt

Getting your website on https can be done in a matter of minutes. So there is no excuse anymore to go without it. Not even on your test and dev websites.

As this example is on CentOS, it really goes for any other linux distro.

Excellent, tailor-made instructions per webserver and OS are found on the website of Certbot:
https://certbot.eff.org/

Here, a short recap of that for my own archive.

You’ll need the repel repository for this. After that, install the certbot software.

$ sudo yum install epel-release
$ sudo yum install certbot-nginx

 

Getting your website secured with SSL is now as simple as answering some questions on the following command.

Note: I’m using a method which takes a bit of downtime because LetsEncrypt is in the middle of an update. Read all about it

$ sudo certbot --authenticator standalone --installer nginx --pre-hook "service nginx stop" --post-hook "service nginx start"

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer nginx
 
Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: yoursite.pauledenburg.com
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 2


Running pre-hook command: service nginx stop
Error output from service:
Redirecting to /bin/systemctl stop nginx.service
 
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for es.git.innospense.com
Waiting for verification...
Cleaning up challenges
Running post-hook command: service nginx start
Error output from service:
Redirecting to /bin/systemctl start nginx.service
 
Deployed Certificate to VirtualHost /etc/nginx/sites-enabled/yoursite.pauledenburg.com.conf for set(['yoursite.pauledenburg.com'])
 
Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/yoursite.pauledenburg.com.conf
 
-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://yoursite.pauledenburg.com
 
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=yoursite.pauledenburg.com
-------------------------------------------------------------------------------
 
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/es.git.innospense.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/es.git.innospense.com/privkey.pem
   Your cert will expire on 2018-04-24. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:
 
   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

 

Things which might throw you an error

python-urllib3 version

First caveat for CentOS7 is that you need specific version 1.21 for urllib3. I had 1.22 installed via yum which gave me the following error.

ImportError: No module named 'requests.packages.urllib3'

You can see the currently installed version with pip:

pip freeze | grep urllib

To resolve this, first remove the old version it with yum and then add it with pip:

sudo yum remove python-urllib3 
sudo pip install -Iv https://github.com/shazow/urllib3/archive/1.21.1.tar.gz

pyOpenSSL version

Just like urllib3, pyOpenSSL was of an unsupported version.

sudo yum remove pyOpenSSL
sudo pip install pyOpenSSL

Error message stating that the CA can’t be satisfied

After running

certbot --nginx

you get the following error:

Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA.

Due to legal reasons there currently is no

From the github certbot website:

If you’re serving files for that domain out of a directory on Nginx, you can run the following command:

# Webroot method
$ sudo certbot --authenticator webroot --installer nginx \
  --webroot-path <path to served directory> -d <domain>

If you’re not serving files out of a directory (for instance if you are using proxy_pass), you can temporarily stop your server while you obtain the certificate and restart it after Certbot has obtained the certificate. This would look like:

# Temporary outage method
$ sudo certbot --authenticator standalone --installer nginx \
  -d <domain> --pre-hook "service nginx stop" --post-hook "service nginx start"

 

Install python pip

From the pip website:

pip is already installed if you’re using Python 2 >=2.7.9 or Python 3 >=3.4 binaries downloaded from python.org, but you’ll need to upgrade pip.

To install pip, run the following.

curl -Lo get-pip.py https://bootstrap.pypa.io/get-pip.py
python get-pip.py

 

SonarQube with Postgres on docker-compose

[updated 2022-08-08]

Struggling to get a working environment with SonarQube and PostgreSQL?

Use the following docker-compose file and be up and running in minutes.

It is as ‘bare’ as possible:

• use of official Docker images for both PostgreSQL and SonarQube
• no other configuration required
• use of volumes so you can backup your data

Recommended system specs

• >= 3GB of RAM

# file: docker-compose.yml

version: "3"

services:
  sonarqube:
    image: sonarqube:9-community
    # platform: linux/amd64  # uncomment this when using Mac M1
    restart: unless-stopped
    environment:
      - SONARQUBE_JDBC_USERNAME=sonar
      - SONARQUBE_JDBC_PASSWORD=v07IGCFCF83Z95NX
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonarqube
    ports:
      - "9000:9000"
      - "9092:9092"
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins

  db:
    image: postgres:14.4
    # platform: linux/amd64  # uncomment this when using Mac M1
    restart: unless-stopped
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=v07IGCFCF83Z95NX
      - POSTGRES_DB=sonarqube
    volumes:
      - sonarqube_db:/var/lib/postgresql
      # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
      - postgresql_data:/var/lib/postgresql/data

volumes:
  postgresql_data:
  sonarqube_bundled-plugins:
  sonarqube_conf:
  sonarqube_data:
  sonarqube_db:
  sonarqube_extensions:

Start this stack with the following command:

# start the containers
docker-compose up -d

You can reach your SonarQube instance at http://localhost:9000

Use the default credentials admin/admin to login.

Useful links:

• SonarQube on Docker Hub: https://hub.docker.com/_/sonarqube/
• PostgreSQL on Docker Hub: https://hub.docker.com/_/postgres/

Change mysql_ to mysqli_ functions

In the process of upgrading PHP5.3 code I had to change all deprecated mysql_* functions to their mysqli_* counterparts.

For a lot of functions the signature stayed the same.

But mysqli_query and mysqli_connect have differences. So you can’t just find and replace them.

Instead of doing this manually, I wanted to find and replace recursively while changing the order of the arguments.

In vim:

# change mysql_query(param1, param2) to: 
# mysqli_query(param2, param1)
:%s/mysql_query(\(.\{-}\),\(.\{-}\))/mysqli_query(\2, \1)/g

Using sed:

# on linux

# mysql_query(param1, param2) to 
# mysqli_query(param2, param1)
sed -i 's|mysql_query(\(.*\),\(.*\))|mysqli_query(\2, \1)|g' devices.php

# on mac (otherwise you get the 'invalid command mode' when 
# you run the sed command)

# mysql_query(param1, param2) to: 
# mysqli_query(param2, param1)
sed -i '' -e 's|mysql_query(\([^,]*\),\([^)]*\))|mysqli_query(\2, \1)|g' devices.php

Recursively changing all files:

# in all files under current directory:
# mysql_query(param1, param2) to: 
# mysqli_query(param2, param1)
fgrep -rl mysql_query . | while read file; do
  sed -i '' -e 's|mysql_query(\([^,]*\),\([^)]*\))|mysqli_query(\2, \1)|g' $file
done

Note that sed cannot do non greedy matching.

That’s why we’re searching for anything but the separator until the separator like this:

# non greedy matching with sed
\([^,]*\),

It basically states: get everything except for the comma until you get a comma (which is the first one to appear).

Disable xdebug for one run

This script disables xdebug for one run. No more error-messages like:

$ composer update
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug

and:

$ php-cs-fixer fix --dry-run .
You are running PHP CS Fixer with xdebug enabled. This has a major impact on runtime performance.
If you need help while solving warnings, ask at https://gitter.im/PHP-CS-Fixer, we will help you!

This is what you’ll get

We’ll create a script which will:

• disable xdebug
• run your command
• enable xdebug

the script we’ll name php-no-xdebug (or whatever you like)

With Xdebug (note the last line)

$ php --version
PHP 7.1.10 (cli) (built: Oct  6 2017 01:08:19) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans

Without Xdebug (note the missing last line)

$ php-no-xdebug --version
PHP 7.1.10 (cli) (built: Oct  6 2017 01:08:19) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies

The script php-no-xdebug

Create the script /usr/local/bin/php-no-xdebug with the following contents.

# file /usr/local/bin/php-no-xdebug
#!/bin/bash

php=$(which php)

# get the xdebug config
xdebugConfig=$(php -i | grep xdebug | while read line; do echo $line; exit; done)

# no xdebug? Nothing to do!
if [ "$xdebugConfig" == "" ]; then
    $php "$@"
    exit
fi

# get the configfile (which should be the first value)
# so strip off everything after the first space of the xdebug-config
xdebugConfigFile=$(php -i | grep xdebug | while read line; do echo $line; exit; done)

# test whether we got it right
if [ ! -f "$xdebugConfigFile" ]; then
    echo "No XDebug configfile found!"
    exit 1
fi

# disable xdebug by renaming the relevant .ini file
mv ${xdebugConfigFile}{,.temporarily-disabled}

# dissect the argument to extract the first one (which should be a script or an application in $PATH) from the rest
index=0
for arg in $(echo $@ | tr ' ' "\n")
do
    if [ "$index" == "0" ]; then
        firstArg=$arg
    else
      restArg="$restArg $arg"
    fi

   ((index++))
done

# check whether the command to be executed is a local PHP file or something in the $PATH like composer or php-cs-fixer
fullPath="$(which $firstArg)"
if [ "$fullPath" == "" ]; then
    # check whether it's a local file
    if [ ! -f  $firstArg ]; then
        echo "Could not find $firstArg. No such file or directory"
        exit 1
    else
        # just run the commands
        $php $@
    fi
else
    # run the command with the fullpath followed by the rest of the arguments provided
    $php $fullPath $restArg
fi

# execute the command
$php "$@"

# re-enable xdebug
mv ${xdebugConfigFile}{.temporarily-disabled,}

# test whether the conf file is restored correctly
if [ ! -f "$xdebugConfigFile" ]; then
    echo "Something went wrong with restoring the configfile for xdebug!"
    exit 1
fi

and make it executable

$ chmod +x /usr/local/bin/php-no-xdebug

That’s it! Run it like this:

$ php-no-xdebug composer update

Switch between multiple PHP versions on your mac

This article is a slimmed-down, firing-from-the-hips, get right to the action version based on the ones listed below. If you miss some background info or want a more spelled out version, please do visit these articles:

• https://getgrav.org/blog/macos-monterey-apache-multiple-php-versions

Install multiple PHP

If you want to install deprecated versions of PHP (< PHP7.2), then you’ll need to add this tap to homebrew:

brew tap shivammathur/php

#versions="php@7.2 php@7.3 php@7.4 php@8.0 php@8.1" # in bash
versions=(php@8.1 php@8.2 php@8.3) # in ZSH
for version in $versions; do
    echo "installing ${version}"
    brew install shivammathur/php/${version}
done

# install xdebug for php >= 7.2
pecl uninstall -r xdebug
pecl install xdebug

If you receive the error configure: error: Cannot find libz you need to install required libraries via XCode:

xcode-select --install 
brew upgrade

Install easy switch-script

To easily switch PHP versions, install the following script.

curl -L https://raw.githubusercontent.com/rhukster/sphp.sh/main/sphp.sh > /usr/local/bin/sphp
$ chmod +x /usr/local/bin/sphp

As I don’t add Apache on my host machine, change
apache_change=1 to apache_change=0 in the script.

vi `which sphp`

Switch PHP-version

Without arguments, this command returns useful information like which versions are installed and which is the active one right now:

$ sphp 7.4

If you need to enter your administrator password, then you probably need to disable the part where Apache is restarted (see chapter above).

Switching to php@7.4
Switching your shell
Unlinking /usr/local/Cellar/php@5.6/5.6.40... 0 symlinks removed
Unlinking /usr/local/Cellar/php@7.0/7.0.33... 0 symlinks removed
Unlinking /usr/local/Cellar/php@7.1/7.1.33... 0 symlinks removed
Unlinking /usr/local/Cellar/php@7.2/7.2.31_1... 0 symlinks removed
Unlinking /usr/local/Cellar/php@7.3/7.3.19... 25 symlinks removed
Unlinking /usr/local/Cellar/php/7.4.7... 0 symlinks removed
Linking /usr/local/Cellar/php/7.4.7... 24 symlinks created

PHP 7.4.7 (cli) (built: Jun 12 2020 00:04:10) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.7, Copyright (c), by Zend Technologies

All done!

NOTE FOR XDEBUG: if you want to use xdebug you’re switch command needs to be expanded to:

sphp 7.4 && pecl uninstall -r xdebug && pecl install xdebug

Now you’ll see that xdebug is enabled for PHP:

php -v
                  
PHP 7.4.7 (cli) (built: Jun 12 2020 00:04:10) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Xdebug v2.9.6, Copyright (c) 2002-2020, by Derick Rethans
    with Zend OPcache v7.4.7, Copyright (c), by Zend Technologies

That’s it!

Again; take a look at the great, kept up-to-date, article of Andy Miller over at his website: https://getgrav.org/blog/macos-monterey-apache-multiple-php-versions

Use events in symfony

The other day I had to add some logic right after an user was saved to the database. I ended up using events to get this done without cluttering the entity.

I first added the logic in the User-entity but I then realised this logic was not really related to the user entity itself. Or the application, for that matters.

In order to seperate concerns, I decided to create a hook after the User save-action. That would allow me to add logic at that particular time without cluttering the User entity with nonrelevant code.

This has a downside though. When you want to debug what the heck happens after the User is saved to the database, you won’t find it in the User entity. This might send you down a long code-hunt. But you will see the dispatching of the event though. So if you’re new to this, remind yourself that there can be a whole different world behind the dispatching of an event.

Setting it up consists of 3 steps:

1. create an Event class (the one that will be dispatched)
2. dispatch the event at the right time and place (after the User is saved to the database)
3. create (and subscribe) the subscriber which will take action upon the dispatched event

Create Event

The event is nothing more than a class.

The event is the object which is passed around. Therefore you want to populate the event with all the information the subscriber(s) need.

For that purpose I create a setter and a getter. The code which will dispatch the event will use the setter, the event subscriber will use the getter.

# file src/AppBundle/Event/UserCreatedEvent.php
<?php
namespace AppBundle\Event;

use AppBundle\Entity\User;
use Symfony\Component\EventDispatcher\Event;

class UserCreatedEvent extends Event
{
    private $user;

    public function setUser(User $user)
    {
        $this->user = $user;
    }

    public function getUser()
    {
        return $this->user;
    }
}

Dispatch Event

Now we decide in what moment of time we’ll dispatch (fire) the event. In our example this will be right after the user is saved to the database.

# file src/AppBundle/Entity/User.php
/* other code */

public function createUser(User $user)
{

    /* more code */

    // save the user to the database
    $this->em->persist($user);
    $this->em->flush();

    // dispatch an event where others can work with the newly created user
    $eventDispatcher = $this->container->get('event_dispatcher');
    $event = new UserCreatedEvent();
    $event->setUser($user);
    $eventDispatcher->dispatch('user.event.created', $event);

    return $user;
}

Act on event with the subscriber

Create subscriber

The subscriber is the class with the method which gets called once the event gets dispatched.

# file src/AppBundle/EventSubscriber/UserCreatedSubscriber.php
<?php
namespace AppBundle\EventSubscriber;

class UserCreatedSubscriber
{
    public function newUserCreated(UserCreatedEvent $event)
    {
        var_dump($event->getUser());
    }
}

Subscribe the subscriber

Now that we have the code for the subscriber, we need to actually subscribe the subcriber to the event. This is the glue between dispatching an event and acting upon it.

You do this in services.yml as you register it as a service.

# create a listener for the UserCreatedEvent
valuation.event.created:
  class: AppBundle\EventSubscriber\UserCreatedSubscriber
  tags:
    - { name: kernel.event_listener, event: valuation.event.created, method: newUserCreated }

That’s it!

Sources

If you want to read more:

• Symfony documentation:
  https://symfony.com/doc/current/event_dispatcher.html
• Nice dense setup: https://stackoverflow.com/a/34162603

behat: element is not clickable at point xxxx

When testing with behat on my laptop I often get the following error shoved up my face:

unknown error: Element <button id="mybutton">...</button> is not clickable at point (126, 698). Other element would receive the click: <span class="sf-toolbar-value sf-toolbar-info-piece-additional">...</span>

Cause: another element is in front of my button. My laptop has a smaller screen what makes this happen. In this case it is the Symfony toolbar which is in front of my button as I am testing it on my development environment.

Solution: scroll! (or maximize your browser window when it pops up)

# file FeatureContext.php

/**
 * Scroll HTML element  into view
 *
 * @Then I scroll element :cssSelector into view
 */
public function iScrollElementIntoView($cssSelector)
{
    // scroll the element
    $this->scrollHtmlElementIntoView($cssSelector);
}

/**
 * Scroll HTML element with the supplied ID in view so that you can click on it (for example)
 */
public function scrollHtmlElementIntoView($cssSelector)
{
    $function = <<<JS
(
    function(){
      let elem = $('$cssSelector');
      $('html, body').animate({scrollTop:elem.offset().top})
    }
)()
JS;
    try {
        $this->getSession()->executeScript($function);
    } catch (Exception $e) {
        throw new \Exception("ScrollIntoView failed");
    }
}

And you can use the following step in your test by using a CSS-selector :

# file: FeatureContext.php

Then I scroll element "button#coiCheckButton" into view

phpunit: method serves different output based argument

I had the issue that my test-double sent incorrect values when invoked with specific arguments.

It returned null on every request.

First I thought this mock would return null on every call, but that’s not the case. Then I stumbled on this post on StackOverflow https://stackoverflow.com/questions/12748607/phpunits-returnvaluemap-not-yielding-expected-results.

The solution: the map-array needs all parameter-values listed in every element. Even the optional ones. I had to add null values for the optional parameters!

use returnValueMap method to map the received arguments to an array.

function myMethod($name, $optional=null){
  // ...
}

// Define which value need to be returned when 
// called with argument 'x'
// first element is argument 'x'; the argument passed, 
// 2nd element is the optional argument of the function
// 3d element is what will be returned by the test-double
$map = [
  ['value1', null, $valueToReturn1],
  ['value2', null, $valueToReturn2]
];

$request->expects($this->any())
  ->method('getRepository')
  ->will($this->returnValueMap($map));

$request->getRepository('value1'); // will return $valueToReturn1

Without the null added to it, it won’t work.

Example for my mock with Symfony. This will return the relevant repository on a request for method getRepository and with returnvalues for arguments AppBundle:Valuation and AppBundle:ObjectInvolvement.

$map = [
    ['AppBundle:Valuation', $valuationRepository],
    ['AppBundle:ObjectInvolvement', $objectInvolvementRepository]
];
$entityManager->expects($this->any())
    ->method('getRepository')
    ->will($this->returnValueMap($map));