Category Archives for programming

Gitlab CI upload artifact fails: too large

Today I wanted to add a package-job to my Gitlab CI as instructed in this nice Gitlab tutorial.

I created the tar-file but when it came to uploading it failed with Request entity too large.

(...)
ERROR: Uploading artifacts to coordinator... too large archive  id=243 responseStatus=413 Request Entity Too Large status=413 Request Entity Too Large token=JYszbA9F
FATAL: Too large                                   
ERROR: Job failed: exit status 1

It took me some digging, but this is how I fixed this (note, the Nginx proxy was the one giving me a hard time).

Step 1: Set the maximum artifacts size

In your gitlab, go to Settings > Continuous Integration and Deployment > Maximum artifacts size (MB) and set it to the desired value. The default is 100MB.

Step 2: Set the nginx upload size

In the gitlab.rb file, mine at /etc/gitlab/gitlab.rb, set or uncomment the following line.

nginx['client_max_body_size'] = '250m'

 

 

And reconfigure gitlab to get this to work.

gitlab-ctl reconfigure

Step 3: (optional) update your proxy(!)

I run gitlab on docker containers. On the server, I run nginx as a proxy to redirect requests for gitlab to these containers.

I failed to update the proxy configuration to allow the POST-ing of data to the containers.

As I use nginx, this is the line I added. For Apache, just google and you’ll find your answer.

client_max_body_size 0;

This will set no limits on clients sending data.

For reference, this is my whole nginx vhost file.

server {
    listen 80;
    server_name git.pauledenburg.com;
    client_max_body_size 0;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Don’t forget to reload nginx.

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ sudo service nginx reload

 

sonarqube

SonarQube with Postgres on docker-compose

[updated 2022-08-08]

Struggling to get a working environment with SonarQube and PostgreSQL?

Use the following docker-compose file and be up and running in minutes.

It is as ‘bare’ as possible:

  • use of official Docker images for both PostgreSQL and SonarQube
  • no other configuration required
  • use of volumes so you can backup your data

Recommended system specs

  • >= 3GB of RAM
# file: docker-compose.yml

version: "3"

services:
  sonarqube:
    image: sonarqube:9-community
    # platform: linux/amd64  # uncomment this when using Mac M1
    restart: unless-stopped
    environment:
      - SONARQUBE_JDBC_USERNAME=sonar
      - SONARQUBE_JDBC_PASSWORD=v07IGCFCF83Z95NX
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonarqube
    ports:
      - "9000:9000"
      - "9092:9092"
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins

  db:
    image: postgres:14.4
    # platform: linux/amd64  # uncomment this when using Mac M1
    restart: unless-stopped
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=v07IGCFCF83Z95NX
      - POSTGRES_DB=sonarqube
    volumes:
      - sonarqube_db:/var/lib/postgresql
      # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
      - postgresql_data:/var/lib/postgresql/data

volumes:
  postgresql_data:
  sonarqube_bundled-plugins:
  sonarqube_conf:
  sonarqube_data:
  sonarqube_db:
  sonarqube_extensions:

Start this stack with the following command:

# start the containers
docker-compose up -d

You can reach your SonarQube instance at http://localhost:9000

Use the default credentials admin/admin to login.

Useful links:

Disable xdebug for one run

This script disables xdebug for one run. No more error-messages like:

$ composer update
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug

and:

$ php-cs-fixer fix --dry-run .
You are running PHP CS Fixer with xdebug enabled. This has a major impact on runtime performance.
If you need help while solving warnings, ask at https://gitter.im/PHP-CS-Fixer, we will help you!

This is what you’ll get

We’ll create a script which will:

  • disable xdebug
  • run your command
  • enable xdebug

the script we’ll name php-no-xdebug (or whatever you like)

With Xdebug (note the last line)

$ php --version
PHP 7.1.10 (cli) (built: Oct  6 2017 01:08:19) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans

Without Xdebug (note the missing last line)

$ php-no-xdebug --version
PHP 7.1.10 (cli) (built: Oct  6 2017 01:08:19) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies

The script php-no-xdebug

Create the script /usr/local/bin/php-no-xdebug with the following contents.

# file /usr/local/bin/php-no-xdebug
#!/bin/bash

php=$(which php)

# get the xdebug config
xdebugConfig=$(php -i | grep xdebug | while read line; do echo $line; exit; done)

# no xdebug? Nothing to do!
if [ "$xdebugConfig" == "" ]; then
    $php "$@"
    exit
fi

# get the configfile (which should be the first value)
# so strip off everything after the first space of the xdebug-config
xdebugConfigFile=$(php -i | grep xdebug | while read line; do echo $line; exit; done)

# test whether we got it right
if [ ! -f "$xdebugConfigFile" ]; then
    echo "No XDebug configfile found!"
    exit 1
fi

# disable xdebug by renaming the relevant .ini file
mv ${xdebugConfigFile}{,.temporarily-disabled}

# dissect the argument to extract the first one (which should be a script or an application in $PATH) from the rest
index=0
for arg in $(echo $@ | tr ' ' "\n")
do
    if [ "$index" == "0" ]; then
        firstArg=$arg
    else
      restArg="$restArg $arg"
    fi

   ((index++))
done

# check whether the command to be executed is a local PHP file or something in the $PATH like composer or php-cs-fixer
fullPath="$(which $firstArg)"
if [ "$fullPath" == "" ]; then
    # check whether it's a local file
    if [ ! -f  $firstArg ]; then
        echo "Could not find $firstArg. No such file or directory"
        exit 1
    else
        # just run the commands
        $php $@
    fi
else
    # run the command with the fullpath followed by the rest of the arguments provided
    $php $fullPath $restArg
fi

# execute the command
$php "$@"

# re-enable xdebug
mv ${xdebugConfigFile}{.temporarily-disabled,}

# test whether the conf file is restored correctly
if [ ! -f "$xdebugConfigFile" ]; then
    echo "Something went wrong with restoring the configfile for xdebug!"
    exit 1
fi

and make it executable

$ chmod +x /usr/local/bin/php-no-xdebug

That’s it! Run it like this:

$ php-no-xdebug composer update

Use events in symfony

The other day I had to add some logic right after an user was saved to the database. I ended up using events to get this done without cluttering the entity.

I first added the logic in the User-entity but I then realised this logic was not really related to the user entity itself. Or the application, for that matters.

In order to seperate concerns, I decided to create a hook after the User save-action. That would allow me to add logic at that particular time without cluttering the User entity with nonrelevant code.

This has a downside though. When you want to debug what the heck happens after the User is saved to the database, you won’t find it in the User entity. This might send you down a long code-hunt. But you will see the dispatching of the event though. So if you’re new to this, remind yourself that there can be a whole different world behind the dispatching of an event.

Setting it up consists of 3 steps:

  1. create an Event class (the one that will be dispatched)
  2. dispatch the event at the right time and place (after the User is saved to the database)
  3. create (and subscribe) the subscriber which will take action upon the dispatched event

symfony events

Create Event

The event is nothing more than a class.

The event is the object which is passed around. Therefore you want to populate the event with all the information the subscriber(s) need.

For that purpose I create a setter and a getter. The code which will dispatch the event will use the setter, the event subscriber will use the getter.

# file src/AppBundle/Event/UserCreatedEvent.php
<?php
namespace AppBundle\Event;

use AppBundle\Entity\User;
use Symfony\Component\EventDispatcher\Event;

class UserCreatedEvent extends Event
{
    private $user;

    public function setUser(User $user)
    {
        $this->user = $user;
    }

    public function getUser()
    {
        return $this->user;
    }
}

Dispatch Event

Now we decide in what moment of time we’ll dispatch (fire) the event. In our example this will be right after the user is saved to the database.

# file src/AppBundle/Entity/User.php
/* other code */

public function createUser(User $user)
{

    /* more code */

    // save the user to the database
    $this->em->persist($user);
    $this->em->flush();

    // dispatch an event where others can work with the newly created user
    $eventDispatcher = $this->container->get('event_dispatcher');
    $event = new UserCreatedEvent();
    $event->setUser($user);
    $eventDispatcher->dispatch('user.event.created', $event);

    return $user;
}

Act on event with the subscriber

Create subscriber

The subscriber is the class with the method which gets called once the event gets dispatched.

# file src/AppBundle/EventSubscriber/UserCreatedSubscriber.php
<?php
namespace AppBundle\EventSubscriber;

class UserCreatedSubscriber
{
    public function newUserCreated(UserCreatedEvent $event)
    {
        var_dump($event->getUser());
    }
}

Subscribe the subscriber

Now that we have the code for the subscriber, we need to actually subscribe the subcriber to the event. This is the glue between dispatching an event and acting upon it.

You do this in services.yml as you register it as a service.

# create a listener for the UserCreatedEvent
valuation.event.created:
  class: AppBundle\EventSubscriber\UserCreatedSubscriber
  tags:
    - { name: kernel.event_listener, event: valuation.event.created, method: newUserCreated }

That’s it!

Sources

If you want to read more:

  • Symfony documentation:
    https://symfony.com/doc/current/event_dispatcher.html
  • Nice dense setup: https://stackoverflow.com/a/34162603

phpunit: method serves different output based argument

I had the issue that my test-double sent incorrect values when invoked with specific arguments.

It returned null on every request.

First I thought this mock would return null on every call, but that’s not the case. Then I stumbled on this post on StackOverflow https://stackoverflow.com/questions/12748607/phpunits-returnvaluemap-not-yielding-expected-results.

The solution: the map-array needs all parameter-values listed in every element. Even the optional ones. I had to add null values for the optional parameters!

use returnValueMap method to map the received arguments to an array.

function myMethod($name, $optional=null){
  // ...
}

// Define which value need to be returned when 
// called with argument 'x'
// first element is argument 'x'; the argument passed, 
// 2nd element is the optional argument of the function
// 3d element is what will be returned by the test-double
$map = [
  ['value1', null, $valueToReturn1],
  ['value2', null, $valueToReturn2]
];

$request->expects($this->any())
  ->method('getRepository')
  ->will($this->returnValueMap($map));

$request->getRepository('value1'); // will return $valueToReturn1

Without the null added to it, it won’t work.

Example for my mock with Symfony. This will return the relevant repository on a request for method getRepository and with returnvalues for arguments AppBundle:Valuation and AppBundle:ObjectInvolvement.

$map = [
    ['AppBundle:Valuation', $valuationRepository],
    ['AppBundle:ObjectInvolvement', $objectInvolvementRepository]
];
$entityManager->expects($this->any())
    ->method('getRepository')
    ->will($this->returnValueMap($map));

 

Too large textareas

The other day I noticed really large textareas on the website. The textarea contained all the text and a lot of whitespace after it. After a short investigation I noticed two libraries fiddling with the height of textareas on the website: autosize and autogrow (a CKEditor plugin).

As the names suggests, they grow your textareas to the size of the text so you don’t have to scroll inside the textarea itself.

Cause

The large textareas reside on tabs which are not visible on pageload. And it were these textareas which grew out of bounds.

The cause is the following:

When an HTML-element is hidden, it is not possible for javascript to accurately get it’s width. And the width is needed to calculate correct dimensions for the textarea.

As noted on the autosize website (at the bottom) there are few things you can do:

  • assign a set width in pixels to the textarea in your stylesheet

  • delay autosizing until the element is revealed:

    var ta = document.querySelector(‘textarea’); ta.addEventListener(‘focus’, function(){ autosize(ta); });

  • use the autosize.update method after the element is revealed

The textareas I was dealing with did not have the issue with autosize, but rather with the autogrow plugin from CKEditor. The cause is the same.

Solution

My solution was to initiate the CKeditor on all visible textareas on the page. By putting this initiation in a function, I could call this function again when the contents of a new (bootstrap3) tab were loaded. But then again, a tab itself can contain hidden elements as well. So as a third I used library VisSense to observe relevant textareas when they become (partly) visible. I added a callback on this function to initiate CKEditor whenever the field becomes visible.

<!-- include the VisSense library -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/vissense/0.10.0/vissense.min.js"></script>

// get all relevant textareas (eg: textareas which have the 
//attribute 'data-provide' with value 'wysiwyg')
let ckeditorElements = $('textarea[data-provide="wysiwyg"]');

// add a listener which instructs to add CKEditor when the 
// field comes into view
$(ckeditorElements).each(function (index, element) {
    VisSense.VisMon.Builder(VisSense(element))
        .on('visible', function (monitor) {
            addCKEditorToElement(element);
        })
        .build()
        .start();
});

 /**
 * Add CKEditor functionality to an HTML-element 
 * (usually a textarea)
 * @param element   HTML-element like $('div.some-class')
 */
function addCKEditorToElement(element)
{
   let ckeditorTemplate = wysiwygConfigSmall;

   CKEDITOR.config.extraPlugins = 'autogrow';

    // instruct CKEditor to takeover this textarea
    CKEDITOR.replace(element);
}
1 2