computers Archives - Paul Edenburg

Category Archives for computers

Restore too large database dumpfile

My backup file of my database was too large to handle.

Separating schema and data

I fixed restoring it by cutting it up in a seperate ‘schema’ and ‘data’ sql file.

# use LC_ALL=C because there are multibyte chars in the dump
LC_ALL=C sed '/^LOCK TABLES/,/^UNLOCK TABLES/d' backup.sql > schema.sql
LC_ALL=C sed '/^LOCK TABLES/,/^UNLOCK TABLES/p' backup.sql > data.sql

I also needed to disable the foreign key check during the import as the order of data entry was incorrect.


echo "SET FOREIGN_KEY_CHECKS=0;" > data_new.sql
cat data.sql >> data_new.sql
echo "SET FOREIGN_KEY_CHECKS=1;" >> data_new.sql
mv data_new.sql data.sql

Separate large tables into their own dump files

The data dump was still large. So I moved out the tables which have a lot of data (like events, audits and telescope_* tables)

# table: events
LC_ALL=C grep -A 1000000 "LOCK TABLES \`events\`" backup.sql | \
  grep -B 1000000 -m 1 "UNLOCK TABLES" > events_data.sql

# table: audits
LC_ALL=C grep -A 1000000 "LOCK TABLES \`audits\`" backup.sql \
  | grep -B 1000000 -m 1 "UNLOCK TABLES" > audits_data.sql

# table telescope_*
LC_ALL=C grep -A 1000000 "LOCK TABLES \`telescope_" backup.sql | grep -B 1000000 "UNLOCK TABLES" > telescope_data.sql

# now remove those tables from the data.sql
cp data.sql data.sql.backup
LC_ALL=C sed -i '' '/^LOCK TABLES `events`/,/^UNLOCK TABLES/d' data.sql
LC_ALL=C sed -i '' '/^LOCK TABLES `audits`/,/^UNLOCK TABLES/d' data.sql
LC_ALL=C sed -i '' '/^LOCK TABLES `telescope_/,/^UNLOCK TABLES/d' data.sql

Format Laravel log messages for Papertrail

You can edit the information that is in your log messages when Laravel writes to the log.

I wanted to add the application name to the log messages.

// in config/logging.php
...

        'papertrail' => [
            'driver' => 'monolog',
            'level' => env('LOG_LEVEL', 'debug'),
            'handler' => SyslogUdpHandler::class,
            'handler_with' => [
                'host' => env('PAPERTRAIL_URL'),
                'port' => env('PAPERTRAIL_PORT'),
            ],
            'tap' => [App\Logging\CustomizeLogFormatter::class],
        ],
...

Then create a new file ./app/Logging/CustomizeLogFormatter.php

<?php

// file ./app/Logging/CustomizeLogFormatter.php
declare(strict_types=1);

namespace App\Logging;

use Monolog\Formatter\LineFormatter;

class CustomizeLogFormatter
{
    public function __invoke($logger): void
    {
        foreach ($logger->getHandlers() as $handler) {
            $handler->setFormatter(new LineFormatter(
                '%channel%.%level_name%: '.config('app.name').': %message% %context% %extra%\n'
            ));
        }
    }
}

Docker helper function to get a shell

Want to get your bash shell in a running docker container?

run: deit application
(deit, as in: ‘Docker Exec -IT’)

and will show you all running containers which have ‘application’ in the name.

Add the following function to your ~/.bashrc or ~/.zshrc

# Function to display matching containers and allow user selection
function deit {
    container_query="$1"
    shift 1
    container_command="${@:-bash}"  # Default to 'bash' if no command is provided

    # Get the list of matching container names
    matching_containers=($(docker ps --format '{{.Names}}' | grep "$container_query"))
    count=${#matching_containers[@]}

    if [ "$count" -eq 0 ]; then
        echo "No containers found with '$container_query' in the name."
        return 1
    elif [ "$count" -eq 1 ]; then
        container_name="${matching_containers[0]}"
        echo "Found container: $container_name"
        echo "Executing: docker exec -it $container_name $container_command"
        docker exec -it "$container_name" $container_command
    else
        echo "Multiple containers found with '$container_query' in the name:"
        for i in "${!matching_containers[@]}"; do
            echo "$((i + 1)). ${matching_containers[i]}"
        done

        while true; do
            read -p "Choose the number of the container you want to use: " choice
            if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "$count" ]; then
                container_name="${matching_containers[$((choice - 1))]}"
                echo "Selected container: $container_name"
                echo "Executing: docker exec -it $container_name $container_command"
                docker exec -it "$container_name" $container_command
                break
            else
                echo "Invalid choice. Please try again."
            fi
        done
    fi
}

Install multiple versions of PHP on Ubuntu 24.04

Works on other versions of Ubuntu as well.

Let’s install PHP8.1, PHP8.2 and PHP8.3 on our Ubuntu machine.

Add the following script as ~/install-multiple-php-versions.sh.

#!/bin/bash

# file: ~/install-multiple-php-versions.sh

# Check if the Ondřej Surý PPA repository is already added
if ! grep -q "ondrej/php" /etc/apt/sources.list /etc/apt/sources.list.d/*; then
    echo "Adding Ondřej Surý PPA repository to get the latest PHP versions..."
    sudo add-apt-repository -y ppa:ondrej/php
    sudo apt update
fi

# Function to install a PHP version
install_php_version() {
    local version="$1"
    sudo apt install -y "php$version" "php$version-cli" "php$version-fpm"  "php$version-common" "php$version-zip" "php$version-mbstring" "php$version-opcache"
}

# Start installation message
echo "Starting installation of PHP versions..."

# Install different PHP versions
versions=("8.0" "8.1" "8.2" "8.3")
for version in "${versions[@]}"; do
    install_php_version "$version"
    echo "PHP $version installed successfully."
done

# Completion message
echo "All PHP versions have been installed successfully."

# Set default PHP version (optional)
# sudo update-alternatives --set php /usr/bin/phpX.X  # Replace X.X with desired version

# Check installed PHP versions
php -v

# Check all installed PHP versions
dpkg -l | grep php | awk '{print $2}' | grep -E "^php[0-9]+\.[0-9]+$"

Allow it to execute and then execute:

chmod +x ~/install-multiple-php-versions.sh
. ~/install-multiple-php-versions.sh

Add the script to switch php versions over at /usr/local/bin/sphp

#!/bin/bash

# file: /usr/local/bin/sphp
# switch PHP versions with: sphp 8.3

# Function to check if a command exists
command_exists() {
    type "$1" &> /dev/null
}

# Function to show the current PHP version
show_current_php() {
    php --version | head -n 1
}

# Main function to change PHP version for the current user
switch_php() {
    if [ -z "$1" ]; then
        echo "Usage: sphp <version>"
        echo "Example: sphp 8.1"
        exit 1
    fi

    VERSION=$1
    PHP_BIN="php$VERSION"

    if ! command_exists $PHP_BIN; then
        echo "PHP $VERSION is not installed. Install it first using apt."
        exit 1
    fi

    # Check if update-alternatives --set-user is supported
    if update-alternatives --help | grep -q "\-\-set-user"; then
        # Use update-alternatives --set-user if it's supported
        update-alternatives --set-user php /usr/bin/$PHP_BIN
    else
        # Alternative method using symlinks if --set-user is not available
        mkdir -p ~/.local/bin
        ln -sf /usr/bin/$PHP_BIN ~/.local/bin/php

        # Add ~/.local/bin to PATH if it's not already there
        if [[ ":$PATH:" != *":$HOME/.local/bin:"* ]]; then
            echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
            export PATH="$HOME/.local/bin:$PATH"
        fi
    fi

    echo "Switched to PHP $VERSION for user $USER"
    show_current_php
    echo "Note: You may need to open a new terminal or run 'source ~/.bashrc' to see the change."
}

# Execute the main function
switch_php "$1"

And make it executable

sudo chmod +x /usr/local/bin/sphp

How to set up your Mac mini to reboot without a monitor and keyboard

If you’re using your Mac mini as a headless server, you might have found that you cannot login using ssh or Screen Sharing after a reboot.

To overcome this, only reboot using the command sudo fdesetup authrestart

This will prompt you for the user to login with after reboot, but it won’t log in to the desktop.
It just shows the login screen on the Mac mini, but now you are able to remotely login to the Mac mini after a reboot without a monitor or keyboard attached.

# to reboot and have remote services accessible:
sudo fdesetup authrestart

# tip: make an alias of 'reboot'
echo 'sudo fdesetup authrestart' >> ~/.bash_profile

# after re-login, you can now just type: reboot

Launch shell in Docker container via command line function

Add the following function somewhere where it will be picked up by your shell.

For example: ~/.bash_profile

Then use it like so:

$ deit api
Multiple containers found with 'api' in the name:
1. srv-captain--strippenkaart-api.1.ytwsvciqerfm3k0821tpowx23
2. srv-captain--verkooprapport-api.1.we0g3exhke86q2k2qgikzkmyk
3. srv-captain--trainingsomgeving-api.1.624q0sm2qkpbqxyn37osiq2er
4. srv-captain--funnelmonitor-api.1.phukkl9slxifjcvg3yx2ykdda
Choose the number of the container you want to use:

This is the bash function:

# in file ~/.bash_profile
# 'deit' as in: docker exec -it 
function deit {
    container_query="$1"
    shift 1
    container_command="${@:-bash}"  # Default to 'bash' if no command is provided

    # Get the list of matching container names
    matching_containers=($(docker ps --format '{{.Names}}' | grep "$container_query"))
    count=${#matching_containers[@]}

    if [ "$count" -eq 0 ]; then
        echo "No containers found with '$container_query' in the name."
        return 1
    elif [ "$count" -eq 1 ]; then
        container_name="${matching_containers[0]}"
        echo "Found container: $container_name"
        echo "Executing: docker exec -it $container_name $container_command"
        docker exec -it "$container_name" $container_command
    else
        echo "Multiple containers found with '$container_query' in the name:"
        for i in "${!matching_containers[@]}"; do
            echo "$((i + 1)). ${matching_containers[i]}"
        done

        while true; do
            read -p "Choose the number of the container you want to use: " choice
            if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "$count" ]; then
                container_name="${matching_containers[$((choice - 1))]}"
                echo "Selected container: $container_name"
                echo "Executing: docker exec -it $container_name $container_command"
                docker exec -it "$container_name" $container_command
                break
            else
                echo "Invalid choice. Please try again."
            fi
        done
    fi
}

npm install: ERR_SSL_CIPHER_OPERATION_FAILED and ERR_SSL_CIPHER_OPERATION_FAILED

I got these errors after trying to run npm install.

# npm install

...
npm ERR! gyp ERR! UNCAUGHT EXCEPTION
npm ERR! gyp ERR! stack TypeError: Cannot assign to read only property 'cflags' of object '#<Object>'
npm ERR! gyp ERR! stack     at createConfigFile (/app/node_modules/node-gyp/lib/configure.js:117:21)
npm ERR! gyp ERR! stack     at /app/node_modules/node-gyp/lib/configure.js:84:9
npm ERR! gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:191:23)
npm ERR! gyp ERR! System Linux 6.6.12-linuxkit
npm ERR! gyp ERR! command "/usr/local/bin/node" "/app/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
npm ERR! gyp ERR! cwd /app/node_modules/node-sass
npm ERR! gyp ERR! node -v v20.11.1
npm ERR! gyp ERR! node-gyp -v v7.1.2
npm ERR! gyp ERR! Node-gyp failed to build your package.
npm ERR! gyp ERR! Try to update npm and/or node-gyp and if it does not help file an issue with the package author.
npm ERR! Build failed with error code: 7

I did the following to fix these

installed system packages
updated node to the current version (not LTS)
update npm to the latest version
update node_gyp to the latest version
updated all versions in package.json to the latest versions
removed package-lock.json file
removed node_modules directory
cleaned npm cache
ran npm install again

I installed the following packages:

# apt-get update && apt-get install -y ca-certificates curl gnupg zip unzip git software-properties-common rsyslog

Upgraded node, npm, node_gyp and cleared cache

 # npm cache clean -f \
  && npm install -g n \
  && n current
# npm install -g npm@latest
# npm update -g node-gyp
# rm -rf node_modules package-lock.json

Then, update all package versions in package.json to their latest version.

And then tried again, this time it worked.

# npm install

Did this work for you as well? Had to do less? More? Let me know and I’ll update this post.

Show your query with values in Laravel 10 Eloquent

When debugging your eloquent model queries, it’s often nice to know which SQL query is executed in the end.

By default, Laravel won’t show you the values used in the query. Instead, it shows you a questionmark ? instead of the actual values. This is because of the parameter bindings in PDO.

Let’s create a ->toSqlWithBindings() macro so you can have the full query.

Create the macro as ServiceProvider

<?php

// file app/Providers/MacroServiceProvder.php

declare(strict_types=1);

namespace App\Providers;

use Carbon\Carbon;
use Illuminate\Database\Eloquent\Builder as EloquentBuilder;
use Illuminate\Database\Query\Builder;
use Illuminate\Support\ServiceProvider;
use Illuminate\Support\Str;

class MacroServiceProvider extends ServiceProvider
{
    public function boot(): void
    {
        Builder::macro('toSqlWithBindings', function () {
            /** @var Builder $this */
            $bindings = array_map(
                fn ($parameter) => (is_string($parameter) || $parameter instanceof Carbon) ? "'{$parameter}'" : $parameter,
                /** @var Builder $this */
                $this->getBindings()
            );

            return Str::replaceArray(
                '?',
                $bindings,
                $this->toSql()
            );
        });

        /** @var Builder $this */
        EloquentBuilder::macro('toSqlWithBindings', fn () => $this->toBase()->toSqlWithBindings());
    }
}

the serviceprovider file containing the macro which gives us the toSqlWithBindings() method on eloquent queries

Use it

Switching from Sanctum to JWT

I’ve been debugging for days to get my Ionic app working with the Laravel API backend.

The issue: logging in worked fine, but every subsequent API request failed with a 401.

The reason (I guess): the Ionic/capacitor app works with a referrer of capacitor://localhost. I think this messes up the cookie that is set to maintain the session.

So I had to switch over to something without cookies. Having already spend days on the issues, I went to work with JWT as I’ve done so before.

I followed this tutorial to get me a head-start: https://www.positronx.io/laravel-jwt-authentication-tutorial-user-login-signup-api/

Generate random values in Postman to use in your tests

When you want your tests to be able to run whenever you want, you should use values which are random.

In Postman, click on the name of Collection and then open the ‘Pre-request Script’ tab.

There, add the following:

// get a random number between a minimum and a maximum
// gives you current datetime with milliseconds like 2022810_171012_174
postman.setGlobalVariable("getCurrentDate",  () => {
  const date=new Date(); 
  return String(date.getFullYear())  
      + String(date.getMonth()+1) 
      + String(date.getDate()) 
      + '_' 
      + String(date.getHours() < 10 ? "0"+date.getHours() : date.getHours()) 
      + String(date.getMinutes() < 10 ? "0"+date.getMinutes() : date.getMinutes()) 
      + String(date.getSeconds() < 10 ? "0" + date.getSeconds() : date.getSeconds())
      + '_' 
      + String(date.getMilliseconds())
})

You can now use this function in your tests. This enables you to make your strings (like emailaddresses) random by adding the current datetime to it.

To use it, open your test, click on the ‘Pre-request Script’ tab and add the following.

var currentDate = eval(pm.globals.get("getCurrentDate"))();
var randomEmail = `postman-${currentDate}@pauledenburg.com`;
pm.environment.set("randomEmail", randomEmail);

You can now use the generated value in the body of your POST-request by referencing it as {{randomEmail}}

1 2 3 … 5 Next »