Linux – Setting “LANG=C LC_ALL=C” in script has no effect on padding length for non-English characters in printf

bashlanguagelinuxtext formatting

I am working on formatting the output of a Bash script into a table such that the pipe characters used to represent the bounds of each column are spaced relative to the longest string in that column (thus lining up all of the pipe characters nicely).

Below is the relevant bit of output. Note how both for the "il2" and "ksp" rows, both pipe characters are aligned, however in the "iceland" row, which contains the non-English character 'ý', the second pipe character is left-indented one space relative to the other rows.

iceland          | Mýrdalssandur Iceland                    | steam://rungameid/1248990                                                                   | iceland-Win64-Shipping.exe
il2              | IL 2 Sturmovik: 1946                      | steam://rungameid/15320                                                                     | il2fb.exe
ksp              | Kerbal Space Program                      | steam://rungameid/220200                                                                    | KSP_x64.exe

At the very beginning of the script, I have the following:

oLang=$LANG oLcAll=$LC_ALL
LANG=C LC_ALL=C

And at the end, right before the final "done" statement, I have:

LANG=$oLang LC_ALL=$oLcAll

There is no other change of language settings throughout the script.

If in a shell I set a $string variable to "Mýrdalssandur Iceland" and run printf "%-14s is %2d char length\n" "'$string'" ${#string}, I get a character length of 21, but when I run LANG=C LC_ALL=C and then printf "%-14s is %2d char length\n" "'$string'" ${#string}, I get a character length of 22, showing that it should in-fact make a difference.

I created a barebones script to test the padding, and in that script, changing LANG does have the desired effect, even when the printf lines are contained in a function as they are in the main script in question.

I'm at a loss at this point as to why the output isn't applying the language change, so any help is appreciated.

Here is the full script:

#!/bin/bash

oLang=$LANG oLcAll=$LC_ALL
LANG=C LC_ALL=C

#Config file location
listpath="/home/$USER/.config/play/"
config="$listpath"config
games="$listpath"games.list

if ! [ -f $config ] ; then
    echo "Config file not found at \"$config\""
    exit
fi

#Vars/arrs used for script
gamecmd="$1"
prgm=""
line=""
pathf=""
ctr=0
rctr=0
appID=0
taskname=""
result=""
task=""
answer=""
longestshort=0
longestlong=0
longestpathf=0
longestservicename=0
ip=`cat $config | sed -n -e 's/^.*ip=//p'`
wr=`cat $config | sed -n -e 's/^.*wr=//p'`
display=`cat $config | sed -n -e 's/^.*display=//p'`
ControlMyMonitor=`cat $config | sed -n -e 's/^.*ControlMyMonitor=//p'`
ControlMyMonitor=${ControlMyMonitor//\\//}
declare -a short
declare -a long
declare -a path
declare -a servicename

#Shortcut locations to make adding games easier
STEAM="steam://rungameid/"
DESKTOP=`cat $config | sed -n -e 's/^.*desktop=//p'`
APPDATA=`cat $config | sed -n -e 's/^.*appdata=//p'`

function check_for_duplicate()
{
    dctr=0

    #Set delimiter to comma
    IFS=','

    while IFS= read -r line
    do
        read -a strarr <<< "$line"

        for (( n=0; n < ${#strarr[*]}; n++))
        do
            if [ "${strarr[n]}" == "$1" ]
            then
                get_record $dctr "$line"
                echo -e "Duplicate entry for \""$1"\" found:"
                echo
                print_verbose_header
                echo
                pathf="${path[$dctr]}"
                format_pathf
                    print_verbose_record "${short[$dctr]}" "${long[$dctr]}" "$pathf" "${servicename[$dctr]}"

                exit
            fi
        done
        ((dctr++))
    done < $games
}
function format_pathf()
{
    pathf="${pathf//\"/}"
    pathf="${pathf//%STEAM%/$STEAM}"
    pathf="${pathf//%DESKTOP%/$DESKTOP}"
    pathf="${pathf//%APPDATA%/$APPDATA}"
}
function get_normal_header_spacing()
{
    for (( IFS=" " i=0; i<$ctr; i++ ))
    do
        if [ "${#short[i]}" -gt "$longestshort" ]
        then
            longestshort="${#short[i]}"
        fi
        if [ "${#long[i]}" -gt "$longestlong" ]
        then
            longestlong="${#long[i]}"
        fi
        done
}
function get_verbose_header_spacing()
{
    for (( IFS=" " i=0; i<$ctr; i++ ))
    do
        if [ "${#short[i]}" -gt "$longestshort" ]
        then
            longestshort="${#short[i]}"
        fi
        if [ "${#long[i]}" -gt "$longestlong" ]
        then
            longestlong="${#long[i]}"
        fi

        pathf="${path[i]}"
        format_pathf

        if [ "${#pathf}" -gt "$longestpathf" ]
        then
            longestpathf="${#pathf}"
        fi
        if [ "${#servicename[i]}" -gt "$longestservicename" ]
        then
            longestservicename="${#servicename[i]}"
        fi
        done
}
function get_record()
{
    short[$1]=$(echo "$2" | grep -Po '([^,]+)' | head -1)
    long[$1]=$(echo "$2" | grep -Po '([^,]+)' | head -2 | tail -1)
    path[$1]=$(echo "$2" | grep -Po '([^,]+)' | head -3 | tail -1)
    servicename[$1]=$(echo "$2" | grep -Po '([^,]+)' | head -4 | tail -1)
}
function print_normal_header()
{
    printf "%-${longestshort}s   %-${longestlong}s\n" "Command" "Game Title"

    for (( i=0; i<$(($longestshort + $longestlong + 2)); i++ ))
    do
        echo -n "-"
    done
}
function print_verbose_header()
{
    printf "%-${longestshort}s   %-${longestlong}s   %-${longestpathf}s   %s\n" "Command" "Game Title" "Path" "Task Name"
        
    for (( i=0; i<$(($longestshort + $longestlong + $longestpathf + $longestservicename + 4)); i++ ))
    do
        echo -n "-"
    done
}
function print_normal_record()
{
    printf "%-${longestshort}s | %-${longestlong}s\n" "$1" "$2"
}
function print_verbose_record()
{
    printf "%-${longestshort}s | %-${longestlong}s | %-${longestpathf}s | %s\n" "$1" "$2" "$3" "$4"
}
function print_help()
{
    echo -e "Launch program on remote Windows PC and change monitor inputs automatically.\n"
    echo -e "USAGE"
    echo -e "-----"
    echo -e
    echo -e "\e[3mplay [COMMAND]\e[0m"
    echo -e "Launch the game defined by the COMMAND parameter.\n"
    echo -e "\e[3mplay -l\e[0m"
    echo -e "Print a formatted table of games saved to the games.list file excluding file paths.\n"
    echo -e "\e[3mplay -lv\e[0m"
    echo -e "Print a formatted table of games saved to the games.list file including file paths and task names.\n"
    echo -e "\e[3mplay -f [PATTERN]\e[0m"
    echo -e "Search for a matching PATTERN in the games.list file and list results excluding file paths.\n"
    echo -e "\e[3mplay -fv [PATTERN]\e[0m"
    echo -e "Search for a matching PATTERN in the games.list file and list results including file paths and task names.\n"
    echo -e "\e[3mplay -a [COMMAND] [TITLE] [PATH] [TASK]\e[0m"
    echo -e "Add a game to the list that is located at the PATH on the Windows machine, is started by using the COMMAND parameter, and can be easily referenced in the -l functionality by the game's formal TITLE, which shows up in task manager as TASK (typically a \".exe\" file).\n"
    echo -e "\tFilepath shortcuts for PATH"
    echo -e "\t---------------------------"
    printf  "\t%-10s %-1s %-50s\n" "%STEAM%" "=" "$STEAM"
    printf  "\t%-10s %-1s %-50s\n" "%DESKTOP%" "=" "$DESKTOP"
    printf  "\t%-10s %-1s %-50s\n\n" "%APPDATA%" "=" "$APPDATA"
    echo -e "\e[3mplay -r [PATTERN]\e[0m"
    echo -e "Search for a matching PATTERN in the games.list file and remove it from the games.list file. This will generate a games.list.bak file, which is the games.list file before the specified program is removed.\n"
    echo -e "\e[3mplay -?\e[0m OR \e[3mplay --help\e[0m"
    echo -e "Print this help text.\n"
}
function list_games()
{
        for (( IFS=" " i=0; i<$ctr; i++ ))
    do
        print_normal_record "${short[i]}" "${long[i]}"
        done
    echo
}
function list_games_verbose()
{
        for (( IFS=" " i=0; i<$ctr; i++ ))
    do
        #Remove/replace backslash, quote marks, and path shortcuts from path and print result in table
        pathf="${path[i]//\\/}"
        format_pathf
            print_verbose_record "${short[i]}" "${long[i]}" "$pathf" "${servicename[i]}"
        done
    echo
}
function run_game()
{
    isRunning=0

    echo "Changing monitor input and launching winrun with parameters \"start $1\""
    sudo chmon $display
    winrun "start $1"

    while [[ 1 ]]
    do
        #Wait for program to actually show up in Task Manager (accounts for user messing around in launcher
        if [ -n $(winrun "tasklist /fi \"SESSIONNAME eq CONSOLE\" /fo list" | grep -i $task) ] && [!isRunning]; then
            isRunning=1

        #Switch monitor inputs when program is no longer running
        elif [ -z $(winrun "tasklist /fi \"SESSIONNAME eq CONSOLE\" /fo list" | grep -i $task) ] && [isRunning]; then
            echo "No longer seeing $task running on Window system, changing display input."
            winrun "run $ControlMyMonitor /SetValue \\\\.\DISPLAY1\Monitor0 60 4"
            break;
        fi
    done
}

#Load games from games.list file
if [ -f $games ]
then

    #Read games list
    while IFS= read -r line
    do
        short[$ctr]=$(echo $line | awk -F, '{print $1}')
        long[$ctr]=$(echo $line | awk -F, '{print $2}')
        path[$ctr]=$(echo $line | awk -F, '{print $3}')
        servicename[$ctr]=$(echo $line | awk -F, '{print $4}')
        ((ctr++))
    done < $games
else
    #Make games list file if it does not exist
    echo $games file  not found, creating!
    touch $games
fi

#Check for script arguments
while [ "$#" != "" ]
do
    case "$1" in
        -l)#List games

            if [ "$2" ]
            then
                echo -e "Detected arguments after \""$1"\". Ignoring, as these are not used...\n"
            fi

            get_normal_header_spacing
            print_normal_header
            list_games | sort
            break;;

        -lv)#Display games list with more info
            
            if [ "$2" ]
            then
                echo -e "Detected arguments after \""$1"\". Ignoring, as these are not used...\n"
            fi

            get_verbose_header_spacing
                        print_verbose_header
            list_games_verbose | sort
                        break;;

        -f)#Find for parameter and display

            if [ ! "$2" ]
            then
                echo "Missing search parameter!" >&2
                break
            fi
            if [ "$3" ]
            then
                echo -e "Detected arguments after \""$2"\". Ignoring, as these are not used...\n"
            fi

            get_normal_header_spacing
            print_normal_header
            echo
            list_games | grep -i "$2" | sort
            break;;

        -fv)#Find for parameter and display with more info

            if [ ! "$2" ]
            then
                echo "Missing search parameter!" >&2
                break
            fi
            if [ "$3" ]
            then
                echo -e "Detected arguments after \""$2"\". Ignoring, as these are not used...\n"
            fi

            get_verbose_header_spacing
            print_verbose_header
            echo
            list_games_verbose | grep -i "$2" | sort
            break;;

        -a)#Add game to list

            if [ ! "$2" ]
            then
                echo "Missing program command string!" >&2
                break

                elif [ ! "$3" ]
                then
                    echo "Missing program title!" >&2
                    break

                    elif [ ! "$4" ]
                    then
                        echo "Missing program path!" >&2
                        break

                        elif [ ! "$5" ]
                        then
                            echo "Missing program task name!" >&2
                            break
            fi
            if [ "$6" ]
            then
                echo -e "Detected arguments after \""$5"\". Ignoring, as these are not used...\n"
            fi

            #Check for duplicate entry
            check_for_duplicate "$2"
            check_for_duplicate "$3"
            check_for_duplicate "$4"

            #If no duplicate is found, add entry
            if [ ! $match ]
            then
                pathf="${4//\\//}"
                echo ""$2","$3","$pathf","$5"" >> "$games"
                echo "Added the following record:"
                echo
                print_verbose_header
                echo
                format_pathf
                    print_verbose_record "$2" "$3" "$pathf" "$5"
            fi

            break;;

        -r)#Attempt to remove game from list

            if [ ! "$2" ]
            then
                echo "Missing search parameter!" >&2
                break
            fi
            if [ "$3" ]
            then
                echo -e "Detected arguments after \""$2"\". Ignoring, as these are not used...\n"
            fi

            rctr=0
            while IFS= read -r line
            do
                if [[ -n $(echo $line | grep "$2") ]]
                then
                    match=1

                    get_record $rctr "$line"
                    echo "Discovered the following matching record:"
                    echo
                    print_verbose_header
                    echo
                    pathf="${path[$rctr]}"
                    format_pathf
                        print_verbose_record "${short[$rctr]}" "${long[$rctr]}" "$pathf" "${servicename[$rctr]}"
                    echo                    

                    while [[ $answer != "y" && $answer != "n" ]]
                    do
                        echo -n "Remove this record? (y/n) "
                        read -r answer </dev/tty
                    done

                    if [ $answer == "n" ]
                    then
                        echo "$games not changed."
                        break
                    fi

                    echo
                    echo "Removed the following record:"
                    print_verbose_header
                    echo
                    print_verbose_record "${short[$rctr]}" "${long[$rctr]}" "$pathf" "${servicename[$rctr]}"

                    #Shift all entries beyond the removed entry down one
                    for (( i=$rctr; i<ctr; i++ ))
                    do
                        short[i]=${short[i+1]}                      
                        long[i]=${long[i+1]}
                        path[i]=${path[i+1]}
                        servicename[i]=${servicename[i+1]}
                    done

                    #Rebuild games.list
                    cp $games $games.bak
                    rm $games
                    for (( i=0; i<(ctr-1); i++ ))
                    do
                        echo "${short[i]}","${long[i]}","${path[i]}","${servicename[i]}" >> $games
                    done

                    break
                fi

                ((rctr++))
            done < $games


            if [ ! $match ]
            then
                echo -e "Did not find match for \""$2"\" in \"$games\""
            fi
            break;;

        -? | --help)#Print help text

            if [ "$2" ]
            then
                echo -e "Detected arguments after \""$1"\". Ignoring, as these are not used...\n"
            fi

            print_help
            break;;

        *)#Search for matching game

            if [ ! "$1" ]
            then
                print_help
                break;
            fi

            if [ "$2" ]
            then
                echo -e "Detected arguments after \""$1"\". Ignoring, as these are not used...\n"
            fi

            echo -n "Checking if \""$1"\" matches a known game in games.list... "
            shopt -s nocasematch
            for (( i=0; i<$ctr; i++ ))
            do
                if [[ "$1" == "${short[i]}" ]]; then
                    path[i]="${path[i]//%STEAM%/$STEAM}"
                    path[i]="${path[i]//%DESKTOP%/$DESKTOP}"
                    path[i]="${path[i]//%APPDATA%/$APPDATA}"
                    prgm="${path[i]}"
                    task="${servicename[i]}"
                    break
                fi
            done
            shopt -u nocasematch

            #If there was a match, run program. Otherwise, run first arg as is.
            if [ "$prgm" != "" ]
            then
                echo "Found!"
                run_game "$prgm"
            else
                echo "Not found!"
                run_game "$1"
            fi

            break;;
    esac
    shift

#LANG=$oLang LC_ALL=$oLcAll

done

Best Answer

Bash's printf field widths work in bytes, not in characters. Because ý is stored as two bytes in the UTF-8 encoding, printf will think it's two columns wide – regardless of your locale setting.

(Yes, that is inconsistent with string operations like ${#var}, most likely because the printf command is just a direct passthrough to the C printf() function which is always byte-oriented and bash has no say in how it works.)

However, if printf worked with characters, then LC_CTYPE=C would be counterproductive – you would want the two bytes c3 bd to be treated as representing a single character, just as they're drawn in a single cell, and therefore you would want to keep LC_CTYPE set to something.UTF-8. (If bash thinks a string is wider than it's actually displayed, then it will add less padding, and so the column will appear to be narrower than it should be, just like in your situation.)

I would recommend using the column tool that is part of most Linux systems – it does understand UTF-8 input, and it would save you from needing to calculate the column widths in general. In fact it would even work with characters that are more than 1 cell wide – for example:

(
    printf '%s\t%s\t%s\t%s\n' "iceland" "Mýrdalssandur Iceland" "steam://rungameid/1248990" "iceland-Win64-Shipping.exe"
    printf '%s\t%s\t%s\t%s\n' "il2" "IL 2 Sturmovik: 1946" "steam://rungameid/15320" "il2fb.exe"
    printf '%s\t%s\t%s\t%s\n' "ksp" "Kerbal Sp🚀ce Pr🚀gram" "steam://rungameid/220200" "KSP_x64.exe"
    printf '%s\t%s\t%s\t%s\n' "ffxiv" "ファイナルファンタジーXIV(FF14)" "steam://rungameid/39210" "ffxiv.exe"
) | column -s $'\t' -t -N SHORT,LONG,URL,EXE -W LONG -o ' │ '

The fancy options like -N or -o require at least version 2.30 of the util-linux package, but the basic column -t -s ... will still work with older versions.

But if you do not want to use any external tools, then you'll need to manually pad the strings – making use of the ${#var} expansion which tells you the number of characters (although not the number of terminal cells, and therefore not as good as column):

printf '%s%*s | %s%*s | %s%*s | %s%*s\n' \
  "$short" $((maxshort - ${#short})) "" \
  "$long"  $((maxlong - ${#long}))   "" \
  "$url"   $((maxurl - ${#url}))     "" \
  "$exe"   $((maxexe - ${#exe}))     "" ;

Note: In both cases above, whether you use 'column' or bash's ${#var}, you need the data to be interpreted as UTF-8 to get the correct character count, and therefore you must not set LANG or LC_ALL to "C".

Related Question