Posts on tag: cli

I started to work on tablizer three years ago and lots of new features and changes have accumulated since. So, here’s the update due.

First of all, I migrated the repo toe Codeberg and left Github behind because Microsoft supports Trump and supporting Trump means to support a fucking fascist asshole. And MS scrapes every repo there to feed their incarnation of the “AI” bubble, Copilot, to which I decline to consent.

So let’s look at some of the innovations since 2022:

Help and Documentation

There’s now a short and detailed help. The short help is shown when you use the parameter -h or when there’s some usage error. As of v1.6.0 it looks like this:

$ tablizer -h
tablizer [regex,...] [-r file] [flags]
-c col,...   show specified columns                  -L  colorize rows
-k col,...   sort by specified columns               -j  read JSON input
-F col=reg   filter field with regexp                -v  invert match
-T col,...   transpose specified columns             -n  numberize columns
-R /from/to/ apply replacement to columns in -T      -N  do not use colors
-y col,...   yank columns to clipboard               -H  do not show headers
--ofs char   output field separator                  -s  specify field separator
-r file      read input from file                    -z  use fuzzy search
-f file      read config from file                   -I  interactive filter mode
-x col,...   use custom headers                      -d  debug
-o char      use char as output separator            -g  auto generate headers

-K /pattern/foreground[:background]/ colorize pattern of output
-O org  -C CSV  -M md  -X ext  -S shell  -Y yaml  -J json  -P template
-a sort by age   -i sort numerically   -t sort by time   -D sort descending order
-m show manual   -v  show version      --help  show detailed help

The detailed help is shown, when you ask for it using --help, which is much longer these days, hence the short help:

$tablizer --help
Manipulate tabular output of other programs

Usage:
  tablizer [regex,...] [-r file] [flags]

Operational Flags:
  -c, --columns string               Only show the speficied columns (separated by ,)
  -v, --invert-match                 select non-matching rows
  -n, --numbering                    Enable header numbering
  -N, --no-color                     Disable pattern highlighting
  -H, --no-headers                   Disable headers display
  -s, --separator <string>           Custom field separator (maybe char, string or :class:)
  -k, --sort-by <int|name>           Sort by column (default: 1)
  -z, --fuzzy                        Use fuzzy search [experimental]
  -F, --filter <field[!]=reg>        Filter given field with regex, can be used multiple times
  -T, --transpose-columns string     Transpose the speficied columns (separated by ,)
  -R, --regex-transposer </from/to/> Apply /search/replace/ regexp to fields given in -T
  -K  --regex-colorizer /from/color/ colorize pattern of output (color: fg[:bg])
  -j, --json                         Read JSON input (must be array of hashes)
  -I, --interactive                  Interactively filter and select rows
  -g, --auto-headers                 Generate headers if there are none present in input
  -x, --custom-headers a,b,...       Use custom headers, separated by comma

Output Flags (mutually exclusive):
  -X, --extended                     Enable extended output
  -M, --markdown                     Enable markdown table output
  -O, --orgtbl                       Enable org-mode table output
  -S, --shell                        Enable shell evaluable output
  -Y, --yaml                         Enable yaml output
  -J, --jsonout                      Enable JSON output
  -C, --csv                          Enable CSV output
  -A, --ascii                        Default output mode, ascii tabular
  -P, --template <tpl>               Enable template mode with template <tpl>
  -L, --hightlight-lines             Use alternating background colors for tables
  -o, --ofs <char>                   Output field separator, used by -A and -C. 
  -y, --yank-columns                 Yank specified columns (separated by ,) to clipboard,
                                     space separated

Sort Mode Flags (mutually exclusive):
  -a, --sort-age                     sort according to age (duration) string
  -D, --sort-desc                    Sort in descending order (default: ascending)
  -i, --sort-numeric                 sort according to string numerical value
  -t, --sort-time                    sort according to time string

Other Flags:
  -r  --read-file <file>             Use <file> as input instead of STDIN
      --completion <shell>           Generate the autocompletion script for <shell>
  -f, --config <file>                Configuration file (default: ~/.config/tablizer/config)
  -d, --debug                        Enable debugging
  -h, --help                         help for tablizer
  -m, --man                          Display manual page
  -V, --version                      Print program version

As you might already know, tablizer has a manual page as well, which can be read when the parameter --man has been supplied. In earlier versions, the text was piped into the external program more. Today we’re using our own builtin tui pager based on bubbletea viewport.

Input processing

Tablizer accepts a wide range of tabular input formats, which you can split using the -s option, which also accepts a regular expression. While this is a powerful feature, not everyone is familiar with regexes. Therefore you can use pre-defined split regexes:

• :tab: Matches a tab and eats spaces around it.
• :spaces: Matches 2 or more spaces.
• :pipe: Matches a pipe character and eats spaces around it.
• :default: Matches 2 or more spaces or tab. This is the default separator if none is specified.
• :nonword: Matches a non-word character.
• :nondigit: Matches a non-digit character.
• :special: Matches one or more special chars like brackets, dollar sign, slashes etc.
• :nonprint: Matches one or more non-printable characters.

Beside tabular input you may also use JSON input, which must consist of an array of hashes using the -j option.

The processing of file input has changed as well. Previously you could specify a pattern and one or more files and tablizer attempted to determine which was what. But this was cumbersome and errorprone code and I ditched it. To specify a file to read from, use the option -r now.

Table Headers

Previously headers has just been numberized and to tell tablizer to use a header, you had to supply it’s number. This has been completely overhauled.

First of all, header numbering is still there and these numbers can still be used, but it’s not being displayed by default anymore. If you want to see numberized headers, add the -n option, which now enables it.

However, in most of cases you’ll not need it and can just refer to the header names. Let’s look at some examples, shall we?

Given the following CSV file (which we’ll tabularize using -s,):

$ cat t/testtable6.csv 
Date,Account Number,Subject,Amount
20250101,968723487,Dogs Medication Invoice 9919292,-450.00
20250103,172747812,Tax return tax id HHD813D/12564H,+912.14
20250105,987122711,Car repair order 020123,-299.45
20250108,731217273,Rent - 12234 Sunset Blvd,-2960.00

we can reduce the number of displayed columns with -c:

$ cat t/testtable6.csv | tablizer -s, -cdate,acc.*,amount
Date      Account Number  Amount    
20250101  968723487       -450.00   
20250103  172747812       +912.14   
20250105  987122711       -299.45   
20250108  731217273       -2960.00

So, -c expects a comma-separated list of column names (case insensitive) and it supports regular expressions as well, which is very handy if there are particular long names etc.

However, you can still use numbers and you can even mix them:

$ cat t/testtable6.csv | tablizer -s, -cdate,acc.*,4
Date      Account Number  Amount    
20250101  968723487       -450.00   
20250103  172747812       +912.14   
20250105  987122711       -299.45   
20250108  731217273       -2960.00 

Sometimes you’ll have input without headers. Let’s simulate it by removing the headers of our CSV input:

$ grep -v Date t/testtable6.csv | tablizer -s, -g -c1,3
1         3                                 
20250101  Dogs Medication Invoice 9919292   
20250103  Tax return tax id HHD813D/12564H  
20250105  Car repair order 020123           
20250108  Rent - 12234 Sunset Blvd

The option -g (--auto-headers) generates numberized headers when specified. Everything else can be done using these numbers.

However, these numbers look awkwards, especially if you want to use the output to generate a report to be sent to upper management or the like. In such cases you can use specify custom headers instead:

$ grep -v Date t/testtable6.csv | tablizer -s, -x Datum,Kontonummer,Verwendungszweck,Betrag -cdatum,betrag
Datum     Betrag    
20250101  -450.00   
20250103  +912.14   
20250105  -299.45   
20250108  -2960.00

Here I have used german headers.

Cell Filtering

A couple of new possibilities to filtering of cells have been added since 2022. It was already possible to filter rows using a regular expression as program argument.

Today you can filter by singular cells (or fields) using the -F option. It takes a field name as argument with a regular expression like:

-F 'field=regexp'

A good example for the benefit of this feature for example is a table like this:

$ cat table
ID                                    VERSION  STATUS   
3cc7b723-7cad-485e-9612-327470df3908  13       Running  
727eb035-8678-43e4-b144-f4024cc4db07  13       Running  
96377897-66bf-4226-8c8a-dd07c351140c  15       Running  
a0bb2d7c-6e67-48e6-9162-4c4c5054c820  14       Running  
e91778cf-5474-492e-9a14-70298732c300  15       Running  
f16caf54-b488-4c58-9d70-140412f6aa62  16       Running

Now, when you want to get a list of rows with version 14 and use the pattern argument, you’ll get:

$ cat table | tablizer 14
ID                                    VERSION  STATUS   
727eb035-8678-43e4-b144-f4024cc4db07  13       Running  
96377897-66bf-4226-8c8a-dd07c351140c  15       Running  
a0bb2d7c-6e67-48e6-9162-4c4c5054c820  14       Running  
e91778cf-5474-492e-9a14-70298732c300  15       Running  
f16caf54-b488-4c58-9d70-140412f6aa62  16       Running 

Not, what we wanted, because the string 14 matches other parts of the rows and we didn’t get only rows with version 14. To achieve this, do:

$ cat table | tablizer -Fversion=14
ID                                    VERSION  STATUS   
a0bb2d7c-6e67-48e6-9162-4c4c5054c820  14       Running

You can use the option multiple times and negation is possible as well, e.g.:

$ cat table | tablizer -Fversion!=15
ID                                    VERSION  STATUS   
3cc7b723-7cad-485e-9612-327470df3908  13       Running  
727eb035-8678-43e4-b144-f4024cc4db07  13       Running  
a0bb2d7c-6e67-48e6-9162-4c4c5054c820  14       Running  
f16caf54-b488-4c58-9d70-140412f6aa62  16       Running

Interactive Filtering

This is a really fun feature! From time to time I stumbled upon a situation, where neither row filter or cell filter were successful to filter the rows I wanted. Therefore there’s now an interactive filter invoked with the option -I:

Here you can see the interactive mode in action. The help is being shown (normally it is hidden, show it with the ? key). As you can see you can filter, sort and manually select rows. Pretty handy!

Cell Modification

Sometimes you might need to modify certain cell contents. Of course you can use external tools such as sed or sd. However, due to the nature of tabular output, patterns to replace just the content of a cell might get very complicated. You can also use the venerable teip to work on cell level, which is a very powerful tool.

However, for small manipulations you can use the new builtin options -R and -T. The option -T marks the columns on which to operate and -R specifies the search/replace operation. Multiple manipulations can be applied.

Given the following CSV input:

$ cat t/testtable6.csv
Date,Account Number,Subject,Amount
20250101,968723487,Dogs Medication Invoice 9919292,-450.00
20250103,172747812,Tax return tax id HHD813D/12564H,+912.14
20250105,987122711,Car repair order 020123,-299.45
20250108,731217273,Rent - 12234 Sunset Blvd,-2960.00

We want to modify the date format:

$ cat t/testtable6.csv | tablizer -s, -Tdate -R '/(....)(..)(..)/$1-$2-$3/'
Date        Account Number  Subject                           Amount    
2025-01-01  968723487       Dogs Medication Invoice 9919292   -450.00   
2025-01-03  172747812       Tax return tax id HHD813D/12564H  +912.14   
2025-01-05  987122711       Car repair order 020123           -299.45   
2025-01-08  731217273       Rent - 12234 Sunset Blvd          -2960.00

As you can see, only the Date column has been touched, not any other columns.

More output options

You can now output the tabular data as JSON (an array of objects).

And there’s an option to use templates, which makes it flexible to create any output you like.

For example:

$ cat t/testtable6.csv | tablizer -s, --template "{{ .amount }} dollars have been transfered on {{ .date }}."
-450.00 dollars have been transfered on 20250101.
+912.14 dollars have been transfered on 20250103.
-299.45 dollars have been transfered on 20250105.
-2960.00 dollars have been transfered on 20250108.

The template syntax is the builtin template language of GO. It also supports the Sprig template functions. Together this is a very powerful toolset. You can create almost any output format with this like HTML or Latex.

The template is applied to every row. To access a field, use the lowercase field (column) name. Numberized field names are not supported in templates. A newline is appended after each template output.

Colors

From the start tablizer has highlighted matching strings in the output using colors.

A new option -L highlights the rows in an alternating color:

You can also highlight parts of the output using regular expressions using the -K option. For instance:

$ cat t/testtable6.csv | tablizer -s, -K '/-[0-9\.]+/white:red/' -K '/\+[0-9\.]+/white:green/'

The parameter to -K has the same syntax as -R but the replacement string must be a color as available by the color module. By default the specified color will be used as foreground color. You can add a background color after a colon as in our example above.

Colorization can be turned off with the option -N or by setting the environment variable NO_COLORS=1.

The Bottom Line

So, lots of things have changed in the past years and more might come in the future as I am using the tool myself daily privately and at work.

If you want to give it a try, go to the tablizer release page and download a release binary for your platform. For feedback or bugs, use the issue tracker.

Prior Art

Last but not least I want to show you some alternatives (from the README):

miller

This is a really powerful tool to work with tabular data and it also allows other inputs as json, csv etc. You can filter, manipulate, create pipelines, there’s even a programming language builtin to do even more amazing things.

csvq

Csvq allows you to query CSV and TSV data using SQL queries. How nice is that? Highly recommended if you have to work with a large (and wide) dataset and need to apply a complicated set of rules.

goawk

Goawk is a 100% POSIX compliant AWK implementation in GO, which also supports CSV and TSV data as input (using -i csv for example). You can apply any kind of awk code to your tabular data, there are no limit to your creativity!

teip

I particularly like teip, it’s a real gem. You can use it to drill “holes” into your tabular data and modify these “holes” using small external unix commands such as grep or sed. The possibilities are endless, you can even use teip to modify data inside a hole created by teip. Highly recommended.