RSS

Command Line for the 21. Century: The Low Hanging Fruit

30 Oct

Updates

  • 2021-04-07: plocate
  • 2021-03-05: ugrep
  • 2021-01-26: diffoscope, procs, tokei, watchexec
  • 2020-09-28: huniq, duf (Disk Usage/Free Utility)
  • 2020-09-14: xoxide, starship, delta, bottom, loop
  • 2020-05-03: xpipe
  • 2019-05-23: moreutils
  • 2019-01-09: exa, hyperfine, PathPicker, and svg-term-cli
  • 2018-11-19: Pipe Viewer

People who use Unix since 1980s, or GNU/Linux since 1990s know that they can rely on the command line and many utilities for a lot of daily automation and data processing tasks. As someone who’s been using Unix and GNU/Linux based systems since 1994, I’m more than happy that I can count on these tools and the stable know-how built on them. Nevertheless, I think the command line and TUIs (Text-based User Interfaces) can be a bit better in 21. century. Therefore, in this post, I’ll list a few recent utilities that helps us to have a better command line experience.

Before you dive into the list, please be aware that I’m after the low-hanging fruit, that is, tools that can make an existing Unix / Linux command line environment a bit better with the least disruption. In other words, I will not touch on how to have brand new, GPU-powered terminal emulators such as alacritty and kitty, and neither will I talk about how nice it’d be if you only changed your shell from Bash into fish, or elvish. (If you really want to know about alternative shells, please read https://github.com/oilshell/oil/wiki/ExternalResources and http://www.oilshell.org/blog/2018/01/28.html.) I also won’t send you down the rabbit hole and make you spend countless hours customizing your shell prompt (that requires an article by itself, but in the meantime you can go and check Go Bullet Train (GBT), you’ve been warned!). Finally, no, I won’t be talking about tmux neither, because it has books dedicated to it such as “The Tao of tmux” and “tmux 2: Productive Mouse-Free Development“.

If you think something is missing and fits within the context of “low-hanging fruit” (see above), please add a comment at the end of this post. Also, for a more specialized domain, see my recent post titled “Data Processing Resources: Command-line Interface (CLI) for CSV, TSV, JSON, and XML“.

So let’s start with cd command, and how it can be enhanced with context and history, together with fuzzy matching:

–  🚀 enhancd v2 : A next-generation cd command with an interactive filter. URL https://github.com/b4b4r07/enhancd (Similar to autojump and fasd.)

fzf: A command-line fuzzy finder. For me, more of a nice Ctrl+r replacement for Bash reverse history search. URL: https://github.com/junegunn/fzf

fzy: A better fuzzy finder. For me, this is more of an integration tool with enhancd v2. URL: https://github.com/jhawthorn/fzy

zoxide: A faster way to navigate your filesystem. This can be combined with fzf.

Quickly finding files is also a very regular task, and we can always rely on the find utility, but we can have a slightly more ergonomic version:

fd: a simple, fast and user-friendly alternative to find. URL: https://github.com/sharkdp/fd/. Also see the screencast.

plocate: a faster locate. It can completely replac mlocate with a much faster (and smaller) index. It is suitable as a default locate on your system. plocate does nearly all I/O asynchronously using io_uring if available (Linux 5.1+), which reduces the impact of seek latency on systems without SSDs.

Whenever I need to use a command line program that I haven’t used for a while, I need to refresh my memory, and probably read at least one man page. But even for commands such as tar, reading a man page can trigger a TL;DR (Too Long; Didn’t Read) reaction. Therefore, tealdeer to the rescue:

tealdeer: A very fast implementation of tldr in Rust. For me, this is the quick go to reference before checking a man page of command line program. URL: https://github.com/dbrgn/tealdeer

Another command that I frequently use is cat, and it’s great that we can rely on an enhanced cat nowadays:

bat: A cat(1) clone with wings. I think this utility is good enough to move the official cat to the attic and use in only rare circumstances. It supports syntax highlighting for a large number of programming and markup languages. It is Git-aware, supports automatic paging for long outputs, can show non-printable characters, and can integrate with other tiny little command line utilities such as fd, ripgrep, xclip, man, tail, and others. Did I say that you can use it as a drop-in replacement for cat in many cases?

Every now and then, I check how much disk space is left, or which directory is consuming the biggest amount. Back in the day, I used df and du for this, and I also like the more interactive version that’s called ncdu, but nowadays I think the following tiny terminal file manager can also be used for that task:

nnn: probably the fastest and most resource-sensitive file manager you have ever used. It integrates seamlessly with your DE and favourite GUI utilities, has a unique navigate-as-you-type mode with auto-select, disk usage analyzer mode, bookmarks, familiar navigation shortcuts, subshell spawning and much more. It also lets you preview a file using whichever previewer you want. You can use bat, or even emacsclient (that is, if you, like me, prefer to have your Emacs server started, first thing after booting).

– Another fine utility to see disk usage and free space at a glance is “duf: Disk Usage/Free Utility (Linux, BSD & macOS)“. It has a very nice human-readable output, but it can also output its reports in JSON:

duf: Disk Usage/Free Utility for Linux, BSD and macOS

One other frequent task, in addition to navigating around your directories, and files is searching for strings and patterns in them. You always have the venerable grep tool, but nowadays, we have nice alternatives such as ripgrep:

ripgrep: a faster grep / ag / git grep / ucg / pt / sift. See for yourself at https://github.com/BurntSushi/ripgrep. Also see: “Faster, RegEx! Match! Match! (Which Regular Expression Utility is the Fastest?)

For those who are looking for something even faster than ripgrep, coupled with interactive search options, there’s also ugrep:

ugrep: “Search for anything in everything… ultra fast. Search file systems, source code, text, binary files, archives (cpio/tar/pax/zip), compressed files (gz/Z/bz2/lzma/xz/lz4), documents and more. A faster, user-friendly and compatible grep replacement. New match algorithms and features beating grep, ripgrep, silver searcher, ack, sift, etc. A lot of optional capabilities to make searching very convenient.”

ugrep in action

If you’re like me, you’ve used sort and uniq, and their combination for too many times. Therefore you might find the following utility useful (for more such utilities you can always visit “Data Processing Resources: Command-line Interface (CLI) for CSV, TSV, JSON, and XML“):

huniq: Replacement for sort | uniq optimized for speed (10x faster) when sorting is not needed. huniq replaces sort | uniq (or sort -u with gnu sort) and huniq -c replaces sort | uniq -c. The order of the output is stable when in normal mode, but it is not stable when in -c/count mode.

Another tool that I and many other people use very frequently is Git, a version-control system for tracking changes in computer files. And when using Git, as well as other similar tools, seeing what changed in a particular file is important. Normally you rely on your favorite Git tool to compare two versions of a file, but if you need to quickly compare two files on the command line, independent of your Git tool, you can rely on:

icdiff: improved, colored diff, for side-by-side diff view. URL: https://github.com/jeffkaufman/icdiff (You can also give diff-so-fancy a try for a human-readable diff view)

delta: a viewer for git and diff output. It has support for diff-highlight and diff-so-fancy emulation modes, too.

diffoscope: when you really need to see the difference between not only same type of files but all kinds of different artifacts. This program helps you for in-depth comparison of files, archives, and directories. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily.

Next in line, we have an alternative for the venerable ping utility. I think having a bit more information from ping wouldn’t hurt:

prettyping: runs the standard ping in background and parses its output, showing ping responses in a graphical way at the terminal (by using colors and Unicode characters).  It’s written in bash and awk, and should work out-of-the-box in most systems (Linux, BSD, Mac OS X, …). It is self-contained in only one file, so it is trivial to install and run. (You can also give gping a try).

bmon: Bandwidth Monitor is a monitoring and debugging tool to capture networking related statistics and prepare them visually in a human friendly way. It features various output methods including an interactive curses user interface and a programmable text output for scripting. I think of this utility in the same category as iptraf and iptraf-ng.

bmon – Bandwidth Monitor

Another relatively less known tool is pv , the pipe viewer: it  is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion:

pv

One thing that I also do at the command line is to trigger an event whenever there’s some change to some files & directories. Of course, you can watch for things, and even utilize inotify subsystem, but maybe you’ll find the following more convenient to quickly build a command line event triggering system:

entr: Run arbitrary commands when files change. URL: http://www.entrproject.org/ (You can also give the following a try: on-change and fswatch)

watchexec: a simple, standalone tool that watches a path and runs a command whenever it detects modifications. Among the use cases are automatically running unit tests and run linters/syntax checkers whenever there’s a pre-defined change in a directory and its sub-directories.

And finally, a few honorable mentions:

glances : a cross-platform, modern system monitoring tool that demonstrates the power of TUI. See https://nicolargo.github.io/glances/ for screenshots and its many features.

procs: is a replacement for ps written in Rust.

bottom: A cross-platform graphical process/system monitor with a customizable interface and a multitude of features. Written in Rust, and supports GNU/Linux, macOS, and Windows. Inspired by both gtop and gotop.

exa: a modern replacement for ls. It’s an improved file lister with more features and better defaults. It uses colours to distinguish file types and metadata. It knows about symlinks, extended attributes, and Git. And it’s small, fast, and just one single binary. Source code is also available at https://github.com/ogham/exa.

hyperfine: A command-line benchmarking tool (inspired by bench). When GNU time, or GNU Bash’s built-in time simply doesn’t cut it.

PathPicker: accepts a wide range of input — output from git commands, grep results, searches — pretty much anything.After parsing the input, PathPicker presents you with a nice UI to select which files you’re interested in. After that you can open them in your favorite editor or execute arbitrary commands.

svg-term-cli: Share terminal sessions as razor-sharp animated SVG everywhere. Useful for demonstrations and information sharing, error-reporting, etc.

moreutils: “a growing collection of the Unix tools that nobody thought to write long ago when Unix was young.” Once you get used to them, you’ll wonder how you managed to live without them.

loop: “UNIX’s missing loop command!” loop lets you write powerful, intuitive looping one-liners in your favorite shell! Finally, loops in Bash that make sense! Imagine combining loop with moreutils and wielding all that automation power into one-liners!

xpipe: “reads input from stdin and splits it by the given number of bytes, lines, or if matching the given pattern. It then invokes the given utility repeatedly, feeding it the generated data chunks as input. You can think of it as a Unix love-child of the split(1), tee(1), and xargs(1) commands.”

starship: ☄🌌️ The minimal, blazing-fast, and infinitely customizable prompt for any shell. Great installation experience with sane out-of-the-box defaults. See the demo below:

starship prompt for Unix and Linux shells

tokei: count your code, quickly. This is a program that displays statistics about your code. Tokei will show the number of files, total lines within those files and code, comments, and blanks grouped by language. Tokei is very fast, and is able to count millions of lines of code in seconds.

About the author: Emre is the co-founder & CTO of TM Data ICT Solutions in Belgium. You can read more about him in About page of this blog.

 
2 Comments

Posted by on October 30, 2018 in Linux

 

Tags: , , ,

2 responses to “Command Line for the 21. Century: The Low Hanging Fruit

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

 
%d bloggers like this: