Marcel – A More Modern Shell for Linux

Marcel is a new shell. It is similar to traditional shells in many ways, but it does a few things differently:

• Piping: All shells use pipes to send a text from the output of one command to the input of another. Marcel pipes structured data instead of strings.
• Python: Marcel is implemented in Python, and exposes Python in a number of ways. If you need a little bit of logic in your commands, marcel allows you to express it in Python.
• Scripting: Marcel takes an unusual approach to scripting. You can, of course, simply write a sequence of marcel commands in a text file and execute them. But Marcel also provides an API in the form of a Python module. You can import this module to do Python scripting in a far more convenient way than is possible with plain Python.

Marcel is licensed under GPLv3.

Installing Marcel Modern Shell in Linux

Marcel requires Python 3.6 or later. It has been developed and tested on Linux, and it mostly works on macOS. (If you’d like to help port to Windows, or to fix the macOS deficiencies, get in touch.)

To install marcel for your own use:

# python3 -m pip install marcel

Or if you want to install for all users (e.g., to /usr/local):

$ sudo python3 -m pip install --prefix /usr/local marcel

Once you have installed marcel, check that it’s working by running the command marcel, and then at the marcel prompt, run the version command:

$ marcel

Customization of Marcel Shell

You can customize marcel in the file ~/.marcel.py, which is read on startup, (and reread when modified). As you can tell from the file’s name, customization of marcel is done in Python.

One thing you probably want to do is to customize the prompt. To do this, you assign a list to the PROMPT variable. For example, if you want your prompt to be the current directory, printed in green, followed by > printed in blue:

PROMPT = [
    Color(0, 4, 0),
    lambda: PWD,
    Color(0, 2, 5),
    '> '
]

The resulting prompt looks like this:

This replaces the inscrutable PS1 configuration that you would need to do in bash. Color(0, 4, 0) specifies green, (the arguments are RGB values, in the range 0-5). PWD is the environment variable representing your current directory and prefixing this variable with lambda: generates a function, evaluated each time the prompt is displayed.

The ~/.marcel.py can also import Python modules. E.g., if you want to use the functions of the math module in your marcel commands:

from math import *

Once you’ve done this, you can refer to symbols from that module, e.g. pi:

Note that pi is parenthesized. In general, marcel uses parentheses to delimit Python expressions. So (pi) evaluates the Python expression that retrieves the value of the variable pi. You can also access traditional environment variables in this way, e.g. (USER) and (HOME), or any valid Python expression relying on symbols in marcel’s namespace.

And you can, of course, define your own symbols. For example, if you put this function definition in ~/.marcel.py:

def factorial(n):
    f = 1
    for i in range(1, n + 1):
        f *= i
    return f

then you can use the factorial function on the command line, e.g.

Marcel Shell Examples

Here, we will learn some examples of commands in the marcel shell.

Find File Sizes by Extension

Explore the current directory recursively, group the files by their extension (e.g. .txt, .py and so on), and compute the total file size for each group.

You can do this in marcel as follows:

The ls operator produces a stream of File objects, (-fr means visit directories recursively, and return only files).

The File objects are piped to the next command, map. The map specifies a Python function, in the outermost parentheses, which maps each file to a tuple containing the file’s extension, and it’s size. (Marcel allows the lambda keyword to be omitted.)

The red (reduce) operator, groups by the first part of the tuple (extension) and then sum up the sizes within each group. The result is sorted by extension.

Host Executables and the Marcel Pipeline

Pipelines may contain a mixture of marcel operators and host executables. Operators pipe objects, but at the operator/executable boundaries, marcel pipes strings instead.

For example, this command combines operators and executables and lists the usernames of users whose shell is /bin/bash.

$ cat /etc/passwd \
| map (line: line.split(':')) \
| select (*line: line[-1] == '/bin/bash') \
| map (*line: line[0]) \
| xargs echo

cat is a Linux executable. It reads /etc/passwd, and marcel pipes its contents downstream to the marcel operator map.

The parenthesized argument to map is a Python function that splits the lines at the : separators, yielding 7-tuples. A select is a marcel operator whose argument is a Python function identifying those tuples in which the last field is /bin/bash.

The next operator, another map keeps the username field of each input tuple. Finally, xargs echo combines the incoming usernames into a single line, which is printed to stdout.

Scripting in Marcel Shell

While Python is sometimes considered to be a scripting language, it doesn’t actually work well for that purpose. The problem is that running shell commands, and other executables from Python is cumbersome. You can use os.system(), which is simple but often inadequate for dealing with stdin, stdout, and stderr. subprocess.Popen() is more powerful but more complex to use.

Marcel’s approach is to provide a module that integrates marcel operators with Python’s language features. To revisit an earlier example, here is the Python code for computing the sum of file sizes by extension:

from marcel.api import *

for ext, size in (ls(file=True, recursive=True)
                  | map(lambda f: (f.suffix, f.size))
                  | red('.', '+')):
    print(f'{ext}: {size})

The shell commands are the same as before, except for syntactic conventions. So ls -fr turns into ls(file=True, recursive=True). The map and red operators are there too, connected with pipes, as in the shell version. The entire shell command (ls … red) yields a Python iterator so that the command can be used with Python’s for a loop.

Database Access with Marcel Shell

You can integrate database access with marcel pipelines. First, you need to configure database access in the config file, ~/.marcel.py, e.g.

define_db(name='jao',
          driver='psycopg2',
          dbname='acme',
          user='jao')

DB_DEFAULT = 'jao'

This configures access to a Postgres database named acme, using the psycopg2 driver. Connections from marcel will be made using the jao user, and the database profile is named jao. (DB_DEFAULT specifies the jao database profile as the one to be used if no profile is specified.) With this configuration done, the database can now be queried using the sql operator, e.g.

sql 'select part_name, quantity from part where quantity < 10' \
| out --csv –-file ~/reorder.csv

This command queries a table named part, and dumps the query result into the file ~/reorder.csv, in CSV format.

Remote Access with Marcel Shell

Similarly to database access, remote access can be configured in ~/.marcel.py. For example, this configures a 4-node cluster:

define_remote(name='lab',
              user='frankenstein',
              identity='/home/frankenstein/.ssh/id_rsa',
              host=['10.0.0.100', 
                    '10.0.0.101',
                    '10.0.0.102',
                    '10.0.0.103'])

The cluster can be identified as a lab in marcel commands. The user and identity parameters specify login information, and the host parameter specifies the IP addresses of the nodes on the cluster.

Once the cluster is configured, all nodes can be operated on at once. For example, to get a list of process pids and command lines across the cluster:

@lab [ps | map (proc: (proc.pid, proc.commandline))]

This returns a stream of (IP address, PID, command line) tuples.

For more information visit:

• https://www.marceltheshell.org/
• https://github.com/geophile/marcel

Marcel is pretty new and under active development. Get in touch if you would like to help out.