Copyright © 2016 Ashok P. Nadkarni. All rights reserved.

1. Introduction

Tip This document assumes the reader is familiar with typed arrays. If not, please glance through the Programmer’s guide before continuing.

The (eXtended typed array language) Xtal (pronounced Crystal[1]) is an alternative to Tcl commands for working with collections (lists, typed array columns and tables). It provides convenient and succint syntax for

  • vector operations

  • searching and sorting

  • flexible indexing

Although geared towards operations on typed arrays, Xtal can be used as a general purpose language in its own right. Note however, that Xtal is not a replacement for Tcl. Rather, it is embedded in Tcl and the two can be intermixed according to whatever syntax suits the task at hand.

2. Quick Tour

This section provides a short tour of Xtal to introduce the reader to the language.

Because the primary purpose of Xtal is to work conveniently with typed arrays - columns and tables - we illustrate its use in that context first.

Working with columns

For purposes of demonstration, let us start off by creating two columns containing the months in a year and rainfall received in a particular year.

Months = @string {"Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", \
    "Oct", "Nov", "Dec"}
Rainfall = @double {
    11.0, 23.3, 18.4, 14.7, 70.3, 180.5, 210.2, 205.8, 126.4, 64.9, 33.1, 19.2
}
→ tarray_column double {11.0 23.3 18.4 14.7 70.3 180.5 210.2 205.8 126.4 64.9 3...

Elements in the column are retrieved using indexing.

Rainfall[0]          # Rainfall in January
→ 11.0

Rainfall[11] = 21.5  # Update the measured amount for December
→ tarray_column double {11.0 23.3 18.4 14.7 70.3 180.5 210.2 205.8 126.4 64.9 3...

Instead of a simple integer index, we may specify a range instead in which case a column of the same type containing the elements in the specified range is returned.

print(Rainfall[9:11])        # Rainfall in the last quarter
→ 64.9, 33.1, 21.5

print(@sum(Rainfall[9:11]))  # Total rainfall in last quarter
→ 119.5

Or we could specify a list of indices of interest.

print(Rainfall[{0, 3, 6, 9}])  # Rainfall in first month of every quarter
→ 11.0, 14.7, 210.2, 64.9

When the indexing operation itself returns a column, the vector and folding operations available for columns can be used in conjunction.

print( Rainfall / 10 )      # Print rainfall in centimeters
→ 1.1, 2.33, 1.8399999999999999, 1.47, 7.029999999999999, ..., 20.5800000000000...

@sum(Rainfall[9:11])        # Total rainfall in the last quarter
→ 119.5

Yet another form that indexing can take is a boolean expression. For example, we can list the rainfall values that lie within a specific range.

print ( Rainfall [Rainfall > 100 && Rainfall < 200] )
→ 180.5, 126.4

print ( Rainfall [@@ > 100 && @@ < 200] )  # @@ is the "current" context
→ 180.5, 126.4

Or perhaps more usefully, list the corresponding months.

print ( Months [Rainfall > 100 && Rainfall < 200] )
→ Jun
  Sep

This form of indexing works because simple boolean expressions involving columns return an integer column containing the indices for elements which satisfy the expression. Thus, we have

print(Rainfall > 100 && Rainfall < 200)
→ 5, 8

This feature can be used in other contexts as well. So for example, we can use the % size operator to count the number of such months.

%(Rainfall > 100 && Rainfall < 200)
→ 2
Working with tables

Having looked at basic column operations, we turn our attention to tables. As for columns, we will use a small sample table containing a simple employee data base for demonstration purposes:

Emps = @table (
    Name string,   Salary uint,   Age uint,   Location string
) {
    {'Sally',      70000,      32,       'Boston'},
    {'Tom',        65000,      36,       'Boston'},
    {'Dick',       80000,      40,       'New York'},
    {'Harry',      45000,      37,       'New York'},
    {'Amanda',     48000,      35,       'Seattle'}
}
print(Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 70000| 32|Boston  |
  +------+------+---+--------+
  |Tom   | 65000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 80000| 40|New York|
  +------+------+---+--------+
  |Harry | 45000| 37|New York|
...Additional lines omitted...

Tables rows can be accessed in the same manner as column elements.

% Emps      # Number of employees
→ 5

print(Emps[3])     # Row 3
→ Harry 45000 37 {New York}

print(Emps[2:4])   # Rows 2 through 4
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Dick  | 80000| 40|New York|
  +------+------+---+--------+
  |Harry | 45000| 37|New York|
  +------+------+---+--------+
  |Amanda| 48000| 35|Seattle |
  +------+------+---+--------+

print(Emps[{3,1}]) # Rows 3 and 1
→ +-----+------+---+--------+
  |Name |Salary|Age|Location|
  +-----+------+---+--------+
  |Harry| 45000| 37|New York|
  +-----+------+---+--------+
  |Tom  | 65000| 36|Boston  |
  +-----+------+---+--------+

Columns within a table are accessed with the . operator and indexed as usual.

print(Emps.Name)       # Column name supplied as literal
→ Sally
  Tom
  Dick
  Harry
  Amanda

colname = 'Salary'
→ Salary
print(Emps.$colname)   # Column name supplied in a variable
→ 70000, 65000, 80000, 45000, 48000

print(Emps.Salary[@@ >= 70000])           # High salaries
→ 70000, 80000

print(Emps.Name[Emps.Salary >= 70000])    # Highly paid employees
→ Sally
  Dick

Table slice operators can be used extract subtables.

print(Emps.(Name, Location))
→ +------+--------+
  |Name  |Location|
  +------+--------+
  |Sally |Boston  |
  +------+--------+
  |Tom   |Boston  |
  +------+--------+
  |Dick  |New York|
  +------+--------+
  |Harry |New York|
...Additional lines omitted...

print(Emps.(Name, Location)[2:4])
→ +------+--------+
  |Name  |Location|
  +------+--------+
  |Dick  |New York|
  +------+--------+
  |Harry |New York|
  +------+--------+
  |Amanda|Seattle |
  +------+--------+

As for columns, boolean expression can be used to query tables.

print(Emps[Emps.Salary > 50000 && Emps.Location != "New York"])
→ +-----+------+---+--------+
  |Name |Salary|Age|Location|
  +-----+------+---+--------+
  |Sally| 70000| 32|Boston  |
  +-----+------+---+--------+
  |Tom  | 65000| 36|Boston  |
  +-----+------+---+--------+

print(Emps[Emps.Salary > 50000 && Emps.Location != "New York"] . (Name, Age))
→ +-----+---+
  |Name |Age|
  +-----+---+
  |Sally| 32|
  +-----+---+
  |Tom  | 36|
  +-----+---+

Tables can be modified in multiple ways.

Emps[% Emps] = {'Mary', 38000, 25, 'Seattle'}    # Add a row

Emps.Salary = Emps.Salary + 2000                 # Modify entire column

Emps.Location[Emps.Name.Sally] = "New York"      # Modify a single cell

print(Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 72000| 32|New York|
  +------+------+---+--------+
  |Tom   | 67000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 82000| 40|New York|
  +------+------+---+--------+
  |Harry | 47000| 37|New York|
...Additional lines omitted...
Variables and literals

Having looked at basic operations on columns and tables, we will quickly go over the general purpose features of the language.

The syntax of Xtal is close to the C family of languages than Tcl. Thus unquoted identifiers are variables, not strings, and do not need a $ prefix as in Tcl and expressions use the usual infix form.

a = 1 ; b = 2
c = 1 + a * b
→ 3

String literals can be expressed in two forms - surrounded by double quotes or by single quotes. In the former case, variable and command substitutions are performed as in Tcl. In the latter case, no variable or command substitutions are done.

puts('The value of b at time [clock seconds] is $b')
→ The value of b at time [clock seconds] is $b

puts("The value of b at time [clock seconds] is $b")
→ The value of b at time 1451826006 is 2
Mixing Tcl and Xtal

An important feature of Xtal is that, being built on top of Tcl, it can be freely intermixed with it. This allows use of the syntax that is most suitable for the task at hand.

Tcl code can be intermixed within Xtal by bracketing it with <>.

now = <clock seconds>

L = {1, 2}
< lappend L 3 >
puts(L)
→ 1 2 3

The Tcl fragment may also be spread across multiple lines.

puts("Xtal statement")
<
puts "First Tcl statement"
puts "Second Tcl statement"
>
→ Xtal statement
  First Tcl statement
  Second Tcl statement

Similarly, Xtal can be mixed with Tcl code with the xtal::xtal command.

% namespace import xtal::xtal
% xtal {
    puts("puts called from Xtal")
    <
    puts "puts called from Tcl called from Xtal"
    xtal {puts("puts called from Xtal called from Tcl called from Xtal")}
    >
}
→ puts called from Xtal
  puts called from Tcl called from Xtal
  puts called from Xtal called from Tcl called from Xtal

As we see later, we can also define procs in Xtal that are called from Tcl like any other Tcl command.

Calling functions

Commands or functions (we use the terms interchangeably) are called with parameters enclosed in parenthesis as in the C family of languages.

hex = format("0x%x", 42)
→ 0x2a
Working with lists

Lists are constructed using braces but unlike in Tcl, elements are general expressions separated with commas.

L = {0, a, b, a + b, tcl::mathfunc::rand()}
→ 0 1 2 3 0.5369003920522054

Note that these lists are just plain Tcl lists. The above is equivalent to the Tcl command

set L [list 0 $a $b [expr {$a+$b}] [tcl::mathfunc::rand]]
→ 0 1 2 3 0.6848892214172004

Thus you can call Tcl’s list related commands, like

llength(L)
→ 5

Many of the operators we saw earlier with columns and tables can also be used with lists. This can be more convenient than the corresponding Tcl command.

The prefix operator % can be used to return the length of a list.

L = {'zero', 'one', 'two', 'three'}
len = % L  # Tcl - llength $L
→ 4

Similarly, the indexing operator [] can be used instead of lindex and this supports the many forms we saw earlier.

L[0]       # Single element. Tcl - lindex $L 0
→ zero

L[0:1]     # Index range. Tcl - lrange $L 0 1
→ zero one

L[{3,1,2}] # Index list - no direct Tcl equivalent.
→ three one two

L[%L] = "four" # Append. Tcl - lappend L four
→ zero one two three four

The index operations can be search expressions as well.

L[L ~ ".*o$"]
→ zero two

The indexing operator can be used in assignments.

L[%L-2 : %L] = {999, 1000, 1001}
→ zero one two 999 1000 1001

L[{3,2}] = {100, 101}
→ zero one 101 100 1000 1001

Other list operations are performed by calling the corresponding Tcl command.

lsort(L)
→ 100 1000 1001 101 one zero
Working with dictionaries

Tcl dictionary access has a similar shorthand using the . lookup postfix operator.

colors = {"red", 0xff0000, "green", 0x00ff00, "blue", 0x0000ff}
→ red 0xff0000 green 0x00ff00 blue 0x0000ff

colors.red
→ 0xff0000

colors.blue = 0x0000ee
→ red 0xff0000 green 0x00ff00 blue 0x0000ee

The lookup postfix operator . used with dictionaries can also be used with columns. In this case, it returns the index of the first matching element.

Months.Apr
→ 3

That ends our quick tour of the basic. Xtal has other advanced features as well as standard programming language constructs such as control structures and built-in functions that we have left out in our quick tour.

The rest of this document describes Xtal in more detail.

3. Running Xtal

Xtal is implemented as the Tcl xtal package.

% package require xtal
→ 0.8

To save some typing we will add both xtal and tarray to our namespace path.

% namespace path [list xtal tarray]

A Xtal code fragment may be run by one of several means.

First, it can be run from within a Tcl script or a Tcl shell using the xtal::xtal command. This is roughly the equivalent of running a Tcl script using eval.

% xtal { a = 1 }
→ 1

The xtal command returns the result of the last statement it executes.

Second, you can define a Xtal procedure from Tcl using xtal::function in a similar fashion as the Tcl proc command. It can then be invoked in the same manner as a Tcl procedure.

% function add {x y} { return x + y }
% add 2 3
→ 5

Third, you can load an entire Xtal file using the xtal::source command which has syntax similar to the Tcl source command.

xtal::source ?-encoding ENCODING? XTALFILE

In all the above cases, the Xtal code is compiled to Tcl on the fly at runtime. A more efficient alternative for larger applications is to compile Xtal to Tcl ahead of time with the xtal::compile command. This has the syntax

compile XTALINPUTFILE TCLOUTPUTFILE ?-encoding ENCODING?

The input file in Xtal is translated to a Tcl script which is written out to TCLOUTPUTFILE. Only the output file needs to be shipped with the application and is sourced like any other Tcl script.

3.1. Running interactively

You can also type Xtal commands directly in interactive mode using the Xtal shell which accepts both Xtal and Tcl syntax. This is described in The Xtal shell.

In this document, for ease of exposition we will simply show the relevant Xtal fragments without any reference to how they are executed.

4. Basic syntax

The basic syntax of Xtal is closer to the C family of languages than to Tcl.

4.1. Statements

Like Tcl, a Xtal script is a sequence of statements separated by either a newline character or a semicolon.

x = 2
y = 3 ; z = x*y
→ 6

A statement may be an assignment as above, or simply an expression:

10*x + y
→ 23

4.1.1. Assignment statement

An assignment statement in Xtal takes the form

LVALUE = RAVALUE

where RVALUE may be a Xtal expression or Tcl fragment.

In the current version of Xtal, LVALUE cannot be an arbitrary expression and must take one of the following forms:

4.2. Mixing Tcl and Xtal

Tcl and Xtal code can be intermixed. To embed Xtal within Tcl, use the xtal::xtal command as

xtal SCRIPT

where SCRIPT is the Xtal fragment which may be spread across multiple lines.

% xtal { x = 2 }
→ 2
% xtal {
    y = 3
    z = x * y
}
→ 6

Conversely, Tcl can be embedded with Xtal by placing the Tcl script within < and > delimiters.

now = <clock seconds>
→ 1451826006

Multi-line scripts are permitted.

<
    set x 2
    set y 3
    set z [expr {$x + $y}]
>
→ 5

The terminating > must be followed by a Xtal statement terminator such as a newline or ;, optionally with intervening whitespace.

Note Embedded Tcl is not an expression and thus cannot be used as part of an expression.

4.3. Comments

Comments begin with # and extend to the end of the line.

Note Unlike Tcl, # marks the beginning of a comment irrespective of where it appears outside of a quoted string and not only if appears at the beginning of a statement.

4.4. Variables

As seen above, variables are syntactically different from Tcl:

  • There is no preceding $ as in Tcl for accessing a variable’s value.

  • Variable names are restricted to starting with an alphabetic character and may only contain alphanumeric characters, _ and :.

The variables themselves are standard Tcl variables and follow the same name resolution rules.

% set v "This is a variable"
→ This is a variable
% xtal { v = "Modified from xtal" }
→ Modified from xtal
% puts $v
→ Modified from xtal

4.5. Literals

Numeric literals are as in Tcl and have the same syntax as in Tcl. For example,

r = 2.0e10
→ 2.0e10
big = 123455678987654321000000000
→ 123455678987654321000000000

String literals are delimited by single or double quotes and follow the backslash substitution rules as in Tcl. The difference between the two is that double quoted strings undergo variable and command substition as in Tcl whereas single quoted strings do not.

Unlike in Tcl, a plain unquoted string is treated as an identifier and not a string literal.

s = unquoted
→ can't read "unquoted": no such variable

s = "Sum of 2+2 is [expr 2+2]"
→ Sum of 2+2 is 4

s = 'Sum of 2+2 is [expr 2+2]'
→ Sum of 2+2 is [expr 2+2]
Warning Braces have different semantics in Xtal and cannot be used to delimit strings.

4.6. Lists

Braces in Xtal are used to construct lists, similar to the list command in Tcl.

l = {"a string", x, tcl::clock::seconds(), y+z}
→ {a string} 2 1451826006 8

The list elements are separated using commas and an element may be a literal, variable, function call or general expression. The list elements may be spread across multiple lines but must still be separated by commas.

l = {
    x,
    "a string",
    y+z
}
→ 2 {a string} 8

As an aside, we could also have invoked the Tcl list command as a function to construct a list.

l = list(x,"a string",y+z)
→ 2 {a string} 8

5. Functions

Functions in Xtal may refer to either Tcl commands or to functions implemented in Xtal through the xtal::function Tcl command or the Xtal function statement.

5.1. Invoking functions

Invocation of functions/commands takes a form similar to function calls in C. Function parameters are wrapped in parenthesis and separated by commas.

hex = format("0x%x", 42)
→ 0x2a

The above calls Tcl’s built-in format command. There is no difference between invoking commands implemented in Tcl or in Xtal.

Warning

There is a subtle point to be noted when invoking commands from Xtal as opposed to Tcl. In Xtal, a parameter that is not a literal is implicitly dereferenced when passed to a command. It is easy to forget this when calling Tcl commands that take variable names as parameters. Thus

incr val -1

is not

xtal { incr(val, -1) }

which lands up passing the value of val to the incr command which actually expects the name val itself to be the argument.

The correct equivalents are either of the following:

xtal { incr("val", -1) }
xtal { incr('val', -1) }

As for list elements, the parameters for a function call may be spread across multiple lines.

hex = format("0x%x",
             42)
→ 0x2a

The function called need not be an identifier; it can be supplied as any expression, for example a function call that returns a function name.

< proc function_returning_function {} {return puts} >
function_returning_function () ("Hello there!")
→ Hello there!

5.1.1. Calling ensemble commands

Ensemble commands such as clock seconds are called from Xtal as

clock.seconds()
→ 1451826006

Note the use of . as the separator between the command and its subcommand. Depending on how the ensemble is defined, you can also directly invoke it.

tcl::clock::seconds()
→ 1451826006

Finally, in the very rare case that the ensemble subcommand does not fit the syntax for a Xtal identifier, you can pass it as the first argument.

clock("seconds")
→ 1451826006

5.1.2. Calling object methods

Object methods are called in a manner similar to ensemble commands.

% oo::class create OExample { method m args {puts [join $args ,]} }
→ ::OExample
% OExample create o
→ ::o
% xtal {
    o.m('astring', 10)
    o.destroy()
}
→ astring,10

However, there is an additional case here where the name of the object is not fixed and the object is accessed through a variable.

% set obj [OExample new]
→ ::oo::Obj117
% xtal {obj.m('astring', 10)}
→ invalid command name "obj"

Function and object names, unlike variables, do not get implicitly dereferenced. In this case, the $ operator is used for dereferencing.

$obj.m('astring', 10)
→ astring,10
$obj.destroy()

We will talk about the dereferencing operator later.

5.1.3. Passing options to commands

When passing options to commands, Xtal supports an additional syntax for arguments where the option name does not have to be quoted or separated from its value by a ,.

Instead of invoking the Tcl subst command as follows

subst("-novariables", 'The value of val is $val')
→ The value of val is $val

it can be invoked as

subst(-novariables 'The value of val is $val')
→ The value of val is $val

Note the option name is not quoted and there is no , separating it from its value.

Xtal recognizes this form based on the argument being composed of two expressions with the first one being a token starting with a - character.

5.2. Defining functions

There are two ways to define functions that implemented in Xtal:

  • Using the function keyword within Xtal code

  • Using the xtal::function command within Tcl code

Both offer the same functionality, differing only because of Tcl and Xtal syntax.

5.2.1. Defining Xtal functions with function

Within Xtal, use function to define a Xtal function. This has the syntax

function NAME (?PARAM, …​?) BODY

Parameters are separated by commas.

function add(a, b) {return a + b}
add(1,2)
→ 3

They can take default values where the value is separated from the parameter name with an = sign.

function add(a, b = 1) {return a + b}
add(10)
→ 11

The default value may be any expression, not necessarily a constant. Note however that the expressions are evaluated at the time the function is defined and not at the time it is invoked.

If the last parameter name is args, it is treated in the same manner as in Tcl’s proc command.

5.2.2. Defining Xtal functions in Tcl with xtal::function

Unlike function, the xtal::function command is meant to be called from Tcl as opposed to being used within Xtal itself. It has the same form as the Tcl proc command except that the body of the procedure is Xtal instead of Tcl.

xtal::function NAME PARAMS BODY

PARAMS takes the same form as in the proc command, including defaults for parameter and variable parameters using the args notation.

% xtal::function add {a b} {return a + b}

5.3. The return statement

The Xtal return statement returns control to the caller from a function or procedure.

return EXPRESSION

EXPRESSION is any valid Xtal expression.

6. Control statements

Xtal implements control statement similar to Tcl but in a slightly different form.

6.1. The if statement

The if statement has the general form

if EXPRESSION {
    STATEMENTBLOCK
} elseif EXPRESSION {
    STATEMENTBLOCK
} else {
    STATEMENTBLOCK
}

There may be zero or more elseif clauses and the else clause is optional. If present, the else and elseif keywords must be on the same logical line as the preceding statement block.

EXPRESSION can be any Xtal expression and does not need to be enclosed in parenthesis.

if %Emps {
    print(Emps)
} else {
    puts("Table is empty")
}
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 72000| 32|New York|
  +------+------+---+--------+
  |Tom   | 67000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 82000| 40|New York|
  +------+------+---+--------+
  |Harry | 47000| 37|New York|
...Additional lines omitted...

6.2. The while statement

The while statement has the syntax

while EXPRESSION { STATEMENTBLOCK }

EXPRESSION can be any Xtal expression and does not need to be enclosed in parenthesis.

i = 0
while i < % Emps {
    puts(Emps.Name[i])
    i = i + 1
}
→ Sally
  Tom
  Dick
...Additional lines omitted...

6.3. The for statement

The for statement is used to iterate over a range of integers with a suitable increment.

for IDENTIFIER LOWEXPR ?: HIGHEXPR? ?: INCREMENT? STATEMENTBLOCK

The loop variable IDENTIFIER is initially assigned a value of LOWEXPR. Thereafter, STATEMENTBLOCK is executed as long as the the value of the loop variable is less than or equal to HIGHEXPR. After each iteration, the loop variable is incremented by INCREMENT which defaults to 1 if unspecified. The lower limit of the range is computed only once, before the loop is iterated. The upper limit is computed on every iteration.

for i 0 : %Emps-1 {
    puts(Emps.Name[i])
}
→ Sally
  Tom
...Additional lines omitted...

If HIGHEXPR is unspecified, the loop will not terminate unless you break out of it with a return, break or similar. Some examples of the various forms:

for i 0:2 { puts(i) }
→ 0
  1
  2
limit = 5
for i 0 : limit : 2 { puts(i) }
→ 0
  2
  4
for i 0 {
    if i >= 2 { break }
    puts(i)
}
→ 0
  1
for i 0::2 {
    puts(i)
    if i > 3 { break }
}
→ 0
  2
  4

6.4. The foreach statement

This statement behaves similar to the Tcl foreach statement where the loop variable takes on values from a collection. The syntax takes one of the forms

foreach  VAR COLLECTION SCRIPT
foreach INDEXVAR , VAR COLLECTION SCRIPT

where COLLECTION is any Xtal expression.

In the first form, SCRIPT is executed for each element in COLLECTION. If COLLECTION is a column, the variable VAR takes the value of each element of the column. If COLLECTION is a table, VAR is assigned each row of the table in list form. Otherwise, COLLECTION is treated as a Tcl list and VAR takes on the value of each element of the list.

puts("Rainy months:")
foreach month Months[Rainfall > 100] {
    puts(month)
}
→ Rainy months:
  Jun
  Jul
...Additional lines omitted...

The second form is similar in that again SCRIPT is executed for each element of COLLECTION. However in addition to VAR being assigned the value of the element, the variable named INDEXVAR is assigned the index of the element.

foreach index, rainfall Rainfall {
    if rainfall > 100 {puts(Months[index])}
}
→ Jun
  Jul
  Aug
...Additional lines omitted...
Tip Both the above examples print the same information. The first one is more succint and clear, and possibly faster, but the latter will use significantly less memory when the data is large.

6.5. The break statement

The break statement behaves like the Tcl break command. It causes the innermost loop to stop execution and continue execution at the following statement.

break

6.6. The continue statement

The continue statement behaves like the Tcl continue command. It aborts the current iteration of the innermost loop containing it without aborting the loop itself which then proceeds with its next iteration.

continue

6.7. The try statement

The try statement closely resembles Tcl’s try command differing only in that the syntax is a little different and the clauses are in Xtal.

The general syntax of the command is

try { BODY } ?HANDLER …​? ?finally { FINALBODY }?

Multiple handlers may be specified. Each HANDLER may be one of two forms:

on EXCEPTIONCODE ?VARNAME …​? { HANDLERBODY }
trap ERRORPREFIX ?VARNAME …​? { HANDLERBODY }

The statement results in the execution of BODY. The exception code resulting from the execution is matched against each handler in turn. The HANDLERBODY corresponding to the first matching handler is executed and propagation of the exception stops in that case. Further handlers are not checked. If no handler matches, the exception is propagated. In all cases, irrespective of the exception code and whether any handler matched, the FINALBODY script is run if present.

In the case of the on handler, EXCEPTIONCODE is an integer value or one of the strings ok, error, continue, break or return. The handler will match if this matches the exception code resulting from the execution of BODY.

The trap handler only matches if the exception code is error. Additionally the appropriate number of leading values from the -errorcode entry in the interpreter status dictionary must match the values specified in ERRORPREFIX which must be of the form used for constructing lists.

In both types of handlers, the handler script body may be preceded by zero or more variable names. If present, the first of these receives the result of the execution of BODY. The second variable, if present, receives the status dictionary.

The following example will help clarify the working.

try {
    x = nosuchvar
} trap {'TCL', 'LOOKUP', 'VARNAME'} message status_dict {
    puts("Oops, variable does not exist")
    puts(message)
} on error message status_dict {
    puts("Some other error")
} finally {
    puts("Oh well, life goes on")
}
→ Oops, variable does not exist
  can't read "nosuchvar": no such variable
  Oh well, life goes on
Warning Make a note of how the ERRORPREFIX is specified as a list construct. The elements may be general expressions and hence string constants have to be quoted as shown in the example.

A more bare-bones version of the above, with a different error this time, would be

try {
    throw 'XTAL', 'TEST', "This is only a test"
} trap {'TCL', 'LOOKUP', 'VARNAME'} {
    puts("Oops, variable does not exist")
} on error message {
    puts(message)
}
→ This is only a test

6.8. The throw statement

The throw statement raises a Tcl error in similar manner to Tcl’s own throw and error statement. Its general syntax is

throw EXPRESSION ?, EXPRESSION …​?

If a single argument is given, it is taken to be the equivalent of the message argument to a Tcl error or throw command. If multiple arguments are given, the last argument is interpreted as the message. Moreover, all arguments are gathered into a list which becomes the value of the error code.

% catch { xtal {throw "This is the error message"} } message
→ 1
% puts $message
→ This is the error message
% puts $::errorCode
→ NONE
%
% catch { xtal {throw 'XTAL', 'ERROR', "This is only a test"} } message
→ 1
% puts $message
→ This is only a test
% puts $::errorCode
→ XTAL ERROR {This is only a test}

7. Columns and tables

We now come to the primary reason for the existence of Xtal - convenient vector operations on columns and tables.

7.1. Creating columns

Columns are created using the built-in operators @boolean, @byte, @int, @uint, @wide, @double, @string and @any. The syntax of the command can take three forms, the first of which is

@TYPE ?INITLIST?

Here TYPE is one of the tarray column types and INITLIST is an optional initializer list. For example,

A = @int
→ tarray_column int {}

creates an empty column of type int and assigns it to A and

A = @any {10, {1, 2, 3}, len}
→ tarray_column any {10 {1 2 3} 4}

creates a column of type any with 3 elements.

The second form is

@TYPE START : STOP ? : STEP ?

This creates a column of the specified type containing elements from START (inclusive) to STOP (excluded) at intervals of STEP which default to 1 if unspecified.

% @double 1.0:10.0
→ tarray_column double {1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0}
% @int 5:-5:-2
→ tarray_column int {5 3 1 -1 -3}

Finally, the third form is that of a function,

@TYPE (?EXPR?)

Again, here TYPE is one of the tarray column types. EXPR, if specified, is an expression that will be used to initialize the column. Thus this form can also be used to ''cast'' a column to a column of a different type.

@int(Rainfall)
→ tarray_column int {11 23 18 14 70 180 210 205 126 64 33 21}

7.2. Creating tables

Tables are defined and created using the @table built-in operator. The syntax for the command is

@table(?COLNAME TYPE?, …​) ?TABLEROWS?

where COLNAME is the name of the column and TYPE is one of the tarray column types. TABLEROWS may be specified to initialize the table and should be a list of rows as for the tarray::table create command. Here is the table definition from our Quick Tour section.

Emps = @table (
    Name string,   Salary uint,   Age uint,   Location string
) {
    {'Sally',      70000,      32,       'Boston'},
    {'Tom',        65000,      36,       'Boston'},
    {'Dick',       80000,      40,       'New York'},
    {'Harry',      45000,      37,       'New York'},
    {'Amanda',     48000,      35,       'Seattle'}
}
Tip

In many cases, you may find it easier to use Tcl, either directly or embedded, to create tables. For example, the above table can be created with embedded Tcl as

Emps = <
  table create {
    Name string   Salary uint    Age uint    Location string
  } {
    {Sally        70000          32          Boston}
    {Tom          65000          36          Boston}
    {Dick         80000          40          "New York"}
    {Harry        45000          37          "New York"}
    {Amanda       48000          35          Seattle}
  }
>
→ tarray_table {Name Salary Age Location} {{tarray_column string {Sally Tom Dic...

7.3. The . table column selection operator

Columns from a table can be selected using the . operator.

TABLE.COLSPEC

The column specification COLSPEC may be the actual name of the column or specified through a variable via the dereferencing operator.

print(Emps.Name)
→ Sally
  Tom
  Dick
  Harry
  Amanda
colname = 'Location'
→ Location
print(Emps . $colname)
→ Boston
  Boston
  New York
  New York
  Seattle

These operators can be used in two contexts:

  • When used in expressions as above, they return column values from the table.

  • When referenced on the left side of an assignment statement, they select the column to be modified in a table.

We will see examples later when we discuss indexing operators.

7.4. The .() table slicing operator

The table slicing operators .() operator return a table containing specific columns from a table.

TABLE.(?COLSPEC,…​?)

Each column is specified using its name or by derefencing a variable containing its name.

colname = "Salary"
→ Salary
print(Emps.(Name, $colname))
→ +------+------+
  |Name  |Salary|
  +------+------+
  |Sally | 70000|
  +------+------+
  |Tom   | 65000|
  +------+------+
  |Dick  | 80000|
  +------+------+
  |Harry | 45000|
...Additional lines omitted...
Note

Note the difference between the following two expressions:

Emps.Name
→ tarray_column string {Sally Tom Dick Harry Amanda}
Emps.(Name)
→ tarray_table {Name} {{tarray_column string {Sally Tom Dick Harry Amanda}}}

The first returns a column. The second returns a table containing a single column.

Like the table column selection operators, the table slicing operator can also be used in two contexts:

  • When used in expressions as above, it returns table slices

  • When referenced on the left side of an assignment statement, it assigns to the corresponding columns in the table

Again, we will see examples later when we discuss indexing operators.

7.5. Converting to lists and dictionaries

Columns and tables can be converted to lists and dictionaries with the @list and @dict functions respectively. In the case of the latter, the key for the dictionary is the index of the value in the column or table.

@list (Emps)
→ {Sally 70000 32 Boston} {Tom 65000 36 Boston} {Dick 80000 40 {New York}} {Har...

@dict (Rainfall)
→ 0 11.0 1 23.3 2 18.4 3 14.7 4 70.3 5 180.5 6 210.2 7 205.8 8 126.4 9 64.9 10 ...

Any expression that results in a column or table may be supplied as the parameter to these functions. If the expression does not result in a column or table, an error is generated.

8. Operators

Xtal operators fall into the following classes:

  • Arithmetic operators are similar to those in Tcl’s expr command but work on both scalars as well as columns.

  • Relational operators and logical operators are similar to those in expr when used with scalar operands. However, when used with column operands they have different semantics that provide very convenient search and indexing functionality.

  • Set operators are applicable only to columns and implement operations like union and intersection.

  • The lookup operator that performs associative lookup on dictionaries and columns.

  • Indexing operators that combine both traditional indexing as well as search operations on collection data types including lists, columns and tables.

  • Other miscellaneous operators that provide functionality like cardinality.

Note that operator precedence follows that of Tcl.

8.1. Arithmetic operators

8.1.1. Unary arithmetic operators

The Xtal unary operators are -, +, ~ and ! have the same semantics as in Tcl for operands that are not columns or tables.

In the case of columns, the - and + operators can be applied to numeric columns (types byte, int, uint, wide and double). The result is a column of the same type with each element being the result of the operator being applied to the corresponding element of the operand.

The ~ operator can be applied to integral columns and columns of type boolean.

For all other combinations of column type and operation, an error is raised.

In the case of operands that are tables, all unary operators will raise an error.

8.1.2. Binary arithmetic operators

The binary operators include +, -, , /, *, &, |, ^.

Rules of operation
  • If either operand is a table or a non-numeric column, an error is raised.

  • If neither operand is a column, they have the same semantics as in Tcl.

  • If an operand is not a column, it is treated as a column of the same size as the other operand with all elements having the operand value.

  • The result is a column each element of which is the result of the operation on the corresponding elements of the operands. In case of differing column types, types are promoted as needed. See the column math command reference for full details of type promotion and intermediate values.

Here is an example of mixed operand types:

I = @int {10, 20, 30, 40}
J = @byte {1, 2, 3, 4}
print(I + 2 * J)
→ 12, 24, 36, 48

Tables do not support any arithmetic operators but their contained columns can be used anywhere that a standalone column is valid.

For example, we can reduce everyone’s salaries by 2K since times are tough.

Emps.Salary = Emps.Salary - 2000
print (Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 68000| 32|Boston  |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 78000| 40|New York|
  +------+------+---+--------+
  |Harry | 43000| 37|New York|
  +------+------+---+--------+
  |Amanda| 46000| 35|Seattle |
  +------+------+---+--------+

8.2. Relational operators

The relational operators include the ==, !=, <, <=, >, >= operators that are present in Tcl.

In addition, they include the operators related to string comparisons that are shown in Xtal relational operators not in Tcl.

Table 1. Xtal relational operators not in Tcl

Operator

Syntax

Description

=^

STRINGA =^ STRINGB

Returns 1 if the two strings are equal in case-insensitive mode, else 0.

!^

STRINGA =^ STRINGB

Returns 1 if the two strings are not equal in case-insensitive mode, else 0.

~

STRING ~ REGEXP

Returns 1 if the string matches the specified regular expression, else 0.

~^

STRING ~^ REGEXP

Returns 1 if the string matches the specified regular expression, else 0. The matching is done in case-insensitive mode.

!~

STRING !~ REGEXP

Returns 1 if the string does not match the specified regular expression, else 0.

!~^

STRING !~^ REGEXP

Returns 1 if the string does not match the specified regular expression, else 0. The matching is done in case-insensitive mode.

Note that for the regular expression operators, the order of the operands is significant - the first indicates the string being matched and the second is the regular expression.

Rules of operation
  • If either operand is a table, an error will be generated.

  • If both operands are columns, an error will be generated.

  • If neither operand is a column, the result will be a boolean value based on standard Tcl operators or the operators shown in Xtal relational operators not in Tcl.

  • If (exactly) one of the operands is a column, each element of the column is individually compared with the other operand. The result is an integer column containing the indices of those elements for which the specified comparison operation returns true.

This last rule is illustrated by the example below which prints the indices for all months where the rainfall is less than 100mm.

print(Rainfall < 100)
→ 0, 1, 2, 3, 4, 9, 10, 11

The real usefulness of this will become apparent when we discuss indexing operators.

Note

Xtal does not allow expressions of the form

a < b < c

on the basis that more often than not these are programming errors. This holds only for all relational operators, not arithmetic or logical operators. In the rare case that you need such expressions, use parenthesis to appropriately group the operands.

8.3. Logical operators

The logical operators include && and || as in Tcl. However, unlike Tcl these operators do not short-circuit the evaluation of their operands.

Rules of operation
  • If either operand is a table, an error will be generated.

  • If neither operand is a column, the result will be a boolean value based on standard Tcl operators.

  • If both operands are columns, the operation is treated as a set operation and described in Set operators.

  • If one operand is a column and the other is not, an error is raised.

8.4. Set operators

The operators && and || behave as set operators when both operands are columns. In this case, both operands must be of the same type.

The && operator returns a column of the same type as the operands and containing elements that are common to both.

The || operator returns a column of the same type as the operands and containing elements that are present in either operand.

8.5. Lookup operator

The operator . looks up keys in a dictionary or column.

In the case of a dictionary, the operators retrieve the value associated with the key.

The key operand is treated as a literal and returns the corresponding element from the dictionary. If the key follows the syntax for an identifier, it does not have to be quoted.

d = <dict create one 1 two 2 three 3 4 four "number five" 5>
→ one 1 two 2 three 3 4 four {number five} 5
d.two                 # String key "two" does not need quoting
→ 2
d . "number five"     # Quoted string literal key
→ 5
d . three             # Note whitespace is optional
→ 3
d.4                   # Numeric literal
→ four

To look up a key that is stored in a variable use the $ dereferencing operator.

key = "number five"
→ number five
d.$key
→ 5
d . $key           # Note whitespace is optional
→ 5

In the case of a column the lookup returns the index of the specified value.

Emps.Name.Sally
→ 0

In the prior example, note the overloading of the . operator When applied to tables, it acts as a column selector. When applied to columns (selected from the table in this case) it acts as a lookup operator.

Warning If the value occurs multiple times in the column, any of the corresponding indices may be returned and may not even be the same in repeated calls.

In all cases, an error is raised if the value being looked up does not exist in the dictionary or column.

These operators can also be used on the left hand side of an assignment to set new values.

d.c = 99
→ one 1 two 2 three 3 4 four {number five} 5 c 99

I = @string {'zero', 'one', 'two', 'three'}
→ tarray_column string {zero one two three}
I.one = 1
→ tarray_column string {zero 1 two three}
Note If a variable does not already exist, the operator creates it as a dictionary.

8.6. Indexing and search operators

Much of the convenience of Xtal in dealing with columns and tables stems from its variety of indexing options. Indices may be simple numerics, ranges, lists or selectors based on boolean expressions.

The flexibility of index selectors also means that they are the primary means of searching tables and columns for elements matching desired criteria. Many indexing and search operators can also be used with lists.

8.6.1. Integer indices

In the simplest case, an index is simply a integer value, specified as a literal or an expression that results in a integer value. As the operand of the [] operator, it retrieves the value of the element at that position.

Rainfall[0]          # Index a column with a constant
→ 11.0

i = 1
→ 1
Emps[i]              # Index a table with a variable
→ Tom 63000 36 Boston

{1, 2, 3, 4}[i+2]    # Index a list with an expression
→ 4

Note that the result is a ''scalar'', the value of the element at that position, not a column or table containing that element.

8.6.2. Index ranges

Alternatively, the index may specify an index range in the column or table using the syntax LOWEXPR : HIGHEXPR. So, the rainfall for the first quarter can be retrieved by

q1_rainfall = Rainfall[0:2]
→ tarray_column double {11.0 23.3 18.4}

In this case the index operator returns a column (or table) containing the specified elements. Thus notice the difference between

Rainfall[0]
→ 11.0

and

Rainfall[0:0]
→ tarray_column double {11.0}

though they both specify a single element.

The range limits may be expressions that result in integer values.

i = 0
print(Emps[i:%Emps-1])
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 68000| 32|Boston  |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 78000| 40|New York|
  +------+------+---+--------+
  |Harry | 43000| 37|New York|
...Additional lines omitted...

The upper limit in the range may be left unspecified in which it defaults to the last element. Thus the above could also be written as

print(Emps[i:])
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 68000| 32|Boston  |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 78000| 40|New York|
  +------+------+---+--------+
  |Harry | 43000| 37|New York|
...Additional lines omitted...

Ranges may be used with lists as well.

{1, 2, 3, 4}[1:2]
→ 2 3

We shall also see later that ranges can also be combined with selectors to further focus search and indexing operations.

8.6.3. Index lists

The third form of an index is an index list. Here you can specify a list of integers or a column of type int that contain the numeric indices of interest.

i = 10
Rainfall[{9, i, 11}]
→ tarray_column double {64.9 33.1 21.5}

We could have specified a column of type int instead.

indices = @int {11, 10, 9}
Rainfall[indices]
→ tarray_column double {21.5 33.1 64.9}

Notice that an index list (or column) does not need to contain the indices in order and may even contain duplicate indices. The values returned are in the order of specified indices.

Again, make a note of the difference with respect to a simple numeric index

Rainfall[0]
→ 11.0

which returns the element, versus

Rainfall[@int {0}]
→ tarray_column double {11.0}

which returns a column containing that single element.

Warning

However, because Tcl cannot distinguish between an integer and a list containing a single integer, the following expression returns a single element and not a column with a single element:

Rainfall[{0}]
→ 11.0

Therefore it is better to supply index lists as integer column as opposed to a list of integers.

Like simple indices and ranges, index lists can be used with lists as well.

{1, 2, 3, 4}[@int{0, 2, 1, 3}]
→ 1 3 2 4

8.6.4. Selectors

The final form an index can take is that of a selector which is a boolean expression involving the column or table.

Selecting from columns

As we noted in column operators comparison of a column with a scalar value results in an int column containing the indices for which the comparison of the corresponding element succeeded. This implies that the boolean expression can be used as an expression that return an index list.

As a simple example of a selector, consider the following expression:

Rainfall > 100
→ tarray_column int {5 6 7 8}

The return value is a column containing the indices corresponding to the elements which have a value greater than 100.

Therefore, we can use such boolean selectors as indices to retrieve the actual values themselves.

Rainfall[Rainfall > 100]
→ tarray_column double {180.5 210.2 205.8 126.4}
Selecting from tables

Selectors can be used for tables as well with the operand being a contained column. Moreover, the selector need not be a simple boolean expression. The following prints all employees who are over 35 but have an income less than 70,000.

print(Emps[Emps.Age > 35 && Emps.Salary < 70000])
→ +-----+------+---+--------+
  |Name |Salary|Age|Location|
  +-----+------+---+--------+
  |Tom  | 63000| 36|Boston  |
  +-----+------+---+--------+
  |Harry| 43000| 37|New York|
  +-----+------+---+--------+
Selecting from lists

Selectors may be used with lists as well. However, because Tcl lists cannot be distinguished from plain strings, the operators behave as selectors (as opposed to plain comparison operators) only when the operand is the list being indexed. It is best to use the @@ context, discussed later, when indexing lists. The example below clarifies this point.

% set L {0 0x10 16 20}
→ 0 0x10 16 20
% xtal::xtal {L[L == 16]} 1
→ 0x10 16
% xtal::xtal {L == 16} 2
→ 0
1 L == 16 results in {1 2} (selector operation)
2 L == 16 results in 0 (string comparison)
Ranges in selectors

Selectors can include ranges as well. For example, the rainfall in the last quarter that exceeds 50mm is given by

Rainfall[9:11 && Rainfall > 50]
→ tarray_column double {64.9}
Warning A selector can include function calls as well. However, note that the function is called just once, not once per element of the column or table.

Because of their flexibility, selectors are the mechanism for doing searches in columns and tables.

The @@ selector context

In previous examples, we supplied the variable containing the table or column, for example Emps, to the selector expression. In some cases, the table or column may not stored in a variable, for example when it is a function return value or when it is generated through an expression. In this case the @@ special token may be used to reference the table instead of storing the generated value in a temporary for naming purposes. This is illustrated in the following example.

% proc get_table {} { return $::Emps }
% xtal { print( get_table() [@@.Age > 35 && @@.Salary < 70000]) }
→ +-----+------+---+--------+
  |Name |Salary|Age|Location|
  +-----+------+---+--------+
  |Tom  | 63000| 36|Boston  |
  +-----+------+---+--------+
  |Harry| 43000| 37|New York|
  +-----+------+---+--------+

8.6.5. Using indices in assignments

The various forms of indexing can also be used in the left side of assignments. We have already seen some examples earlier. Below are some slightly more involved examples.

To give an increment only to low-paid employees make use of selectors.

Emps.Salary[Emps.Salary < 50000] = Emps.Salary[Emps.Salary < 50000] + 2000;
print (Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 68000| 32|Boston  |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Dick  | 78000| 40|New York|
  +------+------+---+--------+
  |Harry | 45000| 37|New York|
  +------+------+---+--------+
  |Amanda| 48000| 35|Seattle |
  +------+------+---+--------+

To modify a subset of the fields for a specific row, index a slice of the table. So to relocate Sally with a raise

Emps.(Salary, Location)[Emps.Name.Sally]  = {70000, 'Seattle'}
print (Emps[Emps.Name == 'Sally'])
→ +-----+------+---+--------+
  |Name |Salary|Age|Location|
  +-----+------+---+--------+
  |Sally| 70000| 32|Seattle |
  +-----+------+---+--------+

Use of indices in assignments is not limited to columns and rows. List elements can be assigned in the same manner.

l = {1, 2, 3}
→ 1 2 3
l[0] = 101
→ 101 2 3
l[{3,4}] = {1000, 1001}
→ 101 2 3 1000 1001

8.7. Miscellaneous operators

8.7.1. The % size operator

The unary operator % returns the size of its operand which must be a list, column or table.

len = % {1,2,3,4}   # Length of a list
→ 4
% Rainfall          # Size of a column
→ 12
% (Rainfall > 100)  # Size of a column returned by an expression
→ 4
nemps = % Emps      # Size of a table
→ 5

8.7.2. The $ dereferencing operator

The operator $ is a unary operator that dereferences the value of its operand. We have already seen an example of its use earlier.

The operator can only be applied to an identifier or a string literal and not to an arbitrary expression. In both cases, it returns the value of the variable whose name is given by the operand value.

The operator is useful in a couple of different situations. One is when you need to reference a variable that does not follow Xtal identifier syntax, e.g. one with spaces in it, as shown in the Tcl code below.

% set "variable with spaces in name" "value of variable with spaces"
→ value of variable with spaces
% set varname "variable with spaces in name"
→ variable with spaces in name

Then the equivalent of the Tcl code

% puts ${variable with spaces in name}
→ value of variable with spaces
% puts [set $varname]
→ value of variable with spaces

would be the following in Xtal

puts($"variable with spaces in name")
→ value of variable with spaces
puts($varname)
→ value of variable with spaces

The other use of the dereferencing operator is when Xtal expects a symbol such as function or object name, a lookup operand, a table column name etc. and you wish to supply it through a variable. We saw examples of this earlier which are repeated below.

obj = OExample.new()
→ ::oo::Obj118
$obj.destroy()            # Indirect object reference

d.$key                    # Indirect dictionary lookup
→ 5

name = "Sally"
→ Sally
Emps.Name.$name           # Indirect column lookup
→ 0

colname = 'Location'
→ Location
print(Emps . $colname)    # Indirect table column reference
→ Seattle
  Boston
  New York
  New York
  Seattle
Note Dereferencing operators cannot be used on the left hand side of an assignment statement.

9. Built-in functions

A number of tarray commands that are not implemented as operators, or for which only a subset of functionality is available through operators, are available as built-in functions. These built-in functions can always be invoked through the normal function call mechanism using their qualified name (for example, column.reverse). However, invoking them as built-ins has some convenience like polymorphism and slightly shorter syntax.

Note The documentation provides a summary of the built-in functions. For full details you can refer to the documentation for the corresponding tarray command.

9.1. The @delete function

The @delete deletes elements from a column or table. It has one of two forms:

@delete(COLUMN_OR_TABLE, FIRST : LAST)
@delete(COLUMN_OR_TABLE, INDICES)

The first form deletes all elements in the index range from FIRST to LAST. The second form deletes the elements whose indices are specified by INDICES which may be a single index, an index column or a list of indices.

So for instance we could get rid of all our high-priced employees.

Emps = @delete(Emps, Emps.Salary > 75000)
print(Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 70000| 32|Seattle |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Harry | 45000| 37|New York|
  +------+------+---+--------+
  |Amanda| 48000| 35|Seattle |
  +------+------+---+--------+

9.2. The @fill function

The @fill fills elements from a column or table with a fixed value. It has one of two forms:

@fill(COLUMN_OR_TABLE, VALUE, FIRST : LAST)
@fill(COLUMN_OR_TABLE, VALUE, _INDICES)

The first form sets all elements in the index range from FIRST to LAST to VALUE. The second form fills the elements whose indices are specified by INDICES which may be a single index, an index column or a list of indices.

We can raise all low paid employees to a minimum salary level.

Emps.Salary = @fill(Emps.Salary, 50000, Emps.Salary < 50000)
print(Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 70000| 32|Seattle |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Harry | 50000| 37|New York|
  +------+------+---+--------+
  |Amanda| 50000| 35|Seattle |
  +------+------+---+--------+

The function can also be used to initialize a column or table with a fixed value.

Col = @fill(@int, 0, 0:9)
→ tarray_column int {0 0 0 0 0 0 0 0 0 0}

9.3. The @inject function

The assignment operators in Xtal cannot be used to insert elements between two elements in a column or rows in a table. The @inject and @insert functions are provided instead for that purpose. The @inject command inserts the contents of a list or column (table) into another and returns the result. The syntax is

@inject(COLUMN_OR_TABLE, SOURCE, START)

where SOURCE is a list or column (table) to be inserted at position START in COLUMN_OR_TABLE. For example, we can add a couple of low priced lackeys to replace the ones we layed off.

Emps = @inject(Emps, {
    {'Tom',      30000,      38,       'Boston'},
    {'Peyton',   30000,      38,       'Denver'}
}, 0)
print(Emps)
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Tom   | 30000| 38|Boston  |
  +------+------+---+--------+
  |Peyton| 30000| 38|Denver  |
  +------+------+---+--------+
  |Sally | 70000| 32|Seattle |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Harry | 50000| 37|New York|
  +------+------+---+--------+
  |Amanda| 50000| 35|Seattle |
  +------+------+---+--------+

We supplied a list of rows to be added above but we could also have specified a compatible table instead.

9.4. The @insert function

The @insert function is similar to @inject but inserts a single repeated value or row into a column or table. It has the syntax

@insert(COLUMN_OR_TABLE, VALUE, START ?, COUNT?)

The function inserts COUNT instances of VALUE at the index position START. COUNT defaults to 1 if unspecified.

@insert (@int {1,2,3}, 99, 0)
→ tarray_column int {99 1 2 3}

@insert (@int {1,2,3}, 100, 1, 2)
→ tarray_column int {1 100 100 2 3}

9.5. The @lookup function

The function returns the index of an element in a column which must be of type string. It has the syntax

@lookup(STRINGCOLUMN ?, KEY?)

The command returns the index of an element in STRINGCOLUMN that exactly matches LOOKUPKEY or -1 if not found. If KEY is not specified, command builds an internal dictionary (see below) and the return value is an empty string.

Unlike the @search command, the returned index is not necessarily that of the first occurence in cases where KEY occurs multiple times in the column.

The primary purpose of this function is to provide fast access to columns that are used as an index. For more details, see the documentation for the column lookup command.

9.6. The @reverse function

The @reverse function reverses columns and tables.

@reverse (@int {1,2,3})
→ tarray_column int {3 2 1}

9.7. The @search command

The @search command searches a specified column and returns matching elements or the corresponding indices. Unlike most of the other built-ins, @search is called as a command and does not use the standard function call notation. It can take one of two forms. In the first form, the search target is specified directly:

@search COLUMNEXPR RELOP EXPRESSION ?SEARCHOPTIONS?

In this form, COLUMNEXPR can be any expression that results in a column value. RELOP is any relational operator. Without any options specified, the command returns the index of the first element of the column that matches EXPRESSION. SEARCHOPTIONS is a space-separated list of options that modify this behaviour. The possible option values are shown in @search options.

Table 2. @search options
Option Description

all

If specified, a column containing all matching elements are returned as opposed to just the first match. The type of the column is int if the inline option is not specified. Otherwise it is the type of the search target.

inline

If specified, the command returns the value, or values if the all option is specified, of the matching element, or elements, as opposed to their indices.

The following examples illustrate the various combinations.

@search Emps.Salary > 40000
→ 2

@search Emps.Salary > 40000 all
→ tarray_column int {2 3 4 5}

@search Emps.Salary > 40000 inline
→ 70000

@search Emps.Salary > 40000 inline all
→ tarray_column uint {70000 63000 50000 50000}

The second form that @search takes is

@search INDEXCOLUMNCOLUMNEXPR _ RELOP EXPRESSION ?SEARCHOPTIONS?

In this case, INDEXCOLUMN is expected to contain indices into the column COLUMNEXPR and the search only examines the corresponding elements for a match. In all other respects, this form of the command behaves the same as the first form. See the next section for an example.

Search versus selectors

Given that selectors perform a similar function to @search, when might one be preferred to the other?

The selector form is more succint, convenient and not limited in its expressive power related to conditions. However, the search command in the current implemenation has some performance advantages illustrated in the example below.

Suppose we want to list the salaries of all employees in Boston who make more than 45000/year. We could write this using selectors.

print (Emps.Name[Emps.Salary > 45000 && Emps.Location == 'Boston'])
→ Tom

Alternatively, using @search,

indices = @search (Emps.Salary > 45000) -> Emps.Location == 'Boston' all
print (Emps.Name[indices])
→ Tom

In the first case, Xtal searches the appropriate columns for salaries greater than the specified amount and then again for the specified location. Finally, it returns the intersection of the two and retrieves those names from the Name column of the table.

In the second case, the list of indices where the salary is greater than the specified amount is retrieved. Then only those elements are searched for the location. Consequently, this method is significantly faster for large data sets.

Note This performance benefit is only true because the current Xtal optimizer is extremely rudimentary and not capable of recognising and transforming the selector into the more efficient form. It is hoped future releases will improve on this.

9.8. The @sort command

The @sort command provides a flexible mechanism for sorting columns. Like @search and unlike most of the other built-ins, @sort is called as a command and does not use the standard function call notation. It can take one of two forms. In the first form, the sort target is specified directly:

@sort COLUMNEXPR ?SORTOPTIONS?

In this form, COLUMNEXPR can be any expression that results in a column value. The command then returns a column of the same type sorted based on the specified options. SORTOPTIONS is a space-separated list of options that control the sorting operations. The possible option values are shown in @sort options.

Table 3. @sort options
Option Description

decreasing

Sorts in order of decreasing value

increasing

Sorts in order of increasing value

indices

Specifies that the command should return the indices of the sorted values instead of the values themselves.

nocase

Specifies that sorting should be done in case-insensitive manner if the column is of type string or any. The default is a case-sensitive sort.

So we might want to get the rainfall in sorted order.

% print (Rainfall)
→ 11.0, 23.3, 18.4, 14.7, 70.3, ..., 205.8, 126.4, 64.9, 33.1, 21.5
%
% print (@sort Rainfall)
→ 11.0, 14.7, 18.4, 21.5, 23.3, ..., 70.3, 126.4, 180.5, 205.8, 210.2

But what is probably more interesting is the order of months so we might instead do the following (this time choosing a decreasing sort order).

% print (@sort Rainfall decreasing indices)
→ 6, 7, 5, 8, 4, ..., 1, 11, 2, 3, 0

This returns a column of type int containing the indices of the sorted values.

The sorted indices are useful in many circumstances. For example, we might display the employee database in multiple windows, one sorted by age, the other by salary. Instead of keeping two copies of the table sorted differently, it is cheaper in terms of memory to keep indices sorted differently. For example,

% EmpsByAge    = @sort Emps.Age    indices
→ tarray_column int {2 5 3 4 0 1}
% EmpsBySalary = @sort Emps.Salary indices decreasing
→ tarray_column int {2 3 4 5 0 1}
%
% print (Emps[EmpsByAge])
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 70000| 32|Seattle |
  +------+------+---+--------+
  |Amanda| 50000| 35|Seattle |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Harry | 50000| 37|New York|
...Additional lines omitted...
%
% print (Emps[EmpsBySalary])
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Sally | 70000| 32|Seattle |
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
  +------+------+---+--------+
  |Harry | 50000| 37|New York|
  +------+------+---+--------+
  |Amanda| 50000| 35|Seattle |
...Additional lines omitted...

The second form that @sort takes is

@sort INDEXCOLUMNCOLUMNEXPR ?SORTOPTIONS?

Here the sorting is done indirectly. INDEXCOLUMN is an integer index column as the ones we computed above. This is treated as containing indices into the column COLUMNEXPR. The return value is INDEXCOLUMN sorted by comparing the corresponding values from the column COLUMNEXPR.

There are a couple of scenarios where this is useful. One is in keeping a stable sort order when sorting successively on multiple keys. This is discussed in detail in Sort stability in the Programmer’s guide.

The other example is when only a subset of a column or table is to be sorted. If we wanted to list salaries in sorted order but only for employees older than 35, we could do the following:

OlderEmployees = Emps.Age > 35
→ tarray_column int {0 1 3 4}

print (Emps[@sort OlderEmployees -> Emps.Salary])
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Tom   | 30000| 38|Boston  |
  +------+------+---+--------+
  |Peyton| 30000| 38|Denver  |
  +------+------+---+--------+
  |Harry | 50000| 37|New York|
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
...Additional lines omitted...

Or if you want to save on lines,

print (Emps[@sort (Emps.Age > 35) -> Emps.Salary])
→ +------+------+---+--------+
  |Name  |Salary|Age|Location|
  +------+------+---+--------+
  |Tom   | 30000| 38|Boston  |
  +------+------+---+--------+
  |Peyton| 30000| 38|Denver  |
  +------+------+---+--------+
  |Harry | 50000| 37|New York|
  +------+------+---+--------+
  |Tom   | 63000| 36|Boston  |
...Additional lines omitted...

This method is a lot cheaper than working directly with the table.

9.9. The @sum function

The @sum function returns the sum of the elements in a column.

@sum(NUMERIC_COLUMNEXPR)

So to get our annual salary expenditure,

@sum(Emps.Salary)
→ 293000

10. Futures

Xtal is still under development. Syntactic and functional changes are likely. The runtime is currently scripted in Tcl for ease of development and therefore performance on scalar operation (as opposed to columns and tables) will lag that of Tcl. It will be reimplemented using Tcl’s C API before final release.


1. Xtal is an abbreviation for crystal in the electronics world.