vd

Section: Misc. Reference Manual Pages (\&1)
Updated: Quick Reference Guide
Page Index

BSD mandoc
Linux/MacOS  

NAME

VisiData - a terminal utility for exploring and arranging tabular data  

SYNOPSIS

vd [options ] [input ... ]
vd [options ] --play cmdlog [-w waitsecs ] [--batch ] [-o output ] [field = value ... ]  

DESCRIPTION

VisiData is a multipurpose tool built on the vdtui platform that can be used to explore, clean, edit, and restructure data. Rows can be selected, filtered, and grouped; columns can be rearranged, transformed, and derived via regex or Python expressions; and workflows can be saved, documented, and replayed.  

REPLAY MODE

-p , --play = cmdlog
replay a saved cmdlog within the interface
-w , --replay-wait = seconds
wait seconds between commands
-b , --batch
replay in batch mode (with no interface)
-o , --output = file
save final visible sheet to file as .tsv
--replay-movement
toggle --play to move cursor cell-by-cell
field = value
replace { field } in cmdlog contents with value

 

Commands During Replay

^U
pause/resume replay
Tab
execute next row in replaying sheet
^K
cancel current replay

 

GLOBAL COMMANDS

In most cases, commands that affect selected rows will affect all rows if no rows are selected.

 

Keystrokes for the Cautious

^H
view this man page
z^H
view sheet of commands and keybindings
^Q
abort program immediately
^C
cancel user input or abort all async threads on current sheet
q
quit current sheet
gq
quit all sheets (clean exit)

 

Cursor Movement

Arrow PgUp Home
move as expected
h j k l
move left/down/up/right
gh gj gk gl
move all the way to the left/bottom/top/right of sheet
G gg
move all the way to the bottom/top of sheet
^B ^F
scroll one page back/forward

^^ (Ctrl-^)
jump to previous sheet (swaps with current sheet)

/ ? regex
search for regex forward/backward in current column
g/ g? regex
search for regex forward/backward over all visible columns
z/ z? expr
search by Python expr forward/backward in current column (with column names as variables)
n N
move to next/previous match from last search

< >
move up/down to next value in current column
z< z>
move up/down to next null in current column
{ }
move up/down to next selected row

c regex
move to next column with name matching regex
r regex
move to next row with key matching regex
zc zr number
move to column/row number (0-based)

H J K L
slide current row/column left/down/up/right
gH gJ gK gL
slide current row/column all the way to the left/bottom/top/right of sheet

zh zj zk zl
scroll one left/down/up/right

 

Column Manipulation

_ (underscore)
adjust width of current column
g_
adjust width of all visible columns
z_ number
adjust width of current column to number

- (hyphen)
hide current column
z- reduce width of current column by half
gv unhide all columns

! z! toggle/unset current column as a key column
~ # % $ @ z#
set type of current column to str/int/float/currency/date/len
^
edit name of current column
g^
set names of all unnamed visible columns to contents of selected rows (or current row)
z^
set name of current column to combined contents of current cell in selected rows (or current row)
gz^
set name of all visible columns to combined contents of current column for selected rows (or current row)

= expr
create new column from Python expr , with column names as variables
g= expr
set current column for selected rows to result of Python expr
gz= expr
set current column for selected rows to the items in result of Python sequence expr
z= expr
evaluate Python expression on current row and show result on status line

' (tick)
add a frozen copy of current column with all cells evaluated
g'
open a frozen copy of current sheet with all visible columns evaluated
z' gz'
reset cache for current/all visible column(s)

: regex
add new columns from regex split; number of columns determined by example row at cursor
; regex
add new columns from capture groups of regex (also requires example row)
z ; expr
add new column from bash expr , with $ columnNames as variables
* regex / subst
add column derived from current column, replacing regex with subst (may include \1 backrefs)
g* gz* regex / subst
modify selected rows in current/all visible column(s), replacing regex with subst (may include \1 backrefs)

( g(
expand current/all visible column(s) of lists (e.g. [3] ) or dicts (e.g. {3} ) fully
z( gz( depth
expand current/all visible column(s) of lists (e.g. [3] ) or dicts (e.g. {3} ) to given depth ( 0 = fully)
)
unexpand current column; restore original column and remove other columns at this level

 

Row Selection

s t u
select/toggle/unselect current row
gs gt gu
select/toggle/unselect all rows
zs zt zu
select/toggle/unselect rows from top to cursor
gzs gzt gzu
select/toggle/unselect rows from cursor to bottom
| \  regex
select/unselect rows matching regex in current column
g| g\  regex
select/unselect rows matching regex in any visible column
z| z\  expr
select/unselect rows matching Python expr in any visible column
, (comma)
select rows matching current cell in current column
g,
select rows matching current row in all visible columns

 

Row Sorting/Filtering

[ ]
sort ascending/descending by current column
g[ g]
sort ascending/descending by all key columns
open duplicate sheet with only selected rows
g
open duplicate sheet with all rows
gz
open duplicate sheet with deepcopy of selected rows

 

Editing Rows and Cells

a za
append a blank row/column
ga gza number
append number blank rows/columns
d gd
delete (cut) current/selected row(s) and move to clipboard
y gy
yank (copy) current/all selected row(s) to clipboard
zy gzy
yank (copy) contents of current column for current/selected row(s) to clipboard
zd gzd
delete (cut) contents of current column for current/selected row(s) and move to clipboard
p P
paste clipboard rows after/before current row
zp gzp
set contents of current column for current/selected row(s) to last clipboard value
Y gY
yank (copy) current/all selected row(s) to system clipboard (using options.clipboard_copy_cmd )
zY gzY
yank (copy) contents of current column for current/selected row(s) to system clipboard (using options.clipboard_copy_cmd )
f
fill null cells in current column with contents of non-null cells up the current column
e text
edit contents of current cell
ge text
set contents of current column for selected rows to text

 

Commands While Editing Input

Enter ^C
accept/abort input
^O
open external $EDITOR to edit contents
^R
reload initial value
^A ^E
move to beginning/end of line
^B ^F
move back/forward one character
^H ^D
delete previous/current character
^T
transpose previous and current characters
^U ^K
clear from cursor to beginning/end of line
Backspace Del
delete previous/current character
Insert
toggle insert mode
Up Down
set contents to previous/next in history
Tab Shift+Tab
autocomplete input (when available)

 

Data Toolkit

o input
open input in VisiData
^S g^S filename
save current/all sheet(s) to filename in format determined by extension (default .tsv)
""
Note: if the format does not support multisave, or the filename ends in a / , a directory will be created.
z^S filename
save key columns and current column only to filename in format determined by extension (default .tsv)
^D filename.vd
save CommandLog to filename.vd file
A number
open new blank sheet with number columns
R number
pushes sheet with random population subset of number rows
T
open new sheet with rows and columns transposed

+ aggregator
add aggregator to current column (see Frequency Table )
z+ aggregator
display result of aggregator over values in selected rows for current column

 

Data Visualization

. (dot)
plot current numeric column vs key columns. The numeric key column is used for the x-axis; categorical key column values determine color.
g.
plot a graph of all visible numeric columns vs key columns.

If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), . plots the current row, and g. plots all selected rows (or all rows if none selected).  

Canvas-specific Commands

+ -
increase/decrease zoom level, centered on cursor
_ (underscore)
zoom to fit full extent
s t u
select/toggle/unselect rows on source sheet contained within canvas cursor
gs gt gu
select/toggle/unselect rows on source sheet visible on screen
d
delete rows on source sheet contained within canvas cursor
gd
delete rows on source sheet visible on screen
Enter
open sheet of source rows contained within canvas cursor
gEnter
open sheet of source rows visible on screen
1 - 9
toggle display of layers
^L
redraw all pixels on canvas
v
toggle show_graph_labels option
mouse scrollwheel
zoom in/out of canvas
left click-drag
set canvas cursor
right click-drag
scroll canvas

 

Other Commands

Q
quit current sheet and remove it from the CommandLog
V
view contents of current cell in a new TextSheet
v
toggle sheet-specific visibility (text wrap on TextSheet, legends/axes on Graph)

Space longname
execute command by its longname

^E
view traceback for most recent error
g^E
view traceback for most recent errors
z^E
view traceback for error in current cell

^L
refresh screen
^R
reload current sheet
z^R
clear cache for current column
^Z
suspend VisiData process
^G
show cursor position and bounds of current sheet on status line
^V
show version and copyright information on status line
^P
open Status History

^Y z^Y g^Y
open current row/cell/sheet as Python object
^X expr
evaluate Python expr and opens result as Python object
z^X expr
evaluate Python expr on current row and shows result on status line
g^X stmt
execute Python stmt in the global scope

 

Internal Sheets List

.
Directory Sheet browse and modify properties of files in a directory
"
Metasheets
.
Columns Sheet (Shift+C) edit column properties
.
Sheets Sheet (Shift+S) jump between sheets or join them together
.
Options Sheet (Shift+O) edit configuration options
.
Commandlog (Shift+D) modify and save commands for replay
.
Error Sheet (^E) view last error
.
Status History (^P) view history of status messages
.
Threads Sheet (^T) view, cancel, and profile asynchronous threads

Derived Sheets
.
Frequency Table (Shift+F) group rows by column value, with aggregations of other columns
.
Describe Sheet (Shift+I) view summary statistics for each column
.
Pivot Table (Shift+W) group rows by key and summarize current column
.
Melted Sheet (Shift+M) unpivot non-key columns into variable/value columns

 

INTERNAL SHEETS

 

Directory Sheet

(sheet-specific commands)
Modifying any cell changes the in-memory value. Changes are only applied to the filesystem with ^S

Enter gEnter
open current/selected file(s) as new sheet(s)
^O g^O
open current/selected file(s) in external $EDITOR
d gd
schedule current/selected file(s) for deletion
^R z^R gz^R
reload information for all/current/selected file(s), undoing any pending changes
z^S ^S
apply all deferred changes to current/all file(s)

 

METASHEETS

 

Columns Sheet (Shift+C)

Properties of columns on the source sheet can be changed with standard editing commands ( e ge g= Del ) on the Columns Sheet . Multiple aggregators can be set by listing them (separated by spaces) in the aggregators column. The 'g' commands affect the selected rows, which are the literal columns on the source sheet.

(global commands)

gC
open Columns Sheet for all visible columns on all sheets

(sheet-specific commands)

&
add column from concatenating selected source columns
g! gz!
toggle/unset selected columns as key columns on source sheet
g+ aggregator
add Ar aggregator No to selected source columns
g- (hyphen)
hide selected columns on source sheet
g~ g# g% g$ g@ gz#
set type of selected columns on source sheet to str/int/float/currency/date/len
Enter
open a Frequency Table sheet grouped by column referenced in current row

 

Sheets Sheet (Shift+S)

(global commands)

gS
open Sheets Graveyard which includes references to closed sheets

(sheet-specific commands)

Enter
jump to sheet referenced in current row
a
add row to reference a new blank sheet
gC
open Columns Sheet with all columns from selected sheets
gI
open Describe Sheet with all columns from selected sheets
g^R
reload all selected sheets
z^C gz^C
abort async threads for current/selected sheets(s)
& jointype
merge selected sheets with visible columns from all, keeping rows according to jointype :

.
inner keep only rows which match keys on all sheets
.
outer keep all rows from first selected sheet
.
full keep all rows from all sheets (union)
.
diff keep only rows NOT in all sheets
.
append keep all rows from all sheets (concatenation)
.
extend copy first selected sheet, keeping all rows and sheet type, and extend with columns from other sheets

 

Options Sheet (Shift+O)

(global commands)

Shift+O
edit global options (apply to all sheets )
zO
edit sheet options (apply to this sheet only)
gO
open ~/.visidatarc

(sheet-specific commands)

Enter e
edit option at current row

 

CommandLog (Shift+D)

(global commands)

gD
open Directory Sheet for options.visidata_dir (default: ~/.visidata/ ), which contains saved commandlogs and macros

(sheet-specific commands)

x
replay command in current row
gx
replay contents of entire CommandLog
^C
abort replay
z^S keystroke
save selected rows to macro mapped to keystroke
""
Macros are saved to .visidata/macro/command-longname.vd . The list of macros is saved at .visidata/macros.vd (keystroke, filename).

 

DERIVED SHEETS

 

Frequency Table (Shift+F)

A Frequency Table groups rows by one or more columns, and includes summary columns for those with aggregators.
(global commands)

gF
open Frequency Table, grouped by all key columns on source sheet
zF
open one-line summary for selected rows

(sheet-specific commands)

s t u
select/toggle/unselect these entries in source sheet
Enter
open sheet of source rows that are grouped in current cell

 

Describe Sheet (Shift+I)

(global commands)

gI
open Describe Sheet for all visible columns on all sheets

(sheet-specific commands)

zs zu
select/unselect rows on source sheet that are being described in current cell
!
toggle/unset current column as a key column on source sheet
Enter
open a Frequency Table sheet grouped on column referenced in current row
zEnter
open copy of source sheet with rows described in current cell

 

Pivot Table (Shift+W)

Set key column(s) and aggregators on column(s) before pressing Shift+W on the column to pivot.
(sheet-specific commands)

Enter
open sheet of source rows aggregated in current pivot row
zEnter
open sheet of source rows aggregated in current pivot cell

 

Melted Sheet (Shift+M)

Open melted sheet (unpivot), with key columns retained and all non-key columns reduced to Variable-Value rows.
(global commands)

gM regex
open melted sheet (unpivot), with key columns retained and regex capture groups determining how the non-key columns will be reduced to Variable-Value rows.

 

Python Object Sheet (^X ^Y g^Y z^Y)

(sheet-specific commands)

Enter
dive further into Python object
e
edit contents of current cell
v
toggle show/hide for methods and hidden properties
gv zv
show/hide methods and hidden properties

 

COMMANDLINE OPTIONS

-f , --filetype = filetype
tsv set loader to use for filetype instead of file extension
-y , --confirm-overwrite = F
True overwrite existing files without confirmation
--diff = base
None add colorizer for all sheets against base

--encoding = str utf-8
encoding passed to codecs.open
--encoding-errors = str surrogateescape
encoding_errors passed to codecs.open
--regex-flags = str I
flags to pass to re.compile() [AILMSUX]
--default-width = int 20
default column width
--wrap False
wrap text to fit window width on TextSheet
--bulk-select-clear False
clear selected rows before new bulk selections
--cmd-after-edit = str go-down
command longname to execute after successful edit
--col-cache-size = int 0
max number of cache entries in each cached column
--quitguard False
confirm before quitting last sheet
--null-value = NoneType None
a value to be counted as null
--force-valid-colnames False
clean column names to be valid Python identifiers
--debug False
exit on error and display stacktrace
--curses-timeout = int 100
curses timeout in ms
--force-256-colors False
use 256 colors even if curses reports fewer
--use-default-colors False
curses use default terminal colors
--note-pending = str 1B
note to display for pending cells
--note-format-exc = str ?
cell note for an exception during formatting
--note-getter-exc = str !
cell note for an exception during computation
--note-type-exc = str !
cell note for an exception during type conversion
--note-unknown-type = str
cell note for unknown types in anytype column
--scroll-incr = int 3
amount to scroll with scrollwheel
--skip = int 0
skip first N lines of text input
--confirm-overwrite = bool True
whether to prompt for overwrite confirmation on save
--safe-error = str #ERR
error string to use while saving
--header = int 1
parse first N rows of certain formats as column names
--delimiter = str
delimiter to use for tsv filetype
--filetype = str
specify file type
--save-filetype = str tsv
specify default file type to save as
--tsv-safe-newline = str
replacement for tab character when saving to tsv
--tsv-safe-tab = str
replacement for newline character when saving to tsv
--clipboard-copy-cmd = str
command to copy stdin to system clipboard
--visibility = int 0
visibility level (0=low, 1=high)
--min-memory-mb = int 0
minimum memory to continue loading and async processing
--replay-wait = float 0.0
time to wait between replayed commands, in seconds
--replay-movement False
insert movements during replay
--visidata-dir = str ~/.visidata/
directory to load and store macros
--rowkey-prefix = str AD
string prefix for rowkey in the cmdlog
--cmdlog-histfile = str
file to autorecord each cmdlog action to
--regex-maxsplit = int 0
maxsplit to pass to regex.split
--show-graph-labels = bool True
show axes and legend on graph
--plot-colors = str
list of distinct colors to use for plotting distinct objects
--zoom-incr = float 2.0
amount to multiply current zoomlevel when zooming
--motd-url = str
source of randomized startup messages
--profile = str
filename to save binary profiling data
--csv-dialect = str excel
dialect passed to csv.reader
--csv-delimiter = str ,
delimiter passed to csv.reader
--csv-quotechar = str
quotechar passed to csv.reader
--csv-skipinitialspace = bool True
skipinitialspace passed to csv.reader
--csv-escapechar = NoneType None
escapechar passed to csv.reader
--safety-first False
sanitize input/output to handle edge cases, with a performance cost
--json-indent = NoneType None
indent to use when saving json
--fixed-rows = int 1000
number of rows to check for fixed width columns
--pcap-internet = str n
(y/s/n) if save_dot includes all internet hosts separately (y), combined (s), or does not include the internet (n)
--graphviz-edge-labels = bool True
whether to include edge labels on graphviz diagrams

 

DISPLAY OPTIONS

Display options can only be set via the Sx Options Sheet or a .visidatarc (see Sx FILES ).

disp_note_none 00
visible contents of a cell whose value is None
disp_truncator 26
indicator that the contents are only partially visible
disp_oddspace B7
displayable character for odd whitespace
disp_unprintable .
substitute character for unprintables
disp_column_sep |
separator between columns
disp_keycol_sep 16
separator between key columns and rest of columns
disp_status_fmt {sheet.name}|
status line prefix
disp_lstatus_max 0
maximum length of left status line
disp_status_sep
separator between statuses
disp_edit_fill _
edit field fill character
disp_more_left <
header note indicating more columns to the left
disp_more_right >
header note indicating more columns to the right
disp_error_val
displayed contents for computation exception
disp_ambig_width 1
width to use for unicode chars marked ambiguous
color_default normal
the default color
color_default_hdr bold underline
color of the column headers
color_current_row reverse
color of the cursor row
color_current_col bold
color of the cursor column
color_current_hdr bold reverse underline
color of the header for the cursor column
color_column_sep 246 blue
color of column separators
color_key_col 81 cyan
color of key columns
color_hidden_col 8
color of hidden columns on metasheets
color_selected_row 215 yellow
color of selected rows
color_keystrokes white
color of input keystrokes on status line
color_status bold
status line color
color_error red
error message color
color_warning yellow
warning message color
color_edit_cell normal
cell color to use when editing cell
disp_pending
string to display in pending cells
color_note_pending bold magenta
color of note in pending cells
color_note_type 226 yellow
cell note for numeric types in anytype columns
disp_date_fmt %Y-%m-%d
default fmtstr to strftime for date values
color_change_pending reverse yellow
color for file attributes pending modification
color_delete_pending red
color for files pending delete
disp_histogram *
histogram element character
disp_histolen 50
width of histogram column
color_working green
color of system running smoothly
disp_replay_play B6
status indicator for active replay
disp_replay_pause 16
status indicator for paused replay
color_status_replay green
color of replay status indicator
disp_pixel_random False
randomly choose attr from set of pixels instead of most common
color_graph_hidden 238 blue
color of legend for hidden attribute
color_graph_selected bold
color of selected graph points
color_graph_axis bold
color for graph axis labels
color_diff red
color of values different from --diff source
color_diff_add yellow
color of rows/columns added to --diff source

 

EXAMPLES

vd foo.tsv
open the file foo.tsv in the current directory

vd -f sqlite bar.db
open the file bar.db as a sqlite database

vd -b countries.fixed -o countries.tsv
convert countries.fixed (in fixed width format) to countries.tsv (in tsv format)

vd postgres:// username : password @ hostname : port / database
open a connection to the given postgres database

vd --play tests/pivot.vd --replay-wait 1 --output tests/pivot.tsv
replay tests/pivot.vd, waiting 1 second between commands, and output the final sheet to test/pivot.tsv

ls -l | vd -f fixed --skip 1 --header 0
parse the output of ls -l into usable data

 

FILES

At the start of every session, VisiData looks for $HOME/.visidatarc , and calls Python exec() on its contents if it exists. For example:
   options.min_memory_mb=100  # stop processing without 100MB free

   bindkey('0', 'go-leftmost')   # alias '0' to move to first column, like vim

   def median(values):
       L = sorted(values)
       return L[len(L)//2]

   aggregator('median', median)

Functions defined in .visidatarc are available in python expressions (e.g. in derived columns).  

SUPPORTED SOURCES

These are the supported sources:

tsv (tab-separated value)

Plain and simple. VisiData writes tsv format by default. See the --delimiter option.

csv (comma-separated value)

.csv files are a scourge upon the earth, and still regrettably common.
See the --csv-dialect , --csv-delimiter , --csv-quotechar , and --csv-skipinitialspace options.
Accepted dialects are excel-tab , unix , and excel

fixed (fixed width text)

Columns are autodetected from the first 1000 rows (adjustable with --fixed-rows ).

json (single object) and jsonl (one object per line).

Cells containing lists (e.g. [3] ) or dicts ( {3} ) can be expanded into new columns with ( and unexpanded with )

yaml / yml (requires PyYAML )

pcap ( requires xpkt , dnslib )

View and investigate captured network traffic in a tabular format.

png (requires pypng )

Pixels can be edited and saved in data form. Images can be plotted with . (dot).

The following URL schemes are supported:

http (requires requests ); can be used as transport for with another filetype
postgres (requires psycopg2 )

The following sources may include multiple tables. The initial sheet is the table directory;
Enter loads the entire table into memory.

sqlite
xlsx (requires openpyxl )
xls (requires xlrd )
hdf5 (requires h5py )
ttf / otf (requires fonttools )
mbtiles (requires mapbox-vector-tile )
htm / html (requires lxml )
xml (requires lxml )

v
show only columns in current row attributes
za
add column for xml attributes

xpt (SAS; requires xport )
sas7bdat (SAS; requires sas7bdat )
sav (SPSS; requires savReaderWriter )
dta (Stata; requires pandas )
shp (requires pyshp )

In addition, .zip , .gz , .bz2 , and .xz files are decompressed on the fly.

VisiData has an adapter for pandas . To load a file format which is supported by pandas , pass -f pandas data.foo . This will call pandas.read_foo()

For example, vd -f pandas data.parquet loads a parquet file. Note that when using the pandas loader, the .fileformat file extension is mandatory  

SUPPORTED OUTPUT FORMATS

These are the supported savers:

tsv (tab-separated value)
csv (comma-separated value)
json (one object with all rows)

All expanded subcolumns must be closed (with ) ) to retain the same structure.
.shp files can be saved as geoJSON

md (org-mode compatible markdown table)
htm / html (requires lxml )
png (requires pypng )

Multisave is supported by html , md , and txt ; g^S will save all sheets into a single output file.

 

AUTHOR

VisiData was made by An Saul Pwanson Aq Mt vd@saul.pw Ns .


 

Index

NAME
SYNOPSIS
DESCRIPTION
REPLAY MODE
Commands During Replay
GLOBAL COMMANDS
Keystrokes for the Cautious
Cursor Movement
Column Manipulation
Row Selection
Row Sorting/Filtering
Editing Rows and Cells
Commands While Editing Input
Data Toolkit
Data Visualization
Canvas-specific Commands
Other Commands
Internal Sheets List
INTERNAL SHEETS
Directory Sheet
METASHEETS
Columns Sheet (Shift+C)
Sheets Sheet (Shift+S)
Options Sheet (Shift+O)
CommandLog (Shift+D)
DERIVED SHEETS
Frequency Table (Shift+F)
Describe Sheet (Shift+I)
Pivot Table (Shift+W)
Melted Sheet (Shift+M)
Python Object Sheet (^X ^Y g^Y z^Y)
COMMANDLINE OPTIONS
DISPLAY OPTIONS
EXAMPLES
FILES
SUPPORTED SOURCES
SUPPORTED OUTPUT FORMATS
AUTHOR
LinuxReviews : manual page archive : man1