Skip to content

Raw Text Parser

The Raw Text Parser measures every text file by its lines and their indentation, whatever the language. Deep indentation is a quick, language-independent hint at nested, hard-to-read code. For the about 300 languages that tokei knows, it also counts code and comment lines. Use this parser for languages the Unified Parser does not support, or next to it.

Terminal window
ccsh rawtextparser path/to/project -o project.cc.json

This writes project.cc.json.gz.

ParameterDescription
FILE or FOLDERfile/project to parse. To merge the result with an existing project piped into STDIN, pass a ’-’ as an additional argument
-bf, --base-file=<baseFile>base cc.json file with checksums to skip unchanged files during analysis
--bypass-gitignoredisable automatic .gitignore-based file exclusion (uses regex-based exclusion of common build folders)
-e, --exclude=<exclude>comma-separated list of regex patterns to exclude files/folders (applied in addition to .gitignore patterns)
-fe, --file-extensions=<fileExtensions>comma-separated list of file-extensions to parse only those files (default: any)
--commit=<ref>analyze the codebase at a specific git commit, tag, branch, or date expression (creates a temporary worktree). Cannot be combined with --local-changes. See Commit-Based Analysis
-h, --helpdisplays this help and exits
-ibf, --include-build-foldersinclude build folders (out, build, dist and target) and common resource folders (e.g. resources, node_modules or files/folders starting with ’.’)
--local-changesonly analyze files that differ from the remote tracking branch (uncommitted, staged, unstaged, untracked). See Local Changes
-m, --metrics=metricscomma-separated list of metrics to be computed: IndentationLevel, LinesOfCode, CodeAndCommentLines (all available metrics are computed if not specified)
--max-indentation-level=<maxIndentLvl>maximum Indentation Level (default 10)
-nc, --not-compressedsave uncompressed output File
-o, --output-file=<outputFile>output File (or empty for stdout)
--tab-width=<tabWidth>tab width used (estimated if not provided)
--verboseverbose mode
--without-default-excludes(“DEPRECATION WARNING: this flag will soon be disabled and replaced by ‘—include-build-folders’”) include build, target, dist, resources and out folders as well as files/folders starting with ’.’
Usage: ccsh rawtextparser [-h] [--bypass-gitignore] [--commit=<ref>] [-ibf]
[--local-changes] [-nc] [--verbose]
[--without-default-excludes]
[-bf=<baseFile>]
[--max-indentation-level=<maxIndentLvl>]
[-o=<outputFile>] [--tab-width=<tabWidth>]
[-e=<specifiedExcludePatterns>]...
[-fe=<fileExtensionsToAnalyse>]... [-m=metrics]...
FILE or FOLDER...
MetricDescription
locall lines of the file, blank lines included
rlocthe lines that hold code, that is anything but whitespace and comments
comment_linesthe lines that hold a comment
indentation_level_<n>+the non-blank lines indented by at least n levels, from indentation_level_0+ up to the --max-indentation-level (default indentation_level_10+)

-m IndentationLevel writes the indentation metrics, -m LinesOfCode writes loc, -m CodeAndCommentLines writes rloc and comment_lines. The names are case-sensitive.

A level is one tab or --tab-width spaces. Without --tab-width, the parser guesses the width per file from the spaces it finds. A line whose indentation is no multiple of the width counts for the next level.

The parser finds a file’s language by its extension or file name in the language table of tokei (v15.0.0, MIT licence), and reads the language’s comment and string markers from it. A file whose language the table does not know, and a prose format such as Markdown or plain text, gets neither rloc nor comment_lines.

The lines count like in the Unified Parser, so both parsers give the same numbers for a language they both support:

  • a line with code and a trailing comment counts for both metrics
  • every row of a block comment or a Python docstring counts as a comment line, blank rows included
  • every row of a multi-line string counts as code, blank rows included
  • a comment marker inside a string starts no comment

On top of the table, the parser knows constructs the table cannot describe:

LanguageCounts as code
Bash, sh, zsh, kshheredoc bodies, and a # that does not start a word, as in $#
Ruby, PHPheredoc and nowdoc bodies
JavaScript, TypeScript, JSX, TSX, Vueregular expressions such as /["'/*]/
PHPattributes such as #[Route]
Pascalcompiler directives such as {$IFDEF}
SQL#>, while # starts a comment as in MySQL
Goraw strings in backticks
Rubyevery line from __END__ on
alla char literal such as '"', and a run of quotes closing a """ string

A C, C++ or Objective-C // comment that ends in \ goes on in the next line, and a Python docstring may start with a prefix such as r""". In VBA and VB6, Rem starts a comment as ' does, and an apostrophe inside a string does not. In a Vue file only the lines inside the first <script> block count, as in the Unified Parser.

The parser reads markers, not the language’s grammar, so a construct neither the table nor these rules know can still miscount.

Terminal window
ccsh rawtextparser foo.txt --max-indentation-level=6 --tab-width=4 --metrics=IndentationLevel,LinesOfCode
ccsh rawtextparser foo -o out.cc.json --exclude='.*\.html' --exclude=bar
ccsh rawtextparser foo -o out.cc.json --bypass-gitignore

To add the raw text metrics to another map, pipe it in and pass - as an extra argument:

Terminal window
ccsh unifiedparser foo | ccsh rawtextparser foo - -o out.cc.json

The merged project keeps the project name of the Raw Text Parser.