crash(8) — Linux manual page

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | COMMANDS | FILES | ENVIRONMENT | NOTES | AUTHOR | SEE ALSO | COLOPHON

CRASH(8)                 System Manager's Manual                CRASH(8)

NAME         top

       crash - Analyze Linux crash dump data or a live system

SYNOPSIS         top

       crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile
       form)
       crash [OPTION]... [NAMELIST]                         (live system
       form)

DESCRIPTION         top

       Crash is a tool for interactively analyzing the state of the
       Linux system while it is running, or after a kernel crash has
       occurred and a core dump has been created by the netdump,
       diskdump, LKCD, kdump, xendump kvmdump or VMware facilities.  It
       is loosely based on the SVR4 UNIX crash command, but has been
       significantly enhanced by completely merging it with the gdb(1)
       debugger. The marriage of the two effectively combines the
       kernel-specific nature of the traditional UNIX crash utility with
       the source code level debugging capabilities of gdb(1).

       In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument
       must be entered.  In the live system form, the NAMELIST argument
       must be entered if the kernel's vmlinux file is not located in a
       known location, such as the /usr/lib/debug/lib/modules/<kernel-
       version> directory.

       The crash utility has also been extended to support the analysis
       of dumpfiles generated by a crash of the Xen hypervisor.  In that
       case, the NAMELIST argument must be that of the xen-syms binary.
       Live system analysis is not supported for the Xen hypervisor.

       The crash utility command set consists of common kernel core
       analysis tools such as kernel stack back traces of all processes,
       source code disassembly, formatted kernel structure and variable
       displays, virtual memory data, dumps of linked-lists, etc., along
       with several commands that delve deeper into specific kernel
       subsystems.  Appropriate gdb commands may also be entered, which
       in turn are passed on to the gdb module for execution.  If
       desired, commands may be placed in either a $HOME/.crashrc file
       and/or in a .crashrc file in the current directory.  During
       initialization, the commands in $HOME/.crashrc are executed
       first, followed by those in the ./.crashrc file.

       The crash utility is designed to be independent of Linux version
       dependencies. When new kernel source code impacts the correct
       functionality of crash and its command set, the utility will be
       updated to recognize new kernel code changes, while maintaining
       backwards compatibility with earlier releases.

OPTIONS         top

       NAMELIST
              This is a pathname to an uncompressed kernel image (a
              vmlinux file), or a Xen hypervisor image (a xen-syms file)
              which has been compiled with the "-g" option.  If using
              the dumpfile form, a vmlinux file may be compressed in
              either gzip or bzip2 formats.

       MEMORY-IMAGE[@ADDRESS]
              A kernel core dump file created by the netdump, diskdump,
              LKCD kdump, xendump kvmdump or VMware facilities.

              If a MEMORY-IMAGE argument is not entered, the session
              will be invoked on the live system, which typically
              requires root privileges because of the device file used
              to access system RAM.  By default, /dev/crash will be used
              if it exists.  If it does not exist, then /dev/mem will be
              used; but if the kernel has been configured with
              CONFIG_STRICT_DEVMEM, then /proc/kcore will be used.  It
              is permissible to explicitly enter /dev/crash, /dev/mem or
              /proc/kcore.

              An @ADDRESS value must be appended to the MEMORY-IMAGE if
              the dumpfile is a raw RAM dumpfile that has no header
              information describing the file contents.  Multiple
              MEMORY-IMAGE@ADDRESS ordered pairs may be entered, with
              each dumpfile containing a contiguous block of RAM, where
              the ADDRESS value is the physical start address of the
              block expressed in hexadecimal.  The physical address
              value(s) will be used to create a temporary ELF header in
              /var/tmp, which will only exist during the crash session.
              If a raw RAM dumpile represents a live memory source, such
              as that specified by the QEMU mem-path argument of a
              memory-backend-file object, then "live:" must be prepended
              to the MEMORY-IMAGE name.

              As VMware facility, the crash utility is able to process
              VMware VM memory dump generated by VM suspend or guest
              core dump. In that case, .vmss or .guest file should be
              used as a MEMORY-IMAGE and .vmem file must be located in
              the same folder.

       mapfile
              If the NAMELIST file is not the same kernel that is
              running (live system form), or the kernel that was running
              when the system crashed (dumpfile form), then the
              System.map file of the original kernel should be entered
              on the command line.

       -h [option]
       --help [option]
              Without an option argument, display a crash usage help
              message.  If the option argument is a crash command name,
              the help page for that command is displayed.  If it is the
              string "input", a page describing the various crash
              command line input options is displayed.  If it is the
              string "output", a page describing command line output
              options is displayed.  If it is the string "all", then all
              of the possible help messages are displayed.  After the
              help message is displayed, crash exits.

       -s     Silently proceed directly to the "crash>" prompt without
              displaying any version, GPL, or crash initialization data
              during startup, and by default, runtime command output is
              not passed to any scrolling command.

       -i file
              Execute the command(s) contained in file prior to
              displaying the "crash>" prompt for interactive user input.

       -d num Set the internal debug level.  The higher the number, the
              more debugging data will be printed when crash initializes
              and runs.

       -S     Use /boot/System.map as the mapfile.

       -e vi | emacs
              Set the readline(3) command line editing mode to "vi" or
              "emacs".  The default editing mode is "vi".

       -f     Force the usage of a compressed vmlinux file if its
              original name does not start with "vmlinux".

       -k     Indicate that the NAMELIST file is an LKCD "Kerntypes"
              debuginfo file.

       -g [namelist]
              Determine if a vmlinux or xen-syms namelist file contains
              debugging data.

       -t     Display the system-crash timestamp and exit.

       -L     Attempt to lock all of its virtual address space into
              memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) during
              initialization.  If the system call fails, an error
              message will be displayed, but the session continues.

       -c tty-device
              Open the tty-device as the console used for debug
              messages.

       -p page-size
              If a processor's page size cannot be determined by the
              dumpfile, and the processor default cannot be used, use
              page-size.

       -o filename
              Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM
              dumpfiles, specifies a filename of a new ELF vmcore that
              will be created and used as the dumpfile.  It will be
              saved to allow future use as a standalone vmcore,
              replacing the original raw RAM dumpfile.

       -m option=value
       --machdep option=value
              Pass an option and value pair to machine-dependent code.
              These architecture-specific option/pairs should only be
              required in very rare circumstances:

              X86_64:
                phys_base=<physical-address>
                irq_eframe_link=<value>
                irq_stack_gap=<value>
                max_physmem_bits=<value>
                kernel_image_size=<value>
                vm=orig       (pre-2.6.11 virtual memory address ranges)
                vm=2.6.11     (2.6.11 and later virtual memory address ranges)
                vm=xen        (Xen kernel virtual memory address ranges)
                vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
                vm=5level     (5-level page tables)
                page_offset=<PAGE_OFFSET-value>
              PPC64:
                vm=orig
                vm=2.6.14     (4-level page tables)
              IA64:
                phys_start=<physical-address>
                init_stack_size=<size>
                vm=4l         (4-level page tables)
              ARM:
                phys_base=<physical-address>
              ARM64:
                phys_offset=<physical-address>
                kimage_voffset=<kimage_voffset-value>
                max_physmem_bits=<value>
                vabits_actual=<value>
              X86:
                page_offset=<CONFIG_PAGE_OFFSET-value>

       -x     Automatically load extension modules from a particular
              directory.  If a directory is specified in the
              CRASH_EXTENSIONS shell environment variable, then that
              directory will be used.  Otherwise
              /usr/lib64/crash/extensions (64-bit architectures) or
              /usr/lib/crash/extensions (32-bit architectures) will be
              used; if they do not exist, then the ./extensions
              directory will be used.

       --active
              Track only the active task on each cpu.

       --buildinfo
              Display the crash binary's build date, the user ID of the
              builder, the hostname of the machine where the build was
              done, the target architecture, the version number, and the
              compiler version.

       --memory_module modname
              Use the modname as an alternative kernel module to the
              crash.ko module that creates the /dev/crash device.

       --memory_device device
              Use device as an alternative device to the /dev/crash,
              /dev/mem or /proc/kcore devices.

       --log dumpfile
              Dump the contents of the kernel log buffer.  A kernel
              namelist argument is not necessary, but the dumpfile must
              contain the VMCOREINFO data taken from the original
              /proc/vmcore ELF header.

       --no_kallsyms
              Do not use kallsyms-generated symbol information contained
              within kernel module object files.

       --no_modules
              Do not access or display any kernel module related
              information.

       --no_ikconf
              Do not attempt to read configuration data that was built
              into kernels configured with CONFIG_IKCONFIG.

       --no_data_debug
              Do not verify the validity of all structure member offsets
              and structure sizes that it uses.

       --no_kmem_cache
              Do not initialize the kernel's slab cache infrastructure,
              and commands that use kmem_cache-related data will not
              work.

       --no_elf_notes
              Do not use the registers from the ELF NT_PRSTATUS notes
              saved in a compressed kdump header for backtraces.

       --kmem_cache_delay
              Delay the initialization of the kernel's slab cache
              infrastructure until it is required by a run-time command.

       --readnow
              Pass this flag to the embedded gdb module, which will
              override its two-stage strategy that it uses for reading
              symbol tables from the NAMELIST.

       --smp  Specify that the system being analyzed is an SMP kernel.

       -v
       --version
              Display the version of the crash utility, the version of
              the embedded gdb module, GPL information, and copyright
              notices.

       --cpus number
              Specify the number of cpus in the SMP system being
              analyzed.

       --osrelease dumpfile
              Display the OSRELEASE vmcoreinfo string from a kdump
              dumpfile header.

       --hyper
              Force the session to be that of a Xen hypervisor.

       --p2m_mfn pfn
              When a Xen Hypervisor or its dom0 kernel crashes, the
              dumpfile is typically analyzed with either the Xen
              hypervisor or the dom0 kernel.  It is also possible to
              analyze any of the guest domU kernels if the
              pfn_to_mfn_list_list pfn value of the guest kernel is
              passed on the command line along with its NAMELIST and the
              dumpfile.

       --xen_phys_start physical-address
              Supply the base physical address of the Xen hypervisor's
              text and static data for older xendump dumpfiles that did
              not pass that information in the dumpfile header.

       --zero_excluded
              If the makedumpfile(8) facility has filtered a compressed
              kdump dumpfile to exclude various types of non-essential
              pages, or has marked a compressed or ELF kdump dumpfile as
              incomplete due to an ENOSPC or other error during its
              creation, any attempt to read missing pages will fail.
              With this flag, reads from any of those pages will return
              zero-filled memory.

       --no_panic
              Do not attempt to find the task that was running when the
              kernel crashed.  Set the initial context to that of the
              "swapper" task on cpu 0.

       --more Use /bin/more as the command output scroller, overriding
              the default of /usr/bin/less and any settings in either
              ./.crashrc or $HOME/.crashrc.

       --less Use /usr/bin/less as the command output scroller,
              overriding any settings in either ./.crashrc or
              $HOME/.crashrc.

       --hex  Set the default command output radix to 16, overriding the
              default radix of 10, and any radix settings in either
              ./.crashrc or $HOME/.crashrc.

       --dec  Set the default command output radix to 10, overriding any
              radix settings in either ./.crashrc or $HOME/.crashrc.
              This is the default radix setting.

       --CRASHPAGER
              Use the output paging command defined in the CRASHPAGER
              shell environment variable, overriding any settings in
              either ./.crashrc or $HOME/.crashrc.

       --no_scroll
              Do not pass run-time command output to any scrolling
              command.

       --no_strip
              Do not strip cloned kernel text symbol names.

       --no_crashrc
              Do not execute the commands in either $HOME/.crashrc or
              ./.crashrc.

       --mod directory
              When loading the debuginfo data of kernel modules with the
              mod -S command, search for their object files in directory
              instead of in the standard location.

       --src directory
              Search for the kernel source code in directory instead of
              in the standard location that is compiled into the
              debuginfo data.

       --kaslr offset|auto
              If an x86, x86_64, s390x or loongarch64 kernel was
              configured with CONFIG_RANDOMIZE_BASE, the offset value is
              equal to the difference between the symbol values compiled
              into the vmlinux file and their relocated KASLR values.
              If set to auto, the KASLR offset value will be
              automatically calculated.

       --reloc size
              When analyzing live x86 kernels that were configured with
              a CONFIG_PHYSICAL_START value that is larger than its
              CONFIG_PHYSICAL_ALIGN value, then it will be necessary to
              enter a relocation size equal to the difference between
              the two values.

       --hash count
              Set the number of internal hash queue heads used for list
              gathering and verification.  The default count is 32768.

       --minimal
              Bring up a session that is restricted to the log, dis, rd,
              sym, eval, set and exit commands.  This option may provide
              a way to extract some minimal/quick information from a
              corrupted or truncated dumpfile, or in situations where
              one of the several kernel subsystem initialization
              routines would abort the crash session.

       --kvmhost [32|64]
              When examining an x86 KVM guest dumpfile, this option
              specifies that the KVM host that created the dumpfile was
              an x86 (32-bit) or an x86_64 (64-bit) machine, overriding
              the automatically determined value.

       --kvmio <size>
              override the automatically-calculated KVM guest I/O hole
              size.

       --offline [show|hide]
              Show or hide command output that is related to offline
              cpus.  The default setting is show.

COMMANDS         top

       Each crash command generally falls into one of the following
       categories:

       Symbolic display
              Displays of kernel text/data, which take full advantage of
              the power of gdb to format and display data structures
              symbolically.

       System state
              The majority of crash commands consist of a set of
              "kernel-aware" commands, which delve into various kernel
              subsystems on a system-wide or per-task basis.

       Utility functions
              A set of useful helper commands serving various purposes,
              some simple, others quite powerful.

       Session control
              Commands that control the crash session itself.

       The following alphabetical list consists of a very simple
       overview of each crash command.  However, since individual
       commands often have several options resulting in significantly
       different output, it is suggested that the full description of
       each command be viewed by executing crash -h <command>, or during
       a crash session by simply entering help command.

       *      "pointer to" is shorthand for either the struct or union
              commands.  It displays the contents of a kernel structure
              or union.

       alias  creates a single-word alias for a command.

       ascii  displays an ascii chart or translates a numeric value into
              its ascii components.

       bpf    provides information on currently-loaded eBPF programs and
              maps.

       bt     displays a task's kernel-stack backtrace.  If it is given
              the -a option, it displays the stack traces of the active
              tasks on all CPUs.  It is often used with the foreach
              command to display the backtraces of all tasks with one
              command.

       btop   translates a byte value (physical offset) to its page
              number.

       dev    displays data concerning the character and block device
              assignments, I/O port usage, I/O memory usage, and PCI
              device data.

       dis    disassembles memory, either entire kernel functions, from
              a location for a specified number of instructions, or from
              the start of a function up to a specified memory location.

       eval   evaluates an expression or numeric type and displays the
              result in hexadecimal, decimal, octal and binary.

       exit   causes crash to exit.

       extend dynamically loads or unloads crash shared object extension
              modules.

       files  displays information about open files in a context.

       foreach
              repeats a specified command for the specified (or all)
              tasks in the system.

       fuser  displays the tasks using the specified file or socket.

       gdb    passes its argument to the embedded gdb module.  It is
              useful for executing gdb commands that have the same name
              as crash commands.

       help   alone displays the command menu; if followed by a command
              name, a full description of a command, its options, and
              examples are displayed.  Its output is far more complete
              and useful than this man page.

       ipcs   displays data about the System V IPC facilities.

       irq    displays data concerning interrupt request numbers and
              bottom-half interrupt handling.

       kmem   displays information about the use of kernel memory.

       list   displays the contents of a linked list.

       log    displays the kernel log_buf contents in chronological
              order.

       mach   displays data specific to the machine type.

       mod    displays information about the currently installed kernel
              modules, or adds or deletes symbolic or debugging
              information about specified kernel modules.

       mount  displays information about the currently-mounted
              filesystems.

       net    display various network related data.

       p      passes its arguments to the gdb "print" command for
              evaluation and display.

       ps     displays process status for specified, or all, processes
              in the system.

       pte    translates the hexadecimal contents of a PTE into its
              physical page address and page bit settings.

       ptob   translates a page frame number to its byte value.

       ptov   translates a hexadecimal physical address into a kernel
              virtual address.

       q      is an alias for the "exit" command.

       rd     displays the contents of memory, with the output formatted
              in several different manners.

       repeat repeats a command indefinitely, optionally delaying a
              given number of seconds between each command execution.

       runq   displays the tasks on the run queue.

       sbitmapq
              dumps the contents of the sbitmap_queue structure and the
              used bits in the bitmap. Also, it shows the dump of a
              structure array associated with the sbitmap_queue.

       search searches a range of user or kernel memory space for given
              value.

       set    either sets a new context, or gets the current context for
              display.

       sig    displays signal-handling data of one or more tasks.

       struct displays either a structure definition or the contents of
              a kernel structure at a specified address.

       swap   displays information about each configured swap device.

       sym    translates a symbol to its virtual address, or a static
              kernel virtual address to its symbol -- or to a symbol-
              plus-offset value, if appropriate.

       sys    displays system-specific data.

       task   displays the contents of a task_struct.

       tree   displays the contents of a red-black tree or a radix tree.

       timer  displays the timer queue entries, both old- and new-style,
              in chronological order.

       union  is similar to the struct command, except that it works on
              kernel unions.

       vm     displays basic virtual memory information of a context.

       vtop   translates a user or kernel virtual address to its
              physical address.

       waitq  walks the wait queue list displaying the tasks which are
              blocked on the specified wait queue.

       whatis displays the definition of structures, unions, typedefs or
              text/data symbols.

       wr     modifies the contents of memory on a live system.  It can
              only be used if /dev/mem is the device file being used to
              access system RAM, and should obviously be used with great
              care.

       When crash is invoked with a Xen hypervisor binary as the
       NAMELIST, the command set is slightly modified.  The *, alias,
       ascii, bt, dis, eval, exit, extend, gdb, help, list, log, p, pte,
       rd, repeat, search, set, struct, sym, sys, union, whatis, wr and
       q commands are the same as above.  The following commands are
       specific to the Xen hypervisor:

       domain displays the contents of the domain structure for
              selected, or all, domains.

       doms   displays domain status for selected, or all, domains.

       dumpinfo
              displays Xen dump information for selected, or all, cpus.

       pcpus  displays physical cpu information for selected, or all,
              cpus.

       vcpus  displays vcpu status for selected, or all, vcpus.

FILES         top

       .crashrc
              Initialization commands.  The file can be located in the
              user's HOME directory and/or the current directory.
              Commands found in the .crashrc file in the HOME directory
              are executed before those in the current directory's
              .crashrc file.

ENVIRONMENT         top

       EDITOR Command input is read using readline(3).  If EDITOR is set
              to emacs or vi then suitable keybindings are used.  If
              EDITOR is not set, then vi is used.  This can be
              overridden by set vi or set emacs commands located in a
              .crashrc file, or by entering -e emacs on the crash
              command line.

       CRASHPAGER
              If CRASHPAGER is set, its value is used as the name of the
              program to which command output will be sent.  If not,
              then command output is sent to /usr/bin/less -E -X by
              default.

       CRASH_MODULE_PATH
              Specifies an alternative directory tree to search for
              kernel module object files.

       CRASH_EXTENSIONS
              Specifies a directory containing extension modules that
              will be loaded automatically if the -x command line option
              is used.

NOTES         top

       If crash does not work, look for a newer version: kernel
       evolution frequently makes crash updates necessary.

       The command set scroll off will cause output to be sent directly
       to the terminal rather than through a paging program.  This is
       useful, for example, if you are running crash in a window of
       emacs.

AUTHOR         top

       Dave Anderson <[email protected]> wrote crash.

       Jay Fenlason <[email protected]> and Dave Anderson
       <[email protected]> wrote this man page.

SEE ALSO         top

       The help command within crash provides more complete and accurate
       documentation than this man page.

       https://github.com/crash-utility  - the home page of the crash
       utility.

       netdump(8), gdb(1), makedumpfile(8)

COLOPHON         top

       This page is part of the crash (Linux crash dump analyzer)
       project.  Information about the project can be found at 
       ⟨http://people.redhat.com/anderson/⟩.  If you have a bug report
       for this manual page, send it to [email protected].  This
       page was obtained from the project's upstream Git repository
       ⟨https://github.com/crash-utility/crash.git⟩ on 2024-06-14.  (At
       that time, the date of the most recent commit that was found in
       the repository was 2024-06-12.)  If you discover any rendering
       problems in this HTML version of the page, or you believe there
       is a better or more up-to-date source for the page, or you have
       corrections or improvements to the information in this COLOPHON
       (which is not part of the original manual page), send a mail to
       [email protected]

                                                                CRASH(8)