abstract.py

ofrak_patch_maker.toolchain.abstract

Toolchain - Python driver for compiler, linker, assembler, and binutils that generates linker scripts for use with PatchMaker.

Toolchain (ABC)

name: str property readonly

Returns:

Type	Description
str	name property that matches the value used in toolchain.conf to access paths

_assembler_path: str private property readonly

Provides path to installed assembler given the ISA.

Returns:

Type	Description
str	filepath to the assembler program

Exceptions:

Type	Description
NotImplementedError	if an assembler for that ISA does not exist

_preprocessor_path: str private property readonly

Returns:

Type	Description
str	path to the toolchain preprocessor - this is usually the compiler.

_compiler_path: str private property readonly

Returns:

Type	Description
str	path to the toolchain compiler

_linker_path: str private property readonly

Returns:

Type	Description
str	path to the toolchain linker

_readobj_path: str private property readonly

Returns:

Type	Description
str	path to the toolchain binary analysis utility

_lib_path: str private property readonly

Returns:

Type	Description
str	path to the toolchain libraries

_linker_script_flag: str private property readonly

Returns:

Type	Description
str	the linker script flag for this toolchain, usually -T

segment_alignment: int property readonly

For example, x86 returns 16. This will most often be used when programmatically allocating memory for code/data.

Returns:

Type	Description
int	required alignment factor for the toolchain/ISA

__init__(self, processor, toolchain_config, logger=<RootLogger root (WARNING)>) special

Responsible for building provided patch source. Also responsible for all flag, and linker script syntax implementations.

Toolchain instances should be considered stateless after initialization; they can only be configured once.

Only certain, very specific flags may be appended at runtime. For instance, the output value of --Map={PATH} will change for every distinct executable that is generated in a project.

Generally, compiler flags should never, ever change throughout the lifetime of a Toolchain instance.

Parameters:

Name	Type	Description	Default
processor	ArchInfo	hardware description	required
toolchain_config	ToolchainConfig	assembler, compiler, linker options	required
logger	Logger		<RootLogger root (WARNING)>

Exceptions:

Type	Description
ValueError	if a binary parser doesn't exist for the filetype provided in the ToolchainConfig

Source code in ofrak_patch_maker/toolchain/abstract.py

def __init__(
    self,
    processor: ArchInfo,
    toolchain_config: ToolchainConfig,
    logger: logging.Logger = logging.getLogger(),
):
    """
    Responsible for building provided patch source. Also responsible for all flag, and linker
    script syntax implementations.

    `Toolchain` instances should be considered stateless after initialization; they can only
    be configured once.

    Only certain, very specific flags may be appended at runtime. For instance,
    the output value of `--Map={PATH}` will change for every distinct executable that is
    generated in a project.

    Generally, compiler flags should never, ever change throughout the lifetime of a `Toolchain`
    instance.

    :param processor: hardware description
    :param toolchain_config: assembler, compiler, linker options
    :param logger:

    :raises ValueError: if a binary parser doesn't exist for the filetype provided in the
        [ToolchainConfig][ofrak_patch_maker.toolchain.model.ToolchainConfig]
    """
    self._processor = processor

    self.file_format = toolchain_config.file_format

    self._parser: AbstractBinaryFileParser = None  # type: ignore
    for parser in self.binary_file_parsers:
        if parser.file_format is toolchain_config.file_format:
            self._parser = parser
            break
    if self._parser is None:
        raise ValueError(
            f"No binary file parser found for format " f"{toolchain_config.file_format.name}!"
        )

    self._preprocessor_flags: List[str] = []
    self._compiler_flags: List[str] = []
    self._assembler_flags: List[str] = []
    self._linker_flags: List[str] = []
    self._config = toolchain_config
    self._logger = logger

    # The keep_list should only contain FUNCTIONALLY important sections
    # (not empty .got.plt, for instance).
    # TODO: Come up with a better system to handle this...
    self._linker_keep_list = [".data", ".rodata", ".text", ".rel"]
    self._linker_discard_list = [
        ".gnu.hash",
        ".comment",
        ".ARM.attributes",
        ".dynamic",
        ".ARM.exidx",
        ".hash",
        ".dynsym",
        ".dynstr",
        ".eh_frame",
        ".altinstructions",
        ".altinstr_replacement",
    ]

    self._assembler_target = self._get_assembler_target(processor)
    self._compiler_target = self._get_compiler_target(processor)

__init_subclass__(**kwargs) classmethod special

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

Source code in ofrak_patch_maker/toolchain/abstract.py

def __init_subclass__(cls, **kwargs):
    Toolchain.toolchain_implementations.append(cls)

_get_assembler_target(self, processor) private

Red Balloon Security strongly recommends all users provide their specific hardware target for best results.

Parameters:

Name	Type	Description	Default
processor	ArchInfo		required

Returns:

Type	Description
Optional[str]	a default assembler target for the provided processor unless one is provided in self._config.

Exceptions:

Type	Description
PatchMakerException	if no target provided and program attributes do not correspond to a default value.

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def _get_assembler_target(self, processor: ArchInfo) -> Optional[str]:
    """
    Red Balloon Security strongly recommends all users provide their specific hardware target
    for best results.

    :param processor:

    :raises PatchMakerException: if no target provided and program attributes do not correspond
        to a default value.
    :return str: a default assembler target for the provided processor unless one is provided
        in `self._config`.
    """
    raise NotImplementedError()

_get_compiler_target(self, processor) private

Returns a default compiler target for the provided processor unless one is provided in self._config.

Red Balloon Security strongly recommends all users provide their specific hardware target for best results.

Parameters:

Name	Type	Description	Default
processor	ArchInfo		required

Returns:

Type	Description
Optional[str]

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def _get_compiler_target(self, processor: ArchInfo) -> Optional[str]:
    """
    Returns a default compiler target for the provided processor unless one is provided
    in `self._config`.

    Red Balloon Security strongly recommends all users provide their specific hardware target
    for best results.

    :param processor:

    :return str:
    """
    raise NotImplementedError()

is_userspace(self)

Provides whether the toolchain is configured for userspace patch generation.

Returns:

Type	Description
bool

Source code in ofrak_patch_maker/toolchain/abstract.py

def is_userspace(self) -> bool:
    """
    Provides whether the toolchain is configured for userspace patch generation.

    :return bool:
    """
    return self._config.userspace_dynamic_linker is not None

is_relocatable(self)

Provides whether the toolchain is configured for relocatable patch generation.

This often means hard failures when trying to assemble instructions that branch to absolute values.

Returns:

Type	Description
bool

Source code in ofrak_patch_maker/toolchain/abstract.py

def is_relocatable(self) -> bool:
    """
    Provides whether the toolchain is configured for relocatable patch generation.

    This often means hard failures when trying to assemble instructions that branch to absolute
    values.

    :return bool:
    """
    return self._config.relocatable

_execute_tool(self, tool_path, flags, in_files, out_file=None, env=None) private

Utility function used to invoke the toolchain subprocess we use.

Parameters:

Name	Type	Description	Default
tool_path	str	path	required
flags	List[str]	various CLI flags	required
in_files	List[str]	input files, usually positional	required
out_file	Optional[str]		None
env	Dict[str, str]	environment variables and stringified values to append to the existing environment.	None

Returns:

Type	Description
str	stdout of the subprocess call

Source code in ofrak_patch_maker/toolchain/abstract.py

def _execute_tool(
    self,
    tool_path: str,
    flags: List[str],
    in_files: List[str],
    out_file: Optional[str] = None,
    env: Dict[str, str] = None,
) -> str:
    """
    Utility function used to invoke the toolchain subprocess we use.

    :param tool_path: path
    :param flags: various CLI flags
    :param in_files: input files, usually positional
    :param out_file:
    :param env: environment variables and stringified values to append to the existing
    environment.

    :return str: `stdout` of the subprocess call
    """
    final_flags = []
    final_flags.extend(flags)

    if out_file is not None:
        final_flags.extend([f"-o{out_file}"])

    args = [tool_path] + final_flags + in_files
    self._logger.info(" ".join(args))
    try:
        if env:
            my_env = os.environ.copy()
            my_env.update(env)
            self._logger.info(f"With env: {my_env}")
            proc = subprocess.run(
                args, stdout=subprocess.PIPE, encoding="utf-8", check=True, env=my_env
            )
        else:
            proc = subprocess.run(args, stdout=subprocess.PIPE, encoding="utf-8", check=True)
    except subprocess.CalledProcessError as e:
        cmd = " ".join(args)
        raise ValueError(f'Command "{cmd}" returned non-zero exit status {e.returncode}')

    return proc.stdout

preprocess(self, source_file, header_dirs, out_dir='.')

Runs the Toolchain's C preprocessor on the input file.

Returns:

Type	Description
str	path to the original source file

Source code in ofrak_patch_maker/toolchain/abstract.py

def preprocess(self, source_file: str, header_dirs: List[str], out_dir: str = ".") -> str:
    """
    Runs the Toolchain's C preprocessor on the input file.

    :return str: path to the original source file
    """
    out_file = join(out_dir, split(source_file)[-1] + ".p")

    self._execute_tool(
        self._preprocessor_path,
        self._preprocessor_flags,
        [source_file] + ["-I" + x for x in header_dirs],
        out_file=out_file,
    )
    return os.path.abspath(out_file)

compile(self, c_file, header_dirs, out_dir='.')

Runs the Toolchain's C compiler on the input file.

Returns:

Type	Description
str	path to the object file

Source code in ofrak_patch_maker/toolchain/abstract.py

def compile(self, c_file: str, header_dirs: List[str], out_dir: str = ".") -> str:
    """
    Runs the Toolchain's C compiler on the input file.

    :return str: path to the object file
    """
    out_file = join(out_dir, split(c_file)[-1] + ".o")

    self._execute_tool(
        self._compiler_path,
        self._compiler_flags,
        [c_file] + ["-I" + x for x in header_dirs],
        out_file=out_file,
    )

    return os.path.abspath(out_file)

assemble(self, asm_file, header_dirs, out_dir='.')

Runs the Toolchain's assembler on the input file

Returns:

Type	Description
	path to the object file

Source code in ofrak_patch_maker/toolchain/abstract.py

def assemble(self, asm_file: str, header_dirs: List[str], out_dir: str = "."):
    """
    Runs the Toolchain's assembler on the input file

    :return str: path to the object file
    """
    out_file = join(out_dir, split(asm_file)[-1] + ".o")

    self._execute_tool(
        self._assembler_path,
        self._assembler_flags,
        [asm_file] + ["-I" + x for x in header_dirs],
        out_file=out_file,
    )

    return os.path.abspath(out_file)

_get_linker_map_flag(exec_path) private staticmethod

Generates the linker map file flag for a linker invocation given the executable path.

Parameters:

Name	Type	Description	Default
exec_path	str	path to executable	required

Returns:

Type	Description
Iterable[str]	path to map file

Source code in ofrak_patch_maker/toolchain/abstract.py

@staticmethod
@abstractmethod
def _get_linker_map_flag(exec_path: str) -> Iterable[str]:
    """
    Generates the linker map file flag for a linker invocation given the executable path.

    :param exec_path: path to executable

    :return str: path to map file
    """
    raise NotImplementedError()

link(self, o_files, exec_path, script=None)

Run's the Toolchain's linker on the input object files.

Parameters:

Name	Type	Description	Default
o_files	List[str]	list of object files to be linked	required
exec_path	str	path to executable output file	required
script	str	path to linker script (usually an .ld file)	None

Source code in ofrak_patch_maker/toolchain/abstract.py

def link(self, o_files: List[str], exec_path: str, script: str = None):
    """
    Run's the `Toolchain`'s linker on the input object files.

    :param o_files: list of object files to be linked
    :param exec_path: path to executable output file
    :param script: path to linker script (usually an `.ld` file)
    """
    flags = []
    flags.extend(self._linker_flags)

    if script is not None:
        flags.append(self._linker_script_flag + script)

    if self._config.create_map_files:
        flags.extend(self._get_linker_map_flag(exec_path))

    self._execute_tool(self._linker_path, flags, o_files, out_file=exec_path)

generate_linker_include_file(self, symbols, out_path)

This utility function receives the generated symbols dictionary that results from preprocessing a firmware image and generates a .inc file for use with linker scripts, enabling direct function calls when using the complete cross compilation toolchain.

This functionality must be defined for each toolchain given potential syntactical differences.

Parameters:

Name	Type	Description	Default
symbols	Mapping[str, int]	mappings of symbol string to effective address	required
out_path	str	the path to the resulting symbol include file (usually .inc)	required

Returns:

Type	Description
str	returns out_path

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def generate_linker_include_file(self, symbols: Mapping[str, int], out_path: str) -> str:
    """
    This utility function receives the generated symbols dictionary that results
    from preprocessing a firmware image and generates a `.inc` file for use
    with linker scripts, enabling direct function calls when using the complete
    cross compilation toolchain.

    This functionality must be defined for each toolchain given potential
    syntactical differences.

    :param symbols: mappings of symbol string to effective address
    :param out_path: the path to the resulting symbol include file (usually `.inc`)

    :return str: returns out_path
    """
    raise NotImplementedError()

add_linker_include_values(self, symbols, path)

Adds linker include entries to a provided file (usually ending in .inc).

For example GNU syntax prescribes PROVIDE(name = 0xdeadbeef);.

Parameters:

Name	Type	Description	Default
symbols	Dict[str, int]	mapping of symbol string to effective address	required
path	str	path to the provided linker include file.	required

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def add_linker_include_values(self, symbols: Dict[str, int], path: str):
    """
    Adds linker include entries to a provided file (usually ending in `.inc`).

    For example GNU syntax prescribes `PROVIDE(name = 0xdeadbeef);`.

    :param symbols: mapping of symbol string to effective address
    :param path: path to the provided linker include file.
    """
    raise NotImplementedError()

ld_generate_region(self, object_path, segment_name, permissions, vm_address, length)

Generates regions for linker scripts.

Returns:

Type	Description
Tuple[str, str]	a string entry for a "memory region" for the toolchain in question.

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def ld_generate_region(
    self,
    object_path: str,
    segment_name: str,
    permissions: MemoryPermissions,
    vm_address: int,
    length: int,
) -> Tuple[str, str]:
    """
    Generates regions for linker scripts.

    :return str: a string entry for a "memory region" for the toolchain in question.
    """
    raise NotImplementedError()

ld_generate_bss_region(self, vm_address, length)

Generates .bss regions for linker scripts.

Returns:

Type	Description
Tuple[str, str]	a .bss memory entry string for the toolchain in question.

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def ld_generate_bss_region(self, vm_address: int, length: int) -> Tuple[str, str]:
    """
    Generates `.bss` regions for linker scripts.

    :return str: a `.bss` memory entry string for the toolchain in question.
    """
    raise NotImplementedError()

ld_generate_section(object_path, segment_name, memory_region_name) staticmethod

Generates sections for linker scripts.

Returns:

Type	Description
str	a string entry for a "section" for the toolchain in question.

Source code in ofrak_patch_maker/toolchain/abstract.py

@staticmethod
@abstractmethod
def ld_generate_section(object_path: str, segment_name: str, memory_region_name: str) -> str:
    """
    Generates sections for linker scripts.

    :return str: a string entry for a "section" for the toolchain in question.
    """
    raise NotImplementedError()

ld_generate_bss_section(memory_region_name) staticmethod

Generates .bss sections for linker scripts.

Returns:

Type	Description
str	a .bss section entry string for the toolchain in question.

Source code in ofrak_patch_maker/toolchain/abstract.py

@staticmethod
@abstractmethod
def ld_generate_bss_section(memory_region_name: str) -> str:
    """
    Generates `.bss` sections for linker scripts.

    :return str: a `.bss` section entry string for the toolchain in question.
    """
    raise NotImplementedError()

ld_generate_placeholder_reloc_sections(self)

Implements functionality to generate additional regions and sections required as placeholders for certain toolchains during the link process when compiling relocatable code.

For instance, GNU seems to create .got.plt and .rel.dyn temporarily during link. These sections require memory regions that will take them during the link process even if the linker ends up leaving them out of the final executable.

We can mock them with the resulting regions/sections.

Returns:

Type	Description
Tuple[Iterable[str], Iterable[str]]	(memory regions, sections)

Source code in ofrak_patch_maker/toolchain/abstract.py

def ld_generate_placeholder_reloc_sections(self) -> Tuple[Iterable[str], Iterable[str]]:
    """
    Implements functionality to generate additional regions and sections required as
    placeholders for certain toolchains during the link process when compiling relocatable code.

    For instance, GNU seems to create `.got.plt` and `.rel.dyn` temporarily during link. These
    sections require memory regions that will take them during the link process even if the
    linker ends up leaving them out of the final executable.

    We can mock them with the resulting regions/sections.

    :return Tuple[Iterable[str], Iterable[str]]: (memory regions, sections)
    """
    return [], []

ld_script_create(self, name, memory_regions, sections, build_dir, symbol_files)

Constructs the linker script for the concrete toolchain class in use.

Uses the provided name, memory region strings, section strings, symbol files, expected entrypoint (if any) to generate a linker script that results in a valid FEM object when used within link.

Parameters:

Name	Type	Description	Default
name	str		required
memory_regions	List[str]		required
sections	List[str]		required
build_dir	str		required
symbol_files	List[str]		required

Returns:

Type	Description
str	path to the generated linker script

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def ld_script_create(
    self,
    name: str,
    memory_regions: List[str],
    sections: List[str],
    build_dir: str,
    symbol_files: List[str],
) -> str:
    """
    Constructs the linker script for the concrete toolchain class in use.

    Uses the provided name, memory region strings, section strings, symbol files,
    expected entrypoint (if any) to generate a linker script that results in a valid
    FEM object when used within `link`.

    :param name:
    :param memory_regions:
    :param sections:
    :param build_dir:
    :param symbol_files:

    :return str: path to the generated linker script
    """
    raise NotImplementedError()

get_bin_file_symbols(self, executable_path)

For now, this utility only searches for global function and data symbols which are actually contained in a section in the file, as opposed to symbols which are referenced but undefined.

Parameters:

Name	Type	Description	Default
executable_path	str	path to the program to be analyzed for symbols	required

Returns:

Type	Description
Dict[str, Tuple[int, ofrak_type.symbol_type.LinkableSymbolType]]	mapping of symbol string to tuple of effective address, symbol type.

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def get_bin_file_symbols(
    self, executable_path: str
) -> Dict[str, Tuple[int, LinkableSymbolType]]:
    """
    For now, this utility only searches for global function and data symbols which are
    actually contained in a section in the file, as opposed to symbols which are referenced
    but undefined.

    :param executable_path: path to the program to be analyzed for symbols

    :return Dict[str, Tuple[int, LinkableSymbolType]]: mapping of symbol string to tuple of
    effective address, symbol type.
    """
    raise NotImplementedError()

get_bin_file_segments(self, path)

Parses all segments found in the executable path provided.

Parameters:

Name	Type	Description	Default
path	str	path to the program to be analyzed for symbols	required

Returns:

Type	Description
Tuple[ofrak_patch_maker.toolchain.model.Segment, ...]	Tuple of Segment objects

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def get_bin_file_segments(self, path: str) -> Tuple[Segment, ...]:
    """
    Parses all segments found in the executable path provided.

    :param path: path to the program to be analyzed for symbols

    :return Tuple[Segment, ...]: Tuple of [Segment][ofrak_patch_maker.toolchain.model.Segment]
    objects
    """
    raise NotImplementedError()

get_bin_file_rel_symbols(self, executable_path)

This utility searches for global function and data symbols which are referenced in a section in the file but are undefined.

Parameters:

Name	Type	Description	Default
executable_path	str	path to the program to be analyzed for symbols	required

Returns:

Type	Description
Dict[str, Tuple[int, ofrak_type.symbol_type.LinkableSymbolType]]	mapping of symbol string to tuple of effective address, symbol type.

Source code in ofrak_patch_maker/toolchain/abstract.py

@abstractmethod
def get_bin_file_rel_symbols(
    self, executable_path: str
) -> Dict[str, Tuple[int, LinkableSymbolType]]:
    """
    This utility searches for global function and data symbols which are
    referenced in a section in the file but are undefined.

    :param executable_path: path to the program to be analyzed for symbols

    :return Dict[str, Tuple[int, LinkableSymbolType]]: mapping of symbol string to tuple of
    effective address, symbol type.
    """
    raise NotImplementedError()