flash.py

ofrak.core.flash

This component is intended to make it easier to analyze raw flash dumps using OFRAK alone. Most flash dumps have "useful" data mixed in with out-of-band (OOB) data. The OOB data often includes some Error Correcting Codes (ECC) or checksums.

There are several dataclasses that categorize the sections of the dump:

• FlashResource is the overarching resource. The component expects the user to add this tag in order for this component to be run.
  • FlashOobResource is the region of FlashResource that has OOB data. In the future, there may be multiple of these children resources.
    • FlashLogicalDataResource is the extracted data only with all of the OOB data removed. This will become a FlashOobResource when packed.
    • FlashLogicalEccResource is the extracted ECC only. No other OOB data is included.

FlashAttributes (ResourceAttributes) dataclass

FlashAttributes is for specifying everything about the specific model of flash. The intent is to expand to all common flash configurations. Every block has a format specifier to show where each field is in the block as well as the length. If there is no OOB data, data_block_format may take this form:

data_block_format = [FlashField(field_type=FlashFieldType.DATA,size=block_size),]

Important Notes: Assumes that the provided list for each block format is ordered. Only define a block_format if they are different from other block formats. - A current workaround is adding FlashField(FlashFieldType.ALIGNMENT, 0) Assumes that there are only one of each block format except the data_block_format

FlashEccAttributes (ResourceAttributes) dataclass

Must be configured if the resource includes ECC ecc_magic is assumed to be contained at the start of the file, but may also occur multiple times

FlashField dataclass

FlashField(field_type: ofrak.core.flash.FlashFieldType, size: int)

FlashFieldType (Enum)

DATA_SIZE is the packed size of the DATA only (excluding MAGIC, CHECKSUM, DELIMITER, ECC, etc) TOTAL_SIZE is the size of the entire region (including all DATA, MAGIC, CHECKSUM, DELIMITER, ECC, etc) ALIGNMENT will pad with bytes by default

FlashLogicalDataResource (GenericBinary) dataclass

This is the final product of unpacking a FlashResource. It contains the data without any ECC or OOB data included. This allows for recursive packing and unpacking.

FlashLogicalDataResourcePacker (Packer)

Packs the FlashLogicalDataResource into a FlashOobResource of the format specified by the FlashAttributes

pack(self, resource, config=None) async

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name	Type	Description	Default
resource	Resource		required
config		Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	None

Source code in ofrak/core/flash.py

async def pack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried packing without FlashAttributes")
    data = await resource.get_data()
    bytes_left = len(data)
    original_size = bytes_left
    packed_data = bytearray()
    data_offset = 0

    for c in flash_attr.iterate_through_all_blocks(original_size, False):
        block_data = b""
        block_data_size = flash_attr.get_field_length_in_block(c, FlashFieldType.DATA)
        if block_data_size != 0:
            # Get the data for the current block
            block_data = data[data_offset : data_offset + block_data_size]
            data_offset += block_data_size
            bytes_left -= block_data_size

        packed_data += _build_block(
            cur_block_type=c,
            attributes=flash_attr,
            block_data=block_data,
            original_data=data,
        )

    # Create child under the original FlashOobResource to show that it packed itself
    parent = await resource.get_parent()
    await parent.create_child(tags=(FlashOobResource,), data=packed_data)

FlashLogicalEccResource (GenericBinary) dataclass

The alternate to FlashLogicalDataResource but just includes ECC. Does not include any other OOB data. Generally less useful on its own but provided anyway.

FlashOobResource (GenericBinary) dataclass

Represents the region containing Oob data.

FlashOobResourcePacker (Packer)

Packs the entire region including Oob data back into a binary blob

pack(self, resource, config=None) async

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name	Type	Description	Default
resource	Resource		required
config		Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	None

Source code in ofrak/core/flash.py

async def pack(self, resource: Resource, config=None):
    # We want to overwrite ourselves with just the repacked version
    packed_child = await resource.get_only_child(
        r_filter=ResourceFilter.with_tags(
            FlashOobResource,
        ),
    )
    if packed_child is not None:
        patch_data = await packed_child.get_data()
        original_size = await resource.get_data_length()
        resource.queue_patch(Range(0, original_size), patch_data)

FlashOobResourceUnpacker (Unpacker)

Unpack a single FlashOobResource dump into logical data using the FlashAttributes.

unpack(self, resource, config=None) async

Unpack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.unpack.

Parameters:

Name	Type	Description	Default
resource	Resource	The resource that is being unpacked	required
config		Optional config for unpacking. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	None

Source code in ofrak/core/flash.py

async def unpack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried unpacking without FlashAttributes")
    ecc_attr: Optional[FlashEccAttributes] = flash_attr.ecc_attributes

    oob_resource = resource
    # Parent FlashEccResource is created, redefine data to limited scope
    data = await oob_resource.get_data()
    data_len = len(data)

    # Now add children blocks until we reach the tail block
    offset = 0
    only_data = list()
    only_ecc = list()
    for block in flash_attr.iterate_through_all_blocks(data_len, True):
        block_size = flash_attr.get_block_size(block)
        block_end_offset = offset + block_size
        if block_end_offset > data_len:
            LOGGER.info(
                f"Block offset {block_end_offset} is {block_end_offset - data_len} larger "
                f"than {data_len}. In this case unpacking is best effort and end of unpacked "
                f"child might not be accurate."
            )
            break
        block_range = Range(offset, block_end_offset)
        block_data = await oob_resource.get_data(range=block_range)

        # Iterate through every field in block, dealing with ECC and DATA
        block_ecc_range = None
        block_data_range = None
        field_offset = 0
        for field_index, field in enumerate(block):
            field_range = Range(field_offset, field_offset + field.size)

            # We must check all blocks anyway so deal with ECC here
            if field.field_type == FlashFieldType.ECC:
                block_ecc_range = field_range
                cur_block_ecc = block_data[block_ecc_range.start : block_ecc_range.end]
                only_ecc.append(cur_block_ecc)
                # Add hash of everything up to the ECC to our dict for faster packing
                block_data_hash = md5(block_data[: block_ecc_range.start]).digest()
                DATA_HASHES[block_data_hash] = cur_block_ecc

            if field.field_type == FlashFieldType.DATA:
                block_data_range = field_range
                # Get next ECC range
                future_offset = field_offset
                block_list = list(block)
                for future_field in block_list[field_index:]:
                    if future_field.field_type == FlashFieldType.ECC:
                        block_ecc_range = Range(
                            future_offset, future_offset + future_field.size
                        )
                    future_offset += future_field.size

                if block_ecc_range is not None:
                    # Try decoding/correcting with ECC, report any error
                    try:
                        # Assumes that data comes before ECC
                        if (ecc_attr is not None) and (ecc_attr.ecc_class is not None):
                            only_data.append(
                                ecc_attr.ecc_class.decode(block_data[: block_ecc_range.end])[
                                    block_data_range.start : block_data_range.end
                                ]
                            )
                        else:
                            raise UnpackerError(
                                "Tried to correct with ECC without providing an ecc_class in FlashEccAttributes"
                            )
                    except EccError:
                        raise UnpackerError("ECC correction failed")
                else:
                    # No ECC found in block, just add the data directly
                    only_data.append(block_data[block_data_range.start : block_data_range.end])
            field_offset += field.size
        offset += block_size

    # Add all block data to logical resource for recursive unpacking
    await oob_resource.create_child(
        tags=(FlashLogicalDataResource,),
        data=b"".join(only_data) if only_data else data,
        attributes=[
            flash_attr,
        ],
    )
    if ecc_attr is not None:
        await oob_resource.create_child(
            tags=(FlashLogicalEccResource,),
            data=b"".join(only_ecc),
            attributes=[
                ecc_attr,
            ],
        )

FlashResource (GenericBinary) dataclass

The overarching resource that encapsulates flash storage. This will contain a FlashOobResource in most cases. In the future, support for multiple FlashOobResource children should be added.

FlashResourcePacker (Packer)

Packs the FlashResource into binary and cleans up logical data representations

pack(self, resource, config=None) async

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name	Type	Description	Default
resource	Resource		required
config		Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	None

Source code in ofrak/core/flash.py

async def pack(self, resource: Resource, config=None):
    # We want to overwrite ourselves with just the repacked version
    # TODO: Add supoort for multiple FlashOobResource in a dump.
    packed_child = await resource.get_only_child(
        r_filter=ResourceFilter.with_tags(
            FlashOobResource,
        ),
    )
    if packed_child is not None:
        patch_data = await packed_child.get_data()
        original_size = await resource.get_data_length()
        resource.queue_patch(Range(0, original_size), patch_data)

FlashResourceUnpacker (Unpacker)

Finds the overarching parent for region that includes OOB data. Identifies the bounds based on the FlashAttributes.

unpack(self, resource, config=None) async

Unpack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.unpack.

Parameters:

Name	Type	Description	Default
resource	Resource	The resource that is being unpacked	required
config	ComponentConfig	Optional config for unpacking. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	None

Source code in ofrak/core/flash.py

async def unpack(self, resource: Resource, config: ComponentConfig = None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried creating FlashOobResource without FlashAttributes")

    start_index = 0
    data = await resource.get_data()
    data_len = len(data)

    if flash_attr.ecc_attributes is not None:
        magic = flash_attr.ecc_attributes.ecc_magic
        if magic is not None:
            ecc_magic_offset = data.find(magic)

            if ecc_magic_offset == -1:
                raise UnpackerError("No header magic found")

            magic_range_in_block = flash_attr.get_field_range_in_block(
                flash_attr.header_block_format, FlashFieldType.MAGIC
            )

            if not magic_range_in_block:
                raise UnpackerError("Did not find offset for magic in header")

            start_index = ecc_magic_offset - magic_range_in_block.start

    # Set fallback, in case the current check for the end of the resource fails
    end_offset = data_len
    header_data_size = flash_attr.get_field_in_block(
        flash_attr.header_block_format, FlashFieldType.DATA_SIZE
    )
    header_total_size = flash_attr.get_field_in_block(
        flash_attr.header_block_format, FlashFieldType.TOTAL_SIZE
    )
    if flash_attr.header_block_format is not None and (
        header_data_size is not None or header_total_size is not None
    ):
        # The header has the size of the entire region including OOB data
        total_ecc_protected_size = 0
        if header_data_size is not None:
            # Found data size in the header, need to calculate expected total size (including OOB)
            data_size_bytes = flash_attr.get_field_data_in_block(
                block_format=flash_attr.header_block_format,
                field_type=FlashFieldType.DATA_SIZE,
                data=data,
                block_start_offset=0,
            )
            oob_size = flash_attr.get_total_oob_size(data_len=data_len, includes_oob=True)
            if data_size_bytes is not None:
                total_ecc_protected_size = oob_size + int.from_bytes(data_size_bytes, "big")
        elif header_total_size is not None:
            # Found total size in header
            total_size_bytes = flash_attr.get_field_data_in_block(
                block_format=flash_attr.header_block_format,
                field_type=FlashFieldType.TOTAL_SIZE,
                data=data,
                block_start_offset=0,
            )
            if total_size_bytes is not None:
                total_ecc_protected_size = int.from_bytes(total_size_bytes, "big")

        if total_ecc_protected_size > data_len:
            raise UnpackerError("Expected larger resource than supplied")

        if total_ecc_protected_size > start_index:
            end_offset = start_index + total_ecc_protected_size

    elif flash_attr.tail_block_format is not None:
        # Tail has either magic, total_size, or data_size to indicate the end
        tail_magic = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.MAGIC
        )
        tail_total_size = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.TOTAL_SIZE
        )
        tail_data_size = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.DATA_SIZE
        )

        if tail_magic is not None:
            # we'll start looking after the header to make sure we don't
            # accidentally find the header magic
            if flash_attr.header_block_format:
                tail_start_index = flash_attr.get_block_size(flash_attr.header_block_format)
            else:
                tail_start_index = start_index
            end_offset = _get_end_from_magic(flash_attr, tail_start_index, data, data_len)
        elif tail_total_size is not None:
            end_offset = _get_end_from_size(
                flash_attr, start_index, data, data_len, FlashFieldType.TOTAL_SIZE
            )
        elif tail_data_size is not None:
            end_offset = _get_end_from_size(
                flash_attr, start_index, data, data_len, FlashFieldType.DATA_SIZE
            )

    # Create the overarching resource
    return await resource.create_child(
        tags=(FlashOobResource,),
        data_range=Range(start_index, end_offset),
        attributes=[
            flash_attr,
        ],
    )