flash.py

`ofrak_components.flash`

This component is intended to make it easier to analyze raw flash dumps using OFRAK alone. Most flash dumps have "useful" data mixed in with out-of-band (OOB) data. The OOB data often includes some Error Correcting Codes (ECC) or checksums.

There are several dataclasses that categorize the sections of the dump:

FlashResource is the overarching resource. The component expects the user to add this tag in order for this component to be run.
- FlashOobResource is the region of FlashResource that has OOB data. In the future, there may be multiple of these children resources.
  - FlashLogicalDataResource is the extracted data only with all of the OOB data removed. This will become a FlashOobResource when packed.
  - FlashLogicalEccResource is the extracted ECC only. No other OOB data is included.

`FlashAttributes (ResourceAttributes)` `dataclass`

FlashAttributes is for specifying everything about the specific model of flash. The intent is to expand to all common flash configurations. Every block has a format specifier to show where each field is in the block as well as the length. If there is no OOB data, data_block_format may take this form:

data_block_format = [FlashField(field_type=FlashFieldType.DATA,size=block_size),]

Important Notes: Assumes that the provided list for each block format is ordered. Only define a block_format if they are different from other block formats. - A current workaround is adding FlashField(FlashFieldType.ALIGNMENT, 0) Assumes that there are only one of each block format except the data_block_format

`FlashEccAttributes (ResourceAttributes)` `dataclass`

Must be configured if the resource includes ECC ecc_magic is assumed to be contained at the start of the file, but may also occur multiple times

`FlashField` `dataclass`

FlashField(field_type: ofrak_components.flash.FlashFieldType, size: int)

`FlashFieldType (Enum)`

DATA_SIZE is the packed size of the DATA only (excluding MAGIC, CHECKSUM, DELIMITER, ECC, etc) TOTAL_SIZE is the size of the entire region (including all DATA, MAGIC, CHECKSUM, DELIMITER, ECC, etc) ALIGNMENT will pad with bytes by default

`FlashLogicalDataResource (GenericBinary)` `dataclass`

This is the final product of unpacking a FlashResource. It contains the data without any ECC or OOB data included. This allows for recursive packing and unpacking.

`FlashLogicalDataResourcePacker (Packer)`

Packs the FlashLogicalDataResource into a FlashOobResource of the format specified by the FlashAttributes

`pack(self, resource, config=None)` `async`

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name	Type	Description	Default
`resource`	`Resource`		required
`config`		Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	`None`

Source code in ofrak_components/flash.py

async def pack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried packing without FlashAttributes")
    data = await resource.get_data()
    bytes_left = len(data)
    original_size = bytes_left
    packed_data = bytearray()
    data_offset = 0

    for c in flash_attr.iterate_through_all_blocks(original_size, False):
        block_data = b""
        block_data_size = flash_attr.get_field_length_in_block(c, FlashFieldType.DATA)
        if block_data_size != 0:
            # Get the data for the current block
            block_data = data[data_offset : data_offset + block_data_size]
            data_offset += block_data_size
            bytes_left -= block_data_size

        packed_data += _build_block(
            cur_block_type=c,
            attributes=flash_attr,
            block_data=block_data,
            original_data=data,
        )

    # Create child under the original FlashOobResource to show that it packed itself
    parent = await resource.get_parent()
    await parent.create_child(tags=(FlashOobResource,), data=packed_data)

`FlashLogicalEccResource (GenericBinary)` `dataclass`

The alternate to FlashLogicalDataResource but just includes ECC. Does not include any other OOB data. Generally less useful on its own but provided anyway.

`FlashOobResource (GenericBinary)` `dataclass`

Represents the region containing Oob data.

`FlashOobResourcePacker (Packer)`

Packs the entire region including Oob data back into a binary blob

`pack(self, resource, config=None)` `async`

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name	Type	Description	Default
`resource`	`Resource`		required
`config`		Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	`None`

Source code in ofrak_components/flash.py

async def pack(self, resource: Resource, config=None):
    # We want to overwrite ourselves with just the repacked version
    packed_child = await resource.get_only_child(
        r_filter=ResourceFilter.with_tags(
            FlashOobResource,
        ),
    )
    if packed_child is not None:
        patch_data = await packed_child.get_data()
        original_size = await resource.get_data_length()
        resource.queue_patch(Range(0, original_size), patch_data)

`FlashOobResourceUnpacker (Unpacker)`

Unpack a single FlashOobResource dump into logical data using the FlashAttributes.

`unpack(self, resource, config=None)` `async`

Unpack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.unpack.

Parameters:

Name	Type	Description	Default
`resource`	`Resource`	The resource that is being unpacked	required
`config`		Optional config for unpacking. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	`None`

Source code in ofrak_components/flash.py

async def unpack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried unpacking without FlashAttributes")
    ecc_attr: Optional[FlashEccAttributes] = flash_attr.ecc_attributes

    # oob_resource = _create_oob_resource(resource=resource)
    oob_resource = resource
    # Parent FlashEccResource is created, redefine data to limited scope
    data = await oob_resource.get_data()
    data_len = len(data)

    # Now add children blocks until we reach the tail block
    offset = 0
    only_data = b""
    only_ecc = b""
    for c in flash_attr.iterate_through_all_blocks(data_len, True):
        block_size = flash_attr.get_block_size(c)
        block_end_offset = offset + block_size
        if block_end_offset > data_len:
            raise UnpackerError("Expected complete block and received less than expected")
        block_range = Range(offset, block_end_offset)
        block_data = await oob_resource.get_data(range=block_range)

        block_ecc_range = flash_attr.get_field_range_in_block(c, FlashFieldType.ECC)
        if block_ecc_range is not None:
            cur_block_ecc = block_data[block_ecc_range.start : block_ecc_range.end]
            only_ecc += cur_block_ecc
            # Add hash of everything up to the ECC to our dict for faster packing
            block_data_hash = md5(block_data[: block_ecc_range.start]).digest()
            DATA_HASHES[block_data_hash] = cur_block_ecc

        # Check if there is data in the block
        block_data_range = flash_attr.get_field_range_in_block(c, FlashFieldType.DATA)
        if block_data_range is not None:
            if block_ecc_range is not None:
                # Try decoding/correcting with ECC, otherwise just add the data anyway
                try:
                    # Assumes that data comes before ECC
                    if ecc_attr is not None and ecc_attr.ecc_class is not None:
                        only_data += ecc_attr.ecc_class.decode(
                            block_data[: block_ecc_range.end]
                        )[block_data_range.start : block_data_range.end]
                    else:
                        raise UnpackerError(
                            "Tried to correct with ECC without providing an ecc_class in FlashEccAttributes"
                        )
                except EccError:
                    only_data += block_data[block_data_range.start : block_data_range.end]
            else:
                only_data += block_data[block_data_range.start : block_data_range.end]
        offset += block_size

    # Add all block data to logical resource for recursive unpacking
    await oob_resource.create_child(
        tags=(FlashLogicalDataResource,),
        data=only_data,
        attributes=[
            flash_attr,
        ],
    )
    if ecc_attr is not None:
        await oob_resource.create_child(
            tags=(FlashLogicalEccResource,),
            data=only_ecc,
            attributes=[
                ecc_attr,
            ],
        )

`FlashResource (GenericBinary)` `dataclass`

The overarching resource that encapsulates flash storage. This will contain a FlashOobResource in most cases. In the future, support for multiple FlashOobResource children should be added.

`FlashResourcePacker (Packer)`

Packs the FlashResource into binary and cleans up logical data representations

`pack(self, resource, config=None)` `async`

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name	Type	Description	Default
`resource`	`Resource`		required
`config`		Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	`None`

Source code in ofrak_components/flash.py

async def pack(self, resource: Resource, config=None):
    # We want to overwrite ourselves with just the repacked version
    # TODO: Add supoort for multiple FlashOobResource in a dump.
    packed_child = await resource.get_only_child(
        r_filter=ResourceFilter.with_tags(
            FlashOobResource,
        ),
    )
    if packed_child is not None:
        patch_data = await packed_child.get_data()
        original_size = await resource.get_data_length()
        resource.queue_patch(Range(0, original_size), patch_data)

`FlashResourceUnpacker (Unpacker)`

Finds the overarching parent for region that includes OOB data. Identifies the bounds based on the FlashAttributes.

`unpack(self, resource, config=None)` `async`

Unpack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.unpack.

Parameters:

Name	Type	Description	Default
`resource`	`Resource`	The resource that is being unpacked	required
`config`		Optional config for unpacking. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.	`None`

Source code in ofrak_components/flash.py

async def unpack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried creating FlashOobResource without FlashAttributes")

    start_index = 0
    data = await resource.get_data()
    data_len = len(data)

    if flash_attr.ecc_attributes is not None:
        magic = flash_attr.ecc_attributes.ecc_magic
        if magic is not None:
            ecc_magic_offset = data.find(magic)

            if ecc_magic_offset == -1:
                raise UnpackerError("No header magic found")

            magic_range_in_block = flash_attr.get_field_range_in_block(
                flash_attr.header_block_format, FlashFieldType.MAGIC
            )

            if not magic_range_in_block:
                raise UnpackerError("Did not find offset for magic in header")

            start_index = ecc_magic_offset - magic_range_in_block.start

    # Set fallback, in case the current check for the end of the resource fails
    end_offset = data_len
    header_data_size = flash_attr.get_field_in_block(
        flash_attr.header_block_format, FlashFieldType.DATA_SIZE
    )
    header_total_size = flash_attr.get_field_in_block(
        flash_attr.header_block_format, FlashFieldType.TOTAL_SIZE
    )
    if flash_attr.header_block_format is not None and (
        header_data_size is not None or header_total_size is not None
    ):
        # The header has the size of the entire region including OOB data
        total_ecc_protected_size = 0
        if header_data_size is not None:
            # Found data size in the header, need to calculate expected total size (including OOB)
            data_size_bytes = flash_attr.get_field_data_in_block(
                block_format=flash_attr.header_block_format,
                field_type=FlashFieldType.DATA_SIZE,
                data=data,
                block_start_offset=0,
            )
            oob_size = flash_attr.get_total_oob_size(data_len=data_len, includes_oob=True)
            if data_size_bytes is not None:
                total_ecc_protected_size = oob_size + int.from_bytes(data_size_bytes, "big")
        elif header_total_size is not None:
            # Found total size in header
            total_size_bytes = flash_attr.get_field_data_in_block(
                block_format=flash_attr.header_block_format,
                field_type=FlashFieldType.TOTAL_SIZE,
                data=data,
                block_start_offset=0,
            )
            if total_size_bytes is not None:
                total_ecc_protected_size = int.from_bytes(total_size_bytes, "big")

        if total_ecc_protected_size > data_len:
            raise UnpackerError("Expected larger resource than supplied")

        if total_ecc_protected_size > start_index:
            end_offset = start_index + total_ecc_protected_size

    elif flash_attr.tail_block_format is not None:
        # Tail has either magic, total_size, or data_size to indicate the end
        tail_magic = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.MAGIC
        )
        tail_total_size = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.TOTAL_SIZE
        )
        tail_data_size = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.DATA_SIZE
        )

        if tail_magic is not None:
            # we'll start looking after the header to make sure we don't
            # accidentally find the header magic
            if flash_attr.header_block_format:
                tail_start_index = flash_attr.get_block_size(flash_attr.header_block_format)
            else:
                tail_start_index = start_index
            end_offset = _get_end_from_magic(flash_attr, tail_start_index, data, data_len)
        elif tail_total_size is not None:
            end_offset = _get_end_from_size(
                flash_attr, start_index, data, data_len, FlashFieldType.TOTAL_SIZE
            )
        elif tail_data_size is not None:
            end_offset = _get_end_from_size(
                flash_attr, start_index, data, data_len, FlashFieldType.DATA_SIZE
            )

    # Create the overarching resource
    return await resource.create_child(
        tags=(FlashOobResource,),
        data_range=Range(start_index, end_offset),
        attributes=[
            flash_attr,
        ],
    )

flash.py

ofrak_components.flash

FlashAttributes (ResourceAttributes) dataclass

FlashEccAttributes (ResourceAttributes) dataclass

FlashField dataclass

FlashFieldType (Enum)

FlashLogicalDataResource (GenericBinary) dataclass

FlashLogicalDataResourcePacker (Packer)

pack(self, resource, config=None) async

FlashLogicalEccResource (GenericBinary) dataclass

FlashOobResource (GenericBinary) dataclass

FlashOobResourcePacker (Packer)

pack(self, resource, config=None) async

FlashOobResourceUnpacker (Unpacker)

unpack(self, resource, config=None) async

FlashResource (GenericBinary) dataclass

FlashResourcePacker (Packer)

pack(self, resource, config=None) async

FlashResourceUnpacker (Unpacker)

unpack(self, resource, config=None) async

`ofrak_components.flash`

`FlashAttributes (ResourceAttributes)` `dataclass`

`FlashEccAttributes (ResourceAttributes)` `dataclass`

`FlashField` `dataclass`

`FlashFieldType (Enum)`

`FlashLogicalDataResource (GenericBinary)` `dataclass`

`FlashLogicalDataResourcePacker (Packer)`

`pack(self, resource, config=None)` `async`

`FlashLogicalEccResource (GenericBinary)` `dataclass`

`FlashOobResource (GenericBinary)` `dataclass`

`FlashOobResourcePacker (Packer)`

`pack(self, resource, config=None)` `async`

`FlashOobResourceUnpacker (Unpacker)`

`unpack(self, resource, config=None)` `async`

`FlashResource (GenericBinary)` `dataclass`

`FlashResourcePacker (Packer)`

`pack(self, resource, config=None)` `async`

`FlashResourceUnpacker (Unpacker)`

`unpack(self, resource, config=None)` `async`