Hardware Map ============ .. contents:: Table of Contents :local: .. _introduction: Introduction ~~~~~~~~~~~~ The Python objects described in :doc:`object_model` are used in running Python sessions. They must be instantiated somehow. When writing short, standalone Python scripts, these objects can be created directly: .. doctest:: introduction-test >>> import pydfmux >>> d = pydfmux.Dfmux(hostname='iceboard004.local') >>> m1 = pydfmux.MGMEZZ04(mezzanine=1, serial='fmc2_001') >>> m2 = pydfmux.MGMEZZ04(mezzanine=2, serial='fmc2_002') >>> d.mezzanines = [ m1, m2 ] >>> hwm = pydfmux.HardwareMap() >>> hwm.add(d) >>> hwm.commit() >>> d = hwm.query(pydfmux.Dfmux).one() >>> print(d.mezzanine[2]) Dfmux(u'iceboard004.local').MGMEZZ04(2,u'fmc2_002') This code block is already somewhat clumsy, with just a single dfmux and 2 mezzanines. In an experiment, we typically need to create a large number (thousands) of a much wider variety of objects: * LCBoards containing LCChannels, * Wafers containing Bolometers, * SQUIDs, and * ChannelMappings to associate these structures and readout channels. Formerly, experiments used combinations of CSV, JSON, XML, and Python documents to generate complete hardware descriptions for experiments. In this document, we introduce a similar scheme using `YAML `_ documents. A Simple Example ---------------- To reproduce the above hardware map in YAML, create a file called "hwm.yaml" with the following contents: .. literalinclude:: examples/example1.yaml :language: yaml This hardware map may be loaded in Python as follows: .. doctest:: yaml1 >>> import pydfmux >>> hwm = pydfmux.load_session(open('software/examples/example1.yaml')) This hardware map is identical to the one created in the introduction_: .. doctest:: yaml1 >>> d = hwm.query(pydfmux.Dfmux).one() >>> print(d.mezzanine[2]) Dfmux(u'iceboard004.local').MGMEZZ04(2,u'fmc2_002') A few notes on YAML syntax: * YAML is a superset of JSON. So, :code:`{ key1: value1, key2: value2 }` expresses a *mapping*; :code:`[ value1, value2, ...]` expresses a *sequence*. The YAML above has been deliberately formatted to look familiar to JSON users; below, we'll introduce other ways to format mappings and sequences that are probably better suited to describing large hardware maps. * Tokens beginning with exclamation marks (:code:`!HardwareMap`, :code:`!Dfmux`, :code:`MGMEZZ04`) are *tags*, which tell the YAML parser to treat the following element specially. In the case of :code:`!Dfmux` and :code:`!MGMEZZ04`, the parser will automatically convert dictionaries into :code:`pydfmux.Dfmux` and :code:`pydfmux.GMEZZ04` objects. In the case of :code:`!HardwareMap`, the parser converts a list of hardware map entries (here, a single :code:`pydfmux.Dfmux`) into a fully-fledged HardwareMap object. * The top-level object in our "hwm.yaml" was a :code:`HardwareMap`. It could have been anything (a dictionary or list, perhaps containing :code:`!HardwareMap`-tagged objects). This can be useful when describing more than just a HardwareMap and the objects it can contain. For example:: validity: [ !!timestamp "2014-10-05t21:59:43.10-05:00", !!timestamp "2014-10-06t21:59:43.10-05:00" ] hardware_map: !HardwareMap [ !Dfmux { hostname: "iceboard004.local" mezzanines: [!MGMEZZ04 {serial: "fmc2_001"}, !MGMEZZ04 {serial: "fmc2_002"}] ] As with :code:`!Dfmux`, the :code:`!!timestamp` tag announces a particular type and triggers special behaviour during parsing; in this case, the :code:`!!timestamp` tag is part of the YAML specification and automatically converts the datestring into a Python :code:`datetime.DateTime` object. This suggests YAML documents can be used to describe more than just hardware maps, e.g. test scenarios, quality-control configurations, et cetera. Mappings and Sequences ---------------------- The following two YAML snippets are identical. We begin with YAML code that is deliberately formatted to look like JSON: .. code-block:: yaml !HardwareMap [ !Dfmux { hostname: "iceboard004.local" mezzanines: [!MGMEZZ04 {serial: "fmc2_001"}, !MGMEZZ04 {serial: "fmc2_002"}] } Particularly in larger hardware maps, the square and curly brackets can clutter up the presentation. YAML provides a number of alternative ways to specify mappings and sequences, such as: .. code-block:: yaml # YAML also lets us write comments! !HardwareMap - !Dfmux hostname: iceboard004.local mezzanines: - !MGMEZZ04 { serial: fmc2_001 } - !MGMEZZ04 { serial: fmc2_002 } Note the following: * Unlike JSON, YAML does not require strings to be quoted unless they would otherwise be ambiguous. So, strings like the dfmux hostname :code:`iceboard004.local` may be written plainly. * Mappings are now indicated with :code:`key: value` pairs and scoped using indentation, instead of curly brackets. * Sequences are now indicated using a single dash, and scoped using indentation instead of square brackets. * The second encoding (which does not surround sequences or mappings with delimiters :code:`[]` or :code:`{}`) may produce fewer merge collisions with SCM tools like git. YAML is a medium-sized standard; please refer to the `YAML homepage `_ for details. It should take a short time to read and write simple YAML documents, and a few days to become relatively proficient. Aliases ------- YAML has one last trick up its sleeve: it's hard to put together a complete hardware map without defining an element and needing to refer to it elsewhere. For this purpose, YAML provides *anchors* and *aliases*. Anchors define a label: .. code-block:: yaml - &my_mezz1 !MGMEZZ04 serial: fmc2_001 - &my_mezz2 !MGMEZZ04 serial: fmc2_002 ...anywhere after that, we can use an alias to refer to the object: .. code-block:: yaml - !Dfmux hostname: iceboard004.local mezzanines: [ *my_mezz1, *my_mezz2 ] For the hardware map above, aliases allow us to group like elements together (rather than mixing dfmuxes with nested mezzanine definitions.) For a larger hardware map, a single element might be referred to multiple times and aliases are more important. Building a Complete Hardware Map ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ So far, we have introduced YAML and described ``!tags`` that we've added to generate specific parts of the hardware map. We now provide a complete list of tags, and introduce a few extra helpers used to pull in data from different sources. Creating Experiment-Specific Subclasses --------------------------------------- Let's say we want to instantiate a Wafer object from YAML. We can quickly do so using the following YAML ``example_subclass.yaml``: .. literalinclude:: examples/example_subclass.yaml :language: yaml We can then load this HardwareMap as follows: .. doctest:: subclass-yaml-loader >>> import pydfmux >>> hwm = pydfmux.load_session(open('software/examples/example_subclass.yaml')) >>> w = hwm.query(pydfmux.Wafer).one() >>> print(w.__class__) As the ``print`` statement shows, this snippet produces a :class:`pydfmux.core.dfmux.Wafer` object. However, in :doc:`object_model`, we suggested subclassing the classes in :py:mod:`pydfmux.core.dfmux` classes in order to allow experiments to add custom columns. We need the YAML loader to create these subclasses, instead of their ancestors. To do so, we tell the SessionLoader what module to use by supplying a ``!flavour`` tag: .. literalinclude:: examples/example_subclass2.yaml :language: yaml .. doctest:: subclass-yaml-loader2 >>> import pydfmux >>> hwm = pydfmux.load_session(open('software/examples/example_subclass2.yaml')) >>> w = hwm.query(pydfmux.Wafer).one() >>> print(w.__class__) The loader will create objects from the module specified as a "flavour". If the flavour module does not implement the specified class, you will get an error. (This is preferrable to falling back on :code:`pydfmux.core.dfmux`, since it's quite easy to provide e.g. :code:`pydfmux.mcgill` as a flavour by accident. An error message is quicker to debug than a HWM with the wrong classes in it.) .. note:: Because HWM elements are parsed in order, you may switch back and forth between different flavours (although it's not clear this is a good idea.) Including external YAML/JSON documents -------------------------------------- Because it permits things like comments, YAML is a great way to describe the top level of an experiment's hardware map. (Later on, we'll see how to pull in CSV data as well.) While YAML is machine-readable, it is not suitable for "round-tripping" --- that is, being loaded, modified, and saved *back* to a YAML file by Python code. It's easy to see why: .. doctest:: yaml-no-roundtrip >>> import yaml >>> y = yaml.safe_load(''' ... # This is a simple piece of vanilla YAML data. We wish we could load it ... # in Python and write it back without altering its formatting. ... top_level_is_a_mapping: True ... with_some_other_elements: [ 3.0, ~, { another_mapping: true } ] ... ''') >>> print(yaml.dump(y)) top_level_is_a_mapping: true with_some_other_elements: - 3.0 - null - {another_mapping: true} The contents of the YAML data are unchanged, but it has been reformatted! #. Comments are discarded during YAML parsing. Since they are not present in the Python representation of the YAML data, they are not reproduced when it is re-serialized. #. The style of formatting (e.g. the encoding used for sequences and mappings) has been changed. (For example, the ``[3.0,~,{...}]`` list has been replaced with a dashed encoding.) #. In YAML, JSON, and Python, key ordering is *not* guaranteed. Although the data emerged here in the same order as they were created, they might not have. It is not safe to assume the output data order resembles the input data order at all. These restrictions mean we cannot easily round-trip an entire YAML hardware map without imposing major restrictions on the type of YAML we write. Rather than accept these restrictions, we permit a hardware map to include external YAML/JSON data that *can* be round-tripped (by accepting that it may be arbitrarily reformatted when it is saved.) To do so, we use the ``!include`` tag: .. literalinclude:: examples/include_top.yaml :language: yaml This file includes a subsidiary YAML file, called ``include_child.yaml``. We've given it some fairly arbitrary contents: .. literalinclude:: examples/include_child.yaml :language: yaml We can load both files as follows: .. doctest:: yaml-roundtrip >>> import pydfmux >>> y = pydfmux.load_session(open("software/examples/include_top.yaml")) We can inspect the included data directly: .. doctest:: yaml-roundtrip >>> print(y['included_data']) IncludedYAMLValue({'parameters': [1, 2, 3]}) We can also (optionally) alter ``included_data`` and round-trip the data back to the original YAML file: .. code-block:: python >>> y['included_data']['parameters'] = [4,5,6] >>> y['included_data'].save() .. NOTE:: Let's say we wanted to replace the entire contents of ``included_data``. The following would not work as expected:: >>> y['included_data'] = {'parameters': [4, 5, 6]} >>> y['included_data'].save() AttributeError: 'dict' object has no attribute 'save' This attempt failed because we replaced the entire IncludedYAMLValue class (which defined the ``save()`` method) with a plain Python dictionary. Instead, we need to ensure the wrapper class is also replaced:: >>> y['included_data'] = pydfmux.core.session.IncludedYAMLValue( ... {'parameters': [4,5,6]}, ... filename=y['included_data'].filename) >>> y['included_data'].save() Including CSV Files ------------------- YAML is a good way to specify an experiment's top-level structure. However, for large collections of objects (e.g. bolometers or LC channels), it's much more convenient to store data in an external CSV document. To do this, we provide the :code:`!csv` tag. .. code-block:: yaml - &wafer-arg1a !Wafer name: arg1a bolometers: !CSVBolometers "wafer_arg1a.csv" Assuming the YAML loader has been configured to generate :class:`pydfmux.mcgill.dfmux` classes (see above), this example produces a :class:`pydfmux.mcgill.dfmux.Wafer` element with the name attribute *arg1a*. This Wafer has attached :class:`pydfmux.mcgill.dfmux.Bolometer` classes taken from the CSV file named ``hwm_complete_arg1a.csv``. The contents of this CSV file could be as follows:: name lc_board_pad lc_board_index y_coord x_coord observing_band polarization_angle 1A.6.X 45 1 -35.846 105.988 90GHz -21.6 1A.6.Y 90 1 -35.846 105.988 90GHz 68.4 1A.5.X 4 1 -31.629 96.242 90GHz -66.6 1A.5.Y 3 1 -31.629 96.242 90GHz 23.4 The first line in the CSV file specifies the column headings, which are translated into columns in the Bolometer objects that are constructed. (The names must match exactly! If not, you will see a semi-informative error message.) CSV data must be tab-separated. (It is probably easy to pass arguments to the CSV parser, but it's simpler if we can standardize on tab-separated CSV.) Looking Up HWM Data ------------------- Above, we showed how to import HWM elements from CSV files using the :code:`!csv` tag. Although this permits compact HWMs, it leaves us with a problem: how do we refer to parts of the HWM, e.g. when constructing a ChannelMapping, when we don't have a YAML anchor to use? For example, let's say we want to create a ChannelMapping for bolometer 1A.6.X (see above). How might a ChannelMapping be created with YAML data? .. code-block:: yaml - &wafer-arg1a !Wafer name: arg1a bolometers: !CSVBolometers "hwm_complete_arg1a.csv" - &lc-003 !LCBoard name: LC003 channels: !CSVLCChannels "hwm_complete_lc003.csv" - !ChannelMapping lc_channel: ??? # How do we get e.g. lc-003.channel[0]? bolometer: ??? # How do we get e.g. arg1a.bolometer['1A.6.X']? readout_channel: ??? # ...and readout channels aren't in the YAML at all! squid: *Sq1SBpol03 # Let's at least pretend we have one of these. To resolve these three missing links, we provide a :code:`!HWMLookup` tag. This tag annotates a sequence and walks through HWM links. The following YAML:: !HWMLookup [*lc-003, channel: 1] ...is equivalent to the following Python code:: >>> lc003.channel[1] Likewise, the lookups can be much deeper. Suppose ``*d`` is an alias for a Dfmux object. Then, we can access a :class:`pydfmux.core.dfmux.ReadoutChannel` as follows:: !HWMLookup [*d, mezzanine:1, module:1, channel:1] This is equivalent to the following Python code (if ``d`` is a Dfmux object):: >>> d.mezzanine[1].module[1].channel[1] A Complete Example ~~~~~~~~~~~~~~~~~~ The following YAML HWM demonstrates "one of everything." The top-level YAML ``hwm_complete.yaml`` contains the following: .. literalinclude:: examples/hwm_complete.yaml :language: yaml :linenos: This YAML file references several additional files: ``hwm_complete_bias_properties.json``: This file contains JSON (or YAML) data with contents that may be useful for system tuning. It contains the following: .. literalinclude:: examples/hwm_complete_bias_properties.json :language: json :linenos: ``hwm_complete_arg1a.csv``: This file contains bolometers associated with the wafer "arg1a", stored in CSV format. The contents are as follows: .. literalinclude:: examples/hwm_complete_arg1a.csv :tab-width: 16 :linenos: ``hwm_complete_lc003.csv``: This file contains LC channel definitions associated with the LC board "lc003", in CSV format. The contents are as follows: .. literalinclude:: examples/hwm_complete_lc003.csv :tab-width: 16 :linenos: We load the hardware map as follows: .. doctest:: yaml_complete >>> import pydfmux >>> s = pydfmux.load_session(open('software/examples/hwm_complete.yaml')) >>> hwm = s['hardware_map'] We can now query data from the hardware map: .. doctest:: yaml_complete >>> bolo = hwm.query(pydfmux.Bolometer).filter(pydfmux.Bolometer.name=='1A.6.X').one() >>> print(bolo) Wafer(u'arg1a').Bolometer(u'1A.6.X') >>> print(bolo.polarization_angle) -21.6 Tag Reference ~~~~~~~~~~~~~ The "stock" Python YAML parser has been augmented with the following tags. Hardware Map Objects -------------------- :code:`!HardwareMap [...]` creates a :code:`core.hardware_map.HardwareMap()` from a list of suitable objects. :code:`!HWMLookup [*some_reference, key:value, ...]` returns a single HWM object, retrieved by accessing attributes from an object already in the hardware map. Direct Class Instantiation -------------------------- :code:`!IceCrate {...}` creates a :code:`IceCrate` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - !IceCrate serial: foo-bar-001 slots: [ *my_dfmux1 ] :code:`!Dfmux {...}` creates a :code:`Dfmux` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - &my_dfmux1 !Dfmux hostname: iceboard004.local serial: 004 mezzanines: [ *my_mezz1, *my_mezz2 ] :code:`!MGMEZZ04 {...}` creates a :code:`MGMEZZ04` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - &my_mezz1 !MGMEZZ04 serial: fmc2_001 squid_controller: *my_sqc :code:`!SQUIDController {...}` creates a :code:`SQUIDController` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - &my_sqc !SQUIDController serial: 06-01 squids: [ *squid1, *squid2, *squid3, *squid4 ] :code:`!Wafer {...}` creates a :code:`Wafer` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - !Wafer name: c1 bolometers: [...] :code:`!Bolometer {...}` creates a :code:`Bolometer` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - !Bolometer { name: C1.B1.16.X } :code:`!LCBoard {...}` creates a :code:`LCBoard` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - !LCBoard name: LC027 channels: [...] :code:`!LCChannel {...}` creates a :code:`LCChannel` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - &lc027-1 !LCChannel { } :code:`!ChannelMapping {...}` creates a :code:`ChannelMapping` object from a mapping. Key/value pairs are passed directly to the object's constructor. For example:: - !ChannelMapping readout_channel: !HWMLookup [*fmc2_001, module: 1, channel: 1] lc_channel: *lc027-1 bolometer: *c1-b1-16-x squid: *Sq1SBpol03 External Files -------------- :code:`!CSVBolometers "filename.csv"` loads :code:`Bolometer` objects from the file ``filename.csv``. This file must be tab-separated, and must have column headings that exactly match the attributes of the bolometers to create. The tag generates a list of Bolometers, suitable for attachment to a :code:`Wafer` object. :code:`!CSVLCChannels "filename.csv"` loads :code:`LCChannel` objects from the file ``filename.csv``. This file must be tab-separated, and must have column headings that exactly match the attributes of the bolometers to create. The tag generates a list of LCChannels, suitable for attachment to a :code:`LCBoard` object. :code:`!include "filename.yaml"` includes YAML or JSON data from ``filename.yaml``. If you don't mind having it reformatted (i.e. losing comments, in the case of YAML data), you can "roundtrip" this data by calling .save() on its handle in Python. References ~~~~~~~~~~ * `YAML homepage `_ .. vim: sts=3 ts=3 sw=3 tw=78 smarttab expandtab