Ignore:
Timestamp:
Aug 7, 2006, 7:40:23 AM (18 years ago)
Author:
Chris Staub <chris@…>
Children:
948056c
Parents:
20f0171
Message:

Updated date

File:
1 edited

Legend:

Unmodified
Added
Removed

  • bootscripts/common/udev.xml

    r20f0171 rd0ed88e8  
    22<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
    33  "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
    4   <!ENTITY % general-entities SYSTEM "../../general.ent">
     4  <!ENTITY % general-entities SYSTEM "../general.ent">
    55  %general-entities;
    66]>
    … …  
    99  <?dbhtml filename="udev.html"?>
    1010
    11   <title>Device and Module Handling on a CLFS System</title>
     11  <title>Device and Module Handling on an LFS System</title>
    1212
    1313  <indexterm zone="ch-scripts-udev">
    1414    <primary sortas="a-Udev">Udev</primary>
    15   <secondary>usage</secondary></indexterm>
     15    <secondary>usage</secondary>
     16  </indexterm>
    1617
    1718  <para>In <xref linkend="chapter-building-system"/>, we installed the Udev
    18   package. Before we go into the details regarding how this works, a brief
    19   history of previous methods of handling devices is in order.</para>
     19  package. Before we go into the details regarding how this works,
     20  a brief history of previous methods of handling devices is in
     21  order.</para>
    2022
    2123  <para>Linux systems in general traditionally use a static device creation
    2224  method, whereby a great many device nodes are created under <filename
    2325  class="directory">/dev</filename> (sometimes literally thousands of nodes),
    24   regardless of whether the corresponding hardware devices actually exist.
    25   This is typically done via a <command>MAKEDEV</command> script, which
    26   contains a number of calls to the <command>mknod</command> program with
    27   the relevant major and minor device numbers for every possible device that
    28   might exist in the world. Using the Udev method, only those devices which
    29   are detected by the kernel get device nodes created for them. Because
    30   these device nodes will be created each time the system boots, they will
    31   be stored on a <systemitem class="filesystem">tmpfs</systemitem> (a virtual
    32   file system that resides entirely in system memory). Device nodes do not
    33   require much space, so the memory that is used is negligible.</para>
     26  regardless of whether the corresponding hardware devices actually exist. This
     27  is typically done via a <command>MAKEDEV</command> script, which contains a
     28  number of calls to the <command>mknod</command> program with the relevant
     29  major and minor device numbers for every possible device that might exist in
     30  the world.</para>
     31
     32  <para>Using the Udev method, only those devices which are detected by the
     33  kernel get device nodes created for them. Because these device nodes will be
     34  created each time the system boots, they will be stored on a <systemitem
     35  class="filesystem">tmpfs</systemitem> file system (a virtual file system that
     36  resides entirely in system memory). Device nodes do not require much space, so
     37  the memory that is used is negligible.</para>
    3438
    3539  <sect2>
    … …  
    3943    class="filesystem">devfs</systemitem> was merged into the 2.3.46 kernel
    4044    and was made available during the 2.4 series of stable kernels. Although
    41     it was present in the kernel source itself, this method of creating
    42     devices dynamically never received overwhelming support from the core
    43     kernel developers.</para>
     45    it was present in the kernel source itself, this method of creating devices
     46    dynamically never received overwhelming support from the core kernel
     47    developers.</para>
    4448
    4549    <para>The main problem with the approach adopted by <systemitem
    46     class="filesystem">devfs</systemitem> was the way it handled
    47     device detection, creation, and naming. The latter issue, that of
    48     device node naming, was perhaps the most critical. It is generally
    49     accepted that if device names are allowed to be configurable, then
    50     the device naming policy should be up to a system administrator, not
    51     imposed on them by any particular developer(s). The <systemitem
     50    class="filesystem">devfs</systemitem> was the way it handled device
     51    detection, creation, and naming. The latter issue, that of device node
     52    naming, was perhaps the most critical. It is generally accepted that if
     53    device names are allowed to be configurable, then the device naming policy
     54    should be up to a system administrator, not imposed on them by any
     55    particular developer(s). The <systemitem
    5256    class="filesystem">devfs</systemitem> file system also suffers from race
    53     conditions that are inherent in its design and cannot be fixed
    54     without a substantial revision to the kernel. It has also been marked
    55     as deprecated due to a lack of recent maintenance.</para>
    56 
    57     <para>With the development of the unstable 2.5 kernel tree, later
    58     released as the 2.6 series of stable kernels, a new virtual filesystem
    59     called <systemitem class="filesystem">sysfs</systemitem> came to be. The
    60     job of <systemitem class="filesystem">sysfs</systemitem> is to export a
    61     view of the system's hardware configuration to userspace processes. With
    62     this userspace-visible representation, the possibility of seeing a
    63     userspace replacement for <systemitem class="filesystem">devfs</systemitem>
    64     became much more realistic.</para>
     57    conditions that are inherent in its design and cannot be fixed without a
     58    substantial revision to the kernel. It has also been marked as deprecated
     59    due to a lack of recent maintenance.</para>
     60
     61    <para>With the development of the unstable 2.5 kernel tree, later released
     62    as the 2.6 series of stable kernels, a new virtual filesystem called
     63    <systemitem class="filesystem">sysfs</systemitem> came to be. The job of
     64    <systemitem class="filesystem">sysfs</systemitem> is to export a view of
     65    the system's hardware configuration to userspace processes. With this
     66    userspace-visible representation, the possibility of seeing a userspace
     67    replacement for <systemitem class="filesystem">devfs</systemitem> became
     68    much more realistic.</para>
    6569
    6670  </sect2>
    … …  
    6973    <title>Udev Implementation</title>
    7074
    71     <para>The <systemitem class="filesystem">sysfs</systemitem> filesystem
    72     was mentioned briefly above. One may wonder how <systemitem
    73     class="filesystem">sysfs</systemitem> knows about the devices present
    74     on a system and what device numbers should be used for them. Drivers
    75     that have been compiled into the kernel directly register their objects
    76     with <systemitem class="filesystem">sysfs</systemitem> as they are
    77     detected by the kernel. For drivers compiled as modules, this
    78     registration will happen when the module is loaded. Once the
    79     <systemitem class="filesystem">sysfs</systemitem> filesystem is mounted
    80     (on <filename class="directory">/sys</filename>), data which the built-in
    81     drivers registered with <systemitem class="filesystem">sysfs</systemitem>
    82     are available to userspace processes and to <command>udev</command> for
    83     device node creation.</para>
    84 
    85     <para>The <command>S10udev</command> initscript takes care of creating
    86     these device nodes when Linux is booted. This script starts by registering
    87     <command>/sbin/udevsend</command> as a hotplug event handler. Hotplug
    88     events (discussed below) are not usually generated during this stage,
    89     but <command>udev</command> is registered just in case they do occur.
    90     The <command>udevstart</command> program then walks through the
    91     <systemitem class="filesystem">/sys</systemitem> filesystem and creates
    92     devices under <filename class="directory">/dev</filename> that match the
    93     descriptions. For example, <filename>/sys/class/tty/vcs/dev</filename>
    94     contains the string <quote>7:0</quote> This string is used by
    95     <command>udevstart</command> to create <filename>/dev/vcs</filename>
    96     with major number <emphasis>7</emphasis> and minor <emphasis>0</emphasis>.
    97     The names and permissions of the nodes created under the <filename
    98     class="directory">/dev</filename> directory are configured according to
    99     the rules specified in the files within the <filename
    100     class="directory">/etc/udev/rules.d/</filename> directory. These are
    101     numbered in a similar fashion to the CLFS-Bootscripts package. If
    102     <command>udev</command> can't find a rule for the device it is creating,
    103     it will default permissions to <emphasis>660</emphasis> and ownership to
    104     <emphasis>root:root</emphasis>.</para>
    105 
    106     <para>Once the above stage is complete, all devices that were already
    107     present and have compiled-in drivers will be available for use. This
    108     leads us to the devices that have modular drivers.</para>
    109 
    110     <para>Earlier, we mentioned the concept of a <quote>hotplug event
    111     handler.</quote> When a new device connection is detected by the kernel,
    112     the kernel will generate a hotplug event and look at the file
    113     <filename>/proc/sys/kernel/hotplug</filename> to determine the userspace
    114     program that handles the device's connection. The <command>udev</command>
    115     bootscript registered <command>udevsend</command> as this handler. When
    116     these hotplug events are generated, the kernel will tell
    117     <command>udev</command> to check the <filename
    118     class="directory">/sys</filename> filesystem for the information
    119     pertaining to this new device and create the <filename
    120     class="directory">/dev</filename> entry for it.</para>
    121 
    122     <para>This brings us to one problem that exists with
    123     <command>udev</command>, and likewise with <systemitem
    124     class="filesystem">devfs</systemitem> before it. It is commonly
    125     referred to as the <quote>chicken and egg</quote> problem. Most
    126     Linux distributions handle loading modules via entries in
    127     <filename>/etc/modules.conf</filename>. Access to a device node causes
    128     the appropriate kernel module to load. With <command>udev</command>,
    129     this method will not work because the device node does not exist until
    130     the module is loaded. To solve this, the <command>S05modules</command>
    131     bootscript was added to the CLFS-Bootscripts package, along with the
    132     <filename>/etc/sysconfig/modules</filename> file. By adding module
    133     names to the <filename>modules</filename> file, these modules will be
    134     loaded when the computer starts up. This allows <command>udev</command>
    135     to detect the devices and create the appropriate device nodes.</para>
    136 
    137     <para>Note that on slower machines or for drivers that create a lot
    138     of device nodes, the process of creating devices may take a few
    139     seconds to complete. This means that some device nodes may not be
    140     immediately accessible.</para>
     75    <sect3>
     76      <title>Sysfs</title>
     77
     78      <para>The <systemitem class="filesystem">sysfs</systemitem> filesystem was
     79      mentioned briefly above. One may wonder how <systemitem
     80      class="filesystem">sysfs</systemitem> knows about the devices present on
     81      a system and what device numbers should be used for them. Drivers that
     82      have been compiled into the kernel directly register their objects with
     83      <systemitem class="filesystem">sysfs</systemitem> as they are detected by
     84      the kernel. For drivers compiled as modules, this registration will happen
     85      when the module is loaded. Once the <systemitem
     86      class="filesystem">sysfs</systemitem> filesystem is mounted (on <filename
     87      class="directory">/sys</filename>), data which the built-in drivers
     88      registered with <systemitem class="filesystem">sysfs</systemitem> are
     89      available to userspace processes and to <command>udevd</command> for device
     90      node creation.</para>
     91
     92    </sect3>
     93
     94    <sect3>
     95      <title>Udev Bootscript</title>
     96
     97      <para>The <command>S10udev</command> initscript takes care of creating
     98      device nodes when Linux is booted. The script unsets the uevent handler
     99      from the default of <command>/sbin/hotplug</command>.  This is done
     100      because the kernel no longer needs to call out to an external binary.
     101      Instead <command>udevd</command> will listen on a netlink socket for
     102      uevents that the kernel raises. Next, the bootscript copies any static
     103      device nodes that exist in <filename
     104      class="directory">/lib/udev/devices</filename> to <filename
     105      class="directory">/dev</filename>. This is necessary because some devices,
     106      directories, and symlinks are needed before the dynamic device handling
     107      processes are available during the early stages of booting a system.
     108      Creating static device nodes in <filename
     109      class="directory">/lib/udev/devices</filename> also provides an easy
     110      workaround for devices that are not supported by the dynamic device
     111      handling infrastructure. The bootscript then starts the Udev daemon,
     112      <command>udevd</command>, which will act on any uevents it receives.
     113      Finally, the bootscript forces the kernel to replay uevents for any
     114      devices that have already been registered and then waits for
     115      <command>udevd</command> to handle them.</para>
     116
     117    </sect3>
     118
     119    <sect3>
     120      <title>Device Node Creation</title>
     121
     122      <para>To obtain the right major and minor number for a device, Udev relies
     123      on the information provided by <systemitem
     124      class="filesystem">sysfs</systemitem> in <filename
     125      class="directory">/sys</filename>.  For example,
     126      <filename>/sys/class/tty/vcs/dev</filename> contains the string
     127      <quote>7:0</quote>. This string is used by <command>udevd</command>
     128      to create a device node with major number <emphasis>7</emphasis> and minor
     129      <emphasis>0</emphasis>. The names and permissions of the nodes created
     130      under the <filename class="directory">/dev</filename> directory are
     131      determined by rules specified in the files within the <filename
     132      class="directory">/etc/udev/rules.d/</filename> directory. These are
     133      numbered in a similar fashion to the LFS-Bootscripts package. If
     134      <command>udevd</command> can't find a rule for the device it is creating,
     135      it will default permissions to <emphasis>660</emphasis> and ownership to
     136      <emphasis>root:root</emphasis>. Documentation on the syntax of the Udev
     137      rules configuration files are available in
     138      <filename>/usr/share/doc/udev-&udev-version;/index.html</filename></para>
     139
     140    </sect3>
     141
     142    <sect3>
     143      <title>Module Loading</title>
     144
     145      <para>Device drivers compiled as modules may have aliases built into them.
     146      Aliases are visible in the output of the <command>modinfo</command>
     147      program and are usually related to the bus-specific identifiers of devices
     148      supported by a module. For example, the <emphasis>snd-fm801</emphasis>
     149      driver supports PCI devices with vendor ID 0x1319 and device ID 0x0801,
     150      and has an alias of <quote>pci:v00001319d00000801sv*sd*bc04sc01i*</quote>.
     151      For most devices, the bus driver exports the alias of the driver that
     152      would handle the device via <systemitem
     153      class="filesystem">sysfs</systemitem>. E.g., the
     154      <filename>/sys/bus/pci/devices/0000:00:0d.0/modalias</filename> file
     155      might contain the string
     156      <quote>pci:v00001319d00000801sv00001319sd00001319bc04sc01i00</quote>.
     157      The rules that LFS installs will cause <command>udevd</command> to call
     158      out to <command>/sbin/modprobe</command> with the contents of the
     159      <envar>MODALIAS</envar> uevent environment variable (that should be the
     160      same as the contents of the <filename>modalias</filename> file in sysfs),
     161      thus loading all modules whose aliases match this string after wildcard
     162      expansion.</para>
     163
     164      <para>In this example, this means that, in addition to
     165      <emphasis>snd-fm801</emphasis>, the obsolete (and unwanted)
     166      <emphasis>forte</emphasis> driver will be loaded if it is
     167      available. See below for ways in which the loading of unwanted drivers can
     168      be prevented.</para>
     169
     170      <para>The kernel itself is also able to load modules for network
     171      protocols, filesystems and NLS support on demand.</para>
     172
     173    </sect3>
     174
     175    <sect3>
     176      <title>Handling Hotpluggable/Dynamic Devices</title>
     177
     178      <para>When you plug in a device, such as a Universal Serial Bus (USB) MP3
     179      player, the kernel recognizes that the device is now connected and
     180      generates a uevent. This uevent is then handled by
     181      <command>udevd</command> as described above.</para>
     182
     183    </sect3>
    141184
    142185  </sect2>
    143186
    144187  <sect2>
    145     <title>Handling Hotpluggable/Dynamic Devices</title>
    146 
    147     <para>When you plug in a device, such as a Universal Serial Bus (USB)
    148     MP3 player, the kernel recognizes that the device is now connected and
    149     generates a hotplug event. If the driver is already loaded (either
    150     because it was compiled into the kernel or because it was loaded via
    151     the <command>S05modules</command> bootscript), <command>udev</command>
    152     will be called upon to create the relevant device node(s) according to
    153     the <systemitem class="filesystem">sysfs</systemitem> data available in
    154     <filename class="directory">/sys</filename>.</para>
    155 
    156     <para>If the driver for the just plugged in device is available as a
    157     module but currently unloaded, the Hotplug package will load the
    158     appropriate module and make this device available by creating the
    159     device node(s) for it.</para>
    160 
    161   </sect2>
    162 
    163   <sect2>
    164     <title>Problems with Creating Devices</title>
    165 
    166     <para>There are a few known problems when it comes to automatically
    167     creating device nodes:</para>
    168 
    169     <para>1) A kernel driver may not export its data to <systemitem
    170     class="filesystem">sysfs</systemitem>.</para>
    171 
    172     <para>This is most common with third party drivers from outside the
    173     kernel tree. Udev will be unable to automatically create device nodes
    174     for such drivers. Use the <filename>/etc/sysconfig/createfiles</filename>
    175     configuration file to manually create the devices. Consult the
    176     <filename>devices.txt</filename> file inside the kernel documentation
    177     or the documentation for that driver to find the proper major/minor
    178     numbers.</para>
    179 
    180     <para>2) A non-hardware device is required. This is most common with
    181     the Advanced Linux Sound Architecture (ALSA) project's Open Sound
    182     System (OSS) compatibility module. These types of devices can be
    183     handled in one of two ways:</para>
    184 
    185     <itemizedlist>
    186       <listitem>
    187         <para>Adding the module names to
    188         <filename>/etc/sysconfig/modules</filename></para>
    189       </listitem>
    190       <listitem>
    191         <para>Using an <quote>install</quote> line in
    192         <filename>/etc/modprobe.conf</filename>. This tells the
    193         <command>modprobe</command> command <quote>when loading this
    194         module, also load this other module, at the same time.</quote>
    195         For example:</para>
    196 
    197 <screen><userinput>install snd-pcm modprobe -i snd-pcm ; modprobe \
    198     snd-pcm-oss ; true</userinput></screen>
    199 
    200         <para>This will cause the system to load both the
    201         <emphasis>snd-pcm</emphasis> and <emphasis>snd-pcm-oss</emphasis>
    202         modules when any request is made to load the driver
    203         <emphasis>snd-pcm</emphasis>.</para>
    204       </listitem>
    205     </itemizedlist>
     188    <title>Problems with Loading Modules and Creating Devices</title>
     189
     190    <para>There are a few possible problems when it comes to automatically
     191    creating device nodes.</para>
     192
     193    <sect3>
     194      <title>A kernel module is not loaded automatically</title>
     195
     196      <para>Udev will only load a module if it has a bus-specific alias and the
     197      bus driver properly exports the necessary aliases to <systemitem
     198      class="filesystem">sysfs</systemitem>. In other cases, one should
     199      arrange module loading by other means. With Linux-&linux-version;, Udev is
     200      known to load properly-written drivers for INPUT, IDE, PCI, USB, SCSI,
     201      SERIO and FireWire devices.</para>
     202
     203      <para>To determine if the device driver you require has the necessary
     204      support for Udev, run <command>modinfo</command> with the module name as
     205      the argument.  Now try locating the device directory under
     206      <filename class="directory">/sys/bus</filename> and check whether there is
     207      a <filename>modalias</filename> file there.</para>
     208
     209      <para>If the <filename>modalias</filename> file exists in <systemitem
     210      class="filesystem">sysfs</systemitem>, the driver supports the device and
     211      can talk to it directly, but doesn't have the alias, it is a bug in the
     212      driver. Load the driver without the help from Udev and expect the issue
     213      to be fixed later.</para>
     214
     215      <para>If there is no <filename>modalias</filename> file in the relevant
     216      directory under <filename class="directory">/sys/bus</filename>, this
     217      means that the kernel developers have not yet added modalias support to
     218      this bus type. With Linux-&linux-version;, this is the case with ISA
     219      busses. Expect this issue to be fixed in later kernel versions.</para>
     220
     221      <para>Udev is not intended to load <quote>wrapper</quote> drivers such as
     222      <emphasis>snd-pcm-oss</emphasis> and non-hardware drivers such as
     223      <emphasis>loop</emphasis> at all.</para>
     224
     225    </sect3>
     226
     227    <sect3>
     228      <title>A kernel module is not loaded automatically, and Udev is not
     229      intended to load it</title>
     230
     231      <para>If the <quote>wrapper</quote> module only enhances the functionality
     232      provided by some other module (e.g., <emphasis>snd-pcm-oss</emphasis>
     233      enhances the functionality of <emphasis>snd-pcm</emphasis> by making the
     234      sound cards available to OSS applications), configure
     235      <command>modprobe</command> to load the wrapper after Udev loads the
     236      wrapped module. To do this, add an <quote>install</quote> line in
     237      <filename>/etc/modprobe.conf</filename>. For example:</para>
     238
     239<screen role="nodump"><literal>install snd-pcm /sbin/modprobe -i snd-pcm ; \
     240    /sbin/modprobe snd-pcm-oss ; true</literal></screen>
     241
     242      <para>If the module in question is not a wrapper and is useful by itself,
     243      configure the <command>S05modules</command> bootscript to load this
     244      module on system boot. To do this, add the module name to the
     245      <filename>/etc/sysconfig/modules</filename> file on a separate line.
     246      This works for wrapper modules too, but is suboptimal in that case.</para>
     247
     248    </sect3>
     249
     250    <sect3>
     251      <title>Udev loads some unwanted module</title>
     252
     253      <para>Either don't build the module, or blacklist it in
     254      <filename>/etc/modprobe.conf</filename> file as done with the
     255      <emphasis>forte</emphasis> module in the example below:</para>
     256
     257<screen role="nodump"><literal>blacklist forte</literal></screen>
     258
     259      <para>Blacklisted modules can still be loaded manually with the
     260      explicit <command>modprobe</command> command.</para>
     261
     262    </sect3>
     263
     264    <sect3>
     265      <title>Udev creates a device incorrectly, or makes a wrong symlink</title>
     266
     267      <para>This usually happens if a rule unexpectedly matches a device. For
     268      example, a poorly-writen rule can match both a SCSI disk (as desired)
     269      and the corresponding SCSI generic device (incorrectly) by vendor.
     270      Find the offending rule and make it more specific.</para>
     271
     272    </sect3>
     273
     274    <sect3>
     275      <title>Udev rule works unreliably</title>
     276
     277      <para>This may be another manifestation of the previous problem. If not,
     278      and your rule uses <systemitem class="filesystem">sysfs</systemitem>
     279      attributes, it may be a kernel timing issue, to be fixed in later kernels.
     280      For now, you can work around it by creating a rule that waits for the used
     281      <systemitem class="filesystem">sysfs</systemitem> attribute and appending
     282      it to the <filename>/etc/udev/rules.d/10-wait_for_sysfs.rules</filename>
     283      file. Please notify the LFS Development list if you do so and it
     284      helps.</para>
     285
     286    </sect3>
     287
     288    <sect3>
     289      <title>Udev does not create a device</title>
     290
     291      <para>Further text assumes that the driver is built statically into the
     292      kernel or already loaded as a module, and that you have already checked
     293      that Udev doesn't create a misnamed device.</para>
     294
     295      <para>Udev has no information needed to create a device node if a kernel
     296      driver does not export its data to <systemitem
     297      class="filesystem">sysfs</systemitem>.
     298      This is most common with third party drivers from outside the kernel
     299      tree. Create a static device node in
     300      <filename>/lib/udev/devices</filename> with the appropriate major/minor
     301      numbers (see the file <filename>devices.txt</filename> inside the kernel
     302      documentation or the documentation provided by the third party driver
     303      vendor). The static device node will be copied to
     304      <filename class="directory">/dev</filename> by the
     305      <command>S10udev</command> bootscript.</para>
     306
     307    </sect3>
     308
     309    <sect3>
     310      <title>Device naming order changes randomly after rebooting</title>
     311
     312      <para>This is due to the fact that Udev, by design, handles uevents and
     313      loads modules in parallel, and thus in an unpredictable order. This will
     314      never be <quote>fixed</quote>. You should not rely upon the kernel device
     315      names being stable. Instead, create your own rules that make symlinks with
     316      stable names based on some stable attributes of the device, such as a
     317      serial number or the output of various *_id utilities installed by Udev.
     318      See <xref linkend="ch-scripts-symlinks"/> and
     319      <xref linkend="ch-scripts-network"/> for examples.</para>
     320
     321    </sect3>
    206322
    207323  </sect2>
    … …  
    214330
    215331    <itemizedlist>
     332
    216333      <listitem>
    217334        <para remap="verbatim">A Userspace Implementation of <systemitem class="filesystem">devfs</systemitem>
    218335        <ulink url="http://www.kroah.com/linux/talks/ols_2003_udev_paper/Reprint-Kroah-Hartman-OLS2003.pdf"/></para>
    219336      </listitem>
     337
    220338      <listitem>
    221339        <para remap="verbatim">udev FAQ
    222340        <ulink url="http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ"/></para>
    223341      </listitem>
     342
    224343      <listitem>
    225         <para remap="verbatim">The Linux Kernel Driver Model
    226         <ulink url="http://public.planetmirror.com/pub/lca/2003/proceedings/papers/Patrick_Mochel/Patrick_Mochel.pdf"/></para>
     344        <para remap="verbatim">The <systemitem class="filesystem">sysfs</systemitem> Filesystem
     345        <ulink url="http://www.kernel.org/pub/linux/kernel/people/mochel/doc/papers/ols-2005/mochel.pdf"/></para>
    227346      </listitem>
     347
    228348    </itemizedlist>
    229349
Note: See TracChangeset for help on using the changeset viewer.