Kevin Deldyckehttps://kevin.deldycke.com/2022-06-10T12:00:00+02:00 — 🩬đŸȘ’đŸ‡đŸ•ł yak-shaving the rabbit holesCloudFlare commands2022-06-10T12:00:00+02:002022-06-10T12:00:00+02:00Kevin Deldycketag:kevin.deldycke.com,2022-06-10:/2022/06/cloudflare-commands/<h2 id="api"><span class="caps">API</span><a class="headerlink" href="#api" title="Permanent link">&para;</a></h2> <h2 id="dns-records"><span class="caps">DNS</span> records<a class="headerlink" href="#dns-records" title="Permanent link">&para;</a></h2> <ul> <li>List <span class="caps">DNS</span> records of a&nbsp;zone:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>curl<span class="w"> </span>-X<span class="w"> </span>GET<span class="w"> </span><span class="s2">&quot;https://api.cloudflare.com/client/v4/zones/&lt;ZONE_ID&gt;/dns_records&quot;</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="2 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer &lt;TOKEN&gt;&quot;</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="3 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type:application/json&quot;</span> </code></pre></div> <ul> <li>List <span class="caps">DNS</span> records IDs of a&nbsp;zone:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>curl<span class="w"> </span>-X<span class="w"> </span>GET<span class="w"> </span><span class="s2">&quot;https://api.cloudflare.com/client/v4/zones/&lt;ZONE_ID&gt;/dns_records&quot;</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="2 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization 
</span></code></pre></div><h2 id="api"><span class="caps">API</span><a class="headerlink" href="#api" title="Permanent link">¶</a></h2> <h2 id="dns-records"><span class="caps">DNS</span> records<a class="headerlink" href="#dns-records" title="Permanent link">¶</a></h2> <ul> <li>List <span class="caps">DNS</span> records of a zone:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>curl<span class="w"> </span>-X<span class="w"> </span>GET<span class="w"> </span><span class="s2">"https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records"</zone_id></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="2 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer <token>"</token></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="3 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type:application/json"</span> </code></pre></div> <ul> <li>List <span class="caps">DNS</span> records IDs of a zone:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>curl<span class="w"> </span>-X<span class="w"> </span>GET<span class="w"> </span><span class="s2">"https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records"</zone_id></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="2 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer <token>"</token></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="3 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type:application/json"</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="4 "></span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">'.result[].id'</span> <span class="linenos" data-linenos="5 "></span><span class="go">"82c881261189dc8b8ddbd756cffccd21"</span> <span class="linenos" data-linenos="6 "></span><span class="go">"324437ed3e1212770edeabb65bb3cd6a"</span> </code></pre></div> <ul> <li>Delete one <span class="caps">DNS</span> record:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>curl<span class="w"> </span>-X<span class="w"> </span>DELETE<span class="w"> </span><span class="s2">"https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records/<record_id>"</record_id></zone_id></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="2 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer <token>"</token></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="3 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type:application/json"</span> </code></pre></div> <ul> <li>Delete all <span class="caps">DNS</span> records of a zone:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>curl<span class="w"> </span>-X<span class="w"> </span>GET<span class="w"> </span><span class="s2">"https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records"</zone_id></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="2 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer <token>"</token></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="3 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type:application/json"</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="4 "></span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>--raw-output<span class="w"> </span><span class="s1">'.result[].id'</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="5 "></span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>-I<span class="w"> </span><span class="s1">'{}'</span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="6 "></span><span class="w"> </span>curl<span class="w"> </span>-X<span class="w"> </span>DELETE<span class="w"> </span><span class="s2">"https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records/{}"</zone_id></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="7 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer <token>"</token></span><span class="w"> </span><span class="se">\</span> <span class="linenos" data-linenos="8 "></span><span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type:application/json"</span> </code></pre></div>Migration from FreeNAS to TrueNAS2020-12-27T00:00:00+01:002020-12-27T00:00:00+01:00Kevin Deldycketag:kevin.deldycke.com,2020-12-27:/2020/12/migration-from-freenas-to-truenas/<p>The upgrade itself is quite straightforward by the way of the <span class="caps">GUI</span>. But things gets complicated if you&rsquo;ve got an encrypted&nbsp;pool.</p> <h2 id="encrypted-pool-migration">Encrypted pool migration<a class="headerlink" href="#encrypted-pool-migration" title="Permanent link">&para;</a></h2> <p>Upgrading from FreeNAS to TrueNAS will render your encrypted pool unavailable. FreeNAS relies on <a href="https://en.wikipedia.org/wiki/Geli_(software)"><code>geli</code></a>&lsquo;s block-layer disk encryption and is considered legacy. TrueNAS 
</p><p>The upgrade itself is quite straightforward by the way of the <span class="caps">GUI</span>. But things gets complicated if you’ve got an encrypted pool.</p> <h2 id="encrypted-pool-migration">Encrypted pool migration<a class="headerlink" href="#encrypted-pool-migration" title="Permanent link">¶</a></h2> <p>Upgrading from FreeNAS to TrueNAS will render your encrypted pool unavailable. FreeNAS relies on <a href="https://en.wikipedia.org/wiki/Geli_(software)"><code>geli</code></a>‘s block-layer disk encryption and is considered legacy. TrueNAS moved to native <span class="caps">ZFS</span> encryption at the dataset level.</p> <p>We have no choice but to convert it. And <a href="https://www.truenas.com/docs/hub/initial-setup/storage/encryption/#conversion-from-geli">there is no easy path</a> but destroying the pool and recreating it.Â đŸ˜±</p> <p>So first things first: <strong><span class="caps">BACKUP</span> <span class="caps">YOUR</span> <span class="caps">POOL</span></strong>!</p> <p>But I’m stupid. So I will attempt the migration without any safe guard.</p> <p>I currently run a <a href="https://kevin.deldycke.com/2020/05/nas-hardware/#raid-array">4 * 6 <span class="caps">TB</span> <span class="caps">RAID</span>-Z2 array</a>, which is half-empty. I feel the risk is worth taking as long as I only manipulate one disk at a time.</p> <p>My plan consist in:</p> <ol> <li>Removing the encryption on the array, disk by disk;</li> <li>Then upgrade the array to the new OpenZFS 2.0;</li> <li>To finally re-activate the encryption later.</li> </ol> <h2 id="logging-in">Logging in<a class="headerlink" href="#logging-in" title="Permanent link">¶</a></h2> <p>After the migration to TrueNAS <code>12.0-U1</code>, the main pool (which I called <code>big</code>), gets locked:</p> <p><img alt="" src="/uploads/2020/legacy-encrypted-truenas-zfs-offline-pool.jpeg"/></p> <p>All system metadata resides there. We can no longer <span class="caps">SSH</span> into the box with our regular users. We’ll hack our way in and create a new temporary one.</p> <p>Go to <code>Accounts</code> &gt; <code>Users</code>, click <code>Add</code> button. Then setup our new user:</p> <ul> <li>Username: <code>kev-tmp</code></li> <li>Password: Get a good random one</li> <li>Home Directory: <code>/nonexistent</code></li> <li>Keep all other defaults as-is</li> </ul> <p>Go to <code>Services</code>. Stop all daemons, but make sure <code>SSH</code> is <code>Running</code>.</p> <p>Click <code>SSH</code><span class="quo">‘</span>s <code>Actions</code>, and in the <code>General Options</code> check <code>Allow Password Authentication</code>.</p> <p>Double check you’re allowed to authenticate with a password from your machine:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>(
) <span class="linenos" data-linenos="2 "></span>Host truenas.local <span class="hll"><span class="linenos" data-linenos="3 "></span> PasswordAuthentication yes </span></code></pre></div> <p>Then login to your <span class="caps">NAS</span> as <code>root</code>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">$ </span>ssh<span class="w"> </span>kev-tmp@truenas.local <span class="linenos" data-linenos=" 2 "></span><span class="go">Host key fingerprint is SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">kev-tmp@truenas.local's password:</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">FreeBSD 12.2-RELEASE-p2 663e6b09467(HEAD) TRUENAS</span> <span class="linenos" data-linenos=" 5 "></span> <span class="linenos" data-linenos=" 6 "></span><span class="go">TrueNAS (c) 2009-2020, iXsystems, Inc.</span> <span class="linenos" data-linenos=" 7 "></span><span class="go">All rights reserved.</span> <span class="linenos" data-linenos=" 8 "></span><span class="go">TrueNAS code is released under the modified BSD license with some</span> <span class="linenos" data-linenos=" 9 "></span><span class="go">files copyrighted by (c) iXsystems, Inc.</span> <span class="linenos" data-linenos="10 "></span> <span class="linenos" data-linenos="11 "></span><span class="go">For more information, documentation, help or support, go here:</span> <span class="linenos" data-linenos="12 "></span><span class="go">http://truenas.com</span> <span class="linenos" data-linenos="13 "></span><span class="go">Welcome to TrueNAS</span> <span class="linenos" data-linenos="14 "></span><span class="go">Could not chdir to home directory /nonexistent: No such file or directory</span> <span class="linenos" data-linenos="15 "></span> <span class="hll"><span class="linenos" data-linenos="16 "></span><span class="go">truenas% sudo su</span> </span><span class="linenos" data-linenos="17 "></span> <span class="linenos" data-linenos="18 "></span><span class="gp">root@truenas[/]#</span> </code></pre></div> <h2 id="decrypt-geli-disks">Decrypt geli disks<a class="headerlink" href="#decrypt-geli-disks" title="Permanent link">¶</a></h2> <p>Get a copy of your pool encryption key. You kept it around from back then, when you created your pool right? 🧐</p> <p>So let’s push a copy of it from your machine to the <span class="caps">NAS</span>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>scp<span class="w"> </span>~/pool_big_encryption.key<span class="w"> </span>kev-tmp@truenas.local:/tmp/pool_big_encryption.key </code></pre></div> <p>Get the list of your partitions IDs:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>glabel<span class="w"> </span>status <span class="linenos" data-linenos="2 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="3 "></span><span class="go">gptid/4e377340-917d-11ea-a640-b42e99bf5e8f N/A ada1p2</span> <span class="linenos" data-linenos="4 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f N/A ada2p2</span> <span class="linenos" data-linenos="5 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f N/A ada3p2</span> <span class="linenos" data-linenos="6 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f N/A ada4p2</span> </code></pre></div> <p>Our pool is composed of the four <code>ada[1,2,3,4]p2</code> partitions.</p> <p>Now decrypt each partition with the key:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_encryption.key<span class="w"> </span>/dev/gptid/4e377340-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="2 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_encryption.key<span class="w"> </span>/dev/gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="3 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_encryption.key<span class="w"> </span>/dev/gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="4 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_encryption.key<span class="w"> </span>/dev/gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f </code></pre></div> <p>Check our partitions are properly active and decrypted:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>status <span class="linenos" data-linenos="2 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="3 "></span><span class="go">gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4e377340-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="4 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="5 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="6 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f</span> </code></pre></div> <p>Notice the <code>.eli</code> suffix indicating the partition is encrypted.</p> <h2 id="activating-the-pool">Activating the pool<a class="headerlink" href="#activating-the-pool" title="Permanent link">¶</a></h2> <p>At this point a couple of notifications will pop-up:</p> <p><img alt="" src="/uploads/2020/truenas-critical-error-volume-status.jpeg"/></p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="x">Failed to check for alert VolumeStatus:</span> <span class="linenos" data-linenos=" 2 "></span><span class="gt">Traceback (most recent call last):</span> <span class="linenos" data-linenos=" 3 "></span> File <span class="nb">"/usr/local/lib/python3.8/site-packages/middlewared/plugins/alert.py"</span>, line <span class="m">706</span>, in <span class="n">__run_source</span> <span class="linenos" data-linenos=" 4 "></span><span class="w"> </span><span class="n">alerts</span> <span class="o">=</span> <span class="p">(</span><span class="k">await</span> <span class="n">alert_source</span><span class="o">.</span><span class="n">check</span><span class="p">())</span> <span class="ow">or</span> <span class="p">[]</span> <span class="linenos" data-linenos=" 5 "></span> File <span class="nb">"/usr/local/lib/python3.8/site-packages/middlewared/alert/source/volume_status.py"</span>, line <span class="m">31</span>, in <span class="n">check</span> <span class="linenos" data-linenos=" 6 "></span><span class="w"> </span><span class="k">for</span> <span class="n">vdev</span> <span class="ow">in</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">middleware</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="s2">"pool.flatten_topology"</span><span class="p">,</span> <span class="n">pool</span><span class="p">[</span><span class="s2">"topology"</span><span class="p">]):</span> <span class="linenos" data-linenos=" 7 "></span><span class="gr"> File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in</span> <span class="linenos" data-linenos=" 8 "></span><span class="gr"> call return await self._call(</span> <span class="linenos" data-linenos=" 9 "></span><span class="gr"> File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1206, in _call</span> <span class="linenos" data-linenos="10 "></span><span class="gr"> return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)</span> <span class="linenos" data-linenos="11 "></span><span class="gr"> File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor</span> <span class="linenos" data-linenos="12 "></span><span class="gr"> return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))</span> <span class="linenos" data-linenos="13 "></span><span class="gr"> File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run</span> <span class="linenos" data-linenos="14 "></span><span class="gr"> result = self.fn(*self.args, **self.kwargs)</span> <span class="linenos" data-linenos="15 "></span><span class="gr"> File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 438, in flatten_topology</span> <span class="linenos" data-linenos="16 "></span><span class="gr"> d = deque(sum(topology.values(), []))</span> <span class="linenos" data-linenos="17 "></span><span class="gr">AttributeError</span>: <span class="n">'NoneType' object has no attribute 'values'</span> </code></pre></div> <p>I chose to simply ignore that one.</p> <p>And now the pool is materialized in the <span class="caps">GUI</span>:</p> <p><img alt="" src="/uploads/2020/legacy-encrypted-offline-zfs-pool.jpeg"/></p> <p>It is offline, but can be activated from the shell:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>import<span class="w"> </span>big </code></pre></div> <p>The pool is now online:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="go"> pool: big</span> <span class="hll"><span class="linenos" data-linenos=" 3 "></span><span class="go"> state: ONLINE</span> </span><span class="linenos" data-linenos=" 4 "></span><span class="go">status: One or more devices has experienced an unrecoverable error. An</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> attempt was made to correct the error. Applications are unaffected.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">action: Determine if the device needs to be replaced, and clear the errors</span> <span class="linenos" data-linenos=" 7 "></span><span class="go"> using 'zpool clear' or replace the device with 'zpool replace'.</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> scan: resilvered 30.6M in 00:00:04 with 0 errors on Thu Dec 10 13:24:46 2020</span> <span class="linenos" data-linenos="10 "></span><span class="go">config:</span> <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="13 "></span><span class="go"> big ONLINE 0 0 0</span> <span class="linenos" data-linenos="14 "></span><span class="go"> raidz2-0 ONLINE 0 0 0</span> <span class="linenos" data-linenos="15 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 2</span> <span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 2</span> <span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span> <span class="linenos" data-linenos="20 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p><img alt="" src="/uploads/2020/legacy-encrypted-online-zfs-pool.jpeg"/></p> <p>We got a couple of notifications, one that is warning us about the <a href="https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P/"><code>ZFS-8000-9P</code> error</a>.</p> <p><img alt="" src="/uploads/2020/zfs-online-pool-unrecoverable-error.jpeg"/></p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>CRITICAL <span class="linenos" data-linenos="2 "></span>Pool big state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. </code></pre></div> <p>The data seems <span class="caps">OK</span> and safe thanks to <span class="caps">RAID</span>-Z2. This issue might just be a side-effect of reviving the pool one disk at a time via the <code>geli</code> commands above. So I choose to ignore this minor warning and <code>clear</code> it:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>clear<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 3 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 4 "></span><span class="go"> state: ONLINE</span> <span class="linenos" data-linenos=" 5 "></span><span class="go">status: Some supported features are not enabled on the pool. The pool can</span> <span class="linenos" data-linenos=" 6 "></span><span class="go"> still be used, but some features are unavailable.</span> <span class="linenos" data-linenos=" 7 "></span><span class="go">action: Enable all features using 'zpool upgrade'. Once this is done,</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> the pool may no longer be accessible by software that does not support</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> the features. See zpool-features(5) for details.</span> <span class="linenos" data-linenos="10 "></span><span class="go"> scan: resilvered 30.6M in 00:00:04 with 0 errors on Thu Dec 10 13:24:46 2020</span> <span class="linenos" data-linenos="11 "></span><span class="go">config:</span> <span class="linenos" data-linenos="12 "></span> <span class="linenos" data-linenos="13 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="14 "></span><span class="go"> big ONLINE 0 0 0</span> <span class="linenos" data-linenos="15 "></span><span class="go"> raidz2-0 ONLINE 0 0 0</span> <span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="20 "></span> <span class="linenos" data-linenos="21 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p><span class="caps">ZFS</span> is also proposing to upgrade the pool, but we will not:</p> <p><img alt="" src="/uploads/2020/zfs-feature-flag-pool-upgrade.jpeg"/></p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>WARNING <span class="linenos" data-linenos="2 "></span>New ZFS version or feature flags are available for pool big. Upgrading pools is a one-time process that can prevent rolling the system back to an earlier TrueNAS version. It is recommended to read the TrueNAS release notes and confirm you need the new ZFS feature flags before upgrading a pool. </code></pre></div> <p>Doing so will prevent us to revert back to FreeNAS if anything bad happens. We’ll do that later.</p> <h2 id="removing-encryption">Removing encryption<a class="headerlink" href="#removing-encryption" title="Permanent link">¶</a></h2> <p>The strategy here consist in replacing each encrypted partition by itself, un-encrypted. The pool will be in a <code>DEGRADED</code> state during the operation.</p> <p>First, we choose one partition (<code>ada1p2</code> in this case), and offline the disk from the web interface:</p> <p><img alt="" src="/uploads/2020/offline-zfs-disk.jpeg"/></p> <p>Double-check the status of the pool with the <span class="caps">CLI</span>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 3 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">status: One or more devices has been taken offline by the administrator.</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> Sufficient replicas exist for the pool to continue functioning in a</span> <span class="linenos" data-linenos=" 6 "></span><span class="go"> degraded state.</span> <span class="linenos" data-linenos=" 7 "></span><span class="go">action: Online the device using 'zpool online' or replace the device with</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> 'zpool replace'.</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> scan: resilvered 30.6M in 00:00:04 with 0 errors on Thu Dec 10 13:24:46 2020</span> <span class="linenos" data-linenos="10 "></span><span class="go">config:</span> <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="13 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="14 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="15 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli OFFLINE 0 0 0</span> </span><span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span> <span class="linenos" data-linenos="20 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p>And verify our disk is no longer referenced by <code>geli</code>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>status <span class="linenos" data-linenos="2 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="3 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="4 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="5 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f</span> </code></pre></div> <p>We can now proceed with the resilvering of that disk into the pool:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>replace<span class="w"> </span>big<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f </code></pre></div> <p>This is going to be a slow operation:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 3 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">status: One or more devices is currently being resilvered. The pool will</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> continue to function, possibly in a degraded state.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">action: Wait for the resilver to complete.</span> <span class="hll"><span class="linenos" data-linenos=" 7 "></span><span class="go"> scan: resilver in progress since Thu Dec 10 16:11:52 2020</span> </span><span class="hll"><span class="linenos" data-linenos=" 8 "></span><span class="go"> 2.85T scanned at 889M/s, 1.29T issued at 402M/s, 12.3T total</span> </span><span class="hll"><span class="linenos" data-linenos=" 9 "></span><span class="go"> 320G resilvered, 10.49% done, 07:57:38 to go</span> </span><span class="linenos" data-linenos="10 "></span><span class="go">config:</span> <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="13 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="14 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="15 "></span><span class="go"> replacing-0 DEGRADED 0 0 0</span> </span><span class="hll"><span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli OFFLINE 0 0 0</span> </span><span class="hll"><span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f ONLINE 0 0 0 (resilvering)</span> </span><span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="20 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="21 "></span> <span class="linenos" data-linenos="22 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p>Wait until it completes. It took around 7 hours for one disk in my situation.</p> <p>Once the resilvering finishes, redo with each disk the whole process from that section.</p> <h2 id="failed-replace">Failed replace<a class="headerlink" href="#failed-replace" title="Permanent link">¶</a></h2> <p>While performing the process above, you’re still at the mercy of any corruption issue. It happened to me, and I woke up with an failed replace operation.Â đŸ˜”</p> <p>The pool looked like this:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 3 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">status: One or more devices is currently being resilvered. The pool will</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> continue to function, possibly in a degraded state.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">action: Wait for the resilver to complete.</span> <span class="linenos" data-linenos=" 7 "></span><span class="go"> scan: resilver in progress since Thu Dec 10 16:11:52 2020</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> 6.63T scanned at 464M/s, 5.63T issued at 394M/s, 12.3T total</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> 954G resilvered, 45.79% done, 04:55:41 to go</span> <span class="linenos" data-linenos="10 "></span><span class="go">config:</span> <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="13 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="14 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="15 "></span><span class="go"> replacing-0 UNAVAIL 0 160 0 insufficient replicas</span> </span><span class="hll"><span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli OFFLINE 0 0 0</span> </span><span class="hll"><span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f REMOVED 0 0 0 (resilvering)</span> </span><span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli REMOVED 0 0 0</span> </span><span class="linenos" data-linenos="20 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="21 "></span> <span class="linenos" data-linenos="22 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p>I don’t know what happened, but I quickly discovered 2 disks were no longer actively decrypted:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>status <span class="linenos" data-linenos="2 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="3 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="4 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f</span> </code></pre></div> <p>And one disk, <code>ada1p2</code>, i.e. the one being replaced, disappeared:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>glabel<span class="w"> </span>status <span class="linenos" data-linenos="2 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="3 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f N/A ada3p2</span> <span class="linenos" data-linenos="4 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f N/A ada4p2</span> <span class="linenos" data-linenos="5 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f N/A ada2p2</span> <span class="linenos" data-linenos="6 "></span><span class="go">gptid/4e8b58ff-917d-11ea-a640-b42e99bf5e8f N/A ada2p1</span> </code></pre></div> <p>So I quickly re-attached the one still encrypted (not the one being replaced):</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_recovery.key<span class="w"> </span>/dev/gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="2 "></span> <span class="linenos" data-linenos="3 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>status <span class="linenos" data-linenos="4 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="5 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="6 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f</span> <span class="hll"><span class="linenos" data-linenos="7 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f</span> </span></code></pre></div> <p>Fortunately, <span class="caps">ZFS</span> re-integrated it to the pool, changing its state from <code>REMOVED</code> to <code>ONLINE</code>, at the price of an automatic resilvering:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 3 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">status: One or more devices is currently being resilvered. The pool will</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> continue to function, possibly in a degraded state.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">action: Wait for the resilver to complete.</span> <span class="hll"><span class="linenos" data-linenos=" 7 "></span><span class="go"> scan: resilver in progress since Fri Dec 11 00:18:45 2020</span> </span><span class="hll"><span class="linenos" data-linenos=" 8 "></span><span class="go"> 1.76T scanned at 139G/s, 324K issued at 24.9K/s, 12.3T total</span> </span><span class="hll"><span class="linenos" data-linenos=" 9 "></span><span class="go"> 0B resilvered, 0.00% done, no estimated completion time</span> </span><span class="linenos" data-linenos="10 "></span><span class="go">config:</span> <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="13 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="14 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="linenos" data-linenos="15 "></span><span class="go"> replacing-0 UNAVAIL 0 160 0 insufficient replicas</span> <span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli OFFLINE 0 0 0</span> <span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f REMOVED 0 0 0</span> <span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> </span><span class="linenos" data-linenos="20 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="21 "></span> <span class="linenos" data-linenos="22 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p>Another 7 hours later, with the resilvering completed, it was time to fix the failed <code>replacing-0</code> operation.</p> <p>I tried to force my way but it did not worked:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>replace<span class="w"> </span>big<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="2 "></span><span class="go">cannot open 'gptid/4e377340-917d-11ea-a640-b42e99bf5e8f': no such device in /dev</span> <span class="linenos" data-linenos="3 "></span><span class="go">must be a full path or shorthand device name</span> </code></pre></div> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>online<span class="w"> </span>big<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="2 "></span><span class="go">cannot online gptid/4e377340-917d-11ea-a640-b42e99bf5e8f: no such device in pool</span> </code></pre></div> <p>That’s when I realized <code>4e377340-917d-11ea-a640-b42e99bf5e8f</code> (a.k.a. <code>ada1p2</code>), was completely kicked-out by the system. I stumbled upon some worrying logs that might hint to a kind of hardware issue:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span>Dec 10 19:18:35 truenas ahcich3: Timeout on slot 31 port 0 <span class="linenos" data-linenos=" 2 "></span>Dec 10 19:18:35 truenas ahcich3: is 00000000 cs 80000000 ss 80000000 rs 80000000 tfd c0 serr 00000000 cmd 0000df17 <span class="linenos" data-linenos=" 3 "></span>Dec 10 19:18:35 truenas (ada2:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 30 6b 94 40 08 02 00 00 00 00 <span class="linenos" data-linenos=" 4 "></span>Dec 10 19:18:35 truenas (ada2:ahcich3:0:0:0): CAM status: Command timeout <span class="linenos" data-linenos=" 5 "></span>Dec 10 19:18:35 truenas (ada2:ahcich3:0:0:0): Retrying command, 3 more tries remain <span class="linenos" data-linenos=" 6 "></span>Dec 10 19:18:35 truenas ahcich5: Timeout on slot 1 port 0 <span class="linenos" data-linenos=" 7 "></span>Dec 10 19:18:35 truenas ahcich5: is 00000000 cs 00000002 ss 00000002 rs 00000002 tfd c0 serr 00000000 cmd 0000c117 <span class="linenos" data-linenos=" 8 "></span>Dec 10 19:18:35 truenas (ada3:ahcich5:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 30 6b 94 40 08 02 00 00 00 00 <span class="linenos" data-linenos=" 9 "></span>Dec 10 19:18:35 truenas (ada3:ahcich5:0:0:0): CAM status: Command timeout <span class="linenos" data-linenos="10 "></span>Dec 10 19:18:35 truenas (ada3:ahcich5:0:0:0): Retrying command, 3 more tries remain <span class="linenos" data-linenos="11 "></span>Dec 10 19:18:52 truenas ada2 at ahcich3 bus 0 scbus3 target 0 lun 0 <span class="hll"><span class="linenos" data-linenos="12 "></span>Dec 10 19:18:52 truenas ada2: <toshiba fs1m="" hdwn160=""> s/n 10LSK1FPFAXG detached </toshiba></span><span class="linenos" data-linenos="13 "></span>Dec 10 19:18:52 truenas GEOM_ELI: g_eli_read_done() failed (error=6) gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli[READ(offset=4469598543872, length=65536)] <span class="linenos" data-linenos="14 "></span>Dec 10 19:18:52 truenas GEOM_MIRROR: Device swap0: provider ada2p1 disconnected. <span class="linenos" data-linenos="15 "></span>Dec 10 19:19:23 truenas ahcich1: Timeout on slot 31 port 0 <span class="linenos" data-linenos="16 "></span>Dec 10 19:19:23 truenas ahcich1: is 00000000 cs 80000000 ss 80000000 rs 80000000 tfd c0 serr 00000000 cmd 0000df17 <span class="linenos" data-linenos="17 "></span>Dec 10 19:19:23 truenas (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 80 38 6b 94 40 08 02 00 00 00 00 <span class="linenos" data-linenos="18 "></span>Dec 10 19:19:23 truenas (ada1:ahcich1:0:0:0): CAM status: Command timeout <span class="linenos" data-linenos="19 "></span>Dec 10 19:19:23 truenas (ada1:ahcich1:0:0:0): Retrying command, 3 more tries remain <span class="linenos" data-linenos="20 "></span>Dec 10 19:19:57 truenas ahcich3: AHCI reset: device not ready after 31000ms (tfd = 00000080) <span class="linenos" data-linenos="21 "></span>Dec 10 19:20:01 truenas ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080) <span class="linenos" data-linenos="22 "></span>Dec 10 19:20:31 truenas ahcich1: Timeout on slot 0 port 0 <span class="linenos" data-linenos="23 "></span>Dec 10 19:20:31 truenas ahcich1: is 00000000 cs 00000000 ss 00000000 rs 00000001 tfd 150 serr 00000000 cmd 0000c017 <span class="linenos" data-linenos="24 "></span>Dec 10 19:20:31 truenas (aprobe1:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="25 "></span>Dec 10 19:20:31 truenas (aprobe1:ahcich1:0:0:0): CAM status: Command timeout <span class="linenos" data-linenos="26 "></span>Dec 10 19:20:31 truenas (aprobe1:ahcich1:0:0:0): Retrying command, 0 more tries remain <span class="linenos" data-linenos="27 "></span>Dec 10 19:21:05 truenas ahcich1: Timeout on slot 1 port 0 <span class="linenos" data-linenos="28 "></span>Dec 10 19:21:05 truenas ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 1d0 serr 00000000 cmd 0000c117 <span class="linenos" data-linenos="29 "></span>Dec 10 19:21:05 truenas (aprobe1:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="30 "></span>Dec 10 19:21:05 truenas (aprobe1:ahcich1:0:0:0): CAM status: Command timeout <span class="linenos" data-linenos="31 "></span>Dec 10 19:21:05 truenas (aprobe1:ahcich1:0:0:0): Error 5, Retries exhausted <span class="linenos" data-linenos="32 "></span>Dec 10 19:21:39 truenas ahcich1: Timeout on slot 2 port 0 <span class="linenos" data-linenos="33 "></span>Dec 10 19:21:39 truenas ahcich1: is 00000000 cs 00000004 ss 00000000 rs 00000004 tfd 1d0 serr 00000000 cmd 0000c217 <span class="linenos" data-linenos="34 "></span>Dec 10 19:21:39 truenas (aprobe1:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="35 "></span>Dec 10 19:21:39 truenas (aprobe1:ahcich1:0:0:0): CAM status: Command timeout <span class="linenos" data-linenos="36 "></span>Dec 10 19:21:39 truenas (aprobe1:ahcich1:0:0:0): Error 5, Retry was blocked <span class="linenos" data-linenos="37 "></span>Dec 10 19:21:39 truenas ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 <span class="hll"><span class="linenos" data-linenos="38 "></span>Dec 10 19:21:39 truenas ada1: <toshiba fs1m="" hdwn160=""> s/n Z9B3K1OAFAXG detached </toshiba></span><span class="linenos" data-linenos="39 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="40 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): CAM status: ATA Status Error <span class="linenos" data-linenos="41 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): ATA status: 31 (DF SERV ERR), error: 04 (ABRT ) <span class="linenos" data-linenos="42 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): RES: 31 04 e0 6a 94 48 08 02 00 00 00 <span class="linenos" data-linenos="43 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): Retrying command, 0 more tries remain <span class="linenos" data-linenos="44 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="45 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): CAM status: ATA Status Error <span class="linenos" data-linenos="46 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): ATA status: 31 (DF SERV ERR), error: 04 (ABRT ) <span class="linenos" data-linenos="47 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): RES: 31 04 e0 6a 94 48 08 02 00 00 00 <span class="linenos" data-linenos="48 "></span>Dec 10 19:22:00 truenas (aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted <span class="linenos" data-linenos="49 "></span>Dec 10 19:22:00 truenas xptioctl: pass driver is not in the kernel <span class="linenos" data-linenos="50 "></span>Dec 10 19:22:00 truenas xptioctl: put "device pass" in your kernel config file <span class="linenos" data-linenos="51 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="52 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): CAM status: ATA Status Error <span class="linenos" data-linenos="53 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): ATA status: 31 (DF SERV ERR), error: 04 (ABRT ) <span class="linenos" data-linenos="54 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): RES: 31 04 e0 6a 94 48 08 02 00 00 00 <span class="linenos" data-linenos="55 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): Error 5, Periph was invalidated <span class="linenos" data-linenos="56 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40 00 00 00 00 00 00 <span class="linenos" data-linenos="57 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): CAM status: ATA Status Error <span class="linenos" data-linenos="58 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): ATA status: 31 (DF SERV ERR), error: 04 (ABRT ) <span class="linenos" data-linenos="59 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): RES: 31 04 e0 6a 94 48 08 02 00 00 00 <span class="linenos" data-linenos="60 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): Error 5, Periph was invalidated <span class="linenos" data-linenos="61 "></span>Dec 10 19:22:00 truenas ahcich1: Error while READ LOG EXT <span class="linenos" data-linenos="62 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 80 38 6b 94 40 08 02 00 00 00 00 <span class="linenos" data-linenos="63 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): CAM status: ATA Status Error <span class="linenos" data-linenos="64 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): ATA status: 00 () <span class="linenos" data-linenos="65 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00 <span class="linenos" data-linenos="66 "></span>Dec 10 19:22:00 truenas (ada1:ahcich1:0:0:0): Error 5, Periph was invalidated <span class="linenos" data-linenos="67 "></span>Dec 10 19:22:02 truenas GEOM_ELI: Device gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli destroyed. <span class="linenos" data-linenos="68 "></span>Dec 10 19:22:02 truenas GEOM_ELI: Detached gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli on last close. <span class="hll"><span class="linenos" data-linenos="69 "></span>Dec 10 19:22:02 truenas (ada2:ahcich3:0:0:0): Periph destroyed </span><span class="linenos" data-linenos="70 "></span>Dec 10 19:22:02 truenas ada2 at ahcich3 bus 0 scbus3 target 0 lun 0 <span class="linenos" data-linenos="71 "></span>Dec 10 19:22:02 truenas ada2: <toshiba fs1m="" hdwn160=""> ATA8-ACS SATA 3.x device <span class="linenos" data-linenos="72 "></span>Dec 10 19:22:02 truenas ada2: Serial Number 10LSK1FPFAXG <span class="linenos" data-linenos="73 "></span>Dec 10 19:22:02 truenas ada2: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes) <span class="linenos" data-linenos="74 "></span>Dec 10 19:22:02 truenas ada2: Command Queueing enabled <span class="linenos" data-linenos="75 "></span>Dec 10 19:22:02 truenas ada2: 5723166MB (11721045168 512 byte sectors) <span class="hll"><span class="linenos" data-linenos="76 "></span>Dec 10 19:22:02 truenas (ada1:ahcich1:0:0:0): Periph destroyed </span></toshiba></code></pre></div> <p>So with no resilvering or any heavy operation pending on the pool, I decided to offline the partitions and restart the machine:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>offline<span class="w"> </span>big<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f </code></pre></div> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 3 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">status: Some supported features are not enabled on the pool. The pool can</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> still be used, but some features are unavailable.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">action: Enable all features using 'zpool upgrade'. Once this is done,</span> <span class="linenos" data-linenos=" 7 "></span><span class="go"> the pool may no longer be accessible by software that does not support</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> the features. See zpool-features(5) for details.</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> scan: resilvered 15.5M in 05:42:26 with 0 errors on Fri Dec 11 06:01:11 2020</span> <span class="linenos" data-linenos="10 "></span><span class="go">config:</span> <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="13 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="14 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="linenos" data-linenos="15 "></span><span class="go"> replacing-0 OFFLINE 0 160 0 all children offline</span> <span class="linenos" data-linenos="16 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli OFFLINE 0 0 0</span> <span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f OFFLINE 0 0 0</span> <span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="20 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="21 "></span> <span class="linenos" data-linenos="22 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>poweroff <span class="linenos" data-linenos=" 2 "></span><span class="go">Shutdown NOW!</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">poweroff: [pid 27282]</span> <span class="linenos" data-linenos=" 4 "></span> <span class="linenos" data-linenos=" 5 "></span><span class="gp">root@truenas[/]#</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">*** FINAL System shutdown message from kev-tmp@truenas.local ***</span> <span class="linenos" data-linenos=" 7 "></span> <span class="linenos" data-linenos=" 8 "></span><span class="go">System going down IMMEDIATELY</span> <span class="linenos" data-linenos=" 9 "></span> <span class="linenos" data-linenos="10 "></span><span class="go">System shutdown time has arrived</span> <span class="linenos" data-linenos="11 "></span><span class="go">Shared connection to truenas.local closed.</span> </code></pre></div> <p><img alt="Hello IT, have you tried turning it off and on again?" src="/uploads/2020/have-you-tried-turning-it-off-and-on-again.jpg"/></p> <p>I got lucky, <code>ada1p2</code> showed up after the reboot:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>glabel<span class="w"> </span>status <span class="linenos" data-linenos="2 "></span><span class="go">Name Status Components</span> <span class="hll"><span class="linenos" data-linenos="3 "></span><span class="go">gptid/4e377340-917d-11ea-a640-b42e99bf5e8f N/A ada1p2</span> </span><span class="linenos" data-linenos="4 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f N/A ada2p2</span> <span class="linenos" data-linenos="5 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f N/A ada3p2</span> <span class="linenos" data-linenos="6 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f N/A ada4p2</span> </code></pre></div> <p>I was able to decrypted all partitions again (but the one being replaced):</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_recovery.key<span class="w"> </span>/dev/gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="2 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_recovery.key<span class="w"> </span>/dev/gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="3 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>attach<span class="w"> </span>-p<span class="w"> </span>-k<span class="w"> </span>/tmp/pool_big_recovery.key<span class="w"> </span>/dev/gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="4 "></span><span class="gp">root@truenas[/]# </span>geli<span class="w"> </span>status <span class="linenos" data-linenos="5 "></span><span class="go">Name Status Components</span> <span class="linenos" data-linenos="6 "></span><span class="go">gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="7 "></span><span class="go">gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f</span> <span class="linenos" data-linenos="8 "></span><span class="go">gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ACTIVE gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f</span> </code></pre></div> <p>And get back my pool in the state I left it:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>import<span class="w"> </span>big <span class="linenos" data-linenos=" 2 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 3 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 4 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 5 "></span><span class="go">status: Some supported features are not enabled on the pool. The pool can</span> <span class="linenos" data-linenos=" 6 "></span><span class="go"> still be used, but some features are unavailable.</span> <span class="linenos" data-linenos=" 7 "></span><span class="go">action: Enable all features using 'zpool upgrade'. Once this is done,</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> the pool may no longer be accessible by software that does not support</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> the features. See zpool-features(5) for details.</span> <span class="linenos" data-linenos="10 "></span><span class="go"> scan: resilvered 15.5M in 05:42:26 with 0 errors on Fri Dec 11 06:01:11 2020</span> <span class="linenos" data-linenos="11 "></span><span class="go">config:</span> <span class="linenos" data-linenos="12 "></span> <span class="linenos" data-linenos="13 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="14 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="15 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="linenos" data-linenos="16 "></span><span class="go"> replacing-0 OFFLINE 0 0 0 all children offline</span> <span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli OFFLINE 0 0 0</span> <span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f OFFLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="20 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="21 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="22 "></span> <span class="linenos" data-linenos="23 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p>This time, I finally fixed the failed <code>replacing-0</code> operation by detaching the old encrypted partition, and resilvering the new, unencrypted one:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>detach<span class="w"> </span>big<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f.eli <span class="linenos" data-linenos=" 2 "></span> <span class="linenos" data-linenos=" 3 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos=" 4 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> state: DEGRADED</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">status: One or more devices has been taken offline by the administrator.</span> <span class="linenos" data-linenos=" 7 "></span><span class="go"> Sufficient replicas exist for the pool to continue functioning in a</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> degraded state.</span> <span class="linenos" data-linenos=" 9 "></span><span class="go">action: Online the device using 'zpool online' or replace the device with</span> <span class="linenos" data-linenos="10 "></span><span class="go"> 'zpool replace'.</span> <span class="linenos" data-linenos="11 "></span><span class="go"> scan: resilvered 15.5M in 05:42:26 with 0 errors on Fri Dec 11 06:01:11 2020</span> <span class="linenos" data-linenos="12 "></span><span class="go">config:</span> <span class="linenos" data-linenos="13 "></span> <span class="linenos" data-linenos="14 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="15 "></span><span class="go"> big DEGRADED 0 0 0</span> <span class="linenos" data-linenos="16 "></span><span class="go"> raidz2-0 DEGRADED 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="17 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f OFFLINE 0 0 0</span> </span><span class="linenos" data-linenos="18 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="19 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="20 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="21 "></span> <span class="linenos" data-linenos="22 "></span><span class="go">errors: No known data errors</span> <span class="linenos" data-linenos="23 "></span> <span class="linenos" data-linenos="24 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>online<span class="w"> </span>big<span class="w"> </span>gptid/4e377340-917d-11ea-a640-b42e99bf5e8f <span class="linenos" data-linenos="25 "></span> <span class="linenos" data-linenos="26 "></span><span class="gp">root@truenas[/]# </span>zpool<span class="w"> </span>status<span class="w"> </span>big <span class="linenos" data-linenos="27 "></span><span class="go"> pool: big</span> <span class="linenos" data-linenos="28 "></span><span class="go"> state: ONLINE</span> <span class="linenos" data-linenos="29 "></span><span class="go">status: One or more devices is currently being resilvered. The pool will</span> <span class="linenos" data-linenos="30 "></span><span class="go"> continue to function, possibly in a degraded state.</span> <span class="linenos" data-linenos="31 "></span><span class="go">action: Wait for the resilver to complete.</span> <span class="linenos" data-linenos="32 "></span><span class="go"> scan: resilver in progress since Fri Dec 11 09:53:51 2020</span> <span class="linenos" data-linenos="33 "></span><span class="go"> 4.46T scanned at 623M/s, 2.78T issued at 388M/s, 12.3T total</span> <span class="linenos" data-linenos="34 "></span><span class="go"> 690G resilvered, 22.63% done, 07:07:37 to go</span> <span class="linenos" data-linenos="35 "></span><span class="go">config:</span> <span class="linenos" data-linenos="36 "></span> <span class="linenos" data-linenos="37 "></span><span class="go"> NAME STATE READ WRITE CKSUM</span> <span class="linenos" data-linenos="38 "></span><span class="go"> big ONLINE 0 0 0</span> <span class="linenos" data-linenos="39 "></span><span class="go"> raidz2-0 ONLINE 0 0 0</span> <span class="hll"><span class="linenos" data-linenos="40 "></span><span class="go"> gptid/4e377340-917d-11ea-a640-b42e99bf5e8f ONLINE 0 0 0 (resilvering)</span> </span><span class="linenos" data-linenos="41 "></span><span class="go"> gptid/4ea9ae2e-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="42 "></span><span class="go"> gptid/4eb3e8fc-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="43 "></span><span class="go"> gptid/4ece25f4-917d-11ea-a640-b42e99bf5e8f.eli ONLINE 0 0 0</span> <span class="linenos" data-linenos="44 "></span> <span class="linenos" data-linenos="45 "></span><span class="go">errors: No known data errors</span> </code></pre></div> <p>And off we went, spending another 7 hours of resilvering
</p> <p>All in all, <span class="caps">RAID</span>-Z2 saved my ass. Lesson learned: <strong>a disk failure during heavy-duty operations is no longer a statistically rare event</strong>.</p> <p>So remember the wise man who once said to <strong><span class="caps">BACKUP</span> <span class="caps">YOUR</span> F#@$% <span class="caps">POOL</span></strong>!</p>How-to revert HP printer firmware ban on 3rd-party toner cartridges2020-11-09T12:00:00+01:002020-11-09T12:00:00+01:00Kevin Deldycketag:kevin.deldycke.com,2020-11-09:/2020/11/revert-hp-printer-ban-on-third-party-ink-cartridges/<p>Hewlett <span class="amp">&amp;</span> Packard, the founders, had great lessons to teach us (managers in high-tech) about culture. To the extend of <a href="https://github.com/kdeldycke/awesome-engineering-team-management/commit/de3e64647c911f78a37b3e54c7e46197acb061e1">quoting them</a> in my <a href="https://github.com/kdeldycke/awesome-engineering-team-management#readme">awesome list on engineering team management</a>.&nbsp;đŸ‘šâ€đŸ’Œ</p> <p><span class="caps">HP</span> Inc., the company, sucks. At least their <a href="https://news.ycombinator.com/item?id=25045024">printer division&rsquo;s business model</a>. They recently pushed a <strong>firmware update to ban 
</strong></p><p>Hewlett <span class="amp">&amp;</span> Packard, the founders, had great lessons to teach us (managers in high-tech) about culture. To the extend of <a href="https://github.com/kdeldycke/awesome-engineering-team-management/commit/de3e64647c911f78a37b3e54c7e46197acb061e1">quoting them</a> in my <a href="https://github.com/kdeldycke/awesome-engineering-team-management#readme">awesome list on engineering team management</a>.Â đŸ‘šâ€đŸ’Œ</p> <p><span class="caps">HP</span> Inc., the company, sucks. At least their <a href="https://news.ycombinator.com/item?id=25045024">printer division’s business model</a>. They recently pushed a <strong>firmware update to ban third-party compatible toner cartridges</strong>. 💔</p> <p>The timeline is straightforward:</p> <ul> <li> <p>2020, March: general lockdown. 🩠 I need a home office. <span class="caps">SO</span> is a scientist and spend her time printing papers for review. Got her an <a href="https://amzn.com/B073R2WVKB/?tag=kevideld-20"><span class="caps">HP</span> Color LaserJet M254dw</a> to keep her productive workflow (<a href="https://en.wikipedia.org/wiki/Publish_or_perish">publish or perish!</a>).</p> </li> <li> <p>2020, October: <span class="caps">HP</span> release a new firmware (versioned <code>20201021</code>).</p> </li> </ul> <p><img alt="" src="/uploads/2020/hp-laserjet-printer-20201021-firmware.jpg"/></p> <ul> <li>2020, November: my printer auto-upgrade. I’m welcomed with this <em>Supply Problem <a href="https://en.wikipedia.org/wiki/Screen_of_death">Screen of Death</a></em>:</li> </ul> <p><img alt="" src="/uploads/2020/hp-laserjet-printer-supply-problem-screen-of-death.jpg"/></p> <p>I can’t print anymore.Â đŸ€Ż</p> <p>8 months. My printer worked for only height months.Â đŸ˜€</p> <p><span class="caps">OK</span>. It’s my fault. I should have spent more money buying certifiedℱ gear. 😑</p> <p><img alt="" src="/uploads/2020/copier-printer-meme-03.jpg"/></p> <p>The solution is to travel back in time when things were working just great, and downgrade to the previous firmware.</p> <h2 id="disable-auto-upgrade">Disable auto-upgrade<a class="headerlink" href="#disable-auto-upgrade" title="Permanent link">¶</a></h2> <p>We will stop this madness for good, and prevent the printer from downloading a firmware behind our back.</p> <p>In the control panel, go to <code>Setup</code> &gt; <code>Service</code> &gt; <code>LaserJet Update</code> &gt; <code>Manage Updates</code>:</p> <p><img alt="" src="/uploads/2020/hp-laserjet-printer-manage-updates-menu.jpg"/></p> <p>Then set these options:</p> <ul> <li>Allow Downgrade: <code>Yes</code></li> <li>Check Automatically: <code>Off</code></li> <li>Prompt Before Install: <code>Always Prompt</code></li> <li>Allow Updates: <code>Yes</code></li> </ul> <p>I’m quite surprised downgrades are allowed. đŸ€” It seems out of character. Therefor, with my <em>Evil Product Manager</em> hat on, I advise <span class="caps">HP</span> to monetize this feature under a monthly Enterprise Subscription of sort. 😈</p> <h2 id="download-old-firmware">Download old firmware<a class="headerlink" href="#download-old-firmware" title="Permanent link">¶</a></h2> <p>I got lucky and found the previous <code>20200612</code> firmware referenced in <a href="https://ftp.hp.com/pub/networking/software/pfirmware/pfirmware.glf"><code>https://ftp.hp.com/pub/networking/software/pfirmware/pfirmware.glf</code></a>.</p> <p>There you’ll get a direct link to the <code>.rfu</code> file (Remote Firmware Update):<a href="http://ftp.hp.com/pub/networking/software/pfirmware/HP_Color_LaserJet_Pro_M254_dw_Printer_series_20200612.rfu"><code>http://ftp.hp.com/pub/networking/software/pfirmware/HP_Color_LaserJet_Pro_M254_dw_Printer_series_20200612.rfu</code></a>.</p> <p>And just in case it disappear from its original location, here is a <a href="/uploads/2020/HP_Color_LaserJet_Pro_M254_dw_Printer_series_20200612.rfu">copy of <code>HP_Color_LaserJet_Pro_M254_dw_Printer_series_20200612.rfu</code></a>.</p> <p>The checksum of that file is:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sha256sum<span class="w"> </span>./HP_Color_LaserJet_Pro_M254_dw_Printer_series_20200612.rfu <span class="linenos" data-linenos="2 "></span><span class="go">91c7f51ceba2386f3b94dcb9da20c669ab10b1ee3a9b1e1f742c40091920188e</span> </code></pre></div> <h2 id="downgrade-firmware">Downgrade firmware<a class="headerlink" href="#downgrade-firmware" title="Permanent link">¶</a></h2> <p>Once you get the <code>.rfu</code> file, list all your printers from a macOS terminal:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>lpstat<span class="w"> </span>-p<span class="w"> </span>-d <span class="linenos" data-linenos="2 "></span><span class="go">printer HP_Color_LaserJet_M254dw_0 is idle. enabled since Fri Nov 6 17:47:06 2020</span> <span class="linenos" data-linenos="3 "></span><span class="go">system default destination: HP_Color_LaserJet_M254dw_0</span> </code></pre></div> <p>And run the firmware downgrade <span class="caps">CLI</span>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>lpr<span class="w"> </span>-P<span class="w"> </span>HP_Color_LaserJet_M254dw_0<span class="w"> </span>/Users/kde/Downloads/HP_Color_LaserJet_Pro_M254_dw_Printer_series_20200612.rfu </code></pre></div> <p>Nothing gets printed to the console.</p> <p>I don’t know what happens here but it seems the <code>.rfu</code> file is pushed to the printer’s queue, and then gets consumed as any other printable document. See, <a href="https://www.jsof-tech.com/unpacking-hp-firmware-updates-part-1/">the <span class="caps">RFU</span> file format is a matryoshka doll</a> embedding printing commands, encoded data and raw <span class="caps">NAND</span> code.</p> <p>After a minute or two, the printers reboots and upgrades itself:</p> <p><img alt="" src="/uploads/2020/hp-laserjet-printer-firmware-updating.jpg"/></p> <p>And we’re back in business!Â đŸ„ł</p> <p>A detour via <code>Setup</code> &gt; <code>Service</code> &gt; <code>Firmware Datecode</code> menu confirm we’re running the the previous firmware:</p> <p><img alt="" src="/uploads/2020/hp-laserjet-printer-20200612-firmware.jpg"/></p> <p>Finally, to lock any possibility of a firmware change, go back to <code>Setup</code> &gt; <code>Service</code> &gt; <code>LaserJet Update</code> &gt; <code>Manage Updates</code> to set the option to:</p> <ul> <li>Allow Updates: <code>No</code></li> </ul> <h2 id="printer-security">Printer security<a class="headerlink" href="#printer-security" title="Permanent link">¶</a></h2> <p>In my research for this article, I found out about <a href="https://github.com/RUB-NDS/PRET"><span class="caps">PRET</span>, a printer exploitation toolkit</a>. It’s a brilliant tool, in a malignant way. It allows for pen-testing and hacking, using the same vectors as the firmware update.Â đŸ€«</p> <p>I’ll probably play with it in the future. For fun, but also to try enhance the security of the printer. In the mean time, I guess a password is the bare minimum. And if my printer get kidnapped by a cyber gang, I now have a way to restore my printer’s firmware! 😬</p>TrueNAS Configuration and Maintenance2020-10-27T00:00:00+01:002020-10-27T00:00:00+01:00Kevin Deldycketag:kevin.deldycke.com,2020-10-27:/2020/10/truenas-configuration/<p>A collection of personal notes on the setup, configuration and maintenance of a home-office TrueNAS&nbsp;appliance.</p> <h2 id="hardware">đŸ–„ Hardware<a class="headerlink" href="#hardware" title="Permanent link">&para;</a></h2> <p>I dedicated a whole post on <a href="https://kevin.deldycke.com/2020/05/nas-hardware/">building the machine</a>, in which you&rsquo;ll get the detailed bill of material, cost breakdown and parts selection&nbsp;process.</p> <h3 id="reset-bmc-password">Reset <span class="caps">BMC</span> password<a class="headerlink" href="#reset-bmc-password" title="Permanent link">&para;</a></h3> <p>If you have one of 
</p><p>A collection of personal notes on the setup, configuration and maintenance of a home-office TrueNAS appliance.</p> <h2 id="hardware">đŸ–„ Hardware<a class="headerlink" href="#hardware" title="Permanent link">¶</a></h2> <p>I dedicated a whole post on <a href="https://kevin.deldycke.com/2020/05/nas-hardware/">building the machine</a>, in which you’ll get the detailed bill of material, cost breakdown and parts selection process.</p> <h3 id="reset-bmc-password">Reset <span class="caps">BMC</span> password<a class="headerlink" href="#reset-bmc-password" title="Permanent link">¶</a></h3> <p>If you have one of iXsystems’ TrueNAS server whose motherboard was produced by SuperMicro, there’s a way to reset the <span class="caps">BMC</span> password with that <span class="caps">CLI</span> to use as <code>root</code>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">[root@truenas] ~# </span>ipmitool<span class="w"> </span>raw<span class="w"> </span>0x3c<span class="w"> </span>0x40 <span class="linenos" data-linenos="2 "></span> <span class="linenos" data-linenos="3 "></span><span class="gp">[root@truenas] ~# </span> </code></pre></div> <h2 id="reports">📊 Reports<a class="headerlink" href="#reports" title="Permanent link">¶</a></h2> <h3 id="disk-temperature">Disk Temperature<a class="headerlink" href="#disk-temperature" title="Permanent link">¶</a></h3> <p>⚠ Do not activate any aggressive power-management parameters on disks. This is the main reasons temperatures are not reported in graphs.</p> <h2 id="package-management">📩 Package Management<a class="headerlink" href="#package-management" title="Permanent link">¶</a></h2> <p>To prevent the administrator to mess up with TrueNAS install, FreeBSD’s package management has been locked down. Installing a package does not work:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>pkg<span class="w"> </span>install<span class="w"> </span>dmg2img <span class="linenos" data-linenos="2 "></span><span class="go">Updating local repository catalogue...</span> <span class="linenos" data-linenos="3 "></span><span class="go">pkg: file:///usr/ports/packages/meta.txz: No such file or directory</span> <span class="linenos" data-linenos="4 "></span><span class="go">repository local has no meta file, using default settings</span> <span class="linenos" data-linenos="5 "></span><span class="go">pkg: file:///usr/ports/packages/packagesite.txz: No such file or directory</span> <span class="hll"><span class="linenos" data-linenos="6 "></span><span class="go">Unable to update repository local</span> </span><span class="linenos" data-linenos="7 "></span><span class="go">Error updating repositories!</span> </code></pre></div> <p>If you know what you’re doing and want to unlock package management, here is the fix:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>sed<span class="w"> </span>-i<span class="w"> </span>.orig<span class="w"> </span><span class="s1">'s/enabled: yes/enabled: no/'</span><span class="w"> </span>/usr/local/etc/pkg/repos/local.conf <span class="linenos" data-linenos="2 "></span><span class="gp">root@truenas[/]# </span>sed<span class="w"> </span>-i<span class="w"> </span>.orig<span class="w"> </span><span class="s1">'s/enabled: no/enabled: yes/'</span><span class="w"> </span>/usr/local/etc/pkg/repos/FreeBSD.conf </code></pre></div> <p>Now you’re ready to update the package index:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/]# </span>pkg<span class="w"> </span>update <span class="linenos" data-linenos="2 "></span><span class="go">Updating FreeBSD repository catalogue...</span> <span class="linenos" data-linenos="3 "></span><span class="go">Fetching meta.conf: 100% 163 B 0.2kB/s 00:01</span> <span class="linenos" data-linenos="4 "></span><span class="go">Fetching packagesite.txz: 100% 6 MiB 6.7MB/s 00:01</span> <span class="linenos" data-linenos="5 "></span><span class="go">Processing entries: 100%</span> <span class="hll"><span class="linenos" data-linenos="6 "></span><span class="go">FreeBSD repository update completed. 31953 packages processed.</span> </span><span class="linenos" data-linenos="7 "></span><span class="go">All repositories are up to date.</span> </code></pre></div> <p>And install your package:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/]# </span>pkg<span class="w"> </span>install<span class="w"> </span>dmg2img <span class="linenos" data-linenos=" 2 "></span><span class="go">Updating FreeBSD repository catalogue...</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">FreeBSD repository is up to date.</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">All repositories are up to date.</span> <span class="linenos" data-linenos=" 5 "></span><span class="go">New version of pkg detected; it needs to be installed first.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">The following 1 package(s) will be affected (of 0 checked):</span> <span class="linenos" data-linenos=" 7 "></span> <span class="linenos" data-linenos=" 8 "></span><span class="go">Installed packages to be UPGRADED:</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> pkg: 1.14.6 -&gt; 1.15.10 [FreeBSD]</span> <span class="linenos" data-linenos="10 "></span> <span class="linenos" data-linenos="11 "></span><span class="go">Number of packages to be upgraded: 1</span> <span class="linenos" data-linenos="12 "></span> <span class="linenos" data-linenos="13 "></span><span class="go">The operation will free 31 MiB.</span> <span class="linenos" data-linenos="14 "></span><span class="go">7 MiB to be downloaded.</span> <span class="linenos" data-linenos="15 "></span> <span class="linenos" data-linenos="16 "></span><span class="go">Proceed with this action? [y/N]: y</span> <span class="linenos" data-linenos="17 "></span><span class="go">[1/1] Fetching pkg-1.15.10.txz: 100% 7 MiB 6.9MB/s 00:01</span> <span class="linenos" data-linenos="18 "></span><span class="go">Checking integrity... done (0 conflicting)</span> <span class="linenos" data-linenos="19 "></span><span class="go">[1/1] Upgrading pkg from 1.14.6 to 1.15.10...</span> <span class="linenos" data-linenos="20 "></span><span class="go">[1/1] Extracting pkg-1.15.10: 100%</span> <span class="linenos" data-linenos="21 "></span><span class="go">Updating FreeBSD repository catalogue...</span> <span class="linenos" data-linenos="22 "></span><span class="go">FreeBSD repository is up to date.</span> <span class="linenos" data-linenos="23 "></span><span class="go">All repositories are up to date.</span> <span class="linenos" data-linenos="24 "></span><span class="go">The following 1 package(s) will be affected (of 0 checked):</span> <span class="linenos" data-linenos="25 "></span> <span class="linenos" data-linenos="26 "></span><span class="go">New packages to be INSTALLED:</span> <span class="linenos" data-linenos="27 "></span><span class="go"> dmg2img: 1.6.7 [FreeBSD]</span> <span class="linenos" data-linenos="28 "></span> <span class="linenos" data-linenos="29 "></span><span class="go">Number of packages to be installed: 1</span> <span class="linenos" data-linenos="30 "></span> <span class="linenos" data-linenos="31 "></span><span class="go">22 KiB to be downloaded.</span> <span class="linenos" data-linenos="32 "></span> <span class="linenos" data-linenos="33 "></span><span class="go">Proceed with this action? [y/N]: y</span> <span class="linenos" data-linenos="34 "></span><span class="go">[1/1] Fetching dmg2img-1.6.7.txz: 100% 22 KiB 22.5kB/s 00:01</span> <span class="linenos" data-linenos="35 "></span><span class="go">Checking integrity... done (0 conflicting)</span> <span class="linenos" data-linenos="36 "></span><span class="go">[1/1] Installing dmg2img-1.6.7...</span> <span class="linenos" data-linenos="37 "></span><span class="go">[1/1] Extracting dmg2img-1.6.7: 100%</span> <span class="linenos" data-linenos="38 "></span> <span class="linenos" data-linenos="39 "></span><span class="gp">root@truenas[/]# </span>dmg2img <span class="linenos" data-linenos="40 "></span> <span class="linenos" data-linenos="41 "></span><span class="go">dmg2img v1.6.7 (c) vu1tur (to@vu1tur.eu.org)</span> <span class="linenos" data-linenos="42 "></span> <span class="linenos" data-linenos="43 "></span><span class="go">Usage: dmg2img [-l] [-p N] [-s] [-v] [-V] [-d] <input.dmg> [<output.img>]</output.img></input.dmg></span> <span class="linenos" data-linenos="44 "></span><span class="go">or dmg2img [-l] [-p N] [-s] [-v] [-V] [-d] -i <input.dmg> -o <output.img></output.img></input.dmg></span> <span class="linenos" data-linenos="45 "></span> <span class="linenos" data-linenos="46 "></span><span class="go">Options: -s (silent) -v (verbose) -V (extremely verbose) -d (debug)</span> <span class="linenos" data-linenos="47 "></span><span class="go"> -l (list partitions) -p N (extract only partition N)</span> </code></pre></div> <h2 id="storage">đŸ’Ÿ Storage<a class="headerlink" href="#storage" title="Permanent link">¶</a></h2> <h3 id="list-all-connected-devices">List all connected devices<a class="headerlink" href="#list-all-connected-devices" title="Permanent link">¶</a></h3> <p><code>nvd0</code> is the system’s NVMe <span class="caps">SSD</span>, <code>ada0</code> a <span class="caps">SATA</span> <span class="caps">HDD</span> and <code>da0</code> a <span class="caps">USB</span> external drive.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/mnt]# </span>geom<span class="w"> </span>disk<span class="w"> </span>list <span class="linenos" data-linenos=" 2 "></span><span class="go">Geom name: nvd0</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">Providers:</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">1. Name: nvd0</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> Mediasize: 250059350016 (233G)</span> <span class="linenos" data-linenos=" 6 "></span><span class="go"> Sectorsize: 512</span> <span class="linenos" data-linenos=" 7 "></span><span class="go"> Mode: r1w1e2</span> <span class="linenos" data-linenos=" 8 "></span><span class="go"> descr: WDC WDS250G2B0C</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> rotationrate: 0</span> <span class="linenos" data-linenos="10 "></span><span class="go"> fwsectors: 0</span> <span class="linenos" data-linenos="11 "></span><span class="go"> fwheads: 0</span> <span class="linenos" data-linenos="12 "></span> <span class="linenos" data-linenos="13 "></span><span class="go">Geom name: ada0</span> <span class="linenos" data-linenos="14 "></span><span class="go">Providers:</span> <span class="linenos" data-linenos="15 "></span><span class="go">1. Name: ada0</span> <span class="linenos" data-linenos="16 "></span><span class="go"> Mediasize: 6001175126016 (5.5T)</span> <span class="linenos" data-linenos="17 "></span><span class="go"> Sectorsize: 512</span> <span class="linenos" data-linenos="18 "></span><span class="go"> Stripesize: 4096</span> <span class="linenos" data-linenos="19 "></span><span class="go"> Stripeoffset: 0</span> <span class="linenos" data-linenos="20 "></span><span class="go"> Mode: r1w1e3</span> <span class="linenos" data-linenos="21 "></span><span class="go"> descr: TOSHIBA HDWN160</span> <span class="linenos" data-linenos="22 "></span><span class="go"> rotationrate: 7200</span> <span class="linenos" data-linenos="23 "></span><span class="go"> fwsectors: 63</span> <span class="linenos" data-linenos="24 "></span><span class="go"> fwheads: 16</span> <span class="linenos" data-linenos="25 "></span> <span class="linenos" data-linenos="26 "></span><span class="go">Geom name: da0</span> <span class="linenos" data-linenos="27 "></span><span class="go">Providers:</span> <span class="linenos" data-linenos="28 "></span><span class="go">1. Name: da0</span> <span class="linenos" data-linenos="29 "></span><span class="go"> Mediasize: 320072933376 (298G)</span> <span class="linenos" data-linenos="30 "></span><span class="go"> Sectorsize: 512</span> <span class="linenos" data-linenos="31 "></span><span class="go"> Mode: r0w0e0</span> <span class="linenos" data-linenos="32 "></span><span class="go"> descr: ST332082 0ACE</span> <span class="linenos" data-linenos="33 "></span><span class="go"> rotationrate: unknown</span> <span class="linenos" data-linenos="34 "></span><span class="go"> fwsectors: 63</span> <span class="linenos" data-linenos="35 "></span><span class="go"> fwheads: 255</span> </code></pre></div> <h3 id="mount-an-ntfs-partition">Mount an <span class="caps">NTFS</span> partition<a class="headerlink" href="#mount-an-ntfs-partition" title="Permanent link">¶</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">root@truenas[/mnt]# </span>kldload<span class="w"> </span>fuse.ko <span class="linenos" data-linenos="2 "></span><span class="gp">root@truenas[/mnt]# </span>mkdir<span class="w"> </span>usb-hdd <span class="linenos" data-linenos="3 "></span><span class="gp">root@truenas[/mnt]# </span>ntfs-3g<span class="w"> </span>/dev/da0p2<span class="w"> </span>/mnt/usb-hdd </code></pre></div> <h3 id="delete-a-partition">Delete a partition<a class="headerlink" href="#delete-a-partition" title="Permanent link">¶</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/mnt]# </span>gpart<span class="w"> </span>show<span class="w"> </span>da0 <span class="linenos" data-linenos=" 2 "></span><span class="go">=&gt; 34 7814037101 da0 GPT (3.6T)</span> <span class="linenos" data-linenos=" 3 "></span><span class="go"> 34 6 - free - (3.0K)</span> <span class="linenos" data-linenos=" 4 "></span><span class="go"> 40 409600 1 efi (200M)</span> <span class="linenos" data-linenos=" 5 "></span><span class="go"> 409640 7813365344 2 apple-hfs (3.6T)</span> <span class="linenos" data-linenos=" 6 "></span><span class="go"> 7813774984 262151 - free - (128M)</span> <span class="linenos" data-linenos=" 7 "></span> <span class="linenos" data-linenos=" 8 "></span><span class="gp">root@truenas[/mnt]# </span>gpart<span class="w"> </span>delete<span class="w"> </span>-i<span class="w"> </span><span class="m">1</span><span class="w"> </span>da0<span class="w"> </span> <span class="linenos" data-linenos=" 9 "></span><span class="go">da0p1 deleted</span> <span class="linenos" data-linenos="10 "></span> <span class="linenos" data-linenos="11 "></span><span class="gp">root@truenas[/mnt]# </span>gpart<span class="w"> </span>show<span class="w"> </span>da0<span class="w"> </span> <span class="linenos" data-linenos="12 "></span><span class="go">=&gt; 34 7814037101 da0 GPT (3.6T)</span> <span class="linenos" data-linenos="13 "></span><span class="go"> 34 409606 - free - (200M)</span> <span class="linenos" data-linenos="14 "></span><span class="go"> 409640 7813365344 2 apple-hfs (3.6T)</span> <span class="linenos" data-linenos="15 "></span><span class="go"> 7813774984 262151 - free - (128M)</span> </code></pre></div> <h3 id="3-pass-usb-hdd-wipe">3-pass <span class="caps">USB</span> <span class="caps">HDD</span> wipe<a class="headerlink" href="#3-pass-usb-hdd-wipe" title="Permanent link">¶</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">root@truenas[/mnt]# </span>bcwipe<span class="w"> </span>-v<span class="w"> </span>-me<span class="w"> </span>-t2<span class="w"> </span>-b<span class="w"> </span>/dev/da0<span class="w"> </span> <span class="linenos" data-linenos=" 2 "></span><span class="go">Multithreading not supported.</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">Run ./configure with --enable-pthreads option, then rebuild BCWipe to enable multithreading.</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">Wiping scheme: US DoE, 3 pass(es)</span> <span class="linenos" data-linenos=" 5 "></span><span class="go">Wipe /dev/da0 (y/[n]/a)?y</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">Wiping char device '/dev/da0'</span> <span class="linenos" data-linenos=" 7 "></span><span class="go">Device '/dev/da0' opened in direct access mode</span> <span class="linenos" data-linenos=" 8 "></span><span class="go">Device size 320072933376 bytes (312571224 kB), method 3</span> <span class="linenos" data-linenos=" 9 "></span><span class="go">Wiping char device '/dev/da0' pass 1/3 [random] started</span> <span class="linenos" data-linenos="10 "></span><span class="go">wipe pass 1/3 : 212992/312571224 kB ( 0%) Rate: 21233 kB/s </span> </code></pre></div> <h3 id="zfs"><span class="caps">ZFS</span><a class="headerlink" href="#zfs" title="Permanent link">¶</a></h3> <ul> <li>List all snaphots of the <code>tank/my-data</code> dataset:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>zfs<span class="w"> </span>list<span class="w"> </span>-r<span class="w"> </span>-t<span class="w"> </span>snapshot<span class="w"> </span>tank/my-data </code></pre></div> <ul> <li>Rename all snaphot’s names prefixes from <code>auto-</code> to <code>daily-</code>, for the <code>tank/my-data</code> dataset and its children:</li> </ul> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span><span class="gp">$ </span>zfs<span class="w"> </span>list<span class="w"> </span>-r<span class="w"> </span>-t<span class="w"> </span>snapshot<span class="w"> </span>tank/my-data<span class="w"> </span> <span class="linenos" data-linenos=" 2 "></span><span class="go">NAME USED AVAIL REFER MOUNTPOINT</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">tank/my-data@auto-2021-09-02_00-00 0B - 209G -</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">tank/my-data@auto-2021-09-03_00-00 0B - 209G -</span> <span class="linenos" data-linenos=" 5 "></span><span class="go">tank/my-data@auto-2021-09-04_00-00 0B - 209G -</span> <span class="linenos" data-linenos=" 6 "></span><span class="gp gp-VirtualEnv">(...)</span> <span class="linenos" data-linenos=" 7 "></span><span class="gp">$ </span>zfs<span class="w"> </span>list<span class="w"> </span>-r<span class="w"> </span>-t<span class="w"> </span>snapshot<span class="w"> </span>-o<span class="w"> </span>name<span class="w"> </span>-H<span class="w"> </span>tank/my-data<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{$2 = $1; sub(/@auto\-/, "@daily-", $2); printf "%s\n%s\n", $1, $2;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span><span class="s1">'\n'</span><span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>-0<span class="w"> </span>-n<span class="w"> </span><span class="m">2</span><span class="w"> </span>-t<span class="w"> </span>zsh<span class="w"> </span>-c<span class="w"> </span><span class="s1">'zfs rename "$0" "$1"'</span> <span class="linenos" data-linenos=" 8 "></span><span class="go">zsh -c zfs rename "$0" "$1" tank/my-data@auto-2021-09-02_00-00 tank/my-data@daily-2021-09-02_00-00</span> <span class="linenos" data-linenos=" 9 "></span><span class="go">zsh -c zfs rename "$0" "$1" tank/my-data@auto-2021-09-03_00-00 tank/my-data@daily-2021-09-03_00-00</span> <span class="linenos" data-linenos="10 "></span><span class="go">zsh -c zfs rename "$0" "$1" tank/my-data@auto-2021-09-04_00-00 tank/my-data@daily-2021-09-04_00-00</span> <span class="linenos" data-linenos="11 "></span><span class="gp gp-VirtualEnv">(...)</span> <span class="linenos" data-linenos="12 "></span><span class="gp">$ </span>zfs<span class="w"> </span>list<span class="w"> </span>-r<span class="w"> </span>-t<span class="w"> </span>snapshot<span class="w"> </span>tank/my-data<span class="w"> </span> <span class="linenos" data-linenos="13 "></span><span class="go">NAME USED AVAIL REFER MOUNTPOINT</span> <span class="linenos" data-linenos="14 "></span><span class="go">tank/my-data@daily-2021-09-02_00-00 0B - 209G -</span> <span class="linenos" data-linenos="15 "></span><span class="go">tank/my-data@daily-2021-09-03_00-00 0B - 209G -</span> <span class="linenos" data-linenos="16 "></span><span class="go">tank/my-data@daily-2021-09-04_00-00 0B - 209G -</span> <span class="linenos" data-linenos="17 "></span><span class="gp gp-VirtualEnv">(...)</span> </code></pre></div> <h2 id="issues">🐛 Issues<a class="headerlink" href="#issues" title="Permanent link">¶</a></h2> <h3 id="freenas-to-truenas-migration">FreeNAS to TrueNAS migration<a class="headerlink" href="#freenas-to-truenas-migration" title="Permanent link">¶</a></h3> <p>I wrote a long article on <a href="https://kevin.deldycke.com/2020/12/migration-from-freenas-to-truenas/">how to migrate an encrypted pool from FreeNAS to TrueNAS</a>.</p> <h3 id="unreachable-network-due-to-multiple-nics">Unreachable Network due to Multiple NICs<a class="headerlink" href="#unreachable-network-due-to-multiple-nics" title="Permanent link">¶</a></h3> <p>My machine was regularly disconnected from the network and couldn’t be cleanly rebooted. This started under <code>FreeNAS 11.3-U4.1</code> and is still reproducible with a fresh <code>TrueNAS 12.0</code>.</p> <p>The issue lies somewhere in all connected interfaces being granted <span class="caps">DHCP</span> active service:</p> <p><a href="https://www.youtube.com/watch?v=vDMooVj-flM">https://www.youtube.com/watch?v=vDMooVj-flM</a></p> <p>The <a href="https://jira.ixsystems.com/browse/NAS-108043">issue is being discussed on iXsystems’ <span class="caps">JIRA</span></a>.</p> <h3 id="validateupdate-error-on-upgrade"><code>ValidateUpdate</code> error on upgrade<a class="headerlink" href="#validateupdate-error-on-upgrade" title="Permanent link">¶</a></h3> <p>Got the following error on upgrade?</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="x">PermissionError: [Errno 13] Permission denied: './ValidateUpdate'</span> </code></pre></div> <p>The fix: temporary <a href="https://www.truenas.com/docs/hub/tasks/advanced/system-dataset/">switch the System Dataset</a> to <code>freenas-boot</code>. Your default system dataset is probably set to your <span class="caps">ZFS</span> pool which might be unavailable for whatever reason. In my case, I was in the middle of a <a href="https://kevin.deldycke.com/2020/12/migration-from-freenas-to-truenas/">pool migration process</a>.</p>Billing pipeline: a critical time sensitive system2020-10-12T00:00:00+02:002020-10-12T00:00:00+02:00Kevin Deldycketag:kevin.deldycke.com,2020-10-12:/2020/10/billing-pipeline-critical-time-sensitive-system/<p>This is an answer to <a href="https://www.linkedin.com/in/antoine-veuiller/">Antoine Veuiller</a>&lsquo;s article, an <a href="https://medium.com/@aveuiller/stories-of-flaky-test-encounters-in-the-wild-a152bf7151f5">Introduction to Flaky Tests by Example</a>, in which he shares his experience on tackling unstable&nbsp;tests.</p> <p>And I&rsquo;m the author of the flaky tests he&rsquo;s talking about!&nbsp;😬</p> <h2 id="story-time">Story time!<a class="headerlink" href="#story-time" title="Permanent link">&para;</a></h2> <p>I started designing and implementing Scaleway&rsquo;s billing pipeline 
</p><p>This is an answer to <a href="https://www.linkedin.com/in/antoine-veuiller/">Antoine Veuiller</a>‘s article, an <a href="https://medium.com/@aveuiller/stories-of-flaky-test-encounters-in-the-wild-a152bf7151f5">Introduction to Flaky Tests by Example</a>, in which he shares his experience on tackling unstable tests.</p> <p>And I’m the author of the flaky tests he’s talking about! 😬</p> <h2 id="story-time">Story time!<a class="headerlink" href="#story-time" title="Permanent link">¶</a></h2> <p>I started designing and implementing Scaleway’s billing pipeline in 2013. This was to be a critical system, given its purpose, collecting money. It was very seriouz bizzness.</p> <p><img alt="Now remember, kids, the internet is serious business -- Isaac Asimov" src="/uploads/2020/internet-is-serious-business.jpeg"/></p> <p>I was worried a lot about both accuracy and precision of the numbers it produced. Strange how things gets real when you add a dollar sign.Â đŸ€‘</p> <p>My system had to <strong>track nano-euros at the milli-seconds time granularity</strong>. Why? Read the <a href="https://aws.amazon.com/lambda/pricing/">pricing page of <span class="caps">AWS</span> Lambda</a>. They’re billed 100ms usage at a time. Here lies my <span class="caps">MVP</span>: to be able to invoice usage-based cloud computing resources.</p> <p>So I wrote extensive unit tests to prove that I knew how to add, subtract and multiply.</p> <p><img alt="Math is Math!" src="/uploads/2020/math-is-math.jpeg"/></p> <p>Well, to be picky, it was more an acceptance suite than proper unit-tests. Whatever.</p> <p>Fast forward 4 years, Scaleway is bigger, and millions of euros are collected by the dual billing <span class="amp">&amp;</span> payment stacks. 📈 I’m <a href="https://kevin.deldycke.com/2020/02/engineering-to-management-transition/">no longer an engineer, but the manager</a> of the team taking care of the system. 👔</p> <p>I hired Antoine. And there he was, fixing the crap I implemented 7 years ago. Not only did he fix the tests nobody had time to work on, but he uncovered more edge-cases and wrote an entire article about it. 😍</p> <p>Thanks a lot Antoine for taking the resolution and effort to fix my old mess! 😁 And I’m sure the definitive absence of flaky tests reduced the overall stress of the team.</p> <p>Now <a href="https://medium.com/@aveuiller/stories-of-flaky-test-encounters-in-the-wild-a152bf7151f5">go read that stuff</a>: it demonstrates how hard it is to reason with bare temporal logic, even for seasoned engineers. On my side, practicing this made me compile that <a href="https://github.com/kdeldycke/awesome-falsehood">awesome list of falsehoods programmers believe in</a>, as a cautionary tale of all things that might go wrong.</p> <p>Let’s continue with some complementary notes to provide context on the system.</p> <h2 id="ignoring-flaky-tests">Ignoring flaky tests<a class="headerlink" href="#ignoring-flaky-tests" title="Permanent link">¶</a></h2> <p>When I was alone behind the billing pipeline, I quickly refrained from pushing code in production during the first few days of each month. That was based on the traumatic (but enlighting) early days of operating a live cloud computing platform. With real customers on the other side of the <span class="caps">API</span>.</p> <p>This dogma was carried on when I <a href="https://kevin.deldycke.com/2020/02/engineering-to-management-transition/">grew the team from 0 to 12 engineers</a>. Even though I repeated ad-nauseum the context of that decision, I feared it might have been lost in a typical case of <a href="https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence">Chesterton’s fence</a>. So here it is, in written form.</p> <p>The main reason was the massive reports being produced in that critical time-window. The consolidation of accounting and financial numbers of our holding company was, among other things, dependent on these reports. The blast radius of our team was suddently inflated, at the billion-dollar level. 😅</p> <p>Hence the compromise. In the grand scheme of things, we had to reach operational excellence before taking care of our flaky tests. And I’m proud to highlight that under my watch, the team always delivered on schedule. Because the <em>money must flow</em>.</p> <p><img alt="The Spice must flow" src="/uploads/2020/the-spice-must-flow-cat-version.jpg"/></p> <p>As a manager, <strong>I’m the only one responsible</strong> for allowing the team to disregard these technical issues.</p> <h2 id="time-arithmetics">Time arithmetics<a class="headerlink" href="#time-arithmetics" title="Permanent link">¶</a></h2> <blockquote> <p>Once again, and unfortunately for us, the project was using <a href="https://github.com/arrow-py/arrow"><code>arrow</code></a></p> </blockquote> <p>The reason I choose <code>arrow</code> at the time was practical. It was the only Python library actively maintained that was providing <a href="https://arrow.readthedocs.io/en/stable/#ranges-spans">date-aware ceiling and flooring methods</a>. I wouldn’t have been able to tame the side-effects of quantization without those.</p> <h2 id="time-reference">Time reference<a class="headerlink" href="#time-reference" title="Permanent link">¶</a></h2> <blockquote> <p>Fortunately, and thanks to some previously well-thought environment on the project, we had a central method to provide the current date, which was initially intended to prevent the use of a wrong timezone.</p> </blockquote> <p>Thanks for the kind words! That was one of the tiny design decision I made in the first few weeks of the project. I came up with that central hook as a place from where to fetch a solid time reference. If it ended up enforcing good usage of timezones, the original intent was more convoluted.</p> <p>I was building a system in a domain for which transactions were basically invented. Try for instance explaining to customers that their credit cards got charged twice because of eventual consistency
</p> <p>Scaleway’s ambition was to have datacenters all over the world (that’s the reason I signed in). So how the codebase would behave in a massively distributed architecture? Time at that scale is hard. And it was at the root of everything, so I created that utility.</p> <p>It was supposed to be a gateway to the local datacenter’s <span class="caps">GPS</span> source, or a fancy atomic clock. I even fantasized having access to <a href="https://cloud.google.com/spanner/docs/true-time-external-consistency">TrueTime</a>-like protocols (not even sure we even needed it). <span class="caps">BTW</span>, did you know <a href="https://twitter.com/kdeldycke/status/1102172902995173376">Google’s TrueTime is staffed by 16 senior SREs</a>, which “<em>had take measures up to and including calling the <span class="caps">USAF</span> and telling them their satellites are fucked up</em>”?</p> <h2 id="busting-coupling">Busting coupling<a class="headerlink" href="#busting-coupling" title="Permanent link">¶</a></h2> <blockquote> <p>By “chance” the tests were always run in the right order for years. This situation could have been detected way earlier by using a random execution order for tests. It happens that python has simple modules to do so.</p> </blockquote> <p>Great minds think alike! I switched over <a href="https://docs.pytest.org"><code>pytest</code></a> in my pet projects (namely <a href="https://github.com/kdeldycke/meta-package-manager"><code>meta-package-manager</code></a> and <a href="https://github.com/kdeldycke/mail-deduplicate"><code>mail-deduplicate</code></a>) and used <a href="https://pypi.org/project/pytest-randomly/"><code>pytest-randomly</code></a> too to great success.</p> <p><span class="caps">BTW</span>, <code>pytest</code> is powerful, but feels magical at times. I spent more time working on it than I anticipated, so a switch to <code>pytest</code> should be seen as an investment to increase the quality of your software.</p> <h2 id="randomize-all-the-things">Randomize all the things!<a class="headerlink" href="#randomize-all-the-things" title="Permanent link">¶</a></h2> <blockquote> <h3 id="a-good-flakiness">A good flakiness<a class="headerlink" href="#a-good-flakiness" title="Permanent link">¶</a></h3> <p>(
) the test was generating numerous instances of the class with randomized inputs.</p> </blockquote> <p>That good flakiness has a name: <a href="https://en.wikipedia.org/wiki/Fuzzing">fuzzing</a>! 😉</p> <p><a href="https://dilbert.com/strip/2001-10-25"><img alt="Dilbert's random number generator" src="https://assets.amuniversal.com/321a39e06d6401301d80001dd8b71c47"/></a></p>