Kevin Deldycke - English Yak ShaverAWS Commands2020-06-16T12:00:00+02:002020-06-16T12:00:00+02:00Kevin,2020-06-16:/2020/06/aws-commands/<p>All commandes below relies on the latest version of <a href="">aws-cli</a>.</p> <ul> <li>macOS install:<div class="highlight"><pre><span></span><code><span class="gp">$</span> brew install awscli <span class="gp gp-VirtualEnv">(...)</span> <span class="gp">$</span> aws --version <span class="go">aws-cli/2.0.19 Python/3.8.3 Darwin/19.5.0 botocore/2.0.0dev23</span> </code></pre></div> </li> </ul> <h2 id="authentication">Authentication<a class="headerlink text-decoration-none small pl-2" href="#authentication" title="Permanent link">&para;</a></h2> <ul> <li> <p>Register default&nbsp;profile:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws configure </code></pre></div> </li> <li> <p>Register additional&nbsp;profile:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws configure --profile bob </code></pre></div> </li> <li> <p>List access&nbsp;keys:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws …</code></pre></div></li></ul><p>All commandes below relies on the latest version of <a href="">aws-cli</a>.</p> <ul> <li>macOS install:<div class="highlight"><pre><span></span><code><span class="gp">$</span> brew install awscli <span class="gp gp-VirtualEnv">(...)</span> <span class="gp">$</span> aws --version <span class="go">aws-cli/2.0.19 Python/3.8.3 Darwin/19.5.0 botocore/2.0.0dev23</span> </code></pre></div> </li> </ul> <h2 id="authentication">Authentication<a class="headerlink text-decoration-none small pl-2" href="#authentication" title="Permanent link">&para;</a></h2> <ul> <li> <p>Register default&nbsp;profile:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws configure </code></pre></div> </li> <li> <p>Register additional&nbsp;profile:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws configure --profile bob </code></pre></div> </li> <li> <p>List access&nbsp;keys:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws iam list-access-keys </code></pre></div> </li> <li> <p>List access keys of another&nbsp;profile:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws iam --profile bob list-access-keys </code></pre></div> </li> </ul> <h2 id="amazon-transcribe">Amazon Transcribe<a class="headerlink text-decoration-none small pl-2" href="#amazon-transcribe" title="Permanent link">&para;</a></h2> <ul> <li> <p>Fetch all names of the first 100 <a href="">transcription jobs</a>:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws transcribe list-transcription-jobs --query <span class="s1">&#39;[TranscriptionJobSummaries[*].TranscriptionJobName]&#39;</span> --max-results <span class="m">100</span> --output text </code></pre></div> </li> <li> <p><a href="">Get <span class="caps">URL</span> of the transcript</a> produced by <code>my_job_name</code> job:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> aws transcribe get-transcription-job --transcription-job-name my_job_name --query <span class="s1">&#39;[TranscriptionJob.Transcript.TranscriptFileUri]&#39;</span> --output text </code></pre></div> </li> <li> <p>Same as above but save the transcript content directly to a local <code>transcript.txt</code> file:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> <span class="nv">AWS_PAGER</span><span class="o">=</span><span class="s2">&quot;&quot;</span> aws transcribe get-transcription-job --transcription-job-name my_job_name --query <span class="s1">&#39;[TranscriptionJob.Transcript.TranscriptFileUri]&#39;</span> --output text <span class="p">|</span> wget -i - -O - <span class="p">|</span> jq --raw-output <span class="s1">&#39;.results.transcripts[0].transcript&#39;</span> &gt; transcript.txt </code></pre></div> </li> <li> <p>Putting it all together, here is how do download all transcripts from all your&nbsp;jobs:</p> <div class="highlight"><pre><span></span><code><span class="k">for</span> JOB_ID in <span class="k">$(</span>aws transcribe list-transcription-jobs --query <span class="s1">&#39;[TranscriptionJobSummaries[*].TranscriptionJobName]&#39;</span> --max-results <span class="m">100</span> --output text<span class="k">)</span><span class="p">;</span> <span class="k">do</span> <span class="nv">AWS_PAGER</span><span class="o">=</span><span class="s2">&quot;&quot;</span> aws transcribe get-transcription-job --transcription-job-name <span class="s2">&quot;</span><span class="nv">$JOB_ID</span><span class="s2">&quot;</span> --query <span class="s1">&#39;[TranscriptionJob.Transcript.TranscriptFileUri]&#39;</span> --output text <span class="p">|</span> wget -i - -O - <span class="p">|</span> jq --raw-output <span class="s1">&#39;.results.transcripts[0].transcript&#39;</span> &gt; <span class="s2">&quot;</span><span class="nv">$JOB_ID</span><span class="s2">&quot;</span>.txt<span class="p">;</span> <span class="k">done</span> </code></pre></div> </li> </ul>10G Network at Home2020-06-12T00:00:00+02:002020-06-12T00:00:00+02:00Kevin,2020-06-12:/2020/06/10g-network-at-home/<p>Into my <a href="./">journey building a <span class="caps">NAS</span> for the home office</a>, I went down the rabbit hole and chased better file transfer performance. The next step would be a network upgrade. And so I started exploring 10G at home. It is fancy, but&nbsp;doable.</p> <h2 id="internet-access">Internet Access<a class="headerlink text-decoration-none small pl-2" href="#internet-access" title="Permanent link">&para;</a></h2> <p>I&rsquo;m already equipped with …</p><p>Into my <a href="./">journey building a <span class="caps">NAS</span> for the home office</a>, I went down the rabbit hole and chased better file transfer performance. The next step would be a network upgrade. And so I started exploring 10G at home. It is fancy, but&nbsp;doable.</p> <h2 id="internet-access">Internet Access<a class="headerlink text-decoration-none small pl-2" href="#internet-access" title="Permanent link">&para;</a></h2> <p>I&rsquo;m already equipped with a <a href="">Freebox Delta Server</a> with 10 Gbps fiber downlink, 600 Mbps uplink. We can plan a multi-step&nbsp;upgrade.</p> <h2 id="nas">Nas<a class="headerlink text-decoration-none small pl-2" href="#nas" title="Permanent link">&para;</a></h2> <p>Link together the Freebox and the <span class="caps">NAS</span>. Both are close to each other, in the same cabinet. Freebox is already equipped with a 10G <span class="caps">SFP</span>+ port. All I need is a <span class="caps">NIC</span> and a&nbsp;cable:</p> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th align="right">Quantity</th> <th align="right">Total (excl. shipping)</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td><span class="caps">NIC</span></td> <td><a href="">10Gtek X520-10G-1S-X8</a>, PCIe x8 single <span class="caps">SFP</span>+ port</td> <td align="right">1</td> <td align="right">€135.99</td> <td>82599 Intel controller supported by FreeBSD.</td> </tr> <tr> <td>Cable</td> <td><a href="">10Gtek <span class="caps">SFP</span>+ Direct Attach Copper - 2m</a></td> <td align="right">1</td> <td align="right">€26.99</td> <td>A <span class="caps">DAC</span> is low-power, low-cost, low-latency and less bulky.</td> </tr> <tr> <td></td> <td></td> <td align="right"><strong>Total</strong></td> <td align="right"><strong>€162.98</strong></td> <td></td> </tr> </tbody> </table> <h2 id="cabinet-switch">Cabinet Switch<a class="headerlink text-decoration-none small pl-2" href="#cabinet-switch" title="Permanent link">&para;</a></h2> <p>Home is wired with <em>Grade 3 Residential Triple Play <span class="caps">4PR</span> F/<span class="caps">FTP</span></em> cables. Not sure what that cable is. In theory <span class="caps">10GBASE</span>-T requires Cat6. The reality is that most equipment are <span class="caps">OK</span> with short-distance Cat5e (less than 30-45 meters). I guess it is worth a&nbsp;try.</p> <p><span class="caps">RJ45</span> sockets are available in each room, everything converging in the router <span class="amp">&amp;</span> <span class="caps">NAS</span> cabinet. Unfortunately the Freebox <span class="caps">SFP</span>+ port is capped at level-2 (&lt;1.5 W), and is not providing enough power for an <span class="caps">RJ45</span> transceiver. So we need an <span class="caps">SFP</span>+&nbsp;switch.</p> <p>There is plenty of <span class="caps">SFP</span>+ modules to choose from. I selected the <a href="">MikroTik S+<span class="caps">RJ10</span></a> to reduce incompatibility and for its good price. Other alternatives are covered in this excellent <a href=""><span class="caps">SFP</span>+ to 10Gbase-T adapter buyers guide</a>.</p> <p>At the end the cabinet was equipped&nbsp;with:</p> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th align="right">Quantity</th> <th align="right">Total (excl. shipping)</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>Switch</td> <td><a href="">MikroTik <span class="caps">CRS305</span>-1G-4S+in</a></td> <td align="right">1</td> <td align="right">€144.01</td> <td>Limited to 2 <span class="caps">RJ45</span> transceiver, seems to get too hot with more.</td> </tr> <tr> <td>Transceiver</td> <td><a href="">MikroTik S+<span class="caps">RJ10</span></a></td> <td align="right">1</td> <td align="right">€66.04</td> <td></td> </tr> <tr> <td>Cable</td> <td><a href="">10Gtek <span class="caps">SFP</span>+ Direct Attach Copper - 1m</a></td> <td align="right">1</td> <td align="right">€24.99</td> <td></td> </tr> <tr> <td></td> <td></td> <td align="right"><strong>Total</strong></td> <td align="right"><strong>€235.04</strong></td> <td></td> </tr> </tbody> </table> <h2 id="office-switch">Office Switch<a class="headerlink text-decoration-none small pl-2" href="#office-switch" title="Permanent link">&para;</a></h2> <p>This one brings 10G from the cabinet to the home office. That where most bandwidth is consumed. We&rsquo;ll bring 10G by the way of classic copper cables, and have some options for another switch to distribute the bandwidth&nbsp;there:</p> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th>10G ports</th> <th>1G ports</th> <th align="right">Price</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>Switch</td> <td><a href=""><span class="caps">QNAP</span> <span class="caps">QSW</span>-308S</a></td> <td>3 <span class="caps">SFP</span></td> <td>8 <span class="caps">RJ45</span></td> <td align="right">$159.00</td> <td>Unmanaged</td> </tr> <tr> <td>Switch</td> <td><a href="">Netgear <span class="caps">GS110MX</span></a></td> <td>1 <span class="caps">RJ45</span></td> <td>8 <span class="caps">RJ45</span></td> <td align="right">€160.65</td> <td>Unmanaged</td> </tr> <tr> <td>Switch</td> <td><a href=""><span class="caps">QNAP</span> <span class="caps">QSW</span>-308-1C</a></td> <td>3 <span class="caps">SFP</span></td> <td>8 <span class="caps">RJ45</span></td> <td align="right">$200.99</td> <td>Unmanaged</td> </tr> <tr> <td>Switch</td> <td><a href="">Asus <span class="caps">XG</span>-U2008</a></td> <td>2 <span class="caps">RJ45</span></td> <td>8 <span class="caps">RJ45</span></td> <td align="right">€199.65</td> <td>Unmanaged</td> </tr> <tr> <td>Switch</td> <td><a href="">Netgear <span class="caps">MS510TX</span></a></td> <td>1 <span class="caps">SFP</span> / 1 <span class="caps">RJ45</span></td> <td>4 <span class="caps">RJ45</span></td> <td align="right">€277.98</td> <td>Managed. <a href="">Review</a>.</td> </tr> <tr> <td>Switch</td> <td><a href="">Netgear <span class="caps">XS505M</span></a></td> <td>1 <span class="caps">SFP</span> / 4 <span class="caps">RJ45</span></td> <td>0</td> <td align="right">€314.99</td> <td>Unmanaged</td> </tr> <tr> <td>Switch</td> <td><a href="">Netgear <span class="caps">XS508M</span></a></td> <td>1 <span class="caps">SFP</span> / 8 <span class="caps">RJ45</span></td> <td>0</td> <td align="right">€383.98</td> <td>Unmanaged</td> </tr> </tbody> </table> <h2 id="external-nics">External NICs<a class="headerlink text-decoration-none small pl-2" href="#external-nics" title="Permanent link">&para;</a></h2> <p>To plug the laptops in the office on the 10G network, we&rsquo;ll rely on external dongles. There a wide variety and price point to choose&nbsp;from:</p> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th>Port</th> <th>Interface</th> <th align="right">Price</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td><span class="caps">NIC</span></td> <td><a href="">Club 3D <span class="caps">CAC</span>-1420</a></td> <td>2.5G <span class="caps">RJ45</span></td> <td><span class="caps">USB</span>-A 3.2 Gen 1</td> <td align="right">$41.57</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href="">Club 3D <span class="caps">CAC</span>-1520</a></td> <td>2.5G <span class="caps">RJ45</span></td> <td><span class="caps">USB</span>-C</td> <td align="right">$45.07</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href="">TRENDnet <span class="caps">USB</span>-C 2.5G</a></td> <td>2.5G <span class="caps">RJ45</span></td> <td><span class="caps">USB</span>-C 3.1</td> <td align="right">$86.90</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href="">StarTech <span class="caps">USB</span> 3 5G</a></td> <td>5G <span class="caps">RJ45</span></td> <td><span class="caps">USB</span>-A 3.0</td> <td align="right">$101.68</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href="">TRENDnet <span class="caps">USB</span>-C 5G</a></td> <td>5G <span class="caps">RJ45</span></td> <td><span class="caps">USB</span>-C 3.1</td> <td align="right">$158.20</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href=""><span class="caps">QNAP</span> <span class="caps">QNA</span>-<span class="caps">UC5G1T</span></a></td> <td>5G <span class="caps">RJ45</span></td> <td><span class="caps">USB</span>-C 3.0</td> <td align="right">$224.00</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href="">Sonnet <span class="caps">SOLO10G</span>-<span class="caps">TB3</span></a></td> <td>10G <span class="caps">RJ45</span></td> <td>Thunderbolt 3</td> <td align="right">$149.00</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href=""><span class="caps">QNAP</span> <span class="caps">QNA</span>-<span class="caps">T310G1T</span></a></td> <td>10G <span class="caps">RJ45</span></td> <td>Thunderbolt 3</td> <td align="right">$178.99</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href=""><span class="caps">QNAP</span> <span class="caps">QNA</span>-<span class="caps">T310G1S</span></a></td> <td>10G <span class="caps">SFP</span>+</td> <td>Thunderbolt 3</td> <td align="right">$169.00</td> <td></td> </tr> <tr> <td><span class="caps">NIC</span></td> <td><a href="">Sonnet <span class="caps">SOLO10G</span>-<span class="caps">TB2</span></a></td> <td>10G <span class="caps">RJ45</span></td> <td>Thunderbolt 2</td> <td align="right">$198.98</td> <td></td> </tr> </tbody> </table> <h2 id="wireless">Wireless<a class="headerlink text-decoration-none small pl-2" href="#wireless" title="Permanent link">&para;</a></h2> <p>Check out this <a href="">Wi-Fi 4-5-6-6E (802.11 n/ac/ax) guide</a>, which is one of the best resource I ever found to make educated wireless router/<span class="caps">AP</span> upgrade decisions, and cut through all the marketing&nbsp;hype.</p> <p>One of the best access point so far to increase coverage seems to be the <a href="">Ubiquiti UniFi nanoHD</a>&nbsp;(€163.88).</p> <h2 id="upgrade-path">Upgrade Path<a class="headerlink text-decoration-none small pl-2" href="#upgrade-path" title="Permanent link">&para;</a></h2> <ul> <li>Upgrade <span class="caps">NAS</span>&rsquo;s motherboard and get one with an onboard 10G controller, to eliminate the <span class="caps">PCI</span>-e&nbsp;card.</li> <li>Get rid of all external <span class="caps">USB</span> NICs and replace them with all-in-one <span class="caps">USB</span>-C hubs with embedded 10G. No solution exists on the market&nbsp;yet.</li> <li>Buy a new house, install optical fiber&nbsp;everywhere.</li> </ul>Home Server Hardware for NAS2020-05-16T00:00:00+02:002020-05-16T00:00:00+02:00Kevin,2020-05-16:/2020/05/nas-hardware/<p>I live with a scientific data-hoarder. Last month I discovered a stash of old HDDs ranging from 320 <span class="caps">GB</span> to 3 <span class="caps">TB</span> spanning a period of 15 years. I was horrified by the high probability of loosing all that data to mechanical failure, bit-rot or shear bad&nbsp;luck.</p> <p>I can …</p><p>I live with a scientific data-hoarder. Last month I discovered a stash of old HDDs ranging from 320 <span class="caps">GB</span> to 3 <span class="caps">TB</span> spanning a period of 15 years. I was horrified by the high probability of loosing all that data to mechanical failure, bit-rot or shear bad&nbsp;luck.</p> <p>I can&rsquo;t help it and instantly felt the urge of proper backups and data management. The first step would be the creation of a <span class="caps">NAS</span> at home. Best excuse to bootstrap a home-lab right? <img alt="😇" class="emojione" src="" title=":innocent:"></p> <h2 id="filesystem">Filesystem<a class="headerlink text-decoration-none small pl-2" href="#filesystem" title="Permanent link">&para;</a></h2> <p>I disqualified integrated <span class="caps">NAS</span> embedding their own <span class="caps">OS</span> and proprietary apps to avoid vendor lock-ins. I aimed for a standard and widely adopted software&nbsp;stack.</p> <p>Silent data corruption being my main concern, I need a robust filesystems. The last time I looked years ago, <span class="caps">BTRFS</span> was a big upcoming contender. But it did not seemed to have caught up in popularity since then. So I <strong>picked <span class="caps">ZFS</span>, for its maturity</strong>, flexibility and native data integrity&nbsp;capabilities.</p> <h2 id="os">Os<a class="headerlink text-decoration-none small pl-2" href="#os" title="Permanent link">&para;</a></h2> <p>I need my appliance to run stable for the next few years. I can only achieve that with a healthy distribution. <strong>Choosing FreeNAS was a no-brainer</strong>: big community, open ecosystem, actively maintained, built upon proven&nbsp;technologies.</p> <p>Especially since it was&nbsp;announced:</p> <ul> <li><a href="">the unification of FreeNAS and&nbsp;TrueNAS</a></li> <li><a href=";px=FreeBSD-ZFS-On-Linux">rebasing of FreeBSD code to <span class="caps">ZFS</span>-on-Linux</a></li> </ul> <p>This will reduce project fragmentation and a concentration of efforts: a strong future for the&nbsp;platform.</p> <h2 id="capacity">Capacity<a class="headerlink text-decoration-none small pl-2" href="#capacity" title="Permanent link">&para;</a></h2> <p>Let&rsquo;s evaluate my raw needs. I need a place to&nbsp;store:</p> <ul> <li>Copies of all external HDDs lying around the&nbsp;house</li> <li>Some entertainment&nbsp;medias</li> <li>Time Machine mount points for macOS&nbsp;backups</li> </ul> <p>Adding all that and rounding up, I end up with at least 8 <span class="caps">TB</span> of immediate storage requirement. With a 20% margin, I&rsquo;ll be <strong>happy with 10 <span class="caps">TB</span></strong>.</p> <h2 id="raid-array"><span class="caps">RAID</span> Array<a class="headerlink text-decoration-none small pl-2" href="#raid-array" title="Permanent link">&para;</a></h2> <p>To provide resiliency, flexibility and allow for future grow, we will rely on a disk array. How should we structure&nbsp;it?</p> <p><span class="caps">RAID</span> 5 or <span class="caps">RAID</span>-Z1 arrays (simple parity) are no longer safe. With drives getting bigger (&gt; 2 <span class="caps">TB</span>), reconstructing an array takes too long. So long the probability of loosing an additional disk is real. Cascading failures is not an&nbsp;option.</p> <p><span class="caps">RAID</span> 10 is a bullet-proof arrangement. But too expensive for our home&nbsp;usage.</p> <p><strong><span class="caps">RAID</span> 6 / <span class="caps">RAID</span>-Z2 is the best compromise</strong> by providing double parity. With 4 drives, we have a similar risk profile as <span class="caps">RAID</span> 10: only half the capacity is available because of redundancy, but we can loose up to two disks. The upside is that any additional drive will contribute its full capacity to the&nbsp;array.</p> <p>Speed, space and safety of different <span class="caps">RAID</span>-Z types are explored in the <a href=""><span class="caps">ZFS</span> Raidz Performance, Capacity and Integrity</a>&nbsp;article.</p> <p>Should we activate <span class="caps">ZFS</span> deduplication feature? No. It consumes too much <span class="caps">RAM</span> and <span class="caps">CPU</span> for minimal gain. Unless you&rsquo;re a company looking to archive multiple years of backups for legal requirements, don&rsquo;t bother with&nbsp;deduplication.</p> <h2 id="drives">Drives<a class="headerlink text-decoration-none small pl-2" href="#drives" title="Permanent link">&para;</a></h2> <p>The first hardware to decide upon should be the drives. This is the core component of a <span class="caps">NAS</span>. Once we have these as a starting point, we can figure out the missing&nbsp;pieces.</p> <h3 id="ssds">SSDs<a class="headerlink text-decoration-none small pl-2" href="#ssds" title="Permanent link">&para;</a></h3> <p>I&rsquo;m not that much interested by their performances. What I&rsquo;m looking for is the peace of mind these bring data-wise, by their virtue of having no mechanical part. And the absence of&nbsp;noise.</p> <p><strong>SSDs are still too expensive</strong> for that use-case. Let&rsquo;s revisit that option in a couple of years as prices gets&nbsp;lower.</p> <h3 id="25-hdds">2.5&rdquo; HDDs<a class="headerlink text-decoration-none small pl-2" href="#25-hdds" title="Permanent link">&para;</a></h3> <p>That was my first choice. While pricier than their 3.5&rdquo; equivalent, I was ready to pay a little premium to reduce the physical space and the&nbsp;noise.</p> <p>Then the &ldquo;<a href=""><span class="caps">SMR</span> scandal</a>&rdquo; broke out. <span class="caps">SMR</span> drives have bad performances and <a href="">artificially increase resilvering time of <span class="caps">RAIDZ</span> array from 17 hours to 10 days</a>!</p> <p>Bad news, all <strong>big 2.5&rdquo; HDDs are SMRs</strong>. I had to revise my plans, forget that option and switch to 3.5&rdquo;&nbsp;drives.</p> <p>Worse, I already ordered a small case (<a href="">SilverStone <span class="caps">SST</span>-<span class="caps">ML05B</span></a>) and all its accessories (fans and cables). This box was now useless as I need more volume to accommodate bulkier 3.5&rdquo; drives. My plans were doomed anyway. If that case had four 2.5&rdquo; slots, you can only cramp there two, maybe three, 15 mm thick&nbsp;drives.</p> <blockquote> <p>Note: <span class="caps">SMR</span> is not bad technology in itself. As long as the software architecture exploiting it is designed to leverage its&nbsp;asymmetry.</p> </blockquote> <h3 id="35-hdds">3.5&rdquo; HDDs<a class="headerlink text-decoration-none small pl-2" href="#35-hdds" title="Permanent link">&para;</a></h3> <p>The most economical option. Here are some resources to hunt down the right disk and avoid&nbsp;SMRs:</p> <ul> <li><a href="">The <span class="caps">HDD</span> Platter Capacity&nbsp;Database</a></li> <li><a href="">List of known <span class="caps">SMR</span>&nbsp;drives</a></li> </ul> <p><span class="caps">TL</span>;<span class="caps">DR</span> safe&nbsp;bets:</p> <ul> <li><a href=""><span class="caps">HGST</span> Ultrastar <span class="caps">HE</span></a>, including shucked&nbsp;ones</li> <li><a href="">Seagate IronWolf</a> / <a href="">IronWolf&nbsp;Pro</a></li> <li><strong><a href="">Toshiba N300</a></strong> / <a href="">X300</a></li> </ul> <p>I chose the later. The <a href="">N300 series</a>&nbsp;has:</p> <ul> <li>Good&nbsp;pricing</li> <li>3 year&nbsp;warranty</li> <li>180 <span class="caps">TB</span>/year&nbsp;workload</li> <li><span class="caps">RV</span>&nbsp;sensors</li> <li>1M hours <span class="caps">MTBF</span></li> <li>A design fit for <span class="caps">NAS</span> and <span class="caps">RAID</span> arrays&nbsp;usage</li> </ul> <p>The purchasing decision was also driven by availability, and the fact it was the first drive announced by its manufacturer to be non-<span class="caps">SMR</span> following the scandal. Which shows how timing and transparency is somewhat critical to get new customers.&nbsp;;)</p> <p>I <strong>went with the 6 <span class="caps">TB</span> models</strong> as they were at the right $/<span class="caps">TB</span> sweet-spot and gave me 12 <span class="caps">TB</span> of useable space. Beware that these drives are loud. Only discovered too late the 6 <span class="caps">TB</span> model is 33 dB while the 12-16 <span class="caps">TB</span> variants are 20 dB (according to specs&nbsp;sheet).</p> <p>I ordered four of them on three different marketplaces, with a couple of days in between to spread the risk of delivery mishandling. This is more superstition than science. These drives are likely to come from the same manufacturing batch given the&nbsp;time-window.</p> <h2 id="case">Case<a class="headerlink text-decoration-none small pl-2" href="#case" title="Permanent link">&para;</a></h2> <p>Went fishing for another case. I increased my budget three times and got a <a href="">SilverStone <span class="caps">SST</span>-<span class="caps">DS380</span></a>:</p> <ul> <li>Plenty of room to grow with its 8 hot-swappable 3.5&rdquo; front&nbsp;bays</li> <li>Mini-<span class="caps">ITX</span> motherboard&nbsp;format</li> <li><span class="caps">SFX</span> <span class="caps">PSU</span> format (already ordered&nbsp;one)</li> <li>Fit in my electrical cabinet (extremely&nbsp;important)</li> </ul> <p>I replaced all its 3 default 120 mm fans by three <a href="">Noctua <span class="caps">NF</span>-<span class="caps">S12A</span> <span class="caps">PWM</span></a>. I choose the S-series over the <a href=""><span class="caps">NF</span>-F12</a> / <a href=""><span class="caps">NF</span>-A12</a> as the latter <a href="">targets radiators</a> and I needed to <a href="">optimize for maximal airflow for case cooling</a>. Also the <span class="caps">NF</span>-<span class="caps">S12A</span> variant is a little quieter than the <a href=""><span class="caps">NF</span>-<span class="caps">S12B</span></a> one. These new fans&nbsp;added:</p> <ul> <li>Absolute&nbsp;silence</li> <li>Variable and dynamic speed, as we moved from 3-pins to 4-pins <span class="caps">PWM</span></li> <li>Reduced <span class="caps">RPM</span>: the basic fans were stuck at constant 1100 <span class="caps">RPM</span>, the new ones allowed the system to lower them to 400~500 <span class="caps">RPM</span> for the same cooling&nbsp;performance</li> </ul> <h2 id="motherboard">Motherboard<a class="headerlink text-decoration-none small pl-2" href="#motherboard" title="Permanent link">&para;</a></h2> <p>Requirements:</p> <ul> <li>Had to be x86-64 for&nbsp;FreeNAS</li> <li>Mini-<span class="caps">ITX</span>, to reduce as much the form&nbsp;factor</li> <li>At least 4 <span class="caps">SATA</span> <span class="caps">III</span>&nbsp;ports</li> </ul> <p>I stumbled upon the <a href="">Gigabyte <span class="caps">GA</span>-J3455-<span class="caps">D3H</span></a> which fits the bill and comes with a pretty fan-less design, thanks to its built-in <a href="">Intel Celeron J3455</a> quad-core (up to 2.3&nbsp;GHz).</p> <p>Some reviews here and there called upon its capricious nature. To play it safe I carefully choose <a href=""><span class="caps">RAM</span> that was certified by the manufacturer</a>, and maxed it out at 16 <span class="caps">GB</span>.</p> <h2 id="realtek-nics-are-shit">Realtek NICs Are Shit<a class="headerlink text-decoration-none small pl-2" href="#realtek-nics-are-shit" title="Permanent link">&para;</a></h2> <p>I tested the setup for a couple of days and everything worked&nbsp;great.</p> <p>But I overlooked one issue: <a href="">Realtek ethernet NICs are shit</a>. It&rsquo;s written everywhere on discussions boards, even on <a href="">FreeNAS hardware requirements</a> page (&ldquo;Intel recommended&rdquo; they said&hellip;). I ignored the warning thinking it was meant for enterprise users looking for&nbsp;performances.</p> <p>Sure enough things went south fast the first time I tried to copy 700 <span class="caps">GB</span> over the network. I left the transfer unattended in the night and woke up with a <span class="caps">NAS</span> nowhere to be found on the <span class="caps">LAN</span>. The Realtek <span class="caps">NIC</span> just froze, bringing the whole motherboard with&nbsp;it.</p> <p>A hard reset later, I found an infinite amount of these in the&nbsp;logs:</p> <div class="highlight"><pre><span></span><code>(...) May 11 13:06:36 freenas kernel: re0: link state changed to DOWN May 11 13:06:36 freenas kernel: re0: link state changed to DOWN May 11 13:06:40 freenas kernel: re0: link state changed to UP May 11 13:06:40 freenas kernel: re0: link state changed to UP May 11 13:18:43 freenas re0: watchdog timeout May 11 13:18:43 freenas kernel: re0: link state changed to DOWN May 11 13:18:43 freenas kernel: re0: link state changed to DOWN May 11 13:18:47 freenas kernel: re0: link state changed to UP May 11 13:18:47 freenas kernel: re0: link state changed to UP May 11 13:25:17 freenas re0: watchdog timeout May 11 13:25:17 freenas kernel: re0: link state changed to DOWN May 11 13:25:17 freenas kernel: re0: link state changed to DOWN May 11 13:25:21 freenas kernel: re0: link state changed to UP May 11 13:25:21 freenas kernel: re0: link state changed to UP May 11 13:28:46 freenas kernel: re0: link state changed to DOWN May 11 13:28:46 freenas kernel: re0: link state changed to DOWN May 11 13:28:49 freenas kernel: re1: link state changed to UP May 11 13:28:49 freenas kernel: re1: link state changed to UP (...) </code></pre></div> <p>I already wasted too much time with hardware. I decided to just throw money at the problem and get a server-grade&nbsp;motherboard.</p> <p>Meanwhile, I resorted to one of the worst hack I ever came up with. Not sure it will even have an effect. I added a cronjob-based watchdog checking connectivity and rebooting the machine if the local router can&rsquo;t be&nbsp;found:</p> <div class="highlight"><pre><span></span><code><span class="gp">$</span> ping -c <span class="m">15</span> -o -t <span class="m">600</span> <span class="m">192</span>.168.0.254 <span class="o">||</span> shutdown -r now </code></pre></div> <h2 id="motherboard-take-2">Motherboard, Take #2<a class="headerlink text-decoration-none small pl-2" href="#motherboard-take-2" title="Permanent link">&para;</a></h2> <p>So I settled on an <a href="">ASRock Rack <span class="caps">E3C246D2I</span></a>:</p> <ul> <li>Intel NICs,&nbsp;finally!</li> <li>8 total <span class="caps">SATA</span> by the way of additional OCuLink&nbsp;port</li> <li><span class="caps">ECC</span>&nbsp;memory</li> <li>PCIe 3.0&nbsp;x16</li> <li>3 <span class="caps">PWM</span> fan (<span class="caps">CPU</span> + front + rear) and thermal sensor header for finer (read quieter)&nbsp;cooling</li> <li>Dedicated <span class="caps">IPMI</span>/<span class="caps">BMC</span> <span class="caps">LAN</span> ethernet socket, with non-<span class="caps">JAVA</span> <span class="caps">UI</span></li> <li>All <span class="caps">USB</span> 3.1 ports are Gen2, not Gen1, so no time wasted worrying on which is&nbsp;which</li> <li>No unnecessary multimedia ports (no audio, no Display Port, no <span class="caps">HDMI</span>)</li> </ul> <h2 id="cooling">Cooling<a class="headerlink text-decoration-none small pl-2" href="#cooling" title="Permanent link">&para;</a></h2> <p>We&rsquo;re no longer fan-less, so I searched for the most silent cooling I could ever found. I went straight to Noctua for their reliability, performance and&nbsp;quieteness.</p> <p>I got the <a href="">Noctua <span class="caps">NH</span>-L9i</a>, for <span class="caps">LGA</span> 1151 sockets CPUs and 4-pins <span class="caps">PWM</span>. Which also perfectly fit the low-profile requirement of my case (&lt; 57&nbsp;mm).</p> <p>Surprisingly the black version was cheaper than <a href="">the regular, brown variant</a>.</p> <h2 id="cpu">Cpu<a class="headerlink text-decoration-none small pl-2" href="#cpu" title="Permanent link">&para;</a></h2> <p>In this section I&rsquo;ve cross-referenced <a href="">all CPUs supported by the motherboard</a> with the <a href="">cooler compatibility list</a>:</p> <p><img alt="" src="/uploads/2020/cpu-compatibility-asrock-e3c246d2i-noctua-nh-l9i.png"></p> <p>Source: <a href="../uploads/2020/cpu-compatibility-asrock-e3c246d2i-noctua-nh-l9i.numbers"><code>cpu-compatibility-asrock-e3c246d2i-noctua-nh-l9i.numbers</code></a></p> <p>All CPUs featured the matrix above shares these&nbsp;specs:</p> <ul> <li>Intel x86-64, 3<sup>rd</sup> generation enhanced 14nm++&nbsp;process</li> <li>Socket <span class="caps">LGA</span>-1151</li> <li><a href=";ASIN=B009VCAJ7W"><span class="caps">TDP</span> &lt;= 65 W to keep cooler within&nbsp;limits</a></li> <li><span class="caps">SMM</span>, <span class="caps">FPU</span>, <span class="caps">NX</span>, <span class="caps">MMX</span>, <span class="caps">SSE</span>, <span class="caps">SSE2</span>, <span class="caps">SSE3</span>, <span class="caps">SSSE3</span>, <span class="caps">SSE4</span>.1, <span class="caps">SSE4</span>.2, <span class="caps">AES</span> (encryption boost), <span class="caps">AVX</span>, <span class="caps">FMA3</span>, and <span class="caps">AVX2</span></li> <li><span class="caps">VT</span>-x, <span class="caps">VT</span>-d</li> <li>Dual-channel&nbsp;memory</li> </ul> <p>But most Coffee Lake-R CPUs are missing from Noctua list (tagged as <code>unreferenced</code>). I have no intention of overlocking, so we can assume they&rsquo;re still safe with our cooler, based on <span class="caps">TDP</span> and filiation to their Coffee Lake S&nbsp;ancestors.</p> <p>All in all, low power 35 W models (T suffix) are safe&nbsp;choice.</p> <p>The no-suffix and F variants are great, and their <span class="caps">TDP</span> are perfectly in range within the cooler capabilities. If we find out 65 W to be too much, we can still artificially put hard-limits to the <span class="caps">CPU</span> in <span class="caps">BIOS</span>.</p> <blockquote> <p>Note: there&rsquo;s an elusive candidate in the name of the <a href="">Core i3-9320, based on the i3-9300</a>, with slightly better turbo. Unfortunately it is not referenced in the motherboard compatibility&nbsp;list.</p> </blockquote> <p>My final shortlist&nbsp;was:</p> <ol> <li><a href="">Core i3-9300</a> to max out cache and&nbsp;frequency</li> <li>Core i3-9300T same as above with lower frequency but safe&nbsp;thermals</li> <li>Core i3-9100 or i3-9100T as a nice&nbsp;fallback</li> </ol> <p>I ended up with a <strong>Core i3-9300T</strong>, probably overpriced at the time of purchase, as it was harder and harder to find any of the reference above in retail (seems like Intel prioritizing higher SKUs to increase&nbsp;margins).</p> <blockquote> <p>I also wasted time trying to get a second-hand <a href="">Intel Celeron <span class="caps">G4900T</span></a> (Coffee Lake-S, 2 cores, 2.9GHz). It was <a href="">stolen during transit</a>. By the time I recovered it I decided upon a Core&nbsp;i3-9300T.</p> </blockquote> <h2 id="bill-of-materials">Bill of Materials<a class="headerlink text-decoration-none small pl-2" href="#bill-of-materials" title="Permanent link">&para;</a></h2> <h3 id="initial-plans">Initial Plans<a class="headerlink text-decoration-none small pl-2" href="#initial-plans" title="Permanent link">&para;</a></h3> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th align="right">Quantity</th> <th align="right">Total (excl. shipping)</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>2.5&rdquo; <span class="caps">HDD</span></td> <td><a href="">Seagate BarraCuda Mobile - 4 <span class="caps">TB</span></a></td> <td align="right">4</td> <td align="right">€759.76</td> <td>Abandonned because of <span class="caps">SMR</span>.</td> </tr> <tr> <td>Small case</td> <td><a href="">SilverStone <span class="caps">SST</span>-<span class="caps">ML05B</span></a></td> <td align="right">1</td> <td align="right">€52.84</td> <td>Refunded: not enough 3.5&rdquo; slots.</td> </tr> <tr> <td>Fans for case</td> <td><a href="">Noctua <span class="caps">NF</span>-A8 <span class="caps">PWM</span></a>, 4 pins, 80 mm</td> <td align="right">2</td> <td align="right">€31.80</td> <td>Refunded: only fit the case above.</td> </tr> <tr> <td>Mini-<span class="caps">ITX</span> motherboard</td> <td><a href="">Gigabyte <span class="caps">GA</span>-J3455-<span class="caps">D3H</span></a></td> <td align="right">1</td> <td align="right">€98.54</td> <td>Refunded because of Realtek NICs.</td> </tr> <tr> <td><span class="caps">RAM</span></td> <td><a href="">Corsair <span class="caps">CMSO16GX3M2C1600C11</span></a> <span class="caps">16GB</span> (2x8GB) <span class="caps">DDR3</span> <span class="caps">SODIMM</span> 1600 MHz</td> <td align="right">1</td> <td align="right">€75.76</td> <td>Refunded: only fit for motherboard above.</td> </tr> <tr> <td>Power supply</td> <td><a href="">Corsair <span class="caps">SF450</span></a>, modular, 450 W, 80+ Platinum</td> <td align="right">1</td> <td align="right">€119.90</td> <td>Reused.</td> </tr> <tr> <td>Flash drive</td> <td><a href="">SanDisk Ultra Fit 16 <span class="caps">GB</span></a>, <span class="caps">USB</span> 3.1</td> <td align="right">1</td> <td align="right">€9.59</td> <td>Reused.</td> </tr> <tr> <td><span class="caps">SATA</span> cable</td> <td><a href=""><span class="caps">SATA</span> 3 - 90° plugs</a></td> <td align="right">6</td> <td align="right">€15.56</td> <td>Refunded: 90° plugs not fit for newer case.</td> </tr> <tr> <td></td> <td></td> <td align="right"><strong>Total</strong></td> <td align="right"><strong>€1163.75</strong></td> <td></td> </tr> </tbody> </table> <h3 id="final-configuration">Final Configuration<a class="headerlink text-decoration-none small pl-2" href="#final-configuration" title="Permanent link">&para;</a></h3> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th align="right">Quantity</th> <th align="right">Total (excl. shipping)</th> <th align="right">Shipping</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>3.5&rdquo; <span class="caps">HDD</span></td> <td><a href="">Toshiba N300 - 6 <span class="caps">GB</span></a></td> <td align="right">4</td> <td align="right">€696.89</td> <td align="right">€12.84</td> <td>Unit price: €168.30~€184.88</td> </tr> <tr> <td>Bigger case</td> <td><a href="">SilverStone <span class="caps">SST</span>-<span class="caps">DS380</span></a></td> <td align="right">1</td> <td align="right">€145.18</td> <td align="right">€17.50</td> <td></td> </tr> <tr> <td>Case fans</td> <td><a href="">Noctua <span class="caps">NF</span>-<span class="caps">S12A</span> <span class="caps">PWM</span></a></td> <td align="right">3</td> <td align="right">€59.70</td> <td align="right">€3.90</td> <td></td> </tr> <tr> <td>Mini-<span class="caps">ITX</span> motherboard</td> <td><a href="">ASRock Rack <span class="caps">E3C246D2I</span></a></td> <td align="right">1</td> <td align="right">€324.84</td> <td align="right">€21.05</td> <td><a href="">Upgrade both <span class="caps">BMC</span> and <span class="caps">BIOS</span></a>.</td> </tr> <tr> <td><span class="caps">CPU</span></td> <td><a href="">Intel Core i3-9300T</a> Coffee Lake-R, 4 cores, 3.2GHz, 8MiB L3 cache</td> <td align="right">1</td> <td align="right">€174.90</td> <td align="right">€9.98</td> <td></td> </tr> <tr> <td><span class="caps">CPU</span> cooler</td> <td><a href="">Noctua <span class="caps">NH</span>-L9i</a></td> <td align="right">1</td> <td align="right">€49.90</td> <td align="right">€0.99</td> <td>Shipped with thermal paste.</td> </tr> <tr> <td><span class="caps">RAM</span></td> <td><a href="">Kingston <span class="caps">KSM26ES8</span>/<span class="caps">8ME</span></a> <span class="caps">8GB</span> 2666MHz <span class="caps">DDR4</span> <span class="caps">ECC</span> <span class="caps">CL19</span></td> <td align="right">2</td> <td align="right">€137.76</td> <td align="right"></td> <td></td> </tr> <tr> <td>Power supply</td> <td><a href="">Corsair <span class="caps">SF450</span></a>, modular, 450 W, 80+ Platinum</td> <td align="right">1</td> <td align="right">€119.90</td> <td align="right"></td> <td>Check serial number: this <a href="">model has been recalled</a>.</td> </tr> <tr> <td>Flash drive</td> <td><a href="">SanDisk Ultra Fit 16 <span class="caps">GB</span></a>, <span class="caps">USB</span> 3.1</td> <td align="right">1</td> <td align="right">€9.59</td> <td align="right"></td> <td></td> </tr> <tr> <td><span class="caps">SATA</span> cable</td> <td><a href=""><span class="caps">SATA</span> 3 - straight plug</a></td> <td align="right">3</td> <td align="right">€6.99</td> <td align="right"></td> <td></td> </tr> <tr> <td>OCuLink cable</td> <td><a href="">Supermicro <span class="caps">CBL</span>-<span class="caps">SAST</span>-0933</a> OCuLink to 4 <span class="caps">SATA</span></td> <td align="right">1</td> <td align="right">€29.76</td> <td align="right">€12.01</td> <td></td> </tr> <tr> <td>Probe</td> <td><a href="">10K thermal wire sensor</a></td> <td align="right">1</td> <td align="right">€0.25</td> <td align="right">€1.16</td> <td>Attached to HDDs.</td> </tr> <tr> <td></td> <td></td> <td align="right"><strong>Total</strong></td> <td align="right"><strong>€1755.66</strong></td> <td align="right">€79.43</td> <td></td> </tr> </tbody> </table> <p>Switching from 2.5&rdquo; to 3.5&rdquo; increased the budget by 50%, but I&nbsp;got:</p> <ul> <li>50% more useable space (<span class="caps">8TB</span> → 12 <span class="caps">TB</span>)</li> <li>Faster disks (5400 rpm → 7200&nbsp;rpm)</li> <li>Twice <span class="caps">SATA</span> slots (4 →&nbsp;8)</li> <li>Practical front-loading bays for up to 8&nbsp;disks</li> <li><span class="caps">ECC</span> <span class="caps">RAM</span></li> <li>Intel&nbsp;NICs</li> <li>A more powerful <span class="caps">CPU</span>:<ul> <li>Newer Coffee Lake R architecture with some in-silicon Meldown&nbsp;mitigations</li> <li>10% more base speed (2.9 GHz → 3.2&nbsp;GHz)</li> <li>3.8 GHz turbo boost ability (65%&nbsp;faster)</li> <li>4x L3&nbsp;cache</li> </ul> </li> </ul> <p>My attempt at a 2.5&rdquo; disk home <span class="caps">NAS</span> was expensive. The <span class="caps">TCO</span> of the whole project is way too high if you add the frustration, fallbacks, and time spend browsing forums, gathering reviews, returning and getting refunds. All because of bad purchase decisions and poor&nbsp;planning.</p> <p>From an economical point of view, a <span class="caps">NAS</span> at home is a stupid project. I should have just pushed all data to a bucket in either <span class="caps">AWS</span>, <span class="caps">GCP</span> or Azure and call it a day. So why bother? Let&rsquo;s say this is a personal reminder of the incentives and economics of running a cloud&nbsp;business.</p> <h2 id="upgrade-path">Upgrade Path<a class="headerlink text-decoration-none small pl-2" href="#upgrade-path" title="Permanent link">&para;</a></h2> <p>This time we&rsquo;ll try to plan ahead future hardware&nbsp;purchase.</p> <h3 id="storage">Storage<a class="headerlink text-decoration-none small pl-2" href="#storage" title="Permanent link">&para;</a></h3> <p>Need more space? We have a couple of&nbsp;options:</p> <ol> <li> <p><strong>Buy new 6 <span class="caps">TB</span> drives</strong> to expand the <span class="caps">RAIDZ</span> array. Plain and simple. The <span class="caps">NAS</span> has been designed this way. Use it. The current motherboard support up to 8 <span class="caps">SATA</span> drives, thanks to our pre-purchased OCuLink cable. And the case already has 4 free 3.5&rdquo;&nbsp;bays.</p> </li> <li> <p>An alternative consist in upgrading the base capacity of each disk. Just <strong>purchase 4 new identical drive, all bigger than 6 <span class="caps">TB</span></strong>. Swap from the active array one, and one only, old disk with a brand new, bigger disk. Resilver. Repeat 4 times. Then grow the <span class="caps">ZFS</span> vdev. You can now sell the old 6 <span class="caps">TB</span> drives. This solution might be more future-proof depending on the $/<span class="caps">TB</span> sweet-spot and age of the <span class="caps">6TB</span>&nbsp;disk.</p> </li> </ol> <h3 id="drives_1">Drives<a class="headerlink text-decoration-none small pl-2" href="#drives_1" title="Permanent link">&para;</a></h3> <ul> <li> <p>Revisit the 2.5&rdquo; form-factor. Mobile non-<span class="caps">SMR</span> HDDs might get bigger but that&rsquo;s unlikely. The industry is kind of stuck with current technology and the way forward is&nbsp;SSDs. </p> </li> <li> <p>So <strong>keep watching the $/<span class="caps">TB</span> economics of SSDs</strong>. In a couple of years, if prices gets reasonable for a home <span class="caps">NAS</span> setup, it might be time to get rid of spinning&nbsp;rust.</p> </li> </ul> <h3 id="cooler">Cooler<a class="headerlink text-decoration-none small pl-2" href="#cooler" title="Permanent link">&para;</a></h3> <p>In my research I discovered stronger cooler, capable of handling higher base <span class="caps">TDP</span>. Here a list of potential&nbsp;upgrades:</p> <table> <thead> <tr> <th>Part</th> <th>Model</th> <th align="right">Max <span class="caps">TDP</span></th> <th align="right">Noise Level</th> <th align="right">Air Flow</th> <th align="right">Air Pressure</th> <th align="right">Price</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>Heatsink + Fan</td> <td><a href="">Noctua <span class="caps">NH</span>-L9i</a></td> <td align="right"><a href="">65 W</a></td> <td align="right">23.6 dBA</td> <td align="right">57.5 m³/h</td> <td align="right"></td> <td align="right">€49.90</td> <td>Current cooler for reference. Shipped with paste.</td> </tr> <tr> <td>Heatsink + Fan</td> <td><a href="">Cryorig C7 Cu</a></td> <td align="right"><a href="">115 W</a></td> <td align="right">30 dBA</td> <td align="right">40.5 <span class="caps">CFM</span></td> <td align="right">2.8 mmH2O</td> <td align="right"></td> <td>Shipped with paste.</td> </tr> <tr> <td>Heatsink + Fan</td> <td><a href="">Cryorig C7 G</a></td> <td align="right"><a href="">125 W</a></td> <td align="right">30 dBA</td> <td align="right">40.5 <span class="caps">CFM</span></td> <td align="right">2.8 mmH2O</td> <td align="right"></td> <td>Shipped with paste.</td> </tr> </tbody> </table> <p>All coolers above&nbsp;are:</p> <ul> <li>4-pins <span class="caps">PWM</span></li> <li>Height lower than 57&nbsp;mm</li> </ul> <p>Airflow can be&nbsp;converted:</p> <ul> <li>1 m³/h is ~0.589 <span class="caps">CFM</span></li> <li>1 <span class="caps">CFM</span> is ~1.699&nbsp;m³/h</li> </ul> <p>Other candidates to check&nbsp;dimensions:</p> <ul> <li><a href="">Alpenföhn® Black&nbsp;Ridge</a></li> <li><a href="">ThermalRight <span class="caps">AXP</span>-100&nbsp;C65</a></li> <li><a href="">ThermalRight <span class="caps">AXP</span>-100-Full&nbsp;Copper</a></li> </ul> <p>Upgrading the cooler will open-up more options for the CPUs&nbsp;below.</p> <h3 id="cpu_1">Cpu<a class="headerlink text-decoration-none small pl-2" href="#cpu_1" title="Permanent link">&para;</a></h3> <p>I don&rsquo;t need more power yet. Unless I&rsquo;ll get to run a couple of virtual machines on the <span class="caps">NAS</span>.</p> <p>If this upgrade is targetting <span class="caps">VM</span> usages, I&rsquo;ll skip the F-variants altogether. <a href=";px=Bhyve-GPU-Passthrough-2019"><span class="caps">GPU</span> passthrough is still highly experimental in FreeBSD&rsquo;s Bhyve hypervisor</a>. But there&rsquo;s still <a href="">some hope</a> to have the <span class="caps">UHD</span> 630 iGPU supported in the far future, and provide accelerated video encoding and decoding to&nbsp;VMs.</p> <p>So we&rsquo;re left with the option of revisiting my <span class="caps">CPU</span> matrix against the new coolers above, and add all the more powerfull Xeon references supported by the&nbsp;motherboard.</p> <h3 id="ram">Ram<a class="headerlink text-decoration-none small pl-2" href="#ram" title="Permanent link">&para;</a></h3> <p>If the motherboard support <span class="caps">DDR4</span>-2666, my <span class="caps">CPU</span> matrix revealed the processors supporting that speed have no <span class="caps">ECC</span> support. It&rsquo;s pointless to pursue faster <span class="caps">RAM</span> and sacrifice data integrity. We&rsquo;ll <strong>keep at 2400 MHz, and purchase <span class="caps">DDR4</span>-2666 if cheaper</strong>.</p> <p>The current 16 <span class="caps">GB</span> are plenty, even with 8 drives in the <span class="caps">RAIDZ</span> array. To support VMs, 32 <span class="caps">GB</span> will be nice to have. Maxing it out to 64 <span class="caps">GB</span> is a luxury. The upgrade will depends on&nbsp;price.</p> <h3 id="network">Network<a class="headerlink text-decoration-none small pl-2" href="#network" title="Permanent link">&para;</a></h3> <p>I dedicated an <a href="./">article on bringing 10G at home</a>.</p> <h3 id="motherboard_1">Motherboard<a class="headerlink text-decoration-none small pl-2" href="#motherboard_1" title="Permanent link">&para;</a></h3> <p>The one I ended up with is good enough for the years to come. Only a dramatic change in drive interfaces can justify the upgrade. Once SSDs will makes sense, I&rsquo;ll probably have to upgrade from <span class="caps">SATA</span> to an array of M.2, MVNe or whatever the right standard will be for these drives in the&nbsp;future.</p> <p>At which point these extra-features might be&nbsp;nice-to-have:</p> <ul> <li>Integrated 10G NICs (latest model being <a href="">Intel X710 controller, supported by FreeBSD</a>), with a preference for <span class="caps">SFP</span>+ port to use a <span class="caps">DAC</span>&nbsp;cable</li> <li>A complete fan-less&nbsp;design</li> <li>The upcoming <span class="caps">AMD</span>&rsquo;s Zen 3 architecture (more cores and L3 cache, better&nbsp;price/performance)</li> </ul> <p>Potential candidates so&nbsp;far:</p> <ul> <li><a href="">ASRock Rack <span class="caps">X570D4I</span>-2T</a></li> <li><a href="">ASRock Rack <span class="caps">EPYC3251D4I</span>-2T</a></li> <li>Keep looking for new models at <a href="">SuperMicro motherboard&nbsp;matrix</a></li> </ul> <h3 id="zfs"><span class="caps">ZFS</span><a class="headerlink text-decoration-none small pl-2" href="#zfs" title="Permanent link">&para;</a></h3> <p>Lots of <span class="caps">ZFS</span> features are on the&nbsp;way:</p> <ul> <li><a href="">dRAID with dual parity</a>. It&rsquo;s like <span class="caps">RAIDZ2</span>, but with better workload distribution across drives on&nbsp;rebuilds.</li> <li><span class="caps">ZFS</span> native dataset encryption (should come with OpenZFS&nbsp;2.0).</li> <li>macOS support seems to being worked on for OpenZFS 3.0. Maybe this opens the way to directly mount remote volumes, bypassing Samba shares&nbsp;altogether.</li> </ul>Transitionning From Software Engineering to Management2020-02-10T00:00:00+01:002020-02-10T00:00:00+01:00Kevin,2020-02-10:/2020/02/engineering-to-management-transition/<p>This interview was <a href="">originally published</a> for the <a href="">Developer to Manager</a> project, which is collecting experience and advice from engineering managers about their transition from software&nbsp;development.</p> <hr> <h2 id="hello-whats-your-background-and-what-do-you-do">Hello! What&rsquo;s Your Background and What Do You Do?<a class="headerlink text-decoration-none small pl-2" href="#hello-whats-your-background-and-what-do-you-do" title="Permanent link">&para;</a></h2> <p>My name is Kevin Deldycke and I was the <span class="caps">VP</span> of Billing, Payment and …</p><p>This interview was <a href="">originally published</a> for the <a href="">Developer to Manager</a> project, which is collecting experience and advice from engineering managers about their transition from software&nbsp;development.</p> <hr> <h2 id="hello-whats-your-background-and-what-do-you-do">Hello! What&rsquo;s Your Background and What Do You Do?<a class="headerlink text-decoration-none small pl-2" href="#hello-whats-your-background-and-what-do-you-do" title="Permanent link">&para;</a></h2> <p>My name is Kevin Deldycke and I was the <span class="caps">VP</span> of Billing, Payment and <a href=""><span class="caps">IAM</span></a> at Scaleway, a cloud infrastructure provider. During my time at Scaleway, I led an engineering team of 11, and was one of the people in charge of the architecture, implementation and operation of the whole stack. If Compute, Storage and Network are the three pillars of the product ecosystem, then we, the Billing, Payment and <span class="caps">IAM</span> team are the core components of the platform. Our motto? Make complexity trivial from the outside, while streamlining the&nbsp;business.</p> <p><img alt="" src="/uploads/2020/001-billing-payment-iam-team.jpeg"></p> <p>The team I formed. I am the overly happy guy with the blue shirt at the&nbsp;center.</p> <p>My story starts in 2003. The world was different back then. The dot-com bubble had just popped up and was still casting its shadow on the industry. After five years studying for a master&rsquo;s degree in software engineering, the safest path was to join a big, traditional <span class="caps">IT</span> consultancy firm. I did the opposite, and joined a risky startup, with values I believed in: the philosophy of free/libre&nbsp;software.</p> <p>That&rsquo;s the first lesson here as a manager in the making: developers are not just looking for an occupational programming activity. They are seeking a culture, shared by like-minded peers. Guess what? <strong>As a manager, it is your role to create the conditions for a sustainable culture to emerge and&nbsp;persist.</strong></p> <p>So that&rsquo;s how I started my 15-year journey as a backend engineer, in what was an <a href="">esoteric language at the time: Python</a>. I got the chance to work with smart professionals in various industries: brick-and-mortar retail, transportation, food processing, financial, energy, &hellip; This wide spectrum of projects made me realize that organizations and management styles are diverse, and mostly situational. But there was a pattern. The <strong>happiest people I encountered were the ones given autonomy, mastery and purpose</strong>, echoing <a href="">Daniel Pink&rsquo;s thesis on what motivates us</a>.</p> <p>The majority will not breed the next unicorn; we statistically have a better likelihood of dealing with enterprise frameworks. That&rsquo;s fine and can really be valuable in a career. But once in a while ambitious startups and business software intersect. Enter <a href="">Scaleway</a>.</p> <h2 id="how-was-your-transition-from-software-development-to-management-like">How Was Your Transition From Software Development to Management Like?<a class="headerlink text-decoration-none small pl-2" href="#how-was-your-transition-from-software-development-to-management-like" title="Permanent link">&para;</a></h2> <p>In 2013 I joined a <a href="">brand new R&amp;D team</a> as a founding engineer. We worked in total secrecy on what would become a cloud computing ecosystem, akin to <span class="caps">AWS</span>, <span class="caps">GCP</span> or&nbsp;Azure.</p> <p><img alt="" src="/uploads/2020/002-scaleway-founding-team.jpeg"></p> <p>Scaleway’s founding team (I was behind the camera). All engineers <span class="amp">&amp;</span> tech leads, no&nbsp;managers.</p> <p>After a <a href="">successful launch</a> and incremental iterations, we got a huge investment to <a href="">scale the whole thing up</a> - both the platform itself and the organization around it. We not only had to grow, but also merge with the 20-year old established hosting company we were attached to. It was explosive, but in a good sense; it put new life in our&nbsp;venture.</p> <p>The original founding group was disbanded and people were spread out in a massive reorganization. It&rsquo;s like the job of the vast majority of employees changed overnight. Everybody was busy structuring the company, planning products and recruiting. There was so much to do. I loved the&nbsp;frenzy.</p> <p>The system I designed and produced alone had to scale as well. And it had to absorb some existing legacy services. All the while keeping the business up and running, making sure those damn millions of euros were flowing right back into the bank account. On time. We&rsquo;d better have a resilient team for that piece of critical pipeline. Right? So I was <strong>naturally propelled as manager of a brand new team, without fanfare, murky negotiations or pompous ceremony</strong>. I guess I was the only one with the vision to see all the fun (both technical and functional) there was in these contraptions!&nbsp;:)</p> <p>I had no one to manage yet. But I felt empowered by a mission of bringing meaning and purpose in a place where everything had to be invented. We had the money, we were in the right industry, we already had customers, and competent engineering talents. <strong>The missing pieces were management and leadership. And I was just empowered to positively influence all that up at my humble&nbsp;level.</strong></p> <p><img alt="" src="/uploads/2020/003-last-photo-before-transitioning.jpeg"></p> <p>The last photo of the initial R&amp;D team, right before it was disbanded and I (on the far left) became&nbsp;manager.</p> <h2 id="what-does-your-day-to-day-work-look-like-and-what-motivates-you-to-do-it-every-day">What Does Your Day-To-Day Work Look Like, and What Motivates You to Do It Every Day?<a class="headerlink text-decoration-none small pl-2" href="#what-does-your-day-to-day-work-look-like-and-what-motivates-you-to-do-it-every-day" title="Permanent link">&para;</a></h2> <p>The first phase of my activities as a front-line manager, was mainly inward-facing to the team. Because there was none&nbsp;yet.</p> <p>So here I went for a&nbsp;year:</p> <ul> <li>Recruiting a team of six&nbsp;developers;</li> <li>Training them to take over engineering and operational duties (including my former&nbsp;ones);</li> <li>Mentoring some into Tech Leads and Product&nbsp;Owners;</li> <li>Trying to make myself optional and reduce the&nbsp;bus-factor;</li> <li>Of course loads of project management was involved too:<ul> <li>Organizing and planning upcoming&nbsp;sprints;</li> <li>Tracking delivered&nbsp;features;</li> <li>Writing newsletters to share achievements and&nbsp;milestones;</li> <li>Keeping in sync and aligns with executive&rsquo;s priorities and fellow&nbsp;managers;</li> </ul> </li> <li>Writing the 3-year roadmap and business&nbsp;plan;</li> <li>Designing a full revaluation process to assess and document engineer&rsquo;s level and pay&nbsp;grade;</li> <li>Helping other managers interview candidates and review profiles in hiring&nbsp;committees;</li> <li>And all the other management tasks:<ul> <li>Leave of absences&nbsp;planning,</li> <li>Frequent 1:1 with&nbsp;everyone,</li> <li>Trial period&nbsp;assessments,</li> <li>Experimenting with remote&nbsp;work,</li> <li>Yearly objectives and target&nbsp;reviews,</li> <li>Bonus and awards&nbsp;distribution,</li> <li>Weekly travel to the regional office (the team was split in&nbsp;two).</li> </ul> </li> </ul> <p>It was not a place for routine, and the day-to-day was a varying mix of all of the things above, depending on the yearly/quarterly/monthly/sprint cycle we were&nbsp;in.</p> <p>Then I was promoted to Vice-President, a fancy title, meaning I was now reporting directly to the C-level executives. The job came with the assignment of adding a layer of management between me and the engineers. It was the only way to get ready for the next stage of growth. I was an <span class="caps">OK</span> manager, I now had to find a better one to replace me. I was lucky to find <a href="">Alexandre</a>. He did, and still does, a stellar job taking care of the team I&nbsp;assembled.</p> <p><img alt="" src="/uploads/2020/004-inheriting-the-team.jpeg"></p> <p>Alex, the manager I managed, inherited a sufficiently hydrated&nbsp;team.</p> <p>Then I started a second phase that was more outward-facing, in which I spent my time on coaching and managing managers, and doubling the team&nbsp;headcount.</p> <p>Where did I get the drive to keep going? <strong>Good people to work with.</strong> I know it sounds dull, but you can&rsquo;t fake&nbsp;it.</p> <h2 id="what-are-the-biggest-challenges-youve-faced-so-far-what-did-you-do-to-overcome-them">What Are the Biggest Challenges You&rsquo;ve Faced So Far? What Did You Do to Overcome Them?<a class="headerlink text-decoration-none small pl-2" href="#what-are-the-biggest-challenges-youve-faced-so-far-what-did-you-do-to-overcome-them" title="Permanent link">&para;</a></h2> <p>One of the biggest challenges, and the first you&rsquo;ll encounter right after your transition, is about managing your own psyche. Before you can improve on that, you’ll have to accept that indeed, <strong>we&rsquo;re all just big bags of complicated, messy emotions</strong>. To make progress you will need to tune your response to others. And essentially to yourself. Be prepared, because it&rsquo;s going to be an energy&nbsp;drain.</p> <p><img alt="" src="/uploads/2020/005-tired-manager.jpeg"></p> <p>A manager with no one to manage yet, trying hard to recruit in a competitive&nbsp;market.</p> <p>The only bearable way to live through&nbsp;that:</p> <ul> <li>Turn off mails and Slack at&nbsp;home,</li> <li>Plan frequent and short&nbsp;holidays,</li> <li>Come out in search of regular outdoor&nbsp;activities.</li> </ul> <p>In other words: <strong>find balance out of work, or you&rsquo;re not going to last long</strong>.</p> <h2 id="what-has-been-the-biggest-surprise-so-far-something-you-didnt-expect">What Has Been the Biggest Surprise So Far? Something You Didn&rsquo;t Expect?<a class="headerlink text-decoration-none small pl-2" href="#what-has-been-the-biggest-surprise-so-far-something-you-didnt-expect" title="Permanent link">&para;</a></h2> <p>The first people I hired were key to our long-term success. I was lucky to have met proficient, dedicated and honest engineers. They cemented the backbone of our culture. I couldn&rsquo;t have made it without them. <strong>It was not <em>my</em> team. It was <em>our</em>&nbsp;team.</strong></p> <p>The biggest surprise is that <strong>once you get the right core of a team, the culture takes care of itself</strong>.</p> <p><img alt="" src="/uploads/2020/006-seeding-the-culture.jpeg"></p> <p>The first members of the team, actively redecorating the brand new regional office. They all supplied the seed of our&nbsp;culture.</p> <p>The payoff came a year later. A bunch of developers outside my authority were met with a case of bad management. It is sad, but in a company large enough, shit happens. They all requested to be moved elsewhere in the org-chart. And asked explicitly to join us, seeking peace, stability and a drama-free environment. You can&rsquo;t dream of a better validation: we were a healthy&nbsp;team.</p> <h2 id="whats-the-best-advice-youve-received-about-being-a-manager">What&rsquo;s the Best Advice You&rsquo;ve Received About Being a Manager?<a class="headerlink text-decoration-none small pl-2" href="#whats-the-best-advice-youve-received-about-being-a-manager" title="Permanent link">&para;</a></h2> <p>I was told that starting from now on, I shouldn&rsquo;t code anymore. That was an important clue. But it goes further. <strong>You&rsquo;re no longer the decision-maker on your technical stack. Your team is responsible for it.</strong> You&rsquo;re barely ratifying these&nbsp;decisions.</p> <p>You probably used to make these decisions as a developer. And now that you switched positions, you have to take a step back, and have to accept the consensus reached by others. All the while being liable for the result and its impact. Now trying to reconcile the managerial accountability (which might have bad consequences for you), from the team responsibility (allowing them to move on), is borderline schizophrenia for ex-engineers. But you have no choice and have to learn to navigate in that constant&nbsp;duality.</p> <h2 id="what-do-you-tell-developers-who-are-considering-making-the-switch-or-new-to-the-role">What Do You Tell Developers Who Are Considering Making the Switch or New to the Role?<a class="headerlink text-decoration-none small pl-2" href="#what-do-you-tell-developers-who-are-considering-making-the-switch-or-new-to-the-role" title="Permanent link">&para;</a></h2> <p>Something along the lines&nbsp;of:</p> <ul> <li>It is not a promotion: it is a change in&nbsp;career.</li> <li>The expectations are&nbsp;different.</li> <li>Reward will come from a place completely different from where you&rsquo;ll put your effort&nbsp;to.</li> <li>The granularity of your calendar will be reduced from half-day to the hour (half-hour,&nbsp;even).</li> </ul> <p>Also, you&rsquo;ll have to play the meta-game&nbsp;of:</p> <ul> <li>Filing&nbsp;paperwork,</li> <li>Attend&nbsp;meetings,</li> <li>Repeat yourself a&nbsp;lot,</li> <li>Use corporate tools (both apps and&nbsp;process).</li> </ul> <p>All that work is not vain if you do it with the one noble goal of making your team&nbsp;productive.</p> <p>The greatest effect you can prompt is providing them with the time and mental space they need to focus on things they do best: come up with outstanding software for the success of the&nbsp;business.</p> <p>The final lesson I&rsquo;ll share is the hardest. The higher you are in the hierarchy, the more likely you are going to step down for situations unrelated to your activity, or for things completely out of your direct control. i.e. things tend to get more political towards the&nbsp;top.</p> <p><img alt="" src="/uploads/2020/007-quarterly-priority-planning.jpeg"></p> <p>Final touch on our plans for the next quarter. Arbitrating priorities when every product depends on you is&nbsp;excruciating.</p> <p>I&rsquo;m happy our team ended up strong, demonstrating its capacity and functioning without me. Making myself obsolete, and still having the team perpetuating the values defining who we are, makes me proud of what we accomplished&nbsp;together.</p> <p>The ultimate achievement during the last 24 months as a manager: nobody ever resigned during my term, while the rest of the company had the turnover expected (and planned for) of a high velocity/fast-paced innovating tech&nbsp;company.</p> <p>My stint at Scaleway is foundational. I never stayed that long in any company before. It was the right opportunity at the right time. I&rsquo;ll be forever grateful for the possibility in these past six years to not only build a crucial system from scratch, but also hire a team to take over it and bring the whole to the next&nbsp;level.</p> <p>My transition is complete. Coding is no longer the most impactful thing I can bring to the table. Enabling engineers and making them feel productive&nbsp;is.</p> <h2 id="final-call-to-action-where-can-we-go-to-learn-more-about-you">Final Call to Action! Where Can We Go to Learn More About You?<a class="headerlink text-decoration-none small pl-2" href="#final-call-to-action-where-can-we-go-to-learn-more-about-you" title="Permanent link">&para;</a></h2> <p>I told you earlier I was super-geeky about the domains I operated right? Before going on to my next adventure, I took the time to summarize that knowledge. Here are the repositories consolidating all the stuff I gathered in the&nbsp;field:</p> <ul> <li><a href="">Awesome Engineering Team&nbsp;Management</a></li> <li><a href="">Awesome Billing and&nbsp;Payment</a></li> <li><a href="">Awesome <span class="caps">IAM</span></a></li> </ul> <p>Feel free to send PRs!&nbsp;:)</p>Unlocking, Rooting and Custom Recovery for Sony Xperia E32019-12-18T00:00:00+01:002019-12-18T00:00:00+01:00Kevin,2019-12-18:/2019/12/sony-xperia-e3-d2203-unlock-root-custom-rom/<p>Got hold of a second-hand <a href="">Sony Xperia E3</a> a couple of weeks ago and was surprised by its general build quality and <span class="caps">IPS</span> screen for a 2014&nbsp;phone.</p> <p>Of course it was loaded with crapware and bloatware both from the <span class="caps">ISP</span> and manufacturer. Time to get my hands dirty back …</p><p>Got hold of a second-hand <a href="">Sony Xperia E3</a> a couple of weeks ago and was surprised by its general build quality and <span class="caps">IPS</span> screen for a 2014&nbsp;phone.</p> <p>Of course it was loaded with crapware and bloatware both from the <span class="caps">ISP</span> and manufacturer. Time to get my hands dirty back in the Android ecosystem, and try to clean that&nbsp;up.</p> <p>This process started with the following&nbsp;devices:</p> <ul> <li>Sony Xperia E3 with stock firmware and <span class="caps">OS</span>:<ul> <li>Model number: <code>D2203</code> (<span class="caps">LTE</span>&nbsp;variant)</li> <li>Android version: <code>4.4.2</code> (KitKat)</li> <li>Baseband version: <code>8926-DAAAANAZQ-00282-04</code> (Europe, France,&nbsp;Orange)</li> <li>Kernel version: <code>3.4.0</code></li> <li>Build number: <code>18.4.C.1.29</code></li> </ul> </li> <li>MacBook Air (macOS 10.14&nbsp;Mojave)</li> </ul> <h2 id="factory-reset">Factory Reset<a class="headerlink text-decoration-none small pl-2" href="#factory-reset" title="Permanent link">&para;</a></h2> <p>First things first, let&rsquo;s clean up that phone using the standard&nbsp;functions:</p> <ol> <li> <p>Connect to local&nbsp;wifi.</p> </li> <li> <p>Update phone&rsquo;s stock firmware, <span class="caps">OS</span> and apps using Sony&rsquo;s <em>Update Center</em>.</p> </li> <li> <p>Format the attached <span class="caps">SD</span> card if any to destroy previous user&rsquo;s personal&nbsp;data.</p> </li> <li> <p>Factory reset the&nbsp;phone.</p> </li> </ol> <h2 id="unlock-bootloader">Unlock Bootloader<a class="headerlink text-decoration-none small pl-2" href="#unlock-bootloader" title="Permanent link">&para;</a></h2> <p>Phone was locked, tied to an Orange contract. Sony is providing everything online to properly unlock the phone, and the warranty has been expired for a long time. So let&rsquo;s&nbsp;proceed!</p> <ol> <li> <p>Enter service mode by dialing <code>*#*#7378423#*#*</code> on the&nbsp;phone.</p> </li> <li> <p>Go to <code>Service info</code> &gt; <code>Configuration</code> &gt; <code>Rooting Status</code>.</p> </li> <li> <p>There, the presence of the <code>Bootloader unlock allowed: Yes</code> status confirm the phone is currently locked, but allowed to be&nbsp;unlocked.</p> </li> <li> <p>Dial <code>*#06#</code> on the phone to get its <span class="caps">IMEI</span> unique <span class="caps">ID</span>. Note that number&nbsp;somewhere.</p> </li> <li> <p>Go to <a href="">Sony developer website to obtain an unlock code</a>.</p> </li> <li> <p>Follow the procedure on that site, and enter your phone&rsquo;s <span class="caps">IMEI</span> to get an unlock key. Write it&nbsp;down.</p> </li> </ol> <p>There are several ways to feed that key to the phone. Here is how I did it under&nbsp;macOS:</p> <ol> <li> <p>We need to activate the developer mode. On the phone, go to phone&rsquo;s <code>Settings</code> app &gt; <code>About phone</code>. At the bottom, click 7 times on the <code>Build number</code> line. Developer mode has been unlocked once you get the <code>You are now a developer</code> notification.</p> </li> <li> <p>Then you can enable it for good in the <code>Settings</code> &gt; <code>System</code> &gt; <code>Developer options</code> menu. There, check the <code>Developer option</code> toggle button is set to <code>ON</code>.</p> </li> <li> <p>We&rsquo;re now going to <a href="">activate <span class="caps">USB</span> access with <span class="caps">CLI</span> tools</a>. Scroll down in the menu to find the <code>Debugging</code> item, and enable the <code>USB debugging</code> option.</p> </li> <li> <p>On a macOS terminal, <a href="">install Homebrew</a>:</p> <div class="highlight"><pre><span></span><code>$ /usr/bin/ruby -e &quot;$(curl -fsSL; </code></pre></div> </li> <li> <p>Now install Android tools with Homebrew from the&nbsp;terminal:</p> <div class="highlight"><pre><span></span><code>$ brew install android-platform-tools </code></pre></div> </li> <li> <p>Plug the phone to your computer with a micro <span class="caps">USB</span>&nbsp;cable.</p> </li> <li> <p>Check your device is recognized by <code>adb</code>:</p> <div class="highlight"><pre><span></span><code>$ adb devices List of devices attached BL4828DXXXXX device </code></pre></div> </li> <li> <p>Reboot device in bootloader&nbsp;mode:</p> <div class="highlight"><pre><span></span><code>$ adb reboot bootloader </code></pre></div> </li> <li> <p>Unlock the device with the unlock key previously fetched from&nbsp;Sony:</p> <div class="highlight"><pre><span></span><code>$ fastboot oem unlock 0x0828XXXXXXXXXXXX </code></pre></div> </li> <li> <p>Reboot the device in standard&nbsp;mode:</p> <div class="highlight"><pre><span></span><code>$ fastboot reboot </code></pre></div> </li> <li> <p>Go back to service mode (dial <code>*#*#7378423#*#*</code>; go to <code>Service info</code> &gt; <code>Configuration</code> &gt; <code>Rooting Status</code>) and confirm the <code>Bootloader unlock : Yes</code> status. If so, your phone is now&nbsp;unlocked!</p> </li> </ol> <p>All the steps above have been performed with the stock ecosystem: firmware, <span class="caps">ROM</span> and <span class="caps">OS</span> are all standard. Now we&rsquo;ll delves into the dark&nbsp;side.</p> <h2 id="root-phone">Root Phone<a class="headerlink text-decoration-none small pl-2" href="#root-phone" title="Permanent link">&para;</a></h2> <p>We&rsquo;ll now root the&nbsp;phone.</p> <ol> <li> <p>Get a microSD card and insert it in the&nbsp;phone.</p> </li> <li> <p>Format the external <span class="caps">SD</span> card from the phone. Unmount it, then physically remove it from the&nbsp;phone.</p> </li> <li> <p>Plug that card to your&nbsp;computer.</p> </li> <li> <p>Download <a href="">King Root</a> for Android (v5.3.7 in my case). Copy the <code>.apk</code> file to the <span class="caps">SD</span>&nbsp;card.</p> </li> <li> <p>Unmount and remove the card from the computer. Insert it back into the phone, and mount&nbsp;it.</p> </li> <li> <p>In <code>Settings</code> &gt; <code>Security</code> menu, allow applications to be installed from unknown&nbsp;sources.</p> </li> <li> <p>Connect to wifi, go to Google Play store and install a file browser. My personal favorite: <a href="">Amaze File Manager</a>.</p> </li> <li> <p>Use the file manager to browse into the <span class="caps">SD</span> card, <a href="">find the <span class="caps">APK</span>, install it and run KingRoot</a>.</p> </li> </ol> <h2 id="flash-custom-recovery">Flash Custom Recovery<a class="headerlink text-decoration-none small pl-2" href="#flash-custom-recovery" title="Permanent link">&para;</a></h2> <p>The custom <span class="caps">ROM</span> I&rsquo;ll provide here only targets Sony Xperia E3 D2203 <span class="caps">LTE</span> model with the stock 4.4.2&nbsp;Android.</p> <p>Before proceeding any further, please double check your device is the right one, with the expected versions. If not you&rsquo;re at risk of bricking your device for&nbsp;good.</p> <ol> <li> <p>It was hard to find an appropriate <span class="caps">ROM</span> so <a href="../uploads/2019/boot_D2203_4.4.2.img">here is a copy of the <code>boot_D2203_4.4.2.img</code> file</a>. I originally found it on an article on <a href=""><em>How to Install Custom Recovery on Sony Xperia E3</em></a>, and the <a href="">image itself is hosted on MediaFire</a>.</p> <p>It is a <span class="caps">CMW</span>-based Recovery v6.0.4.7 uniquely built for the D2203 <span class="caps">LTE</span>&nbsp;model.</p> <p>Download it and compare its&nbsp;checksum:</p> <div class="highlight"><pre><span></span><code>$ wget -P ~/Downloads/ (...) $ sha1sum ~/Downloads/boot_D2203_4.4.2.img 7fdf48efa91167ee20282b6595e11fe5afef72f2 boot_D2203_4.4.2.img </code></pre></div> </li> <li> <p>Make sure the phone is plugged into your computer, and reboot the device in bootloader&nbsp;mode:</p> <div class="highlight"><pre><span></span><code>$ adb devices List of devices attached BL4828DXXXXX device $ adb reboot bootloader </code></pre></div> </li> <li> <p>Flash the custom <span class="caps">ROM</span>:</p> <div class="highlight"><pre><span></span><code>$ fastboot flash boot ~/Downloads/boot_D2203_4.4.2.img </code></pre></div> <p>Here you might stumble upon the following&nbsp;error:</p> <div class="highlight"><pre><span></span><code>Writing &#39;recovery&#39; FAILED (remote: &#39;partition table doesn&#39;t exist&#39;) </code></pre></div> <p>If that the case it&rsquo;s because you tried to flash the <span class="caps">ROM</span> into the recovery partition (as generally advised in most tutorials out there), with a command similar&nbsp;to:</p> <div class="highlight"><pre><span></span><code>$ fastboot flash recovery boot.img </code></pre></div> <p>This doesn&rsquo;t cut it: as <a href=";p=68056829&amp;postcount=3"><em>foxite</em> explained in 2016</a>, the custom <span class="caps">ROM</span> provided here targets the boot&nbsp;partition.</p> </li> <li> <p>Reboot the&nbsp;device:</p> <div class="highlight"><pre><span></span><code>$ fastboot reboot </code></pre></div> <p>The boot process should be eventless, as if nothing&nbsp;happened.</p> </li> <li> <p>Now if you reboot your device once more, you&rsquo;ll notice a slight&nbsp;difference.</p> <p>When the Sony Xperia logo show up, at the very start of the boot process, the front notification <span class="caps">LED</span> gets turned on in pink for a couple of seconds while the phone briefly&nbsp;vibrate.</p> <p>And right at that time you press alternatively the <span class="keys"><kbd class="key-arrow-up">Up</kbd></span> and <span class="keys"><kbd class="key-arrow-down">Down</kbd></span> volume buttons a couple of times, while the <span class="caps">LED</span> last, you&rsquo;ll boot into the custom recovery <span class="caps">ROM</span> you just&nbsp;flashed:</p> <p><img alt="" src="/uploads/2019/sony-xperia-e3-d2203-lte-cmw-recovery.jpg"></p> </li> </ol> <h2 id="custom-os">Custom <span class="caps">OS</span><a class="headerlink text-decoration-none small pl-2" href="#custom-os" title="Permanent link">&para;</a></h2> <p>In this section we will install a new <span class="caps">OS</span> to refresh the dated&nbsp;one.</p> <p>I choose a tweaked yet stable distribution based on the stock Android 4.4.4 and 18.5.C.0.19 firmware. The main reason being that more experimental ports of Android 5.0 exists but often lacks support for the&nbsp;camera.</p> <ol> <li> <p>Download the so called <a href=""><em>De-bloated, Tweaked, Pre-Rooted, <span class="caps">ROM</span> for the Sony Xperia E3</em> from <span class="caps">XDA</span> forums</a>. The original <span class="caps">ROM</span> being long gone, a <a href="">copy is available online</a> (<span class="caps">SHA1</span> checksum: <code>d2e949d15aea97c64414ae2ef493f7b18e32dd78</code>).</p> </li> <li> <p>Unzip that file. You should end up with that&nbsp;structure:</p> <div class="highlight"><pre><span></span><code>[ 96] clockworkmod └── [ 128] backup └── [ 352] Version_1 ├── [ 20M] boot.img ├── [ 0] cache.ext4.tar ├── [1.1M] cache.ext4.tar.a ├── [ 0] data.ext4.tar ├── [2.5K] data.ext4.tar.a ├── [ 343] nandroid.md5 ├── [232K] recovery.log ├── [ 0] system.ext4.tar └── [871M] system.ext4.tar.a </code></pre></div> </li> <li> <p>Copy the <code>clockworkmod</code> folder and its content at the root of the <span class="caps">SD</span> card. Unmount the card and place it into the&nbsp;phone.</p> </li> <li> <p>Reboot the phone and enter the recovery <span class="caps">ROM</span>.</p> </li> <li> <p>Go to <code>wipe data/factory reset</code> &gt; <code>Yes - Wipe all user data</code> to clean up&nbsp;cache.</p> </li> <li> <p>Repeat same thing but this time go to <code>wipe cache partition</code> &gt; <code>Yes - Wipe Cache</code>.</p> </li> <li> <p>Again, go to <code>advanced</code> &gt; <code>wipe dalvik cache</code> &gt; <code>Yes - Wipe Dalvik Cache</code>.</p> </li> <li> <p>Go back to the main recovery menu, then mount the <span class="caps">SD</span> card: <code>mounts and storage</code> &gt; <code>mount /storage/sdcard1</code>.</p> </li> <li> <p>Then go back, <code>backup and restore</code> &gt; <code>restore from /storage/sdcard1</code> &gt; <code>Version_1</code> (i.e. the name of the first subfolder from the <span class="caps">ZIP</span> file we downloaded above) &gt; <code>Yes - Restore</code>.</p> </li> <li> <p>Wait while the installation is performed. At the end you can reboot the device and you should have a brand new and clean&nbsp;phone:</p> <p><img alt="" src="/uploads/2019/sony-xperia-e3-d2203-android-444.jpg"></p> </li> </ol> <h2 id="epilogue">Epilogue<a class="headerlink text-decoration-none small pl-2" href="#epilogue" title="Permanent link">&para;</a></h2> <p>I wrote this article along the way, as I was (re)discovering the whole Android development scene. Brings back memories. But I&rsquo;m a little rusty which explain the somewhat convoluted&nbsp;process.</p> <p>At the end, I realized there&rsquo;s no need to go through the extra hoops of rooting the phone and copying APKs on <span class="caps">SD</span> card when you can simply download and execute them from the phone&rsquo;s&nbsp;browser.</p> <p>Another huge detour I took was the hunt for a custom <span class="caps">CMW</span> recovery image. I could have instead unzip the <code></code> file and flash the <code>clockworkmod/backup/Version_1/boot.img</code> image right away, instead of <code>boot_D2203_4.4.2.img</code>. The benefit being a slightly newer version (, the <span class="caps">LED</span> being turned on in white instead of pink, and an appropriate screen&nbsp;size:</p> <p><img alt="" src="/uploads/2019/sony-xperia-e3-cmw-recovery-6051.jpg"></p>