Jekyll2019-05-27T08:51:48+00:00https://adamatan.github.io//feed.xmlAdam MatanAdam Matan's tech blogHow to rsync dotfiles between Macs2019-05-21T00:00:00+00:002019-05-21T00:00:00+00:00https://adamatan.github.io//how-to-rsync-dotfiles-between-macs<p>Many CLI apps - like <code class="highlighter-rouge">git</code>, <code class="highlighter-rouge">vim</code>, <code class="highlighter-rouge">zsh</code> and <code class="highlighter-rouge">tmux</code> - use dot files to save their configuration. A <em>dot file</em> is a file or directory located in the user’s home directory that begins with the “<code class="highlighter-rouge">.</code>” character.</p>
<p>Moving these files to a new laptop can save a lot of configuration time and keep a consistent development experience accross devices. <strong>However, we want to avoid accidentally deleting existing dotfiles, and avoid copying junk dirs like <code class="highlighter-rouge">.Trash</code>.</strong>.</p>
<h1 id="dotfiles">dotfiles</h1>
<ol>
<li>Make sure your computers are on the same network.</li>
<li><a href="https://apple.stackexchange.com/a/2425/17965">Enable remote login on the source laptop</a>. The remote login window should list your username and address, e.g. <code class="highlighter-rouge">adam@adams-mbp</code> or <code class="highlighter-rouge">adam@192.168.120.130</code>.
<img src="/images/remote_login.png" alt="My helpful screenshot" /></li>
<li>Create a temporary directory in the new computer and <code class="highlighter-rouge">cd</code> into it - you don’t want to accidentally override existing dot files on the new computer.</li>
<li>
<p>Test:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">$ </span>rsync <span class="nt">--archive</span> <span class="nt">--verbose</span> <span class="nt">--dry-run</span> <span class="nt">-f</span><span class="s2">"- */"</span> <span class="s1">'adam@adams-mbp:~/.*'</span> <span class="nb">.</span>
</code></pre></div> </div>
<p>Giving something like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> receiving file list ... <span class="k">done</span>
...
.zcompdump-Adam<span class="se">\#</span>200<span class="se">\#</span>231s MacBook Pro-5.2
.zcompdump-Adam<span class="se">\#</span>200<span class="se">\#</span>231s MacBook Pro-5.3
.zsh-update
.zsh_history
.zshrc
</code></pre></div> </div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> * `--archive`: rsync [archive mode](https://serverfault.com/questions/141773/what-is-archive-mode-in-rsync) (essentially, preserve attributes).
* `--verbose`: verbose: list files.
* `--dry-run`: dry run. Just list, don't copy yet.
* `-f"- */"`: Copy files, but not directories. This is important to avoid cache dirs like `.npm` or `.Trash`.
* `'adam@adams-mbp:~/.*'`: Change `adam@adams-mbp` to the address given on the remote login screen. `.*` means all files beginning with `.`.
</code></pre></div> </div>
</li>
<li>Once you’re happy with the list, remove the <code class="highlighter-rouge">--dry-run</code> flag and copy the files. Selectively copy the dot files to your home directory. For example, you probably want to copy the <code class="highlighter-rouge">.zshrc</code> and <code class="highlighter-rouge">.zsh_history</code> file but avoid <code class="highlighter-rouge">.zcompdump</code> and <code class="highlighter-rouge">.zsh-update</code>.</li>
</ol>
<h1 id="dot-dirs">dot dirs</h1>
<p>Some users have meaningful data in dot directories, for example <code class="highlighter-rouge">.aws</code> or <code class="highlighter-rouge">.ssh</code>. To selectively copy these directories, but avoid digital atrocities like copying <code class="highlighter-rouge">.Trash</code>, start with:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> rsync <span class="nt">--archive</span> <span class="nt">--verbose</span> <span class="nt">--dry-run</span> <span class="s1">'adam@adams-mbp:~/.*'</span> <span class="nb">.</span>
</code></pre></div></div>
<p>Start excluding junk directories, till you reach a reasonable list:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> rsync <span class="nt">--archive</span> <span class="nt">--verbose</span> <span class="nt">--dry-run</span> <span class="nt">--exclude</span> <span class="s1">'.Trash'</span> <span class="s1">'adam@adams-mbp:~/.*'</span> <span class="nb">.</span>
rsync <span class="nt">--archive</span> <span class="nt">--verbose</span> <span class="nt">--dry-run</span> <span class="nt">--exclude</span> <span class="s1">'.Trash'</span> <span class="nt">--dry-run</span> <span class="nt">--exclude</span> <span class="s1">'.vscode'</span> <span class="s1">'adam@adams-mbp:~/.*'</span> <span class="nb">.</span>
</code></pre></div></div>
<p>When done, copy the relevant dirs with <code class="highlighter-rouge">cp -R <dirname> ~</code> to your home (<code class="highlighter-rouge">~</code>) directory. <strong>Double-check before overriding existing dirs, especially <code class="highlighter-rouge">.ssh</code></strong>.</p>Many CLI apps - like git, vim, zsh and tmux - use dot files to save their configuration. A dot file is a file or directory located in the user’s home directory that begins with the “.” character.Dockerized ffmpeg video conversion2019-05-09T00:00:00+00:002019-05-09T00:00:00+00:00https://adamatan.github.io//dockerized-ffmpeg-video-conversion<p>Easily convert large <code class="highlighter-rouge">.MOV</code> files to a compressed VP9 format using the <a href="https://hub.docker.com/r/jrottenberg/ffmpeg/"><code class="highlighter-rouge">jrottenberg/ffmpeg</code></a> docker image.</p>
<h1 id="code">Code</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Replace the following:</span>
<span class="c"># * Mounted directory (mine is /tmp/ff)</span>
<span class="c"># * Source filename (mine is is MVI_7493.MOV)</span>
<span class="c"># * CRF (quality) value - I used 30</span>
<span class="c"># First pass output.webm</span>
docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">--mount</span> <span class="nb">type</span><span class="o">=</span><span class="nb">bind</span>,source<span class="o">=</span>/tmp/ff,target<span class="o">=</span>/tmp/workdir <span class="se">\</span>
<span class="nt">--rm</span> <span class="se">\</span>
jrottenberg/ffmpeg <span class="se">\</span>
<span class="nt">-y</span> <span class="se">\ </span>
<span class="nt">-i</span> /tmp/workdir/MVI_7493.MOV <span class="se">\</span>
<span class="nt">-c</span>:v libvpx-vp9 <span class="nt">-b</span>:v 0 <span class="nt">-crf</span> 30 <span class="se">\</span>
<span class="nt">-pass</span> 1 <span class="se">\</span>
<span class="nt">-f</span> webm <span class="se">\</span>
/dev/null
<span class="c"># Second pass</span>
docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">--mount</span> <span class="nb">type</span><span class="o">=</span><span class="nb">bind</span>,source<span class="o">=</span>/tmp/ff,target<span class="o">=</span>/tmp/workdir <span class="se">\</span>
<span class="nt">--rm</span> <span class="se">\</span>
jrottenberg/ffmpeg <span class="se">\</span>
<span class="nt">-y</span> <span class="se">\</span>
<span class="nt">-i</span> /tmp/workdir/MVI_7493.MOV <span class="se">\</span>
<span class="nt">-c</span>:v libvpx-vp9 <span class="nt">-b</span>:v 0 <span class="nt">-crf</span> 30 <span class="se">\</span>
<span class="nt">-pass</span> 2 <span class="se">\</span>
<span class="nt">-f</span> webm <span class="se">\</span>
output.webm
</code></pre></div></div>
<h1 id="file-size">File size</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1.5G May 9 21:23 MVI_7493.MOV
2.2M May 9 23:09 ffmpeg2pass-0.log
121M May 10 02:18 output.webm
</code></pre></div></div>
<p>The conversion took about 3 hours. The original is 1.5GB, the temporary log file is 2.2M, and the output is 121M.</p>
<h1 id="discussion">Discussion</h1>
<p>My <a href="https://www.dpreview.com/products/canon/slrs/canon_eos550d">Canon 550d DSLR</a> saves video files using the <code class="highlighter-rouge">h264</code> format. The video files are quite big - 4.5 minutes take about 1.5 GB, and I want to share a smaller version in a royalty-free format.</p>
<p>I am not a professional video editor, so I decided to avoid the complexity of <code class="highlighter-rouge">ffmpeg</code> installation on my Mac and choose a proper Docker container instead. <a href="https://github.com/jrottenberg/ffmpeg">Julien Rottenberg’s image</a> seemed like a great choice - popular, starred, and <a href="https://hub.docker.com/r/jrottenberg/ffmpeg/dockerfile">its crazy Dockerfile</a> is exactly what I was trying to avoid doing myself.</p>
<p>I took some <a href="https://github.com/jrottenberg/ffmpeg/blob/master/docker-images/4.1/alpine/Dockerfile#L14">digging into the Dockerfile</a> to understand the mount point (which is <code class="highlighter-rouge">/tmp/workdir</code>), and from that point it’s pretty much running the ffmpeg commands.</p>
<h1 id="two-pass">Two pass</h1>
<p>Quoting the manual:</p>
<blockquote>
<p>Two-pass is the recommended encoding method for libvpx-vp9 as some quality-enhancing encoder features are only available in 2-pass mode.</p>
</blockquote>
<p>In Two-pass mode, the first pass creates a temporary log file, and the second pass uses the log file and the original video to generate the final output. The only difference between the first and second pass commands are the output file (<code class="highlighter-rouge">/dev/null</code> in the first pass) and the <code class="highlighter-rouge">-pass 1</code>, <code class="highlighter-rouge">-pass 2</code> arguments respectively.</p>
<h1 id="links">Links</h1>
<ul>
<li><a href="https://hub.docker.com/r/jrottenberg/ffmpeg/">Docekrhub, jrottenberg/ffmpeg</a></li>
<li><a href="https://trac.ffmpeg.org/wiki/Encode/VP9#constrainedq">FFmpeg and VP9 Encoding Guide</a></li>
</ul>Easily convert large .MOV files to a compressed VP9 format using the jrottenberg/ffmpeg docker image.