Jeremy W. Sherman https://jeremywsherman.com/ Recent content on Jeremy W. Sherman Hugo -- gohugo.io en-us All rights reserved. Sat, 10 Oct 2020 21:37:25 -0400 Turning a PDF into a Coptic-bound book https://jeremywsherman.com/blog/2022/03/25/turning-a-pdf-into-a-coptic-bound-book/ Fri, 25 Mar 2022 15:54:31 +0000 https://jeremywsherman.com/blog/2022/03/25/turning-a-pdf-into-a-coptic-bound-book/ It&rsquo;s sometimes convenient to turn an ebook into a paper book. My running example will be Skirmish: Wallet Friendly Wargaming by Biscuit Fund Games, which I wanted to have on hand at the table. But what&rsquo;s a satisfying way to do this ebook to book conversion? I wound up creating a Coptic-bound volume that lays flat using a couple free tools I hadn&rsquo;t encountered before and some stuff I already had lying around the house. <p>It&rsquo;s sometimes convenient to turn an ebook into a paper book. My running example will be <a href="https://biscuitfund.itch.io/skirmish"><em>Skirmish: Wallet Friendly Wargaming</em> by Biscuit Fund Games</a>, which I wanted to have on hand at the table.</p> <p>But what&rsquo;s a satisfying way to do this ebook to book conversion?</p> <p>I wound up creating a Coptic-bound volume that lays flat using a couple free tools I hadn&rsquo;t encountered before and some stuff I already had lying around the house. This post records my thoughts leading up to and notes during the process.</p> <h2 id="options">Options</h2> <p>Here are the options I considered:</p> <ul> <li>Treat it like a short paper: Print, staple the corner, done. <ul> <li>Pros: <ul> <li>Dead easy.</li> </ul> </li> <li>Cons: <ul> <li>Can eat a lot of paper.</li> <li>Not the most portable form-factor.</li> <li>Could be hard to staple for larger works. Not terribly durable.</li> </ul> </li> </ul> </li> <li>Treat it like a final paper, and stick that sucker in a plastic cover with snap-on binding. <ul> <li>Pros: <ul> <li>Also pretty easy.</li> <li>Much more durable.</li> </ul> </li> <li>Cons: <ul> <li>Those covers are surprisingly expensive for what they are.</li> <li>I don&rsquo;t have any on hand.</li> <li>They don&rsquo;t seem to actually hold all that many pages - 12-20 max.</li> <li>Still a huge letter-size volume.</li> <li>That snap-on / slide thing can pop off, and then your papers get all mixed up.</li> </ul> </li> </ul> </li> <li>Get fancier and do some bookbinding <ul> <li>Pros: <ul> <li>Doesn&rsquo;t really require much in the way of materials: If you have a sewing kit on hand for mending, you&rsquo;re probably good to go.</li> <li>You can readily wind up with a smaller, half-letter–size volume.</li> <li>It&rsquo;s not gonna fall apart on you.</li> <li>You get to learn something new.</li> </ul> </li> <li>Cons: <ul> <li>Printing gets a lot more complicated.</li> <li>Probably not spillproof.</li> <li>Takes more time.</li> </ul> </li> </ul> </li> </ul> <p>I chose the bookbinding option. I actually had everything I needed on-hand already, and the crafting sounded fun.</p> <h2 id="bookbinding-flavors">Bookbinding Flavors</h2> <p>There&rsquo;s a lot of <a href="https://undercoverprint.com/book-binding-types/">ways to get a book to stick together</a>. The main tradeoffs are around:</p> <ul> <li>Do you need to do any sewing? <ul> <li>Staple-bound (think of most magazines) and perfect-bound books (line &lsquo;em up, run glue down one side - most paperbacks and thicker magazines like <em>Asimov&rsquo;s Science Fiction</em>) can do without any sewing.</li> </ul> </li> <li>Is stuff held together using glue? <ul> <li>Perfect-bound books rely entirely on glue.</li> <li>Case-bound books use sewing and gluing to create a very solid binding. But they tend to assume you have a bookpress on hand and special ribbon and book cloth for the covers and such.</li> </ul> </li> <li>What size book does it work for? <ul> <li>Staple binding runs into limits on stapler size and power, for example.</li> </ul> </li> <li>Does the book lie flat? <ul> <li>Most bindings don&rsquo;t pull this off. But Coptic binding, which has an open back and knots the signatures together, does.</li> </ul> </li> </ul> <p>A couple decades ago, I did some very simple Japanese stab book binding of a volume I just printed off full letter-size double-sided using some random yard I had lying around and a drill press I found in my parent&rsquo;s basement. The book had no cover, but it held up well. (I think I got the method from a Boy Scout magazine book at the time.)</p> <p>But I really wanted the lie-flat behavior for this at-table reference work, and I wanted to do it using stuff I had lying around the house. So I wound up selecting <strong>Coptic binding.</strong></p> <h2 id="printing-for-binding">Printing for Binding</h2> <p>The main complications of printing are generally:</p> <ul> <li>Figuring out signature sizes</li> <li>Rearranging the pages so they print 2-up in such a way to be in order when gathered into signatures. The outermost sheet has the first two pages and the last two pages of the signature on it. This might require adding extra blank pages to page at the end.</li> </ul> <h3 id="carl-mctagues-signature-optimizer">Carl McTague&rsquo;s Signature Optimizer</h3> <p>Luckily, that is entirely handled for me in a very nice way for Coptic binding by Carl McTague&rsquo;s <a href="https://www.mctague.org/carl/fun/bookbinding/optimizer.html?start=1&amp;landmarks=1,3,4,5,6,7,8,9,12,14,16,18,20,34,46,66,82,5,6,14,16,18,66&amp;end=85&amp;lambda=0.5&amp;pdfStart=1">Signature Optimizer</a>.</p> <p>If you weren&rsquo;t doing Coptic binding, you could still use the LaTeX output as a pointer in the right direction to generate your signatures, but you&rsquo;d probably go for regularly-sized signatures, rather than the variably-sized ones.</p> <h3 id="briss">Briss</h3> <p>Unluckily, the source PDF for Skirmish is already 2-up, so I had to disassemble it before I could reassemble it for signature printing.</p> <p>Fortunately, McTague mentions <a href="http://briss.sourceforge.net/">Briss</a> in passing, and it proved to be just the thing to slice and dice the PDF back into 1-up for me. It was just a <code>brew install briss</code> away. The interface is a bit clunky; the key for me was realizing that you just click-and-sweep to add a new rect.</p> <h3 id="reversed-pages">Reversed Pages?</h3> <p>I was worried after imposition using the LaTeX that the Signature Optimizer spit out that some pages were going to come out upside down. But a test printing of one signature showed that the usual long-edge flip printing worked just fine, so those fears proved unfounded.</p> <p>I also was worried I&rsquo;d messed something up when I found the odd-numbered pages were on the left, but that turned out to be the case in the source PDF as well, and I didn&rsquo;t want to change it - there were a lot of well-designed two-page spreads that would only be a spread if you have the even numbers on the right-hand side. Unconventional, but not really a problem in the end.</p> <h3 id="latex-errors">LaTeX Errors?</h3> <p>I was also worried about a warning that was emitted over and over:</p> <pre><code>pdfTeX warning: pdflatex (file ./Skirmish-v1-1-1up-briss.pdf): PDF inclusion: m ultiple pdfs with page group included in a single page &gt;] [4 &lt;./Skirmish-v1-1-1up-briss.pdf&gt; &lt;./Skirmish-v1-1-1up-briss.pdf </code></pre><p>It turned out to be <a href="https://tex.stackexchange.com/questions/76273/multiple-pdfs-with-page-group-included-in-a-single-page-warning">an ignorable error</a> in most cases, including this one, as determined by reviewing the output. (It would be an issue if the page groups were configured differently, which they weren&rsquo;t.)</p> <h2 id="folding-the-signatures">Folding the Signatures</h2> <p>I saw some advice to fold each sheet individually and then nest them. That sounded like a great way for me to wind up with a bunch of subtly different sheets within the signature. I just stacked &lsquo;em up, squared &lsquo;em up, and then folded the whole lot.</p> <p>I did not trim the signature to have an even end (the inner ones poke out more the deeper in you go, since they&rsquo;re stacked atop the outer). I don&rsquo;t regret this in the finished product: it&rsquo;s not terribly noticeable, since my cover overhangs, and the signatures slide around a bit due to the way the binding is done anyway, so even if the pages were cut within the signature, the signatures would not line up into a perfect textblock most of the time anyway.</p> <h2 id="making-the-cover">Making the Cover</h2> <p>I was divided on even adding a cover. It definitely took more fiddling and time (overnight drying!) than I liked at the time, even with me cutting corners (like eyeballing the size and shape of the cardboard I chopped out of a spare cardboard box). But the firm covers contribute a lot to making the book feel like a book to me, so I think it was probably worth it.</p> <p>I unfolded a signature, held it onto the side of a box, marked some points around it to provide some overhang by eye, used a straightedge to rule between the lines, then went at it with a pair of scissors. It was not actually square, but it was close enough. I then chopped it in half.</p> <p>I tweaked some images from the book to make a front and back cover, then printend them on some nice report/resume paper still sitting around the house from college thesis times.</p> <p>I stuck the cardboard in the middle of the paper, then cut lines straight up and down to the edge at the corners, then diagonally back and up. This is to enable folding it over onto the cardboard to wrap it up. (This would normally be done with book cloth, but I didn&rsquo;t have any, and I wasn&rsquo;t making any, either.) I made the cuts with a boxcutter without marking the lines first.</p> <p>I used some spare kid gluesticks (the purple ones that dry clear) to coat the paper, dropped on the cardboard, folded the edges down, then smoothed it all out. One side of the cardboard was more obviously ribbed than the other, but I noticed this too late, and due to my cardboard rectangles being more trapezoidal, I couldn&rsquo;t really fix this after the cuts had been made, so I just sucked it up. Luckily it was the backside, and also, it&rsquo;s not really that big a deal, either.</p> <p>I found some green cardstock to cover up the inside part, cut it in half with a boxcutter, coated it in glue, and pasted it on over the edges of the outer wrapper. The paper tried to curl, which made lining it up a bit hard. I then left these to dry overnight in a stack of books.</p> <h2 id="pricking">Pricking</h2> <p>At this point, it was time to prep for sewing. I decided to do 3 groups of 2 holes each. I eyeballed it based on where I&rsquo;d placed binder clips before, moved the binder clips to the outside edges of the stacked-up signatures, then ruled a line along each intended hole position straight across the binding side. (I found out later that this hadn&rsquo;t hit one of the signatures very well at all, and that one of the hole lines for some reason had scarcely marked any of them. It slowed down actually making the holes.)</p> <p>I then transferred the markings to the edge of a spare piece of paper to use as a jig to mark the cover holes. The cover holes are inset a bit from the edge of the board, though not much, since I needed to ensure some overlap. The 1/8-inch inset has proven to be enough.</p> <p>After that, I flipped each signature open in turn, lay it inside-down so the markings were visible, and pricked it straight through with a scratch awl down the crease. I used the paper jig to put the holes through the covers. I also flipped each signature over and made the holes from the other side, since the depth of the signatures was not great, so only the smallest bit of the scratch awl went through - not a very big hole.</p> <h2 id="sewing">Sewing</h2> <p>I&rsquo;d been convinced that coating the thread in beeswax would make things a lot easier. (It did - the stiffer thread didn&rsquo;t try to knot up against itself as badly as it would otherwise, and that&rsquo;s important when you&rsquo;re working with a thread that&rsquo;s long enough to sew the whole binding to start with.) So I melted some beeswax pastilles we had around in a silicone cupcake liner in the microwave and let it set before I started.</p> <p>Then I measured enough sewing thread length (in a nice green color to match the book cover insides) as long as each signature, plus as long as each cover. Then I doubled that, because I wanted to work with doubled thread for strength. I held it against the beeswax with my thumb, then pulled the whole length through between my thumb and the beeswax to coat it.</p> <p>Thread the needle, knot the end, and then I just followed the guide I&rsquo;d found, <a href="https://www.clothpaperscissors.com/wp-content/uploads/Coptic-Stitch-PDF.pdf">Sharilyn Miller&rsquo;s <em>The Coptic Stitch: Instructions and Illustrations</em> (PDF)</a>. The last signature plus cover was a bit odd, and I deviated a bit in a way that mirrored more the first signature, with a separate knot between the signature and the cover, except for the first and last holes, which I did as directed.</p> <p>I found I needed to take some care not to let the thread knot up on itself at first - it was a lot of thread to start! This got easier as more thread had been used up, and once I knew to watch out for it, I only fouled it up a bit a couple times more.</p> <p>I sometimes missed one or two pages in the middle of the signature when poking through; it was easy to notice this and correct by just poking it through the missed holes. This is a pretty forgiving binding technique.</p> <p>Using a curved needle proved to be sound advice. It made looping around the earlier stitches much, much easier.</p> <p>The sewing felt like it went fairly quickly. It&rsquo;s pretty easy to get into a rhythm, and aside from the start and end, it&rsquo;s all the same thing over and over, so very straightforward and easy to leave and come back to.</p> <h2 id="notes-for-next-time">Notes for Next Time</h2> <p>Cover:</p> <ul> <li>It is worth doing a cover.</li> <li>It&rsquo;s probably worth taking a bit more time on the cover and making sure it&rsquo;s square.</li> <li>If I had a white box (like the kind Hallmark send their stuff in), I would probably feel OK not covering it and just pasting a label on.</li> </ul> <p>Overall technique:</p> <ul> <li>The one-needle approach works great; I don&rsquo;t know what <a href="https://makezine.com/projects/build-a-sturdy-notebook-with-coptic-book-binding/">Make Magazine&rsquo;s Coptic binding notebook project</a> was on about with the two-needles per pair of holes thing. (Or the wood covers. That seems like the opposite of portable!)</li> <li>Binding would go pretty quick (just folding, pricking, and sewing) if you didn&rsquo;t do a cover; I can see why the author of the Signature Optimizer gets a lot of mileage out of this binding technique.</li> <li>I don&rsquo;t see much need for a bone folder; my fingernail works as well as ever it did when playing at origami.</li> </ul> <h2 id="key-resources">Key Resources</h2> <ul> <li><a href="https://www.mctague.org/carl/fun/bookbinding/optimizer.html?start=1&amp;landmarks=1,3,4,5,6,7,8,9,12,14,16,18,20,34,46,66,82,5,6,14,16,18,66&amp;end=85&amp;lambda=0.5&amp;pdfStart=1">Signature Optimizer</a></li> <li><a href="http://briss.sourceforge.net/">Briss to trim PDFs</a></li> <li><a href="https://www.clothpaperscissors.com/wp-content/uploads/Coptic-Stitch-PDF.pdf">How-To with pictures</a></li> </ul> Improving a testing-library test https://jeremywsherman.com/blog/2021/02/04/improving-a-testing-library-test/ Thu, 04 Feb 2021 13:12:31 +0000 https://jeremywsherman.com/blog/2021/02/04/improving-a-testing-library-test/ Test clarity helps in understanding the claims being made and the various ways the test might fail. With JavaScript/TypeScript, asynchrony can make this syntactically more confusing. So I saw this code recently: // Test for official email address await waitFor(() =&gt; expect(screen.getByTestId(&#34;officialEmail&#34;).getAttribute(&#34;href&#34;)).toBe( &#34;mailto:OFFICIAL_any_string@email.com&#34; ) ); This is using testing-library&lsquo;s explicit polling waitFor to repeatedly test the predicate. If it keeps failing till a timeout, then it concludes the test failed. <p>Test clarity helps in understanding the claims being made and the various ways the test might fail. With JavaScript/TypeScript, asynchrony can make this syntactically more confusing.</p> <p>So I saw this code recently:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#75715e">// Test for official email address </span><span style="color:#75715e"></span><span style="color:#a6e22e">await</span> <span style="color:#a6e22e">waitFor</span>(() =&gt; <span style="color:#a6e22e">expect</span>(<span style="color:#a6e22e">screen</span>.<span style="color:#a6e22e">getByTestId</span>(<span style="color:#e6db74">&#34;officialEmail&#34;</span>).<span style="color:#a6e22e">getAttribute</span>(<span style="color:#e6db74">&#34;href&#34;</span>)).<span style="color:#a6e22e">toBe</span>( <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string@email.com&#34;</span> ) ); </code></pre></div><p>This is using <a href="https://testing-library.com/">testing-library</a>&lsquo;s explicit polling <a href="https://testing-library.com/docs/dom-testing-library/api-async/#waitfor"><code>waitFor</code></a> to repeatedly test the predicate. If it keeps failing till a timeout, then it concludes the test failed.</p> <p>Two problems:</p> <ul> <li>it could do a better job of matching how users would see the content</li> <li>it&rsquo;s hard to read &amp; slow</li> </ul> <h2 id="matching-how-users-would-see-the-content">Matching how users would see the content</h2> <p>For this, I&rsquo;ll point to Testing Library&rsquo;s advice on which query to use, <a href="https://testing-library.com/docs/queries/about#priority">&ldquo;About Queries: Priority&rdquo;</a>. The short version is &ldquo;accessibility APIs, then visible stuff, then invisible stuff&rdquo;. A test ID is firmly in the &ldquo;invisible&rdquo; category; this could be improved by instead searching by link, or by searching just for the text in question.</p> <p>That&rsquo;s not actually what I want to focus on here, though, and Testing Library covers that well enough.</p> <h2 id="hard-to-read--slow">Hard to read &amp; slow</h2> <p>Yes, I&rsquo;m counting this as one issue, because I&rsquo;m blaming <code>waitFor</code>, and rewriting the test not to use it naturally leads to fixing both readability and slowness.</p> <h3 id="nesting--overhead">Nesting &amp; Overhead</h3> <p>It&rsquo;s hard to read because of the nesting. That&rsquo;s a lot of syntax to do an attribute check.</p> <h3 id="waiting-for-test-success-not-just-element-presence">Waiting for test success, not just element presence</h3> <p>It slow because it mixes up synchronizing with rendering (&ldquo;is this thing here yet that we need to exist before the test makes sense?&quot;) and passing the test (&ldquo;ok but is that thing <em>right</em>?&quot;). It should only be waiting for the element to appear. But by including the test expectation within the <code>waitFor</code>'d predicate, when the element has rendered but the test concerning the element fails, <code>waitFor</code> will keep polling long past the time the test outcome could have changed: It&rsquo;ll run out the clock on its timeout.</p> <h3 id="switch-to-findby">Switch to findBy</h3> <p>So, we separate syncing on the element&rsquo;s presence from checking its content. The cleanest way to do this is to use one of <a href="https://testing-library.com/docs/dom-testing-library/api-async/#findby-queries">the async <code>findBy…</code> queries</a>, which handles the waiting on our behalf.</p> <h4 id="syntactic-gotcha-wrt-what-to-await">Syntactic gotcha WRT what to await</h4> <p>But there&rsquo;s a syntactic gotcha; if you write:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#75715e">/* DOES NOT COMPILE */</span> <span style="color:#a6e22e">await</span> <span style="color:#a6e22e">waitFor</span>(() =&gt; <span style="color:#a6e22e">expect</span>(<span style="color:#a6e22e">await</span> <span style="color:#a6e22e">screen</span>.<span style="color:#a6e22e">findByTestId</span>(<span style="color:#e6db74">&#39;officialEmail&#39;</span>).<span style="color:#a6e22e">getAttribute</span>(<span style="color:#e6db74">&#39;href&#39;</span>)).<span style="color:#a6e22e">toBe</span>( <span style="color:#e6db74">&#39;mailto:OFFICIAL_any_string@email.com&#39;</span> ) ) <span style="color:#75715e">/* DOES NOT COMPILE */</span> </code></pre></div><p>then it won&rsquo;t work. With TypeScript, it won&rsquo;t even compile:</p> <blockquote> <p>error TS2339: Property <code>'getAttribute'</code> does not exist on type <code>'Promise&lt;HTMLElement&gt;'</code>.</p> </blockquote> <p>This error is informative, though:</p> <p><code>findByTestId</code> returns a <code>Promise</code> of some type. <code>Promise</code> does not have a <code>getAttribute</code> function. But the promised type does. So narrow the scope of what you&rsquo;re awaiting to that particular expression using parentheses:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#a6e22e">expect</span>( (<span style="color:#a6e22e">await</span> <span style="color:#a6e22e">screen</span>.<span style="color:#a6e22e">findByText</span>(<span style="color:#e6db74">&#34;OFFICIAL_any_string@email.com&#34;</span>)).<span style="color:#a6e22e">getAttribute</span>( <span style="color:#e6db74">&#34;href&#34;</span> ) ).<span style="color:#a6e22e">toBe</span>(<span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string@email.com&#34;</span>); </code></pre></div><p>The more readable way to write this would be to extract the expression to a named variable:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#66d9ef">const</span> <span style="color:#a6e22e">emailElement</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">await</span> <span style="color:#a6e22e">screen</span>.<span style="color:#a6e22e">findByText</span>(<span style="color:#e6db74">&#34;OFFICIAL_any_string@email.com&#34;</span>); <span style="color:#a6e22e">expect</span>(<span style="color:#a6e22e">emailElement</span>.<span style="color:#a6e22e">getAttribute</span>(<span style="color:#e6db74">&#34;href&#34;</span>)).<span style="color:#a6e22e">toBe</span>( <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string@email.com&#34;</span> ); </code></pre></div><p>Now it compiles, and we&rsquo;ve fixed what we&rsquo;re syncing on.</p> <p>But we&rsquo;re not done yet.</p> <h3 id="getting-the-most-out-of-test-failure">Getting the most out of test failure</h3> <p>The next step is to realize that, if this fails, it&rsquo;s going to provide very poor context, because it&rsquo;s just a string comparison.</p> <p>TDD&rsquo;s &ldquo;watch it fail&rdquo; step serves two purposes:</p> <ul> <li>Make sure you&rsquo;re actually sensing what you think you&rsquo;re sensing, and that the test isn&rsquo;t trivially passing.</li> <li>Tune the test failure output so it&rsquo;s obvious from the failure alone what went wrong, how, and where to fix it.</li> </ul> <p><strong>Those concerns apply whenever you&rsquo;re writing tests; TDD just frontloads addressing them.</strong> In test-after coding, you need to ensure your automated test in fact catches what you were manually testing for before automating. With test-after, you watch it fail by breaking it on purpose:</p> <ul> <li>Break the implementation and make sure your test detects it.</li> <li>Trigger test failures so you can tune the failure output</li> </ul> <p>I often find the test-failure tuning easier when corrupting the expectation in the test code rather than the actual implementation. It&rsquo;s just easier to break all the links in the test chain that way, rather than doing it at a distance by breaking the implementation. If you started by breaking the implementation, you know the core claim of the test will be checked, so I feel it&rsquo;s OK to just go straight to breaking the test itself when tuning output.</p> <h4 id="use-higher-level-matchers-to-inject-more-context-into-the-failure-message">Use higher-level matchers to inject more context into the failure message</h4> <p>So back to the example. Currently, a failed expectation isn&rsquo;t all that helpful:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#a6e22e">expect</span>(<span style="color:#a6e22e">received</span>).<span style="color:#a6e22e">toBe</span>(<span style="color:#a6e22e">expected</span>); <span style="color:#75715e">// Object.is equality </span><span style="color:#75715e"></span> <span style="color:#a6e22e">Expected</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string!@email.com&#34;</span>; <span style="color:#a6e22e">Received</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string@email.com&#34;</span>; </code></pre></div><p>This spawns some immediate questions:</p> <ul> <li>Where did these strings come from?</li> <li>Where in the app is the string wrong?</li> <li>Why is the string wrong?</li> </ul> <p>It&rsquo;s a <code>mailto:</code> scheme, so you can think a bit and work out it&rsquo;s probably an <code>href</code> attribute, but that takes thinking. You don&rsquo;t want to spend time and effort inferring that. Push that context into the test!</p> <p>Fix that by using a more contextual matcher:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#a6e22e">expect</span>( <span style="color:#a6e22e">await</span> <span style="color:#a6e22e">screen</span>.<span style="color:#a6e22e">findByText</span>(<span style="color:#e6db74">&#34;OFFICIAL_any_string@email.com&#34;</span>) ).<span style="color:#a6e22e">toHaveAttribute</span>(<span style="color:#e6db74">&#34;href&#34;</span>, <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string@email.com&#34;</span>); </code></pre></div><p>And then it fails with enough information to start debugging just from the error alone:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#a6e22e">expect</span>(<span style="color:#a6e22e">element</span>).<span style="color:#a6e22e">toHaveAttribute</span>(<span style="color:#e6db74">&#34;href&#34;</span>, <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string!@email.com&#34;</span>) <span style="color:#75715e">// element.getAttribute(&#34;href&#34;) === &#34;mailto:OFFICIAL_any_string!@email.com&#34; </span><span style="color:#75715e"></span><span style="color:#a6e22e">Expected</span> <span style="color:#a6e22e">the</span> <span style="color:#a6e22e">element</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">have</span> <span style="color:#a6e22e">attribute</span><span style="color:#f92672">:</span> <span style="color:#a6e22e">href</span><span style="color:#f92672">=</span><span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string!@email.com&#34;</span> <span style="color:#a6e22e">Received</span><span style="color:#f92672">:</span> <span style="color:#a6e22e">href</span><span style="color:#f92672">=</span><span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string@email.com&#34;</span> </code></pre></div><p>You&rsquo;ll note that I was triggering the failure case by intentionally corrupted the expected value, by injecting a <code>!</code> into it.</p> <h4 id="narrow-the-scope-of-element-search-so-failed-searches-dump-the-dom-you-care-about">Narrow the scope of element search so failed searches dump the DOM you care about</h4> <p>What if the element isn&rsquo;t even there? How helpful is the test failure?</p> <p>I corrupted the matcher and confirmed it provides some context to help, but probably I&rsquo;d want to scope down to the specific component, so the HTML output is less likely to truncate before hitting the relevant part I&rsquo;d want to see, and so the person debugging has less logspew to wade through.</p> <p>Let&rsquo;s say this was checking links in a &ldquo;Contact Info&rdquo; section on a &ldquo;Profile&rdquo; page.</p> <p>In the context of a whole page, the clean way to focus the failure info would be to introduce a <code>&lt;section&gt;</code> and then pull it out using a search for the <code>region</code> role:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">diff --git a/src/components/Profile/ProfileCard.tsx b/src/components/Profile/ProfileCard.tsx index 9931d8d..19ab3d9 100644 <span style="color:#f92672">--- a/src/components/Profile/ProfileCard.tsx </span><span style="color:#f92672"></span><span style="color:#a6e22e">+++ b/src/components/Profile/ProfileCard.tsx </span><span style="color:#a6e22e"></span><span style="color:#75715e">@@ -242,11 +242,15 @@ const ContactInfo: React.VoidFunctionComponent&lt;{ </span><span style="color:#75715e"></span> const addressLabel = formatAddress(address) return ( <span style="color:#f92672">- &lt;div className={classes.contactInfo}&gt; </span><span style="color:#f92672"></span><span style="color:#a6e22e">+ &lt;section </span><span style="color:#a6e22e">+ className={classes.contactInfo} </span><span style="color:#a6e22e">+ aria-labelledby=&#34;contactinfo-header&#34; </span><span style="color:#a6e22e">+ &gt; </span><span style="color:#a6e22e"></span> &lt;Typography className={classes.contactInfoHeader} variant=&#34;h6&#34; component=&#34;h3&#34; <span style="color:#a6e22e">+ id=&#34;contactinfo-header&#34; </span><span style="color:#a6e22e"></span> &gt; {t(&#39;ProfileCard.ContactInfo.Header&#39;, &#39;Contact Information&#39;)} &lt;/Typography&gt; <span style="color:#75715e">@@ -302,7 +306,7 @@ const ContactInfo: React.VoidFunctionComponent&lt;{ </span><span style="color:#75715e"></span> className={classes.listItem} /&gt; &lt;/List&gt; <span style="color:#f92672">- &lt;/div&gt; </span><span style="color:#f92672"></span><span style="color:#a6e22e">+ &lt;/section&gt; </span><span style="color:#a6e22e"></span> ) } </code></pre></div><p><em>(That would probably benefit from using some flavor of unique ID generator in case multiple contact info sections got rendered, but let&rsquo;s ignore that for now.)</em></p> <p>(Another way to improve the error output would be to narrow the scope of the test: instead of testing the component as part of a whole page, test the component directly. Then, when <code>screen.findBy</code> barfs, the whole screen is precisely the info we want to see. You might want to do that if this component gets reused elsewhere, but for now, assume it&rsquo;s an implementation detail.)</p> <p>Now we can use a matcher scoped to just that region:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#66d9ef">const</span> <span style="color:#a6e22e">contactInfo</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">within</span>( <span style="color:#a6e22e">await</span> <span style="color:#a6e22e">screen</span>.<span style="color:#a6e22e">findByRole</span>(<span style="color:#e6db74">&#34;region&#34;</span>, { <span style="color:#a6e22e">name</span><span style="color:#f92672">:</span> <span style="color:#e6db74">/contact info/i</span> }) ); <span style="color:#a6e22e">expect</span>( <span style="color:#a6e22e">await</span> <span style="color:#a6e22e">contactInfo</span>.<span style="color:#a6e22e">findByText</span>(<span style="color:#e6db74">&#34;OFFICIAL_any_string?@email.com&#34;</span>) ).<span style="color:#a6e22e">toHaveAttribute</span>(<span style="color:#e6db74">&#34;href&#34;</span>, <span style="color:#e6db74">&#34;mailto:OFFICIAL_any_string!@email.com&#34;</span>); </code></pre></div><p>This uses <a href="https://testing-library.com/docs/dom-testing-library/api-within"><code>within</code></a> to narrow the queried region, which also narrows the &ldquo;nothing found, here&rsquo;s what <em>is</em> there&rdquo; output usefully. Now, a failed element search dumps the entire Contact Info region to the test log, rather than the entire blessed page, so you can plainly see what&rsquo;s what.</p> <h2 id="conclusion">Conclusion</h2> <ul> <li>Be careful what you sync on, because this can needlessly slow your whole test suite. <ul> <li>Syntactically, pay attention to what, precisely, you are <code>await</code>ing.</li> </ul> </li> <li>Make sure the answers to your first-order debugging questions make it into your test failure messaging! (Tack stuff onto the failure exception message if worse comes to worst.) <ul> <li>To do this, break your tests, observe the info they provide about how they broke, and then tune that output so all your immediate first questions are answered. Otherwise you&rsquo;ll find yourself needing to dig up file and line and rummage around, or maybe you&rsquo;ll be running a 15-minute build just to see new debug info because you can&rsquo;t repro the issue locally. (You could also automate breaking your tests with mutation testing as with <a href="https://stryker-mutator.io/">Stryker</a>, but that&rsquo;s a topic for some other day.)</li> </ul> </li> </ul> A Month of Terraform https://jeremywsherman.com/blog/2020/11/21/a-month-of-terraform/ Sat, 21 Nov 2020 16:55:43 +0000 https://jeremywsherman.com/blog/2020/11/21/a-month-of-terraform/ I took Heroku for granted, and a month into setting up my own infra, I now know how much it bought me. A lot of my past work has been infrastructure-adjacent. I often find myself filling in the Build &amp; Integration role - the person that gets continuous integration off the ground and keeps it actually continuing rather than falling flat on its face. But often I&rsquo;ve just been building one of a constellation of services, so the core infrastructure was already there, or I&rsquo;ve been targeting something like Heroku, where you basically pick your poison, git push, and bob&rsquo;s your uncle. <p>I took Heroku for granted, and a month into setting up my own infra, I now know how much it bought me.</p> <p>A lot of my past work has been infrastructure-adjacent. I often find myself filling in the Build &amp; Integration role - the person that gets continuous integration off the ground and keeps it actually continuing rather than falling flat on its face. But often I&rsquo;ve just been building one of a constellation of services, so the core infrastructure was already there, or I&rsquo;ve been targeting something like Heroku, where you basically pick your poison, git push, and bob&rsquo;s your uncle.</p> <p>This time, I&rsquo;m putting the pieces together using the AWS toolkit. And to smoosh them all together, I&rsquo;m using Terraform, because heck if I&rsquo;m going to be hand-writing YAML or JSON and praying it&rsquo;s formatted right. Plus there&rsquo;s more I want to orchestrate than just AWS, like, say, GitLab.</p> <p>I don&rsquo;t wanna talk about AWS just now. It reminds me of learning Foundation &amp; Cocoa - you look at one piece, and it can do so much, and then you gotta put all those individually deep &amp; complex pieces together to do more stuff. I figure if I put in the hours reading docs, learning what&rsquo;s all there, and getting stabbed by the pointy bits, it&rsquo;ll probably all come out fine in the end.</p> <p>So, Terraform.</p> <h2 id="the-good">The Good</h2> <ul> <li>It mostly works!</li> <li>When it doesn&rsquo;t, it generally fails in a useful way, and then I can fix it and try again.</li> <li>There are docs for most things.</li> <li>Autoformatting works great.</li> <li>Linting works pretty well.</li> <li><em>Terraform: Up &amp; Running</em> is excellent, and Terragrunt makes it even easier. Huge thanks to their team for providing the <a href="https://rachelbythebay.com/w/2018/03/23/ducttape/">duct tape</a> we need. 🙌</li> </ul> <h2 id="the-not-so-good">The Not So Good</h2> <ul> <li>terraform-lsp is supposed to provide autocomplete, but it mostly doesn&rsquo;t, in my experience. First it flipped its lid that I dared to have a repo with multiple root modules in it, so I just aimed VS Code at the folder with a single root module. Then the language server says it&rsquo;s all hunky dory AFAICT, and yet it autocompletes nothing beyond bare language syntax. As a result, I&rsquo;m manually referencing docs and writing stuff down and wasting tons of time that tools like autocomplete and integrated linting ought to be saving me from.</li> <li>State files contain secrets in plaintext. (You might enjoy the <a href="https://github.com/hashicorp/terraform/issues/516">six-year-old GitHub issue about the plaintext secrets problem</a>.) You can mark outputs as secret, so they don&rsquo;t get printed at the end of applying your infra spec, but run <code>terraform show</code> instead of <code>terraform apply</code>, and there they are, staring back at you. At least you can lock down and encrypt the S3 bucket holding the state. <ul> <li><a href="https://www.pulumi.com/docs/intro/concepts/config/#secrets">Pulumi&rsquo;s secrets management</a> is far more satisfying. But Pulumi is even more cutting-edge than v0.whatever Terraform, and I expect Hashicorp to keep TF running for a good while, while I&rsquo;m not so confident in Pulumi, so I&rsquo;m using TF. (Hashicorp of course would recommend <a href="https://www.hashicorp.com/products/vault">Vault</a>.)</li> </ul> </li> <li>Annoying asymmetries in the language about how you *<em>declare and reference</em> things in slightly variant ways - I trip over these over and over as a beginner: <ul> <li>You declare locals in a <code>locals</code> block, but you reference them as <code>local.thing</code>, not <code>locals.thing</code>.</li> <li>You declare a variable in a <code>variable</code> block, but you reference it as <code>var.thing</code>.</li> <li>You declare data sources as <code>data &quot;provider_thingy&quot; &quot;my_name_for_this_data&quot;</code>, and then you have to access it as <code>data.provider_thingy.my_name_for_this_data</code>. (This is actually pretty darn consistent, at least. Though, like, why the quotes around the provider thingy?)</li> <li>You declare resources as <code>resource &quot;provider_thingy&quot; &quot;my_name&quot;</code>. But you do NOT reference them as <code>resource.provider_thingy.my_name</code>. Nope, you just reference them as bare <code>provider_thingy.my_name</code>.</li> </ul> </li> <li>For that matter, there are other oddities as well. Pieces of syntax that seem like they should be orthogonal just aren&rsquo;t. <code>for_each</code> stands out here: <ul> <li>You can generate multiple resources by just dropping a <code>for_each</code> in the block: <code>resource &quot;provider_thing&quot; &quot;mine&quot; {}</code> becomes <code>resource &quot;provider_thing&quot; &quot;mine&quot; { for_each = of_these }</code></li> <li>But nested <em>argument</em> blocks require conversion from like <code>setting { namespace = &quot;blah&quot; }</code> to <code>dynamic &quot;setting&quot; { for_each = thingy; content { namespace = &quot;blah&quot; }}</code>. Have fun looking that up a few times.</li> <li>And you can&rsquo;t even use the <code>for_each</code> trick with module imports. It just isn&rsquo;t supported. Sorry, sucks to be you.</li> </ul> </li> <li>Annoying gaps in the docs: <ul> <li><strong>Required vs optional parameters</strong> are not very clearly called out and are not at all segregated. So you get to play the game of &ldquo;what is the minimal skeleton to declare this resource&rdquo;. Actually running it a few times to see what you screwed up takes longer than just looking at the docs and puzzling it out, due to the lengthy iteration times in infra-land (see below).</li> <li><strong>Types are not shown in the docs!!!</strong> All the outputs and arguments are typed. You have to declare those types. It&rsquo;s right there in the code. But the docs don&rsquo;t say what any of the types are. You just hit a type error at runtime. Fun fun!</li> <li><strong>The HCL language is doc&rsquo;d under the CLI tool, not in and of itself.</strong> It was really hard to actually find the docs since my first thought when I have syntax questions isn&rsquo;t &ldquo;let&rsquo;s look at the docs for the tool.&rdquo; It&rsquo;d be like pulling up the manpage for GCC (carefully draw your triangle of art first) when you have a question about C syntax.</li> </ul> </li> <li>Annoying asymmetries in the AWS provider: <ul> <li><strong>Missing links:</strong> Sometimes you get into a &ldquo;can&rsquo;t get there from here&rdquo; situation. Like trying to find the zone ID for an Elastic Beanstalk environment&rsquo;s CNAME so you can aim a Route 53 alias at it. (Hint, you need a completely different resource, the <code>aws_elastic_beanstalk_hosted_zone</code>.)</li> <li><strong>Irregular naming:</strong> <ul> <li>Sometimes something is <code>zone_id</code>, but other times it&rsquo;s maybe just <code>id</code>.</li> <li>Sometimes you can fish stuff out by <code>arn</code>, or maybe by <code>id</code>, or maybe it&rsquo;s by <code>name</code> - good luck. Keep the docs close to hand.</li> <li>(It&rsquo;s totally possible this is inherited from the AWS APIs themselves, but the whole point of an abstraction layer is to make things better and more usable, dangit.)</li> </ul> </li> </ul> </li> </ul> <h2 id="the-different">The Different</h2> <ul> <li><strong>Iteration times are way longer than with even mobile apps.</strong> Like, &ldquo;you&rsquo;re liable to task-switch while waiting to see plan output&rdquo; longer.</li> <li><strong>Testing is a pain.</strong> I haven&rsquo;t pulled in <a href="https://terratest.gruntwork.io">Terratest</a> yet, because anyone maintaining this after me is unlikely to have Go experience, and my focus here isn&rsquo;t building reusable infra anyway - it&rsquo;s building <em>this</em> infra – so I&rsquo;ve just been using <code>bats</code> and Bash shell scripts (with <a href="https://github.com/koalaman/shellcheck/blob/master/README.md">shellcheck</a>, which is amazing) for some after-the-fact sanity checking using the AWS CLI. (Pro tip: Use the community-maintained fork <a href="https://github.com/bats-core/bats-core"><code>bats-core</code></a> rather than the no-longer-maintained sstephenson original.) <ul> <li>Policy assertions feel like a different flavor of test, but the tooling here seems to be fairly immature, with perhaps the exception of if you&rsquo;re targeting Kubernetes.</li> </ul> </li> </ul> <h2 id="summary">Summary</h2> <p>I expect I&rsquo;ll get used to most of the rough edges of the syntax in another month. And Terraform is still v0, so hey, maybe some breaking changes will clear all this mess away. 🤞</p> <p>I&rsquo;m intentionally not getting sucked into hacking around the docs frustrations just now. Or even the <a href="https://github.com/gruntwork-io/terragrunt/issues/432#issuecomment-371467507">very tempting open issue about silencing all the Terragrunt logspew</a>.</p> <p>I do plan to spend a bit of time trying to get autocomplete working for resource and data source types and their arguments/attributes from the language server, at least. That would be a huge help.</p> <p>It still feels like magic to run a command and have infrastructure just…happen. You hit return, wait a bit, and suddenly servers are serving and domains are aliasing and a whole constellation of systems are interoperating. It kinda reminds me of the magic of home automation with blinkenlights, only without any of that messy &ldquo;hardware&rdquo; stuff to break on you.</p> Collected TILs https://jeremywsherman.com/blog/2020/10/10/collected-tils/ Sat, 10 Oct 2020 21:05:39 +0000 https://jeremywsherman.com/blog/2020/10/10/collected-tils/ This post collects a number of &ldquo;Today I Learned&rdquo; messages I&rsquo;d previously sent into a Slack workspace. Git TIL: Two ways to change git&rsquo;s default comment handling so I can write Markdown headings: Use the &ndash;cleanup flag for git commit to set a different cleanup approach, like git commit &ndash;cleanup=scissors Change the comment character to something else with git config core.commentChar ; https://stackoverflow.com/questions/2788092/start-a-git-commit-message-with-a-hashmark TIL: You can ask git to show a file as of a specific version REVSPEC, and less to add line numbers and jump right to your target line, like so: <p>This post collects a number of &ldquo;Today I Learned&rdquo; messages I&rsquo;d previously sent into a Slack workspace.</p> <h2 id="git">Git</h2> <ul> <li> <p>TIL: Two ways to change git&rsquo;s default comment handling so I can write Markdown headings:</p> <ul> <li>Use the &ndash;cleanup flag for git commit to set a different cleanup approach, like git commit &ndash;cleanup=scissors</li> <li>Change the comment character to something else with git config core.commentChar ; <a href="https://stackoverflow.com/questions/2788092/start-a-git-commit-message-with-a-hashmark">https://stackoverflow.com/questions/2788092/start-a-git-commit-message-with-a-hashmark</a></li> </ul> </li> <li> <p>TIL: You can ask git to show a file as of a specific version <code>REVSPEC</code>, and <code>less</code> to add line numbers and jump right to your target line, like so:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">git show REVSPEC:PATH_RELATIVE_TO_GIT_ROOT <span style="color:#ae81ff">\ </span><span style="color:#ae81ff"></span>| less -N +164g </code></pre></div></li> <li> <p>TIL: how to see a merge&rsquo;s conflicts &amp; resolutions: if <code>hash</code> is the merge commit, run either <code>git diff hash hash^1 hash^2</code> or <code>git diff-tree --cc hash</code>. (for me, former will paginate, latter will not.)</p> </li> <li> <p>TIL: I could have just aimed <code>core.hooksPath</code> to a shared hook folder, rather than linking-in my <code>prepare-commit-message</code> hook script to many folders. (Though I do wonder if Husky handles that bit of git configurability at all.)</p> </li> <li> <p>TIL: git comes with a perl module, and git add &ndash;interactive (and so &ndash;patch) is a perl script.</p> <p>why did i learn this? because i wondered how to sniff the comment char in use. even the git perl module punts and falls back to <code>#</code> when it&rsquo;s <code>auto</code>, though.</p> </li> <li> <p>TIL: <code>git whatchanged</code> is a thing. it&rsquo;s like <code>git log --name-status</code> only less typing, and with file modes in your face.</p> </li> <li> <p>TIL: VSCode&rsquo;s GitLens has a &ldquo;heat map&rdquo; feature that shows <a href="http://hci.ucsd.edu/media/uploads/hci_papers/JH1992-3.pdf">Edit Wear</a> in your editor.</p> </li> <li> <p>TIL: to force an update with pull without it trying to do some derpy merge, i can <code>git fetch</code> then <code>git reset --hard FETCH_HEAD</code>, without needing to worry about what the actual branch name is that i&rsquo;m on.</p> </li> <li> <p>TIL: Mercurial has a nifty &ldquo;ignore this reformatting&rdquo; cue to its blame engine. throw <code># skip-blame reason</code> in the commit message, and <code>hg praise</code> will ignore that commit.</p> <blockquote> <p>a sub-issue of adding all the <code>b</code> prefixes and reformatting code is that it would break annotate/blame more than was tolerable. The latter issue was addressed by teaching Mercurial&rsquo;s annotate/blame feature to skip revisions. The project now has a convention of annotating commit messages with <code># skip-blame &lt;reason&gt;</code> so structural only changes can easily be ignored when performing an annotate/blame.</p> </blockquote> <p><a href="https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/">https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/</a></p> </li> <li> <p>TIL: <code>git blame</code> since v2.23 can ignore a list of commits. pass the list one-by-one with <code>--ignore-rev=REV</code>, or en masse with <code>--ignore-revs-file FILE</code>, which admits comments. you can automate the process by configuring <code>blame.ignoreRevsFile</code>; a popular filename is <code>.git-blame-ignore-revs</code>. (i went looking for git functionality after the hg one and found <a href="https://www.moxio.com/blog/43/ignoring-bulk-change-commits-with-git-blame">this summary</a>.)</p> </li> <li> <p>TIL: git log at last applies the <code>.mailmap</code> rewrites previously only applied by shortlog. ah, a tidy history.</p> </li> <li> <p>TIL: to list your branches ordered from most-recently committed to to least:</p> <pre><code>git branch --sort=-committerdate </code></pre><p>(handy when you need to go 2 branches back, so can&rsquo;t just use <code>git checkout -</code>)</p> </li> <li> <p>TIL: how to word-wrap commit messages in Fork: <strong>Preferences &gt; Show page guideline</strong> then Right click on commit description field -&gt; <strong>Wrap Paragraph at Ruler</strong> <a href="https://github.com/ForkIssues/Tracker/issues/879">https://github.com/ForkIssues/Tracker/issues/879</a></p> </li> </ul> <h2 id="github">GitHub</h2> <ul> <li> <p>TIL: you can use .gitattributes to fix up weird language use listings on GitHub <a href="https://github.com/github/linguist/blob/master/README.md#overrides">https://github.com/github/linguist/blob/master/README.md#overrides</a></p> <ul> <li>fixes: x is language Y: someGlob linguist-language=Y</li> <li>ignores: treat x as vendored, generated, or docs: someGlob linguist-{vendored,generated,documentation}=true - or just skip detecting it for whatever reason: linguist-detectable=false</li> <li>missed: x should be considered detectable linguist-detectable=true</li> </ul> </li> <li> <p>TIL: there are a lot of restrictions on code search on github, which is why its code search always seems so lousy next to cloning and searching locally <a href="https://help.github.com/articles/searching-code/#considerations-for-code-search">https://help.github.com/articles/searching-code/#considerations-for-code-search</a></p> </li> <li> <p>TIL: if you use AsciiDoc on GitHub, you can just write :toc: to get a table of contents. beats the pants off manual doctoc&rsquo;ing!</p> </li> <li> <p>TIL: github knows how to attribute commits to multiple authors via a <code>\r\rCo-authored-by: NAME2 &lt;EMAIL2&gt;\rCo-authored-by: NAME3 &lt;EMAIL3&gt;\r…</code> convention in the commit message body. <a href="https://help.github.com/en/articles/creating-a-commit-with-multiple-authors">https://help.github.com/en/articles/creating-a-commit-with-multiple-authors</a></p> <p>via: <a href="https://tuple.app/pair-programming-guide/template#4-configure-git-to-share-credit">https://tuple.app/pair-programming-guide/template#4-configure-git-to-share-credit</a></p> </li> <li> <p>TIL: github has pretty nice web preview for CSV files. renders as a table, bolds the first line, and lets you search within the table.</p> </li> <li> <p>TIL: github now supports multiple issue &amp; pr templates. but really, i&rsquo;m just here to link you to a CYOA template builder: <a href="https://www.talater.com/open-source-templates/#/">https://www.talater.com/open-source-templates/#/</a></p> </li> <li> <p>TIL: if you fork a GitHub repo using large file storage, the parent repo gets to pay for all the storage &amp; bandwidth.</p> <ul> <li>note to self: never GLFS in a public repo.</li> </ul> <blockquote> <p>In forks, bandwidth and storage usage count against the root of the repository network. (<a href="https://help.github.com/en/articles/about-storage-and-bandwidth-usage">https://help.github.com/en/articles/about-storage-and-bandwidth-usage</a>)</p> </blockquote> </li> </ul> <h2 id="trivia">Trivia</h2> <ul> <li> <p>TIL: The origin story of Hamcrest (an anagram of &ldquo;matchers&rdquo;): Folks wanted to use jMock2&rsquo;s Matchers in production code, but felt weird having to pull in a test-centric lib as part of their prod code. <a href="http://www.natpryce.com/articles/000662.html">http://www.natpryce.com/articles/000662.html</a></p> </li> <li> <p>TIL: Lewis Carroll found a workaround for burning the midnight oil. &ldquo;Carroll invented Nyctography because he was often awakened during the night with thoughts that needed to be written down immediately, and didn&rsquo;t want to go through the lengthy process of lighting a lamp just to have to extinguish it shortly thereafter. (<a href="https://en.wikipedia.org/wiki/Nyctography)%22">https://en.wikipedia.org/wiki/Nyctography)&quot;</a></p> </li> <li> <p>TIL: the US actually minted some half-cent coins <a href="https://en.wikipedia.org/wiki/Half_cent_(United_States_coin)">https://en.wikipedia.org/wiki/Half_cent_(United_States_coin)</a></p> </li> <li> <p>TIL: JIRA is a shortened form of &ldquo;Gojira&rdquo; derived after riffing on Bugzilla <a href="https://confluence.atlassian.com/pages/viewpage.action?pageId=223219957">https://confluence.atlassian.com/pages/viewpage.action?pageId=223219957</a></p> </li> <li> <p>TIL: Poor ventilation in conf rooms can have cognitive impacts:</p> <blockquote> <p>three people quietly sitting in a mid-size conference room produced CO2 levels that within 60 minutes, reached concentrations high enough to impair their ability to make the right decisions. In a corporate world where Directors regularly pile into closed-door conference rooms for hours on end, making the most important planning decisions for their companies, this was a disturbing realization. (<a href="http://blog.gigabase.org/en/contents/132">http://blog.gigabase.org/en/contents/132</a>)</p> </blockquote> </li> </ul> <p>-TIL: inst, ult, and prox <a href="http://worldwidewords.org/weirdwords/ww-ult2.htm">http://worldwidewords.org/weirdwords/ww-ult2.htm</a> (this month, last month, next month). they were disturbingly common in 19th century business letters, and shorthand learning materials of the same period. &ldquo;i am in receipt of your esteemed favor of 13th ult.&rdquo;</p> <ul> <li> <p>TIL: you can get little bumpers for your keys to deafen the thunderous typing. everyone i video call with will benefit from this. &gt;.&lt;</p> </li> <li> <p>TIL: “to be across” is Aussie biztalk for “know about, be informed of, understand” or similar <a href="https://english.stackexchange.com/questions/122379/usage-of-to-be-across">https://english.stackexchange.com/questions/122379/usage-of-to-be-across</a></p> </li> <li> <p>TIL: How to use Seyès paper: caps go up 3, loops go up 3, sticks go up 2, x-height is 1, stick descenders down 1, loop descenders down 2. <a href="https://www.crapaud-chameau.com/2017/04/apprendre-a-bien-ecrire-en-cursive.html">https://www.crapaud-chameau.com/2017/04/apprendre-a-bien-ecrire-en-cursive.html</a> (the vertical lines are tabstops, if you&rsquo;re laying out tables)</p> </li> <li> <p>TIL: sometimes a pug is a moth <a href="https://en.wikipedia.org/wiki/Foxglove_pug">https://en.wikipedia.org/wiki/Foxglove_pug</a></p> <p>ominous:</p> <blockquote> <p>Soon after hatching it seals the mouth of the flower with silk and feeds on the reproductive parts of the flower.</p> </blockquote> </li> <li> <p>TIL: &ldquo;After many incidents of serious flooding throughout the 19th century, the owners of the two-, three-, and four-story buildings in Sacramento simply abandoned the ground floors and constructed raised sidewalks level with the first floor. Over the years the roadways were also raised to just below the new sidewalk level, effectively elevating the entire town.&rdquo; <a href="https://www.atlasobscura.com/places/the-original-street-level-of-sacramento-sacramento-california">https://www.atlasobscura.com/places/the-original-street-level-of-sacramento-sacramento-california</a></p> <p>h/t David M. Deller</p> </li> <li> <p>TIL: some new sales jargon:</p> <ul> <li>PBO: positive business outcome</li> <li>EB: economic buyer - the person who can release funding to pay for your services</li> <li>RC: required capabilities - skills/resources needed to reach a PBO</li> <li>proof point: a fact you can point to to support a claim about your capabilities and differentiators, such as a case study or customer reference</li> </ul> <p>(most of these seem pretty self-explanatory to me, but that someone is motivated to abbreviate these specific things vs others makes me re-evaluate the importance of those terms, and my fluency with them, in sales thinking.)</p> </li> <li> <p>TIL: <a href="https://en.wikipedia.org/wiki/Kin-Ball">https://en.wikipedia.org/wiki/Kin-Ball</a> is a thing. 48-inch ball, three teams on the floor.</p> </li> <li> <p>TIL: there&rsquo;s a book of quotes roasting french authors, the &ldquo;dictionary of literary insults&rdquo; <a href="https://www.amazon.com/Dictionnaire-Injures-Litteraires-French-Chalmin/dp/2253162361/">https://www.amazon.com/Dictionnaire-Injures-Litteraires-French-Chalmin/dp/2253162361/</a></p> <p>&ldquo;Mallarmé, untranslatable, even in French&rdquo;</p> </li> <li> <p>TIL: &ldquo;ratioing&rdquo;: when you get twice as many replies as retweets. <a href="https://www.merriam-webster.com/words-at-play/words-were-watching-ratio-ratioed-ratioing">https://www.merriam-webster.com/words-at-play/words-were-watching-ratio-ratioed-ratioing</a></p> <p>(if anything ever said &ldquo;twitter is not a platform for discussion&rdquo;, considering replies as a negative may be that…)</p> </li> </ul> <h2 id="cli-fun">CLI Fun</h2> <ul> <li> <p>TIL: a bash <code>for</code> loop defaults to iterating over the positional arguments if you omit the <code>in</code> part. thanks, perplexing example from getopt(1)!</p> <p>also, argparsing in bash is bad enough that <a href="https://argbash.io/">https://argbash.io/</a> to generate all the boilerplate for you exists.</p> <p>there&rsquo;s also a bash-builtin <code>getopts</code> vs standalone <code>getopt</code> utility low-stakes holy war apparently. i&rsquo;ll chuck that in the bin next to emacs vs vi and tabs vs spaces for a later date.</p> </li> <li> <p>TIL: Bash will only load aliases in an interactive shell, not just a login shell:</p> <ul> <li><code>bash -lc &quot;some-alias&quot;</code> gives &ldquo;bash: some-alias: command not found&rdquo;.</li> <li><code>bash -ilc &quot;some-alias&quot;</code> works a treat.</li> </ul> </li> <li> <p>TIL: the &ldquo;run alpine/socat to bridge to an exposed but unpublished container port&rdquo; Docker trick. lets you basically retroactively publish it without having to re-run the container with new config. See: <a href="https://stackoverflow.com/a/49903374/72508">https://stackoverflow.com/a/49903374/72508</a> (which may have my edits applied already)</p> </li> <li> <p>TIL the xnu kernel has 3 flavors of use-after-free detection in addition to its own guard malloc support. neat subsystem. :)</p> </li> <li> <p>TIL: If all else fails for entering something into vim, you can use <code>Ctrl-V u codepoint</code> to enter it interactively, or if you&rsquo;re composing a regex, you can encode it using something like <code>\%u1234</code>. (That last one is handy for copy-paste or scripted use.)</p> <p>and you can re-learn the first one in <code>:help unicode</code> or more specifically <code>:help utf-8-typing</code>, and the second one in <code>:help regex</code>, specifically <code>:help pattern-overview</code>. (as i have done several times now. here&rsquo;s hoping TILing it makes it stick. &gt;.&lt;)</p> </li> <li> <p>TIL: ld (the linker) still has cool tricks up its sleeve. ld can just…wrap a binary into a .o with start and end pointers as its symbols: <a href="https://flak.tedunangst.com/post/embedding-binary-objects-in-c">embedding binary objects in c</a></p> </li> <li> <p>TIL: Powershell aliases <code>wget</code> and <code>curl</code> to <code>iwr</code> aka <code>Invoke-Web-Request</code>. That plus help is enough to nudge you the right direction.</p> </li> </ul> <h2 id="postgres">Postgres</h2> <ul> <li> <p>TIL: <a href="https://dbfiddle.uk/?rdbms=postgres_11">DBFiddle</a> is a thing. gives you an online workbench/demo of querying using a specific DB engine, like jsfiddle, but for databases.</p> </li> <li> <p>TIL: <code>allballs</code> is a special value that Postgres hardcodes to the UTC time of day 00:00:justkeepwriting0 <a href="https://www.postgresql.org/docs/8.3/datatype-datetime.html#AEN5025">https://www.postgresql.org/docs/8.3/datatype-datetime.html#AEN5025</a> (edited)</p> </li> <li> <p>TIL: using <code>psql</code> variables (h/t Liv Vitale for prompting me to review this)</p> <p><code>\set variable [blah…]</code> &ndash; create or update a variable. multiple args will be concatenated. no arg sets to empty string or, for control variables, is equivalent to <code>\set VAR on</code>. <code>\echo :variable</code> &ndash; view the value of that variable <code>\set</code> &ndash; view all variables and their values <code>:{?variable}</code> &ndash; returns TRUE if variable exists, FALSE if not <code>:'variable'</code> &ndash; quoted as a literal, such as a string or number <code>:&quot;variable&quot;</code> &ndash; quoted as an identifier, such as a table name <code>\unset variable</code></p> <p>NOTE: psql reserves variables comprising only uppercase alphanumerics. stick to lowercase and you&rsquo;ll be fine.</p> <p>Bulk insert trick:</p> <pre><code class="language-psql" data-lang="psql">\set content `cat my_file.txt` INSERT INTO my_table VALUES (:'content'); </code></pre><p>References: <a href="https://www.postgresql.org/docs/11/app-psql.html#APP-PSQL-VARIABLES">https://www.postgresql.org/docs/11/app-psql.html#APP-PSQL-VARIABLES</a> <a href="https://www.postgresql.org/docs/11/app-psql.html#APP-PSQL-INTERPOLATION">https://www.postgresql.org/docs/11/app-psql.html#APP-PSQL-INTERPOLATION</a></p> </li> </ul> <h2 id="webdev">Webdev</h2> <ul> <li> <p>TIL: TypeScript&rsquo;s <code>Record&lt;KeyType, ValueType&gt;</code> can be used to statically require an enum lookup table be exhaustive. <a href="https://www.typescriptlang.org/docs/handbook/advanced-types.html#mapped-types">https://www.typescriptlang.org/docs/handbook/advanced-types.html#mapped-types</a></p> <p>so this enum decl + incomplete map declared as a Record:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-typescript" data-lang="typescript"><span style="color:#66d9ef">export</span> <span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">EventCode</span> { <span style="color:#a6e22e">Click</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;email_click&#39;</span>, <span style="color:#a6e22e">Reply</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;email_reply&#39;</span>, } <span style="color:#66d9ef">export</span> <span style="color:#66d9ef">const</span> <span style="color:#a6e22e">Color</span>: <span style="color:#66d9ef">Record</span><span style="color:#f92672">&lt;</span><span style="color:#a6e22e">EventCode</span>, <span style="color:#66d9ef">string</span><span style="color:#f92672">&gt;</span> <span style="color:#f92672">=</span> { [<span style="color:#a6e22e">EventCode</span>.<span style="color:#a6e22e">Click</span>]<span style="color:#f92672">:</span> <span style="color:#e6db74">&#39;purple&#39;</span>, } </code></pre></div><p>produces a compiler error due to the missing enum case:</p> <blockquote> <p>EventCode.ts:24:14 - error TS2741: Property <code>'email_reply'</code> is missing in type <code>'{ [EventCode.Click]: string; }'</code> but required in type <code>'Record&lt;EventCode, string&gt;'</code>.</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-typescript" data-lang="typescript"> <span style="color:#ae81ff">24</span> <span style="color:#66d9ef">export</span> <span style="color:#66d9ef">const</span> <span style="color:#a6e22e">Color</span>: <span style="color:#66d9ef">Record</span><span style="color:#f92672">&lt;</span><span style="color:#a6e22e">EventCode</span>, <span style="color:#66d9ef">string</span><span style="color:#f92672">&gt;</span> <span style="color:#f92672">=</span> { <span style="color:#f92672">~~~~~</span> </code></pre></div></blockquote> </li> <li> <p>TIL: you can do all kinds of evil things with VSCode tasks (like rig &lsquo;em up to ssh into a vagrant vm and run a docker exec command to trigger your tests)</p> </li> <li> <p>TIL: android chrome, when typed into using google&rsquo;s software keyboard, doesn&rsquo;t on providing any meaningful info on the key pressed to <code>onkeydown</code> and <code>onkeyup</code>, which sure seem like they ought to be getting some info about a key. the events are still sent, but not anything about the keys involved. further eroding the foundations on which my precarious frontend web reality is built. <a href="https://bugs.chromium.org/p/chromium/issues/detail?id=118639#c261">https://bugs.chromium.org/p/chromium/issues/detail?id=118639#c261</a></p> </li> <li> <p>TIL: typescript can smartly narrow the type after type-checking the discriminator in a discriminated union, but <strong>only if</strong> the discriminator is a <strong>single type</strong> in each branch. it&rsquo;s not smart enough to handle a union in one case.</p> <p>so this plays nice:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-typescript" data-lang="typescript"><span style="color:#66d9ef">@typedef</span> { {<span style="color:#a6e22e">flavor</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#39;date&#39;</span>, <span style="color:#a6e22e">value</span>: <span style="color:#66d9ef">Date</span>} <span style="color:#f92672">|</span> {<span style="color:#a6e22e">flavor</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#39;time&#39;</span>, <span style="color:#a6e22e">value</span>: <span style="color:#66d9ef">Date</span>} <span style="color:#f92672">|</span> {<span style="color:#a6e22e">flavor</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#39;text&#39;</span>, <span style="color:#a6e22e">value</span>: <span style="color:#66d9ef">string</span>} } <span style="color:#a6e22e">Filter</span> </code></pre></div><p>and you get this desirable smartcast behavior: <code>if (it.flavor === 'date') { /* now it.value is a Date here */ }</code></p> <p>but this otherwise equivalent version does not:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-typescript" data-lang="typescript"><span style="color:#66d9ef">@typedef</span> { {<span style="color:#a6e22e">flavor</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#39;date&#39;</span> <span style="color:#f92672">|</span> <span style="color:#e6db74">&#39;time&#39;</span>, <span style="color:#a6e22e">value</span>: <span style="color:#66d9ef">Date</span>} <span style="color:#f92672">|</span> {<span style="color:#a6e22e">flavor</span><span style="color:#f92672">:</span> <span style="color:#e6db74">&#39;text&#39;</span>, <span style="color:#a6e22e">value</span>: <span style="color:#66d9ef">string</span>} } <span style="color:#a6e22e">Filter</span> </code></pre></div><p>in this case, you get: <code>if (it.flavor === 'date') {/* it.value is still 'date' | 'time' | 'text' :( */ }</code></p> </li> <li> <p>TIL: OpenAPI specs can handle XML pretty well these days. you can rename something by adding a child <code>{xml: {name: 'whatevs'}}</code>, and flag a property as an attribute with <code>{xml: {attribute: true}}</code>. <a href="https://swagger.io/docs/specification/data-models/representing-xml/">https://swagger.io/docs/specification/data-models/representing-xml/</a></p> </li> <li> <p>TIL: how YAML&rsquo;s many string syntaxes play out in practice through a convenient single-issue site: <a href="https://yaml-multiline.info/">https://yaml-multiline.info/</a></p> <p>(In the past, I&rsquo;ve looked at <a href="https://camel.readthedocs.io/en/latest/yamlref.html">https://camel.readthedocs.io/en/latest/yamlref.html</a>. That&rsquo;s still really valuable for YAML in general, but for string formatting, this new site is much more to the point.)</p> </li> </ul> <h2 id="slack">Slack</h2> <ul> <li> <p>TIL: Slack now reminds you of the other party&rsquo;s current time in a DM. (Not in a group DM, though, so ping the heck out of a combined Pacific and Mountain Time crowd at 0900 Eastern if you want.)</p> </li> <li> <p>TIL: In Slack, <code>/remind snooze DURATION</code> is a thing. should be handy for my next pto.</p> </li> <li> <p>TIL: slack will keep a specific whitelist of apps alive even after the member who wired them into a workspace leaves. this whitelist includes mission-critical apps like CI, performance alerting, communication tooling, and the one your workspace clearly cannot survive without for even a second: giphy. &gt;.&gt; <a href="https://slack.com/help/articles/360000446446-Manage-deactivated-members-apps-and-integrations-Manage-deactivated-members-apps-and-integrations">https://slack.com/help/articles/360000446446-Manage-deactivated-members-apps-and-integrations-Manage-deactivated-members-apps-and-integrations</a></p> </li> <li> <p>TIL: that Slack finally added actual hyperlinks. (at least with their new text editor - not with markdown.) the niftiest way is to select a region of the message you&rsquo;re editing, then paste a URL to linkify it.</p> </li> <li> <p>TIL: creating a scheduled time and date workflow in Slack is dead easy. (but you can&rsquo;t use the fancy new markup in the messages you send from a workflow, so no pretty hyperlinks, alas. and unlike with reminders, mentions don&rsquo;t work, so no @here to wake up the channel for a scheduled meeting.)</p> </li> <li> <p>TIL: the secret slack shortcut to pick up a search where you left off is Cmd-g:</p> <blockquote> <p>When you find yourself in a situation where you need to return to a recent search, a function you may find useful would be to use the keyboard shortcut: CMD+G. This shortcut opens the previous search results screen if there was a cached search within the last 5 minutes. If there is no recent search cached, the search window will be pulled up blank for a quick clean search.</p> </blockquote> </li> <li> <p>TIL: Cmd-. opens/closes the right sidebar in Slack. (Not what I would have guessed, being used to that keycombo&rsquo;s &ldquo;cancel current operation&rdquo; classic Mac mapping.)</p> <p>(There&rsquo;s no menu item, but at least this one is actually doc&rsquo;d in the in-app Cmd-? list, unlike the Cmd-g one to reopen the search dialog with the contents of a recent search.)</p> </li> </ul> <h2 id="other-software">Other Software</h2> <ul> <li> <p>TIL: If you click on a label swatch in Trello, they expand to show the text for the label on the cards. So much more readable at a glance!</p> </li> <li> <p>TIL: Exchange encrypted messaging is darn clever. &ldquo;Non-Office 365 message recipients can authenticate and read protected messages using their consumer Google or Yahoo accounts, in addition to a One-Time Passcode and a Microsoft account.&rdquo; (you click a link in the email, get bounced to a webpage, auth, and boom, there&rsquo;s your message) <a href="https://products.office.com/en-us/exchange/office-365-message-encryption">https://products.office.com/en-us/exchange/office-365-message-encryption</a></p> </li> <li> <p>TIL: how to boss jira around with commit messages like i do pivotal: <a href="https://confluence.atlassian.com/jirasoftwarecloud/processing-issues-with-smart-commits-788960027.html">https://confluence.atlassian.com/jirasoftwarecloud/processing-issues-with-smart-commits-788960027.html</a></p> <p>(didn&rsquo;t know you could do that with pivotal, either? check out <a href="https://www.pivotaltracker.com/help/articles/github_integration/#using-the-github-integration-commits">https://www.pivotaltracker.com/help/articles/github_integration/#using-the-github-integration-commits</a> )</p> <p>want to automate that by copying it in from the branch name? look into the <code>prepare-commit-msg</code> git hook</p> </li> <li> <p>TIL: Vivaldi has named tab session support <a href="https://help.vivaldi.com/article/session-management/">https://help.vivaldi.com/article/session-management/</a> (resembles tab groups in firefox when uing the <a href="https://addons.mozilla.org/en-US/firefox/addon/simple-tab-groups/">simple tab groups extension</a>)</p> </li> <li> <p>TIL: i misunderstood SalesForce relative date filters - the relative times are ranges, not instants:</p> <ul> <li>mistaken instant interpretation: i thought &ldquo;date &lt; NEXT 3 WEEKS&rdquo; meant &ldquo;date is before 3 weeks from now&rdquo;, which would include events happening during the next 3 weeks. :x:</li> <li>but actually &ldquo;date &lt; NEXT 3 WEEKS&rdquo; means &ldquo;date is before the time range beginning midnight of the first day of next week and stretching for the 3 weeks after&rdquo;, so &ldquo;&lt; NEXT 3 WEEKS&rdquo; actually means &ldquo;does not happen during the next 3 weeks.&rdquo; !</li> </ul> <p>(= with these ranges acts like &ldquo;falls in&rdquo; or &ldquo;is during&rdquo;, so trading &lt; out for &lt;= fixed my reporting bug.)</p> <p>Read more: <a href="https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select_dateformats.htm">https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select_dateformats.htm</a></p> </li> <li> <p>TIL: you can get focus follows mouse on macOS if you use voiceover. Open VoiceOver Utility, pick Navigation from the sidebar, then check &ldquo;Synchronize keyboard focus and VoiceOver cursor&rdquo; and pick &ldquo;Mouse pointer: Moves VoiceOver cursor&rdquo;.</p> </li> <li> <p>TIL: zoom supports chatbots for zoom chat <a href="https://marketplace.zoom.us/docs/guides/chatbots/sending-messages">https://marketplace.zoom.us/docs/guides/chatbots/sending-messages</a></p> </li> <li> <p>TIL: if you search for &ldquo;followup:actionitems&rdquo; in Drive, it&rsquo;ll list all the docs with comments assigned to you for action.</p> </li> <li> <p>TIL: Pro Mouse&rsquo;s overlays survive screensharing, so if you regularly need to do callouts on your screen, that might be handy. h/t Jacob Bullock</p> </li> <li> <p>TIL: how to submit a Cigna Vision claim online. &ldquo;Find a Form&rdquo; only gives you a paper PDF. instead:</p> <ul> <li>log into cigna.com</li> <li>select Coverage &gt; Vision</li> <li>Visit Cigna Vision</li> <li>select tab Claims &amp; Reimbursement</li> <li>expand the third accordion, &ldquo;Customer Reimbursement Form&rdquo;</li> <li>click &ldquo;Continue&rdquo;</li> </ul> <p>if you&rsquo;re lucky, this&rsquo;ll just jump you straight to the start of the form: <a href="https://cigna.vsp.com/out-of-network1.html">https://cigna.vsp.com/out-of-network1.html</a> there&rsquo;s an attachment limit of 3 files with a max of 5 MB per file and some filename restrictions (they don&rsquo;t like $). but it&rsquo;s way easier than mailing a paper form.</p> </li> <li> <p>TIL: Pivotal Tracker can be told about deployments. <a href="https://www.pivotaltracker.com/blog/2020-01-24-ci-cd-integrations">https://www.pivotaltracker.com/blog/2020-01-24-ci-cd-integrations</a> (The setup tool will write Jenkins Pipeline or Bash snippets for you, then you just paste &lsquo;em into the right places.)</p> </li> </ul> <h2 id="bad-jokes">Bad Jokes</h2> <ul> <li>TIL: <a href="https://meet.google.com/ia-cthulhu-phtagn">https://meet.google.com/ia-cthulhu-phtagn</a> 404s. alas, He slumbers still.</li> </ul> Google Room and The Definitive Guide to SQLite https://jeremywsherman.com/blog/2018/12/01/google-room-and-the-definitive-guide-to-sqlite/ Sat, 01 Dec 2018 23:27:39 +0000 https://jeremywsherman.com/blog/2018/12/01/google-room-and-the-definitive-guide-to-sqlite/ Why read a book on SQLite? The short answer: Because I needed to write a migration for a Google Room database. Limited past exposure I&rsquo;ve mostly worked on Mac &amp; iOS apps: iOS: The data all lives server-side, and persisting it is someone else&rsquo;s problem. macOS: A database, even an embedded one like SQLite, just never was the right tool. What do you mean, not the right tool? Some folks reach for a database straight out of the gate. <h2 id="why-read-a-book-on-sqlite">Why read a book on SQLite?</h2> <p>The short answer: Because I needed to write a migration for a Google Room database.</p> <h3 id="limited-past-exposure">Limited past exposure</h3> <p>I&rsquo;ve mostly worked on Mac &amp; iOS apps:</p> <ul> <li>iOS: The data all lives server-side, and persisting it is someone else&rsquo;s problem.</li> <li>macOS: A database, even an embedded one like SQLite, just never was the right tool.</li> </ul> <h4 id="what-do-you-mean-not-the-right-tool">What do you mean, not the right tool?</h4> <p>Some folks reach for a database straight out of the gate. (Most backend frameworks, like Rails, sure seem pretty keen on baking in that architectural decision.)</p> <p>But <strong>simple file serialization is often a great choice:</strong></p> <ul> <li>easy to work with (cat, ls, rg)</li> <li>simple to manage (rm, cp, mv, ed)</li> <li>easy to backup and restore</li> <li>no DBA headaches</li> <li>good enough surprisingly often</li> </ul> <p>Most static blog generators seem to agree with this sentiment.</p> <p>Even in cases where you think you might eventually need a database, if you don&rsquo;t jump to conclusions, you may just never get pushed into that corner. One example is as Robert Martin tells of the acceptance testing tool <strong>FitNesse:</strong> they put off picking a DB for a couple years, then ultimately realized the project didn&rsquo;t need one after all. It shipped as, and still is, a flat-file wiki.</p> <h4 id="yes-sqlite-would-still-have-dba-headaches">Yes, SQLite would still have DBA headaches</h4> <p>You&rsquo;d think it wouldn&rsquo;t. You might think the same thing about a Berkeley DB key–value store. But the problems still sneak up on ya. Don&rsquo;t take my word for it – consider <a href="https://blog.plover.com/prog/bug/big-bug.html">MJD&rsquo;s thoughts</a> at the end of a bug hunt undertaken to get his blog generator populating the &ldquo;subtopics&rdquo; sidebar again:</p> <blockquote> <p>I am sick of DB files. I am never using them again. I have been bitten too many times. From now on I am doing the smart thing, by which I mean the dumb thing, the worse-is-better thing: I will read a plain text file into memory, modify it, and write out the modified version whem I am done. It will be simple to debug the code and simple to modify the database.</p> </blockquote> <h2 id="what-about-coredata">What about CoreData?</h2> <p>Cocoa&rsquo;s CoreData defaults to using SQLite as an implementation detail in its quest to simultaneously provide object graph persistence. But that&rsquo;s an implementation detail: the file format is undocumented and subject to change at Apple&rsquo;s whim. (As a bonus, you get platform data-lock from this. I&rsquo;ve had cases where CoreData would have made sense, except that the file format needed to work outside of Apple platforms. Ah, well.) And the file format issues are in addition to the fun with thread containment, though that&rsquo;s gotten to be substantially less of a problem over the years. So I had no reason to get cozy with SQLite to date, due to CoreData abstracting it entirely, and due to even CoreData winding up not being the right tool for me entirely too often (did I mention I like flat files and avoiding data lock-in?).</p> <h2 id="but-now-room">But Now, Room</h2> <p>But Google&rsquo;s blessed persistence framework is <a href="https://developer.android.com/training/data-storage/room/">Room</a>. Unlike CoreData, Room is explicitly a SQLite wrapper. It aims to smooth some rough edges (writing DDL, un/marshaling data between rows and POJOs) and codify best practices around using SQLite on a (maybe pretty crummy) mobile device.</p> <h3 id="some-rough-edges-smoothed">Some Rough Edges Smoothed</h3> <ul> <li>It&rsquo;ll write DDL for you, based on your entity classes, so you don&rsquo;t have to. (You can even crib from this when writing your migration.)</li> <li>It&rsquo;ll check your query syntax at compile-time, so you don&rsquo;t have to wait till runtime for an attempted query to blow up in your face.</li> <li>It&rsquo;ll save off the schema description for you as a simple numbered JSON file, so you can look at it, and so the provided test tooling can help you easily test your migrations.</li> <li>It&rsquo;ll un/marshal data between database rows and POJOs (or POKOs, I suppose, with Kotlin).</li> <li>It&rsquo;ll vend a reactive stream for your query (by watching the table, rerunning the query, and pushing out the new value), so you can easily bind your UI to the DB. You got your choice of receiving your reactive stream as the very rich RxJava 2 data types or as Google&rsquo;s simpler LiveData.</li> </ul> <h3 id="some-best-practices-codified">Some Best Practices Codified</h3> <ul> <li>You can&rsquo;t hang yourself with an N+1 query, because it just won&rsquo;t do them for you.</li> <li>Room will throw an exception if you try to work with the DB on the UI thread. No <a href="https://developer.apple.com/library/archive/technotes/tn2124/_index.html#//apple_ref/doc/uid/DTS10003391-CH1-SECCOREDATA">magic debugging preference</a> is required; it just does this, all the time. (Though I&rsquo;ll grant that its exception doesn&rsquo;t have anywhere near the panache of <code>__Multithreading_Violation_AllThatIsLeftToUsIsHonor__</code>.)</li> <li>Making sure you don&rsquo;t accidentally change something and forget to bump your schema version and provide a migration</li> </ul> <h3 id="some-rough-edges-remain">Some Rough Edges Remain</h3> <p>It&rsquo;ll generate the schema for you, but to move between schemas, you&rsquo;ve gotta write the up-migration by hand. (You don&rsquo;t have to write a down-migration. Room doesn&rsquo;t support down-migration. Ever onward!)</p> <p>So that&rsquo;s what led me to read through a book on SQLite over a couple evenings. What follows are the notes I jotted down when I finished using <a href="https://www.logsit.com/">LogsIt</a>.</p> <h2 id="aside-rooms-design-is-swell">Aside: Room&rsquo;s Design Is Swell</h2> <p>I really like Room&rsquo;s design. It builds on top of a rock-solid and well-understood piece of tech (SQLite), there&rsquo;s zero magic, you can readily dump the DB and poke around using the <code>sqlite3</code> CLI tool to test and explore your queries, it codifies rather than prescribes best practice…</p> <p>My favoritest thing, though, is that it vends POJOs. You don&rsquo;t have to wed the heart of your app to a vendor framework just to get easy queries and streaming UI updates for free. You don&rsquo;t have thread-bound crash-bombs lobbing through 99% of your app, or fight to keep it at arm&rsquo;s length (as Dave DeLong advocates in point 8 of <a href="https://davedelong.com/blog/2018/05/09/the-laws-of-core-data/">&ldquo;The Laws of Core Data&rdquo;</a>) to avoid that. There&rsquo;s no &ldquo;will it fault? will it boom?&rdquo; concern. They&rsquo;re just objects. Plain, simple objects.</p> <p>Anyway, on to the reading notes.</p> <h1 id="reading-notes">Reading Notes</h1> <p>Read a Book: November 14, 2018 at 21:57 Notes: Mike Owens, Grant Allen. <em>The Definitive Guide to SQLite, Second Edition.</em> Apress, November 2010. 9781430232254. Via Safari.</p> <p>Read for background when writing migrations for Android Room.</p> <p>Skipped in-depth coverage of C API, of other language bindings, and of the iOS &amp; Android walkthroughs. Also kinda glazed over the shared cache stuff.</p> <p>New to me:</p> <ul> <li> <p>Manifest typing and coercions. Odd middle ground between strict and duck typing that ends up as “will store anything you throw at it, but acts like strict if you do too”.</p> </li> <li> <p>Blobs degrade to a linked list when they exceed the db pagesize.</p> </li> <li> <p>Gory details of locking schemes and failure modes. You can deadlock yourself if you use multiple connections. Fun times!</p> </li> <li> <p>Limited use of indexes. Only plays in with equality tests in order. Skipping an indexed column or testing a different relation will push the rest to linear scans.</p> </li> <li> <p>You prolly want to “begin immediate” for write workloads.</p> </li> <li> <p>.dump to backup a db and .read to restore.</p> </li> <li> <p>Some SQL I wasn’t super aware of: case, is/not null, nullif() and coalesce(), create table as select, everywhere you can ab/use subqueries, and the overall “fixed function pipeline” (to borrow an idea from older OpenGL) for queries, esp wrt group-by and having and aggregate functions. And how the temp_store supports that.</p> </li> <li> <p>The execSql wrapper can take a list of semicolon-separated statements.</p> </li> <li> <p>Enabling WAL can fail: Maybe the fs lacks a required feature.</p> </li> <li> <p>WAL vs rollback journal tradeoffs. WAL reduces contention to writer-writer, though a checkpoint can trip a longer than desired pause, and an old but still open read transaction can prevent flushing data to the actual DB (and elevate read times due to seeking through scads of pages).</p> </li> <li> <p>Don’t access a SQLite DB over a network filesystem. Just don’t.</p> </li> <li> <p>Bump your cache to match largest write workload to keep exclusive lock used only for flushing. Use the analyzer to see how many pages that is.</p> </li> <li> <p>Maybe match the DB pagesize to OS. It’s 1024 by default, or half a Darwin page.</p> </li> <li> <p>You can easily throw in custom functions, aggregates, and collations. SQLite is even more flexible than I knew! (Pair with check constraints for superpowers.)</p> </li> <li> <p>Android CursorWindow maxes out at 1 MB of data. You can crash if you try to work with too much data.</p> </li> <li> <p>Harder to discover introspection features:</p> <ul> <li>sqlite_master_table (not listed in .tables)</li> <li>pragmas table_info, index_list, index_info, database_list</li> <li>hooks and especially sqlite_trace()</li> </ul> </li> <li> <p>Many things about bound params:</p> <ul> <li>Manually numbered bound params: ?1, ?2. Lets you reuse an arg without having to bind it multiple times.</li> <li>Named bound params. :name or @name. You have to resolve them to numbers with an API call, but that’s way more readable. And plays nice with dictionaries.</li> <li>TCL bound params. $name. Says “capture from scope”. May only be used by the TCL library?</li> </ul> </li> </ul> What could MPI's evolution say about when to use CSP-style channels? https://jeremywsherman.com/blog/2018/09/27/what-could-mpis-evolution-say-about-when-to-use-csp-style-channels/ Thu, 27 Sep 2018 14:17:11 +0000 https://jeremywsherman.com/blog/2018/09/27/what-could-mpis-evolution-say-about-when-to-use-csp-style-channels/ The reasons MPI is moving from two-sided to one-sided communication may be interesting in light of programs that may use CSP-style channels for communication rather than synchronization/signaling: The main communication paradigm for MPI point-to-point communication has been two-sided communication, where a send call at the source is matched by a receive call at the destination. This paradigm has weaknesses: The complex matching rules of sends to receives result in significant software overheads, especially for receive operations; overlap of communication and computation requires the presence of an asynchronous communication agent that can poll queues concurrently with ongoing computation; and send-receive communication either requires an extra copying of messages (eager protocol) or extra handshakes between sender and receiver (rendezvous protocol). <p>The reasons MPI is moving from two-sided to one-sided communication may be interesting in light of programs that may use CSP-style channels for communication rather than synchronization/signaling:</p> <blockquote> <p>The main communication paradigm for MPI point-to-point communication has been two-sided communication, where a send call at the source is matched by a receive call at the destination. This paradigm has weaknesses: The complex matching rules of sends to receives result in significant software overheads, especially for receive operations; overlap of communication and computation requires the presence of an asynchronous communication agent that can poll queues concurrently with ongoing computation; and send-receive communication either requires an extra copying of messages (eager protocol) or extra handshakes between sender and receiver (rendezvous protocol).</p> </blockquote> <p>(Marc Snir in <a href="https://cacm.acm.org/magazines/2018/10/231376-technical-perspective-the-future-of-mpi/fulltext">&ldquo;Technical Perspective: The Future of MPI&rdquo;</a>, CACM Oct 2018)</p> Mixed-Language Testing: Fixing "fatal error: 'App-Swift.h' file not found" https://jeremywsherman.com/blog/2018/05/07/mixed-language-testing-fixing-fatal-error-app-swift.h-file-not-found/ Mon, 07 May 2018 18:05:21 +0000 https://jeremywsherman.com/blog/2018/05/07/mixed-language-testing-fixing-fatal-error-app-swift.h-file-not-found/ If you have an Obj-C project that&rsquo;s grown some Swift, you might find yourself with some Obj-C test code that needs to talk to some Swift code. You&rsquo;re likely to run into just one little problem. You can&rsquo;t import the header that actually declares the Obj-C interfaces to all that Swift code. More precisely, the compiler can&rsquo;t find the header, even though you know it&rsquo;s there, because Obj-C code in your main app is using it left and right. <p>If you have an Obj-C project that&rsquo;s grown some Swift, you might find yourself with some Obj-C test code that needs to talk to some Swift code.</p> <p>You&rsquo;re likely to run into just one little problem. You can&rsquo;t import the header that actually declares the Obj-C interfaces to all that Swift code.</p> <p>More precisely, the compiler can&rsquo;t find the header, even though you know it&rsquo;s there, because Obj-C code in your main app is using it left and right.</p> <p>You&rsquo;re looking at an error like this:</p> <pre><code>In file included from /Users/jeremy/Downloads/MixedLanguageTesting/MixedLanguageTestingTests/ObjCTests.m:10: /Users/jeremy/Downloads/MixedLanguageTesting/MixedLanguageTesting/ObjCClassUsingSwiftClass.h:15:9: fatal error: 'MixedLanguageTesting-Swift.h' file not found #import &quot;MixedLanguageTesting-Swift.h&quot; ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 error generated. </code></pre><p>No good, that. Luckily, the Usual Fix works.</p> <h2 id="the-usual-fix-fix-your-header-search-path">The Usual Fix: Fix your header search path!</h2> <p>A file is there. A compiler cannot find the file. You need to tell the compiler where to look to find the file. That is what search paths do.</p> <p>In particular, you probably want <code>USER_HEADER_SEARCH_PATH</code> here, which tells the compiler what paths to hunt through when searching for user, rather than system, headers. (Those would be the ones you import like <code>&quot;user.h&quot;</code> versus <code>&lt;system.h&gt;</code>, hence the actual build flag&rsquo;s name, <code>-iquote</code>.)</p> <h2 id="that-path-the-app-targets-derived-sources-directory">That path: The app target&rsquo;s derived sources directory</h2> <p>You need your test target to have a look through the app target&rsquo;s derived sources directory.</p> <p>To do that, edit your project settings. (Or each and every test target&rsquo;s settings. But that just sounds painful.) Search for the <code>USER_HEADER_SEARCH_PATH</code> setting, open up the setting, and add this line:</p> <blockquote> <p><strong> &ldquo;$(CONFIGURATION_TEMP_DIR)/MixedLanguageTesting.build/DerivedSources&quot;</strong></p> </blockquote> <p>Only, <strong>rewrite that &ldquo;MixedLanguageTesting&rdquo; bit to match <em>your</em> app target&rsquo;s name.</strong></p> <h2 id="behind-the-scenes">Behind the Scenes</h2> <p>If you want to see how I arrived at this solution, I happen to have checkpointed my debugging steps using git. You&rsquo;ll find my trail over at GitHub at <a href="https://github.com/jeremy-w/MixedLanguageTesting">jeremy-w/MixedLanguageTesting</a>.</p> Downtime & Recovery: Bludgeoning DigitalOcean Arch into Working Again https://jeremywsherman.com/blog/2018/05/01/downtime-recovery-bludgeoning-digitalocean-arch-into-working-again/ Tue, 01 May 2018 15:09:10 +0000 https://jeremywsherman.com/blog/2018/05/01/downtime-recovery-bludgeoning-digitalocean-arch-into-working-again/ This blog had some downtime yesterday. I have a DigitalOcean Droplet that&rsquo;s an Arch x86-64 system, from back before they dropped Arch. When I went to poke certbot into renewing my LetsEncrypt cert, I found the system was down. Power cycle, and…it&rsquo;s still down. Console Spew: Not Promising What follows are my notes from getting things working again. I ran into a couple dark corners where I found no search hits, so maybe this&rsquo;ll help the next poor sap. <p>This blog had some downtime yesterday.</p> <p>I have a DigitalOcean Droplet that&rsquo;s an Arch x86-64 system, from back before they dropped Arch.</p> <p>When I went to poke certbot into renewing my LetsEncrypt cert, I found the system was down. Power cycle, and…it&rsquo;s still down.</p> <figure> <img src="images/posts/2018/console-systemd-errors.png" alt="Screenshot of Web console for server ending with systemd logging that it is &#39;Freezing execution.&#39;"/> <figcaption> <h4>Console Spew: Not Promising</h4> </figcaption> </figure> <p>What follows are my notes from getting things working again. I ran into a couple dark corners where I found no search hits, so maybe this&rsquo;ll help the next poor sap.</p> <h2 id="goal-blog-accessible-from-phone">Goal: Blog Accessible from Phone</h2> <p>The main thing the VPS runs is my blog. The other main thing is ZNC (an IRC bouncer). I really just care about getting the blog back up immediately, though. As far as I&rsquo;m concerned, that means my phone connected to the cellular network can resolve the domain and see the latest version of the blog content.</p> <!-- more --> <h2 id="culprit-systemd-is-hosed">Culprit: Systemd Is Hosed</h2> <p>My systemd got hosed somehow when the system tried to boot back up. I reach this conclusion from the fact that all the console errors are from systemd, and that a systemd that freezes execution means a system bootstrap that froze execution.</p> <p>This looks to be <a href="https://www.digitalocean.com/community/questions/archlinux-not-startup-after-update">the same issue described by a DigitalOcean Community user.</a></p> <p>Bad news: No-one had a solution beyond &ldquo;nuke &amp; pave&rdquo;. Hooray.</p> <p>For added fun, I&rsquo;m traveling and away from my backups. All I have is an ancient snapshot from 2014.</p> <p>Maybe that&rsquo;s all I need?</p> <h2 id="nuke--pave-ancient-droplet-strikes-again">Nuke &amp; Pave? Ancient Droplet Strikes Again</h2> <p>Bah, enough with an unsupported platform! I don&rsquo;t get any of the new DO goodies with Arch.</p> <p>What about just switching to FreeBSD?</p> <p>I tried to flash with FreeBSD, but turns out you can&rsquo;t rebuild it with a modern Droplet image unless it was seeded with an SSH key to begin with. You get a fun flash message at a weird place on the screen (or eventually, off the screen, if you keep trying long enough):</p> <blockquote> <p>&ldquo;Data image requires at least one SSH key&rdquo;</p> </blockquote> <p>I sure as heck have an SSH key on file with them, and I confirmed the (MD5? not SHA-256? Huh) fingerprint as matching the one I have on disk here still.</p> <p>But that wasn&rsquo;t quite what was meant. As DigitalOcean support says:</p> <blockquote> <p>Sadly unless the Droplet was originally created using the SSH key in your Cloud panel, you won&rsquo;t be able to rebuild it in place with the FreeBSD image. Instead, you&rsquo;ll have to create a new Droplet.</p> </blockquote> <p>Guess the key got baked in somewhere deep. Or it didn&rsquo;t, in my case. Lame sauce.</p> <p>So back to Arch for now, because I don&rsquo;t want to have to repoint my DNS and do a bunch more clicky-clicky to rig up an entirely new host.</p> <h2 id="recovery-successfully-time-traveled-to-2014">Recovery: Successfully Time-Traveled to 2014</h2> <p>Restore from ancient snapshot of 2014-03-19 using Digital Ocean Web UI. (Yeah, probably should have taken a snapshot a bit more recently. Oh well.)</p> <p>ssh in. I can&rsquo;t <a href="https://certbot.eff.org/lets-encrypt/arch-nginx">install certbot.</a></p> <h2 id="get-certbot-installed">Get Certbot Installed</h2> <p>Let&rsquo;s try to update to a modern toolchain so I can follow those recommended steps.</p> <h3 id="system-update-fails-gpg-key-import-fails">System Update Fails: GPG Key Import Fails</h3> <p>Keys are out of date, so system update with <code>pacman -Syyu</code> fails after restoring from 4-year-old snapshot.</p> <p>Let&rsquo;s refresh those keys:</p> <pre><code>pacman-key --refresh-keys pacman -S archlinux-keyring </code></pre><p>Now try to update system again. Continues failing out when I tell it, &ldquo;Sure, import the key.&rdquo; I can&rsquo;t find anything helpful searching the Web.</p> <p>Yank pacman source and rip-grep for the error message and wind up staring at <a href="https://git.archlinux.org/pacman.git/tree/lib/libalpm/signing.c?id=77986af9b7ae9b2cc701a30bcc1cac51b6f1b5be#n460">libalpm/signing.c:460</a>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#75715e">/** </span><span style="color:#75715e"> * Import a key defined by a fingerprint into the local keyring. </span><span style="color:#75715e"> * @param handle the context handle </span><span style="color:#75715e"> * @param fpr the fingerprint key ID to import </span><span style="color:#75715e"> * @return 0 on success, -1 on error </span><span style="color:#75715e"> */</span> <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">_alpm_key_import</span>(alpm_handle_t <span style="color:#f92672">*</span>handle, <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> <span style="color:#f92672">*</span>fpr) <span style="color:#75715e">/* SNIP */</span> <span style="color:#66d9ef">if</span>(key_import(handle, <span style="color:#f92672">&amp;</span>fetch_key) <span style="color:#f92672">==</span> <span style="color:#ae81ff">0</span>) { ret <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; } <span style="color:#66d9ef">else</span> { _alpm_log(handle, ALPM_LOG_ERROR, _(<span style="color:#e6db74">&#34;key </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">%s</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> could not be imported</span><span style="color:#ae81ff">\n</span><span style="color:#e6db74">&#34;</span>), fetch_key.uid); } <span style="color:#75715e">/* SNIP */</span> </code></pre></div><p>Maybe requires a newer version of GPG or something to be able to import the key that I need to install that newer version of GPG. Hard to say: The error message has no other info than &ldquo;import failed&rdquo;, with no info about <em>why</em> or <em>how</em> it failed.</p> <h3 id="disable-signature-verification">Disable Signature Verification</h3> <p>That&rsquo;s a dead-end: There&rsquo;s no clue what&rsquo;s wrong, so there&rsquo;s no clue how to move past it.</p> <p>Bypass the failing step entirely with:</p> <pre><code>vi /etc/pacman.conf /SigLevel f= d$ = Never ^[ :wq </code></pre><h3 id="system-update-fails-file-conflicts">System Update Fails: File Conflicts</h3> <p>Now we hit conflicting files:</p> <pre><code>error: failed to commit transaction (conflicting files) ca-certificates-utils: /etc/ssl/certs/ca-certificates.crt exists in filesystem lzo: /usr/include/lzo/lzo1.h exists in filesystem lzo: /usr/include/lzo/lzo1a.h exists in filesystem lzo: /usr/include/lzo/lzo1b.h exists in filesystem lzo: /usr/include/lzo/lzo1c.h exists in filesystem lzo: /usr/include/lzo/lzo1f.h exists in filesystem lzo: /usr/include/lzo/lzo1x.h exists in filesystem lzo: /usr/include/lzo/lzo1y.h exists in filesystem lzo: /usr/include/lzo/lzo1z.h exists in filesystem lzo: /usr/include/lzo/lzo2a.h exists in filesystem lzo: /usr/include/lzo/lzo_asm.h exists in filesystem lzo: /usr/include/lzo/lzoconf.h exists in filesystem lzo: /usr/include/lzo/lzodefs.h exists in filesystem lzo: /usr/include/lzo/lzoutil.h exists in filesystem lzo: /usr/include/lzo/minilzo.h exists in filesystem lzo: /usr/lib/liblzo2.so exists in filesystem lzo: /usr/lib/liblzo2.so.2 exists in filesystem lzo: /usr/lib/liblzo2.so.2.0.0 exists in filesystem lzo: /usr/lib/libminilzo.so exists in filesystem lzo: /usr/lib/libminilzo.so.0 exists in filesystem lzo: /usr/share/doc/lzo/AUTHORS exists in filesystem lzo: /usr/share/doc/lzo/COPYING exists in filesystem lzo: /usr/share/doc/lzo/LZO.FAQ exists in filesystem lzo: /usr/share/doc/lzo/LZO.TXT exists in filesystem lzo: /usr/share/doc/lzo/LZOAPI.TXT exists in filesystem lzo: /usr/share/doc/lzo/NEWS exists in filesystem lzo: /usr/share/doc/lzo/THANKS exists in filesystem Errors occurred, no packages were upgraded. </code></pre><p>Let&rsquo;s see what owns those:</p> <pre><code>[root@gateway-arch jeremy]# pacman -Qo /usr/include/lzo/lzo1.h /usr/include/lzo/lzo1.h is owned by lzo2 2.06-3 [root@gateway-arch jeremy]# pacman -Qo /etc/ssl/certs/ca-certificates.crt error: No package owns /etc/ssl/certs/ca-certificates.crt </code></pre><p>But when when I try to <code>pacman -U lzo2</code>, it acts like no such thing exists. Sigh.</p> <p>OK, the cert one is a <a href="https://www.archlinux.org/news/ca-certificates-utils-20170307-1-upgrade-requires-manual-intervention/">known issue.</a> Gotta love rolling releases. The fix is to delete before upgrading.</p> <p>For lzo, I ultimately did <code>pacman -S --force core/lzo</code>. It runs a risk of clobbering the wrong thing and hosing all the things, but it seemed a calculated risk, since it&rsquo;s basically &ldquo;clobber lzo as installed with an older package name with lzo as installed with the new package name&rdquo;. The risk paid off, so.</p> <h3 id="ok-now-really-update">OK, Now Really, Update</h3> <p>Then I could <code>pacman -Su</code>. Finally. And all the things updated.</p> <p>Rebuilding the man-db database took a century. I worried something had gone wrong, but it hadn&rsquo;t.</p> <h2 id="install-certbot">Install Certbot</h2> <pre><code>[root@gateway-arch jeremy]# pacman -S certbot-nginx error: failed to initialize alpm library (database is incorrect version: /var/lib/pacman/) try running pacman-db-upgrade [root@gateway-arch jeremy]# pacman-db-upgrade ==&gt; Pre-4.2 database format detected - upgrading... [root@gateway-arch jeremy]# pacman -S certbot-nginx </code></pre><p>OK, now we can follow the docs on LetsEncrypt.</p> <h2 id="run-certbot">Run Certbot</h2> <h3 id="unicode-issues-set-lc_allen_usutf_8">Unicode Issues: Set LC_ALL=en_US.utf_8</h3> <p>Except that Unicode is fun fun fun:</p> <pre><code>[root@gateway-arch jeremy]# sudo certbot --nginx Saving debug log to /var/log/letsencrypt/letsencrypt.log An unexpected error occurred: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 10453: ordinal not in range(128) Please see the logfiles in /var/log/letsencrypt for more details. </code></pre><p>Matching issue: <a href="https://github.com/certbot/certbot/issues/5236">https://github.com/certbot/certbot/issues/5236</a></p> <p>Looks like there are smart quotes in the default template for nginx.conf, perhaps? Not in my config that I can see.</p> <p>And that byte is at a gibberish offset.</p> <p>And I confirmed that Python is happy to read in my nginx.conf as ascii.</p> <p>OK, whatever. Mucking around with PYTHONIOENCODING=utf8 didn&rsquo;t help.</p> <p>So instead, play with locale. <code>locale</code> reports we&rsquo;re in C. <code>locale -a</code> gives us a utf-8 option. Set that.</p> <p>Now it runs.</p> <h3 id="https-certificate-trust-anchor-missing">HTTPS Certificate Trust Anchor: Missing</h3> <p>Can&rsquo;t get an HTTPS cert because of HTTPS certs. Delicious.</p> <pre><code>[root@gateway-arch jeremy]# certbot --nginx Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator nginx, Installer nginx Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel): ******** An unexpected error occurred: OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt Please see the logfiles in /var/log/letsencrypt for more details. </code></pre><p>That path definitely exists. It is a symlink, though, to be fair.</p> <p>Spewed out when <code>requests</code> is trying to check the cert on the connection to the ACME backend, per that debug logfile:</p> <pre><code>2018-05-01 03:46:44,110:DEBUG:certbot.plugins.selection:Selected authenticator &lt;certbot_nginx.configurator.NginxConfigurator ob ject at 0x7f11271de5c0&gt; and installer &lt;certbot_nginx.configurator.NginxConfigurator object at 0x7f11271de5c0&gt; 2018-05-01 03:46:44,110:INFO:certbot.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx 2018-05-01 03:48:40,618:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory. </code></pre><p>Oh, huh. The file pointed to by the symlink? That one doesn&rsquo;t exist.</p> <p>And <code>pacman -Ql ca-certificates</code>, which should list all the files installed by that package, shows zip. Nada. Nothing: <strong>ca-certificates is an empty package.</strong></p> <h3 id="install-more-certificates">Install More Certificates</h3> <p>Luckily, it&rsquo;s not the only <code>ca-certificates</code> package available:</p> <pre><code>pacman -S ca-certificates-cacert ca-certificates-mozilla [root@gateway-arch jeremy]# pacman -Ss ca-cert core/ca-certificates 20170307-1 [installed] Common CA certificates (default providers) core/ca-certificates-cacert 20140824-4 [installed] CAcert.org root certificates core/ca-certificates-mozilla 3.36.1-1 [installed] Mozilla's set of trusted CA certificates core/ca-certificates-utils 20170307-1 [installed] Common CA certificates (utilities) </code></pre><p>Now let&rsquo;s try again.</p> <h3 id="success">Success!</h3> <p>Modulo the fact this is an old nginx.conf that wasn&rsquo;t updated for http2. I&rsquo;ll just fish those updates out of my backup later. (I don&rsquo;t recall it being difficult at all to update from spdy to http2, but I&rsquo;m out of steam at this point.)</p> <pre><code>[root@gateway-arch jeremy]# certbot --nginx Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator nginx, Installer nginx Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel): ******* /usr/lib/python3.6/site-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. signer = key.signer(self.padding, self.hash) ------------------------------------------------------------------------------- Please read the Terms of Service at https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must agree in order to register with the ACME server at https://acme-v01.api.letsencrypt.org/directory ------------------------------------------------------------------------------- (A)gree/(C)ancel: a ------------------------------------------------------------------------------- Would you be willing to share your email address with the Electronic Frontier Foundation, a founding partner of the Let's Encrypt project and the non-profit organization that develops Certbot? We'd like to send you email about EFF and our work to encrypt the web, protect its users and defend digital rights. ------------------------------------------------------------------------------- (Y)es/(N)o: n Which names would you like to activate HTTPS for? ------------------------------------------------------------------------------- 1: jeremywsherman.com 2: www.jeremywsherman.com ------------------------------------------------------------------------------- Select the appropriate numbers separated by commas and/or spaces, or leave input blank to select all options shown (Enter 'c' to cancel): Obtaining a new certificate Performing the following challenges: http-01 challenge for jeremywsherman.com http-01 challenge for www.jeremywsherman.com 2018/05/01 03:59:59 [warn] 18919#18919: invalid parameter &quot;spdy&quot;: ngx_http_spdy_module was superseded by ngx_http_v2_module in /etc/nginx/nginx.conf:65 2018/05/01 03:59:59 [warn] 18919#18919: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size 2018/05/01 03:59:59 [notice] 18919#18919: signal process started Waiting for verification... /usr/lib/python3.6/site-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. signer = key.signer(self.padding, self.hash) Cleaning up challenges 2018/05/01 04:00:06 [warn] 18922#18922: invalid parameter &quot;spdy&quot;: ngx_http_spdy_module was superseded by ngx_http_v2_module in /etc/nginx/nginx.conf:55 2018/05/01 04:00:06 [warn] 18922#18922: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size 2018/05/01 04:00:06 [notice] 18922#18922: signal process started /usr/lib/python3.6/site-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. signer = key.signer(self.padding, self.hash) Deploying Certificate to VirtualHost /etc/nginx/nginx.conf Deploying Certificate to VirtualHost /etc/nginx/nginx.conf 2018/05/01 04:00:11 [warn] 18924#18924: invalid parameter &quot;spdy&quot;: ngx_http_spdy_module was superseded by ngx_http_v2_module in /etc/nginx/nginx.conf:55 2018/05/01 04:00:11 [warn] 18924#18924: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size 2018/05/01 04:00:11 [notice] 18924#18924: signal process started Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access. ------------------------------------------------------------------------------- 1: No redirect - Make no further changes to the webserver configuration. 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for new sites, or if you're confident your site works on HTTPS. You can undo this change by editing your web server's configuration. ------------------------------------------------------------------------------- Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1 ------------------------------------------------------------------------------- Congratulations! You have successfully enabled https://jeremywsherman.com and https://www.jeremywsherman.com You should test your configuration at: https://www.ssllabs.com/ssltest/analyze.html?d=jeremywsherman.com https://www.ssllabs.com/ssltest/analyze.html?d=www.jeremywsherman.com ------------------------------------------------------------------------------- IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/jeremywsherman.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/jeremywsherman.com/privkey.pem Your cert will expire on 2018-07-30. To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the &quot;certonly&quot; option. To non-interactively renew *all* of your certificates, run &quot;certbot renew&quot; - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le </code></pre><p>I declined the redirect because I already have that in place.</p> <p>The new lines added to the server stanza are:</p> <pre><code> ssl_certificate /etc/letsencrypt/live/jeremywsherman.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/jeremywsherman.com/privkey.pem; # managed by Certbot </code></pre><h2 id="restart-nginx">Restart Nginx</h2> <pre><code>systemctl restart nginx </code></pre><p>And now my blog is up and running again.</p> <p>Total time cost: 2 hours-ish. Guess that coulda been worse.</p> <h2 id="addendum-keeping-cron-running">Addendum: Keeping Cron Running</h2> <p>I kept seeing cron failing out on me, and my LetsEncrypt certs not auto-renewing. (On the bright side, that&rsquo;s how I found out everything was hosed this time.)</p> <p>Let&rsquo;s see if we can fix that.</p> <h3 id="which-cron">Which Cron?</h3> <p>What cron are we running?</p> <pre><code>[root@gateway-arch jeremy]# pacman -Qs cron local/cronie 1.5.1-1 Daemon that runs specified programs at scheduled times and related tools </code></pre><p>Cronie.</p> <h3 id="whos-that-to-systemd">Who&rsquo;s That to Systemd?</h3> <p>And how&rsquo;s it wired into systemd?</p> <pre><code>[root@gateway-arch jeremy]# pacman -Ql cronie | grep systemd cronie /usr/lib/systemd/ cronie /usr/lib/systemd/system/ cronie /usr/lib/systemd/system/cronie.service </code></pre><p>As unit cronie.service.</p> <h3 id="status-shows-errors">Status Shows Errors</h3> <p>Let&rsquo;s flip it on and check its status.</p> <pre><code>[root@gateway-arch jeremy]# systemctl enable cronie [root@gateway-arch jeremy]# systemctl status cronie * cronie.service - Periodic Command Scheduler Loaded: loaded (/usr/lib/systemd/system/cronie.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2018-05-01 02:57:13 UTC; 11h ago Main PID: 169 (crond) CGroup: /system.slice/cronie.service `-169 /usr/bin/crond -n May 01 13:01:01 gateway-arch crond[22224]: PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libpam.so.0: version &gt; May 01 13:01:01 gateway-arch crond[22224]: PAM adding faulty module: /usr/lib/security/pam_unix.so May 01 13:05:01 gateway-arch crond[22250]: PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libpam.so.0: version &gt; May 01 13:05:01 gateway-arch crond[22250]: PAM adding faulty module: /usr/lib/security/pam_unix.so May 01 14:01:01 gateway-arch crond[22590]: PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libpam.so.0: version &gt; May 01 14:01:01 gateway-arch crond[22590]: PAM adding faulty module: /usr/lib/security/pam_unix.so May 01 14:05:01 gateway-arch crond[22616]: PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libpam.so.0: version &gt; May 01 14:05:01 gateway-arch crond[22616]: PAM adding faulty module: /usr/lib/security/pam_unix.so May 01 14:51:01 gateway-arch crond[169]: (root) CAN'T OPEN (/etc/crontab): No such file or directory May 01 14:51:01 gateway-arch crond[169]: (root) RELOAD (/var/spool/cron/root) </code></pre><p>Point one, vendor preset is disabled. So that explains why I kept seeing crond fizzle out on me. I wanted to use cron but then never enabled it with systemd. (And I&rsquo;m sure it&rsquo;s disabled because I ought to be writing a systemd unit file. But I&rsquo;m not going to just now.)</p> <p>Point two, there are a couple errors.</p> <p>The /etc/crontab error is weird - looking at the full <code>-Ql</code> output shows stuff in /etc/anacrontab, not /etc/crontab. I&rsquo;m betting this is an older cron process that started before I updated all my packages from their 2014 vintage flavors.</p> <p>The PAM error looks grungy and noisy. A quick search through <code>man systemctl</code> didn&rsquo;t show me how to change the character width so I could see the rest of the PAM module error, but searching found a good hit for <a href="https://bbs.archlinux.org/viewtopic.php?id=200977">dlopen failure of pam_unix.so</a>.</p> <p>Consensus is: You had a glibc update. Now a process still using the older glibc is trying to open a shared object using a newer one.</p> <p>Fix: <code>systemctl restart cronie</code></p> <h3 id="restarting-the-service-fixes-all-errors">Restarting the Service Fixes All Errors</h3> <p>And indeed, that does it:</p> <pre><code>[root@gateway-arch jeremy]# systemctl status cronie * cronie.service - Periodic Command Scheduler Loaded: loaded (/usr/lib/systemd/system/cronie.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2018-05-01 14:54:35 UTC; 17s ago Main PID: 22985 (crond) Memory: 612.0K CGroup: /system.slice/cronie.service `-22985 /usr/bin/crond -n May 01 14:54:35 gateway-arch systemd[1]: Stopped Periodic Command Scheduler. May 01 14:54:35 gateway-arch systemd[1]: Started Periodic Command Scheduler. May 01 14:54:35 gateway-arch crond[22985]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 96% if used.) May 01 14:54:35 gateway-arch crond[22985]: (CRON) INFO (running with inotify support) May 01 14:54:35 gateway-arch crond[22985]: (CRON) INFO (@reboot jobs will be run at computer's startup.) </code></pre><p>No more errors. Good to go!</p> <p>Total time cost: 20 minutes.</p> Cards: Closing Out a Wikipedia Crawl https://jeremywsherman.com/blog/2017/05/20/cards-closing-out-a-wikipedia-crawl/ Sat, 20 May 2017 00:00:00 +0000 https://jeremywsherman.com/blog/2017/05/20/cards-closing-out-a-wikipedia-crawl/ <p>I don&rsquo;t recall how it started, but I&rsquo;m at the tail end of a several-day Wikipedia crawl through playing cards and card games. Read on for fun trivia, neat games you might not have heard of, and, if your playing card experience is primarily American, probably the destruction of a lot of your assumptions around what a &ldquo;standard deck of cards&rdquo; is.</p> <p>I don&rsquo;t recall how it started, but I&rsquo;m at the tail end of a several-day Wikipedia crawl through playing cards and card games. Read on for fun trivia, neat games you might not have heard of, and, if your playing card experience is primarily American, probably the destruction of a lot of your assumptions around what a &ldquo;standard deck of cards&rdquo; is.</p> <h1 id="fun-trivia">Fun Trivia</h1> <p>These mostly reflect my interest in the how we ended up with the 52-card, four-suit deck I grew up playing cards with.</p> <ul> <li>The Tarot trumps started life as a dedicated trump suit for Italian playing cards. France imported them from Italy, some folks started using them for cartomancy in the XVIIIth Century, and then that got exported to England.</li> <li>Those &ldquo;exotic&rdquo; suits of Wands, Cups, Pentacles/Coins, and Swords? Yeah, those turn out to just be the standard &ldquo;Latin&rdquo; suits in use in Italy, and still used some today in Italy, Spain, and thereabouts.</li> <li>Corner indices - a small notation of the suit and rank at the corners of the cards, to let you easily read your hand with the cards fanned, without having to study the whole face over - are kinda recent. They seem most common with Anglo/French cards, and you can still find other flavors of decks without them.</li> <li>Corner indices are what drove the adoption in English of &ldquo;Jack&rdquo; as the name for the lowest court card. Before that, &ldquo;Jack&rdquo; was kinda slightly scandalous/lower class, and everyone called them &ldquo;Knaves&rdquo;, even up into the XIXth Century it seems. But it turned out to be way too easy to confuse the &ldquo;K&rdquo; and the &ldquo;Kn&rdquo; when you had your cards fanned, so, out with the Knave, and in with the Jack!</li> <li>French suits - hearts, diamonds, clubs, and spades - partly took off because they are such simple shapes that they make it easy to just stamp out the pip cards. This made them a lot cheaper and to produce. No fancy crossed seven clubs art needed!</li> <li>The English names for French suits reflect that English card playing started with the Latin suits. What we call &ldquo;clubs&rdquo; and &ldquo;spades&rdquo;, the French call &ldquo;clovers&rdquo; and &ldquo;pikes&rdquo;. The corresponding Latin suits actually are clubs and swords, and &ldquo;spades&rdquo; is just a name for a kind of sword.</li> <li>The French suits actually evolved out of the German suits of Hearts, Bells, Acorns, and Leaves. (The Swiss swap Hearts for Roses and Leaves for Shields, because they&rsquo;re cool like that.)</li> <li>You can still buy German-suited decks today! And some of the German games get played with both systems, and there&rsquo;s even a hybrid, compromise deck (used in tournaments for the German card game Skat) that uses French suit symbols with German suit colors.</li> <li>Cards are used to play card games, and card games can be used for gambling, so people sometimes banned cards. So you&rsquo;d sometimes see tiles pop up, or a renewed interest in Dominos, or, if you&rsquo;re purebred American Puritan, you&rsquo;ve got your XXth Century Rook deck, with 4 colored suits (just colors: black, red, yellow, and green) of 14 cards (which matches a tarot deck having four court cards per suit of jack, knight, queen, king, or similar), plus a blue Rook card that acts kinda like a trump. It&rsquo;s mostly just used nowadays (and maybe ever?) to play a card game named after the deck.</li> <li>The &ldquo;red suits&rdquo; (vs the &ldquo;black suits&rdquo;, or if you fancy the Latin decks, the &ldquo;long suits&rdquo; vs &ldquo;round suits&rdquo;) used to rank their pip cards the other way around, where 1 (ace) is high and 10 is low, rather than 10 high and 1 low. This old-school inverted ranking is still used when playing modern-day French Tarot!</li> <li>Jokers started life as a &ldquo;super trump&rdquo; for Euchre. &ldquo;The Imperial Bower&rdquo; takes all comers. Then they spread from there, till you can find some games having six jokers in play. Because they&rsquo;re such a recent card, their design is not standardized, but usually, in two-joker decks, there&rsquo;s a &ldquo;greater&rdquo; and a &ldquo;lesser&rdquo; joker, where the lesser either is uncolored or has the deck&rsquo;s guarantee printed on it or whatever. This allows to rank the two jokers against each other for cases where they&rsquo;re at the top of the trumps.</li> <li>The French court cards are considered as depicting specific historical/literary characters, rather than just being anonymous stand-ins. Decks sometimes have the character&rsquo;s name written vertically along the edge.</li> <li>&ldquo;Stripped decks&rdquo;, where you play with less than 52 cards, sometimes way less, by throwing out whole ranks across all the suits, are actually pretty darn common! You see this in some games played with an Anglo-French deck, like Euchre, but for some other-suited decks, you&rsquo;d be hard pressed to find anyone selling a deck with all 52 cards, simply because the most popular games played with those decks (like Skat with the German deck) don&rsquo;t need that many cards. This goes for, well, basically every deck flavor other than the Anglo-French – at least, that&rsquo;s the impression I&rsquo;ve retained after browsing around. That includes most Tarot card decks.</li> </ul> <h1 id="interesting-games">Interesting Games</h1> <p>Most of these stuck out for me based on national popularity. Like, &ldquo;Whoah, there&rsquo;s this game I never knew existed, and it&rsquo;s the biggest card game ever in (some country/region)?&rdquo;</p> <ul> <li> <p>The big German games are Skat, Doppelkopf, and Schafkopf. Skat especially is big, with an active playing community and standardized rules for competition.</p> </li> <li> <p>This <a href="https://en.wikipedia.org/w/index.php?title=Schafkopf&amp;oldid=779599180#Introduction">snippet of Wikipedia</a> just tickled my fancy:</p> <blockquote> <p>In Germany, Schafkopf is not deemed a gambling game and can therefore be legally played for money. Especially in Bavaria it is normally played for small amounts of money to make it more interesting and the players more focused. Normal rates are 10 Euro cents for normal and 50 for solo games.</p> </blockquote> </li> <li> <p>You need like 5-7 riffle shuffles to properly randomize a deck. Using a machine saves your hands and gives you a seriously randomized deck, which is why you see them used so often in gambling games.</p> </li> <li> <p>True randomness isn&rsquo;t even really desirable in some card games! The Grand Skat Authority (not their actual name) actually ruled against using a shuffling machine, since it&rsquo;d introduce too much randomness in the deck. On the far side of &ldquo;eh, random shuffling, meh&rdquo;, the rules of Belote straight up bar shuffling between hands: You just cut the deck before beginning dealing for the hand, instead.</p> </li> <li> <p>The big French games look to be Belote and French Tarot these days. Formerly, Piquet was pretty big, and it had its hey-day in England, as well.</p> </li> <li> <p>The big Eastern European game is Durak, or &ldquo;Fool&rdquo;, which is an interesting shedding-type game (your aim: end up holding no cards, as fast as you can). In characteristic Eastern European style, it seems the game has no winners, only a designated loser, the fool who&rsquo;s the last one left holding cards at the end of the game.</p> </li> <li> <p>I didn&rsquo;t touch on East Asia in the card deck bit, but they&rsquo;ve got their own thing going on, with games you mostly won&rsquo;t find in the US, aside from a Hanafuda import popular in Hawaii called Koi-Koi. Japanese deck evolution was driven by the Tokugawa Shogunate&rsquo;s anti-Western bent, though there are also games that derive from a kind of &ldquo;finish the poem&rdquo; matching game mold. I often found the descriptions on Wikipedia of these games kind of hard to follow, probably because they&rsquo;ve received less attention - I <em>was</em> reading the English-language Wikipedia, after all, and it has its biases.</p> </li> <li> <p>The play style I&rsquo;m familiar with in Euchre is pretty similar to a class of games related to Whist. Hearts is a &ldquo;negative game&rdquo; where the goal is to <em>not</em> score points (&ldquo;card golf&rdquo;), while Spades seems like it might be treated as Bridge with Training Wheels?</p> </li> <li> <p>Euchre has a less confusing recent variant called Bacon, which doesn&rsquo;t have the suit-shifting fun times of the Left Bower.</p> </li> <li> <p>The game Ombre (&ldquo;I&rsquo;m l&rsquo;Hombre, err, the man!&quot;) introduced auctions into card games.</p> </li> <li> <p>Whist eventually evolved into Contract Bridge, which seems completely bonkers from the outside, especially once you run into the notion of Brown-Sticker and Yellow-Sticker bidding conventions/tendencies.</p> </li> <li> <p>Scarto is a three-player Tarot game that&rsquo;s supposed to be an easy starting point if you want to get into that class of game. Loser of the game buys the next round of beers. What&rsquo;s not to love?</p> </li> <li> <p>Cribbage is supposedly kinda big in England? I know of the game, but haven&rsquo;t really ever played it, though I recall puzzling over a cribbage board my mom had once or twice as a kid.</p> </li> <li> <p>Cribbage is also apparently kinda &ldquo;The Game&rdquo; for US submariners, and they keep a WWII-vintage cribbage board in the wardroom of the oldest submarine, and pass it on to the now-oldest whenever that one gets decommissioned.</p> </li> <li> <p>Scopa is a major Italian game, also apparently kinda popular in Brazil and some parts of the US. It&rsquo;s a kind of matching/capturing game.</p> </li> <li> <p>I grew up playing &ldquo;plain trick games&rdquo;, where scoring is basically just who got how many tricks, with maybe some bonuses for a shut-out or playing alone, but nothing fancy. Loads of European games, though, are what are called &ldquo;point-trick&rdquo; games, where the scoring depends on the specific cards captured and their assigned point-values. Sounds like kind of a pain to track to me, but I&rsquo;m sure it becomes easy-peasy after a few evenings, especially given their widespread popularity!</p> </li> <li> <p>OK, in case point-trick games weren&rsquo;t complicated enough, a ton of games also include &ldquo;declarations&rdquo;, where you can score points for holding certain combos of cards, at the price of revealing info about your hand to your opponents. And some games (OK, mostly just the German games!) have several different &ldquo;game modes&rdquo;, where the flavor of the game varies based on deal or sometimes choice, or sometimes it just cycles through the game flavors. Forcing cycling seems more common in competitive play, with stuff like, &ldquo;everyone has to trigger this one game mode at least once&rdquo; popping up across several different games.</p> </li> </ul> <h1 id="conclusions">Conclusions</h1> <ul> <li> <p>There are some really cool looking decks with suits I knew nothing about out there. Many wouldn&rsquo;t be too useful for playing the card games I&rsquo;m used to, due to their starting life as a stripped deck for playing the games most popularly played in their region.</p> </li> <li> <p>The big regional games are probably worth looking at if you&rsquo;re looking for something new to play:</p> <ul> <li>Skat</li> <li>Scopa</li> <li>Belote</li> <li>French Tarot</li> <li>Durak</li> <li>Cribbage</li> </ul> </li> <li> <p>Contract Bridge is, uh, shall we say, &ldquo;baroque&rdquo;. Think &ldquo;C++ template metaprogramming&rdquo;. It&rsquo;s at the end of a long evolution begun with Whist, but it seems frozen as a result of competitive international play. If you really want to lose yourself in detail, though, this might be the game for you!</p> </li> </ul> Microservices vs Distributed Objects https://jeremywsherman.com/blog/2017/03/08/microservices-vs-distributed-objects/ Wed, 08 Mar 2017 00:00:00 +0000 https://jeremywsherman.com/blog/2017/03/08/microservices-vs-distributed-objects/ Distributed objects died out eventually; you can&rsquo;t really hide the network layer without changing your system design to match. Here&rsquo;s a Cocoa take. And here&rsquo;s a Martin Fowler take found via the article below, with a sidebar suggesting a remote façade (to provide a coarse API as a remote endpoint) and data transfer objects (to provide coarse data transfer also as a way around slow remote communication times). So, if DO sucks, why are microservices any different? <p>Distributed objects died out eventually; you can&rsquo;t really hide the network layer without changing your system design to match. <a href="https://mikeash.com/pyblog/friday-qa-2009-02-20-the-good-and-bad-of-distributed-objects.html">Here&rsquo;s a Cocoa take.</a> And <a href="http://www.drdobbs.com/errant-architectures/184414966">here&rsquo;s a Martin Fowler take</a> found via the article below, with a sidebar suggesting a remote façade (to provide a coarse API as a remote endpoint) and data transfer objects (to provide coarse data transfer also as a way around slow remote communication times).</p> <p>So, if DO sucks, why are microservices any different?</p> <p>Enter Phil Calçado&rsquo;s <a href="http://philcalcado.com/2017/03/02/microservices_vs_1st_law_distributed_objects.html">Microservices and the First Law of Distributed Objects</a>:</p> <blockquote> <p>Objects are not a good unit of distribution. They are too small and chatty to be good network citizens. Services, on the other hand, are meant to be more coarse-grained.</p> </blockquote> <p>Group terms by affinity; grab out your connected components; now you have bounded contexts. Make those your services.</p> <p>Early on, not worth the trouble - monoliths make sense. At huge scale, performance considerations (not going down) dwarf maintenance. In the middle, though, this rule of thumb ain&rsquo;t bad.</p> <p><em>Found via <a href="http://devopsweekly.com/">Devops Weekly</a>.</em></p> The Gist of Regex https://jeremywsherman.com/blog/2017/01/14/the-gist-of-regex/ Sat, 14 Jan 2017 00:00:00 +0000 https://jeremywsherman.com/blog/2017/01/14/the-gist-of-regex/ Regular expressions scare some people. They&rsquo;re really quite warm and cuddly, or at least, conceptually very neat and tidy. If you don&rsquo;t feel that way, this post is for you! Here&rsquo;s how I think about regexen, in a nutshell. I use this conception on a regular basis; when it comes to writing regex, I think about what I want to do in this model, then translate it into whatever regex notation the system I&rsquo;m using at the time gives me. <p>Regular expressions scare some people. They&rsquo;re really quite warm and cuddly, or at least, conceptually very neat and tidy. If you don&rsquo;t feel that way, this post is for you! Here&rsquo;s how I think about regexen, in a nutshell.</p> <p>I use this conception on a regular basis; when it comes to writing regex, I think about what I want to do in this model, then translate it into whatever regex notation the system I&rsquo;m using at the time gives me. (I do the same thing with distributed version control and relation databases, but let&rsquo;s stick to regexen for now.)</p> <h2 id="regex-is-tiny-machines">Regex Is Tiny Machines!</h2> <p>Regular expressions are a <strong>compact description of a symbol-matching machine.</strong> Like, &ldquo;If you see an <code>a</code>, then maybe a <code>b</code>, and then one or more <code>c</code>, it&rsquo;s a match!&rdquo; for <code>ab?c+</code>.</p> <p>But <strong>the machines can nest</strong>, so you can instead say stuff like, &ldquo;If you see one thing matched by this machine, then maybe one thing matched by that one, followed by one or more things matched by that other, it&rsquo;s a match!&rdquo; So the <code>a</code>, <code>b</code>, and <code>c</code> from the last bit could actually be bigger regular expressions themselves.</p> <p>But you have <strong>no variables</strong> in regex! So, instead, you plop the whole machine descriptions in there in <strong>parentheses</strong>, like <code>(…)(…)?(…)+</code> And repeat the description if you need the same machine twice.</p> <p>Pitch in self-referentiality - &ldquo;if you see exactly the same thing as you ended up matched back there&rdquo; - by using <strong>backrefs</strong> to parenthesized machines, you&rsquo;re in our modern world of extended &ldquo;regular&rdquo; expressions. At that point, what we&rsquo;re talking about is no longer actually expressions describing what&rsquo;s technically known as a <a href="https://en.wikipedia.org/wiki/Regular_language">regular language</a>, but they&rsquo;re exceedingly useful extensions of the notation, so no-one cares. ;)</p> <h2 id="compact-notation-effective-expressive-power">Compact Notation, Effective Expressive Power</h2> <p>What makes regular expressions so useful is:</p> <ul> <li><strong>Reach:</strong> A lot of stuff we want to match against can actually be described by them, especially when you pitch in a lot of the extended power-ups</li> <li><strong>Compactness:</strong> They&rsquo;re a marvelously compact notation for what would otherwise be a lot of very boring code! Instead of writing that code, we dash of a regex, and we leave the translation into code to the regular expression engine.</li> </ul> <h2 id="for-the-more-curious">For the More Curious</h2> <p>If that&rsquo;s whet your whistle, Friedl&rsquo;s <a href="https://www.amazon.com/Mastering-Regular-Expressions-Jeffrey-Friedl/dp/0596528124/ref=as_li_ss_tl?ie=UTF8&amp;qid=1484450477&amp;sr=8-1&amp;keywords=mastering+regular+expressions&amp;linkCode=ll1&amp;tag=jerwshe-20&amp;linkId=e3ed2732f8f5a0dd548b294c435c25c3"><em>Mastering Regular Expressions</em></a> is excellent. And, as a bonus, you can probably just read the first few chapters and emerge enlightened. :)</p> <p>P.S. You can also look at regular expressions as definitions of regular languages - as <strong>generators rather than consumers</strong> of text. Running them backwards like this can be a good way to think about whether a regex you&rsquo;re writing captures exactly what you&rsquo;re aiming at, or whether it might include a bit more than you intended!</p> <p>P.P.S. And if you think about them in terms of machines, it&rsquo;s really easy to start thinking about how to write <em>fast</em> regular expressions.</p> <p>P.P.P.S. <a href="https://alpha.app.net/jws/post/70880896">Hat-tip to @bazbt3 over at App.Net.</a> What is dead can never die!</p> Iterative Development https://jeremywsherman.com/blog/2017/01/14/iterative-development/ Sat, 14 Jan 2017 00:00:00 +0000 https://jeremywsherman.com/blog/2017/01/14/iterative-development/ <p>&ldquo;At last, my current practice of writing no automated tests has the blessing of science! See, TDD doesn&rsquo;t do anything!&rdquo; That&rsquo;s how Fucci et al.&lsquo;s 2016 conference paper <a href="http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf">An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach</a> was introduced to me.</p> <p>And, indeed, it concludes like so:</p> <blockquote> <p>Despite adopting such countermeasures, aimed at reducing researchers&rsquo; bias [when replicating a prior, baseline study], we confirmed the baseline results: TDD does not affect testing effort, software external quality, and developers’ productivity.</p> </blockquote> <p>Takeaways:</p> <ul> <li>All coding is debugging <ul> <li>Work in small steps</li> <li>Stay grounded in observed outputs</li> <li>Keep good notes (tests or REPL session logs)</li> </ul> </li> <li>TDD won&rsquo;t slow you down at steady state <ul> <li>Changing how you code to be more intentional and iterative might to start.</li> <li>What will definitely slow you down: Learning your tooling and the impact of that iterative approach on the code you produce (expose those probe points for external testing! add indicator LEDs via assertions!)</li> </ul> </li> </ul> <p>Now that we&rsquo;ve got the conclusion out of the way, keep reading to see how I got there. :)</p> <p>&ldquo;At last, my current practice of writing no automated tests has the blessing of science! See, TDD doesn&rsquo;t do anything!&rdquo; That&rsquo;s how Fucci et al.&lsquo;s 2016 conference paper <a href="http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf">An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach</a> was introduced to me.</p> <p>And, indeed, it concludes like so:</p> <blockquote> <p>Despite adopting such countermeasures, aimed at reducing researchers&rsquo; bias [when replicating a prior, baseline study], we confirmed the baseline results: TDD does not affect testing effort, software external quality, and developers’ productivity.</p> </blockquote> <p>Takeaways:</p> <ul> <li>All coding is debugging <ul> <li>Work in small steps</li> <li>Stay grounded in observed outputs</li> <li>Keep good notes (tests or REPL session logs)</li> </ul> </li> <li>TDD won&rsquo;t slow you down at steady state <ul> <li>Changing how you code to be more intentional and iterative might to start.</li> <li>What will definitely slow you down: Learning your tooling and the impact of that iterative approach on the code you produce (expose those probe points for external testing! add indicator LEDs via assertions!)</li> </ul> </li> </ul> <p>Now that we&rsquo;ve got the conclusion out of the way, keep reading to see how I got there. :)</p> <h2 id="martin-it-tells-us-nothing-keep-on-keeping-on-with-tdd">Martin: It Tells Us Nothing: Keep on keeping on with TDD!</h2> <p>In the time since I&rsquo;ve had &ldquo;I should really write up my response to this&rdquo; on my to-do list (a few months…), Bob Martin wrote up his take: <a href="http://blog.cleancoder.com/uncle-bob/2016/11/10/TDD-Doesnt-work.html">TDD Doesn&rsquo;t Work</a>.</p> <p>His article concludes the study made a distinction without difference and so naturally found no difference: Basically, that the folks involved were still practicing TDD, but where they actually wrote the tests – rather than conceived of and directed their coding efforts to them – was altered slightly.</p> <h2 id="me-iterativeness-is-the-key">Me: Iterativeness Is The Key!</h2> <p>But, check this:</p> <blockquote> <p>Control treatment: the baseline experiment and its replication compared TDD to a really similar approach, labelled as TLD. Under this [sic] circumstances, <strong>we might be focusing on the incorrect part of development process (i.e., whether write tests first or not), and disregard the part of the process in which the a substantial effect might lie (i.e., the <em>iterativeness</em> of the process).</strong> Accordingly, the tasks used for both experiments were designed to fit the iterative nature of both treatments — i.e., isolate the process itself from the cognitive effort required to break down a complex problem into sub-problems. Pančur and Ciglarič [33] made a similar claim reporting the inconclusive results of a similar experiment. (bold emphasis added)</p> </blockquote> <h3 id="fail-fast-focus-on-observed-outputs">Fail Fast, Focus on Observed Outputs</h3> <p>The authors are on to something: I&rsquo;ve seen people new to TDD stumble over learning to work in checkable baby steps. That <strong>iterative, fail-fast approach</strong> is where a lot of the time savings comes from; the other savings comes from being very <strong>intentional and focused</strong> about the concrete change you&rsquo;re attempting to effect, or the specific knowledge you&rsquo;re trying to elicit through experimentation. This same mindset also pays off in spades in <a href="https://www.bignerdranch.com/blog/thoughts-on-debugging-2/">debugging</a>.</p> <h3 id="tdd-or-repl-just-use-one">TDD or REPL, Just Use One!</h3> <p>We know you can <strong>learn this mindset via TDD,</strong> but a dev loop based around a <strong>REPL</strong> can work just as well. It ends up as TDD without the durable byproduct – once the session scrolls away, all those tests you wrote during bring-up are gone.</p> <h3 id="tdd-wont-slow-you-down">TDD Won&rsquo;t Slow You Down</h3> <p>There&rsquo;s another positive takeaway from the paper&rsquo;s conclusion of no substantive difference:</p> <ul> <li>TDD does not affect testing effort, software external quality, and developers’ productivity. <strong>As long as you&rsquo;re working iteratively and actually writing tests, you&rsquo;re going to write working software and be as productive as you can under the circumstances.</strong></li> </ul> <p>If you don&rsquo;t practice TDD already, fear not: TDD is not going to slow you down.</p> <h3 id="but-learning-to-use-test-frameworks-might-to-start">…But Learning to Use Test Frameworks Might, To Start</h3> <p>Getting the hang of working iteratively, and actually writing tests yourself, on the other hand – those will take a bit of time. And then save you far more over time.</p> <h2 id="missing">Missing</h2> <h3 id="the-long-view-maintenance-burden">The Long View: Maintenance Burden</h3> <p>It would be interesting to see experiments comparing TDD or ITLD (iterative, test-last development) and development where one of those two constraints is relaxed:</p> <ul> <li>Either drop the iterative bit, or</li> <li>drop the test-writing bit.</li> </ul> <p>This was a small dev task, so I bet you&rsquo;d see productivity go up as quality goes down. Put another way, <strong>I bet small tasks naturally lead people to ditch both of the things that make up TDD/ITLD.</strong></p> <p>This short scale approach doesn&rsquo;t assess two real-world challenges that we are concerned with as software maintainers:</p> <ul> <li>Responding to changes over time.</li> <li>Not breaking stuff on timescales longer than a single workday in codebases larger than what one person can turn out in a workday.</li> </ul> <h3 id="style-internal-code-quality">Style: Internal Code Quality</h3> <p>The study also did not assess internal quality (is it readable, navigable, maintainable code?) in any way. Out of scope for their purposes, rather important for those of many professional developers, as the wide spread of PR-based code review processes and the flourishing of adjuvants like <a href="https://github.com/realm/SwiftLint">SwiftLint</a> and <a href="https://github.com/danger/danger/">Danger</a> reflect.</p> <h2 id="conclusion">Conclusion</h2> <p>Reread the intro. (Or, as the grimly satirical <a href="http://scarfolk.blogspot.com/">Scarfolk Council</a> would say: &ldquo;For more information, please reread.&quot;) I&rsquo;m saving you having to read the abstract and then page to the conclusion this way. Go forth, and do the same for others. ;)</p> How to Work Around an Empty Zenfolio Zip File https://jeremywsherman.com/blog/2016/11/28/how-to-work-around-an-empty-zenfolio-zip-file/ Mon, 28 Nov 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/11/28/how-to-work-around-an-empty-zenfolio-zip-file/ <p>My family recently had some holiday photos taken. The photographer was using Zenfolio to host their photos. I loved the photos and wanted to archive the originals on my laptop (and NAS, and Amazon Photos, and Time Machine, and Carbon Copy Cloner clone, and…). But every time I tried to download an original – of one photo, of all the photos, makes no difference – <strong>the server always sent me an empty zipfile!</strong></p> <p>I emailed the photographer to let them know, but I wasn&rsquo;t going to wait.</p> <p>Rather than work around this manually by visiting each page and right-clicking to Save As each photo – and I&rsquo;m not sure that would show me the full-size image , anyway! – I figured Zenfolio would have an API.</p> <p>Sure enough, there&rsquo;s a well-enough documented <a href="http://www.zenfolio.com/zf/help/api">Zenfolio API</a>. I was in business!</p> <p>I was able to lash together some shell commands to grab my full photoset. To save you some fumbling, here&rsquo;s how I did it.</p> <p>My family recently had some holiday photos taken. The photographer was using Zenfolio to host their photos. I loved the photos and wanted to archive the originals on my laptop (and NAS, and Amazon Photos, and Time Machine, and Carbon Copy Cloner clone, and…). But every time I tried to download an original – of one photo, of all the photos, makes no difference – <strong>the server always sent me an empty zipfile!</strong></p> <p>I emailed the photographer to let them know, but I wasn&rsquo;t going to wait.</p> <p>Rather than work around this manually by visiting each page and right-clicking to Save As each photo – and I&rsquo;m not sure that would show me the full-size image , anyway! – I figured Zenfolio would have an API.</p> <p>Sure enough, there&rsquo;s a well-enough documented <a href="http://www.zenfolio.com/zf/help/api">Zenfolio API</a>. I was in business!</p> <p>I was able to lash together some shell commands to grab my full photoset. To save you some fumbling, here&rsquo;s how I did it.</p> <h2 id="walkthrough">Walkthrough</h2> <h3 id="grab-the-photo-details-for-the-photoset">Grab the Photo Details for the Photoset</h3> <p><strong>Get the photoset ID.</strong> You can grab this from the URL you&rsquo;re using to view the photos on the photographer&rsquo;s website. If you view your photos at <code>http://www.example.com/p544941453</code>, then your photoset ID is <code>544941453</code>.</p> <p><strong>Fetch the list of photos in that photoset using <code>curl</code></strong> and save the JSON response to disk for the next step:</p> <pre><code>curl -v \ -H'Content-Type: application/json' \ api.zenfolio.com/api/1.8/zfapi.asmx \ -d '{ &quot;method&quot;: &quot;LoadPhotoSetPhotos&quot;, &quot;params&quot;: [544941453, 0, 100], &quot;id&quot;: 1 }' \ &gt; photoset.json </code></pre><p>This grabs the photos in photoset <code>544941453</code> starting from index <code>0</code> and returns at most <code>100</code> photos. <strong>Tweak those values to match your photoset and number of photos.</strong></p> <p>Also, I&rsquo;m using <a href="http://fishshell.com/">fish</a> as my shell. You might need to tweak that command line to make your shell happy, especially with the multiline string literal.</p> <p>See: <a href="http://www.zenfolio.com/zf/help/api/ref/methods/loadphotosetphotos">LoadPhotoSetPhotos method documentation</a></p> <h3 id="download-each-originalurl">Download Each OriginalUrl</h3> <p><strong>Grab the <code>OriginalUrl</code> field</strong> from the photo objects in the photoset response using <a href="https://stedolan.github.io/jq/"><code>jq</code>, the JSON multitool</a>:</p> <pre><code>jq '.result[].OriginalUrl' photoset.json </code></pre><p><strong>Download each file at those URLs</strong> by feeding them to <code>curl</code> via <code>xargs</code>:</p> <pre><code>jq '.result[].OriginalUrl' photoset.json \ | xargs -n 1 curl -O </code></pre><p>(The <code>-n 1</code> is there so that curl sees one <code>-O</code> for each file argument. Without it, <code>xargs</code> would run <code>curl -O url1 url2 url3…</code>. This causes <code>curl</code> to download only the first URL to a matching file on disk; the rest, it starts piping out to stdout. I couldn&rsquo;t work out a good way to get <code>xargs</code> to repeat the <code>-O</code> per argument, so I just throttled it to calling <code>curl -O justASingleURL</code> repeatedly.)</p> <p><strong>Enjoy your photos!</strong></p> <h2 id="caveat-assumes-public-photos">Caveat: Assumes Public Photos</h2> <p><strong>This walkthrough assumes no authentication is required to download your photos.</strong> I lucked out: All my photos had an <code>AccessDescriptor.AccessType</code> of <code>Public</code>.</p> <p>If the originals are password-protected, you&rsquo;ll find a walkthrough of the hoops to jump through in <a href="http://www.zenfolio.com/zf/help/api/guide/download">&ldquo;Downloading Original Files&rdquo;</a>.</p> <p>If things are more locked down, you might need to sort out the authentication flow before you can even grab the photoset details. I didn&rsquo;t need to do any of that, so I can&rsquo;t walk you through how. Sorry!</p> A Practical Example of FlatMap https://jeremywsherman.com/blog/2016/09/22/a-practical-example-of-flatmap/ Thu, 22 Sep 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/09/22/a-practical-example-of-flatmap/ <p>The Swift standard library introduces some unfamiliar concepts if you&rsquo;re coming from Obj-C and Cocoa. <code>map</code> is one thing, but for some, <code>flatMap</code> seems a bridge too far. It&rsquo;s a question of taste, and of background, if something comes across as a well-chosen, expressive phrase or if it just seems like status signaling, high-falutin&rsquo; bullshit.</p> <p>Well, I&rsquo;m not going to sort that all out, but I did find myself rewriting an expression using a mix of <code>if let/else</code> into a <code>flatMap</code> chain recently, so I thought I&rsquo;d share how I rewrote it and why.</p> <p>If you&rsquo;re mystified by <code>Optional.flatMap</code>, read on, and you should have a good feel for what that does in a couple minutes.</p> <p>The Swift standard library introduces some unfamiliar concepts if you&rsquo;re coming from Obj-C and Cocoa. <code>map</code> is one thing, but for some, <code>flatMap</code> seems a bridge too far. It&rsquo;s a question of taste, and of background, if something comes across as a well-chosen, expressive phrase or if it just seems like status signaling, high-falutin&rsquo; bullshit.</p> <p>Well, I&rsquo;m not going to sort that all out, but I did find myself rewriting an expression using a mix of <code>if let/else</code> into a <code>flatMap</code> chain recently, so I thought I&rsquo;d share how I rewrote it and why.</p> <p>If you&rsquo;re mystified by <code>Optional.flatMap</code>, read on, and you should have a good feel for what that does in a couple minutes.</p> <p>I&rsquo;m not going to demystify everything: You still won&rsquo;t know why it&rsquo;s called <code>flatMap</code>.</p> <p>But then, why do we use <code>+</code> for addition? And how do you implement it in terms of a fixed number of bits?</p> <p>Just because you don&rsquo;t know a symbol&rsquo;s etymology or a function&rsquo;s implementation, that doesn&rsquo;t mean you can&rsquo;t make it do useful work for you. If you treat <code>flatMap</code> as an operator written using Roman letters, you can get good value out of it!</p> <h2 id="duck-duck-goose">Duck, Duck, Goose</h2> <p>Here&rsquo;s what some deserialization code looked like to start:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">init</span>?(json: JsonApiObject) { <span style="color:#66d9ef">guard</span> <span style="color:#66d9ef">let</span> name = json.attributes[<span style="color:#e6db74">&#34;name&#34;</span>] <span style="color:#66d9ef">as</span>? String , <span style="color:#66d9ef">let</span> initials = json.attributes[<span style="color:#e6db74">&#34;initials&#34;</span>] <span style="color:#66d9ef">as</span>? String <span style="color:#66d9ef">else</span> { <span style="color:#66d9ef">return</span> <span style="color:#66d9ef">nil</span> } <span style="color:#66d9ef">self</span>.name = name <span style="color:#66d9ef">self</span>.initials = initials <span style="color:#66d9ef">self</span>.building = json.attributes[<span style="color:#e6db74">&#34;building&#34;</span>] <span style="color:#66d9ef">as</span>? String <span style="color:#66d9ef">self</span>.office = json.attributes[<span style="color:#e6db74">&#34;office&#34;</span>] <span style="color:#66d9ef">as</span>? String <span style="color:#66d9ef">self</span>.mailStop = json.attributes[<span style="color:#e6db74">&#34;mailStop&#34;</span>] <span style="color:#66d9ef">as</span>? String <span style="color:#66d9ef">if</span> <span style="color:#66d9ef">let</span> base64 = json.attributes[<span style="color:#e6db74">&#34;photoBase64&#34;</span>] <span style="color:#66d9ef">as</span>? String , <span style="color:#66d9ef">let</span> data = Data(base64Encoded: base64) { <span style="color:#66d9ef">self</span>.photo = UIImage(data: data) } <span style="color:#66d9ef">else</span> { <span style="color:#66d9ef">self</span>.photo = <span style="color:#66d9ef">nil</span> } } </code></pre></div><p>Notice how you&rsquo;re trucking along reading, &ldquo;OK, we set this field, set that field, set that other field, and WHAT THE HECK IS THAT.&rdquo; The <code>if let</code> bit comes out of left field, breaks your ability to quickly skim the code, and takes some puzzling to sort out. It also leads to repeating the assignment in both branches.</p> <h2 id="cleaning-this-up">Cleaning This Up</h2> <h3 id="extract-intention-revealing-method">Extract Intention-Revealing Method</h3> <p>To start with, we can take the existing code as-is, yank it out into a helper method, and call that:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">self</span>.photo = image(fromBase64: json.attributes[<span style="color:#e6db74">&#34;photoBase64&#34;</span>] <span style="color:#66d9ef">as</span>? String) </code></pre></div><p>This makes the call site in <code>init?</code> read fine, but we&rsquo;ve just moved the ugly somewhere else.</p> <h3 id="take-advantage-of-guard">Take Advantage of Guard</h3> <p>Shifting it into a method dedicated to returning an image does open up using <code>guard let</code> to make the unhappy path clear:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">image</span>(fromBase64 string: String?) -&gt; UIImage? { <span style="color:#66d9ef">guard</span> <span style="color:#66d9ef">let</span> base64 = string , <span style="color:#66d9ef">let</span> data = Data(base64Encoded: base64) , <span style="color:#66d9ef">let</span> photo = UIImage(data: data) <span style="color:#66d9ef">else</span> { <span style="color:#66d9ef">return</span> <span style="color:#66d9ef">nil</span> } <span style="color:#66d9ef">return</span> photo } </code></pre></div><h4 id="still-too-noisy">Still Too Noisy!</h4> <p>But that&rsquo;s no real improvement:</p> <ul> <li>The return values just restate our return type. They&rsquo;re noise.</li> <li>The reader has to manually notice that we&rsquo;re threading each <code>let</code>-bound name into the computation that&rsquo;s supposed to produce the next one.</li> <li>We&rsquo;re forced to name totally uninteresting intermediate values just so we have a handle to them to feed into the next computation.</li> </ul> <p>All told, that&rsquo;s a lot of noise for something that&rsquo;s conceptually simple and that should be eminently skimmable.</p> <h3 id="a-pipeline-with-escape-hatch">A Pipeline with Escape Hatch</h3> <p>The pipeline we have is:</p> <ul> <li>feed in a string</li> <li>transform it into data by decoding it as base64</li> <li>transform that into an image by feeding it into <code>UIImage</code></li> <li>spit out the image</li> </ul> <p>The trick is, if any of these steps fails – that is, if any step spits out a <code>nil</code> – we just want to bail out and send back a <code>nil</code> immediately. It&rsquo;s like each step has an escape hatch that short circuits the rest of the pipeline.</p> <h4 id="pipeline-with-escape-hatch-is-just-flatmap">Pipeline with Escape Hatch Is Just FlatMap</h4> <p>Well, that&rsquo;s exactly the behavior that sequencing all these with <code>Optional.flatMap</code> would buy you! Have a look:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">image</span>(fromBase64 string: String?) -&gt; UIImage? { <span style="color:#66d9ef">return</span> string .flatMap { Data(base64Encoded: $0) } .flatMap { UIImage(data: $0) } } </code></pre></div><p>And if you inlined it, it&rsquo;d still be eminently readable, because it puts the topic first (&ldquo;hey, y&rsquo;all, we&rsquo;re going to set <code>photo</code>!&quot;), which preserves the flow of the code and its skimmability, and you can quickly skim the pipeline to see how we get that value.</p> <h3 id="conclusion">Conclusion</h3> <p>Flatmap very clearly expresses a data transformation pipeline, without extraneous syntax and temporary variables.</p> <p>We backed into using it in this example for reasons of readability, not for reasons of &ldquo;I have a hammer! Everything is a nail!&rdquo;</p> <p>Sometimes, the new tool really <em>is</em> the right tool.</p> <h2 id="appendix-similar-rewrites">Appendix: Similar Rewrites</h2> <p>This &ldquo;assign something depending on something/s else&rdquo; situation happens a lot. And it can shake out a lot of different ways.</p> <p>If the expression had been simpler, we could have rewritten it using <code>?:</code> to eliminate the repeated assignment target. This often shows up with code like:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#f92672">- if haveThing { </span><span style="color:#f92672">- x = thing </span><span style="color:#f92672">- } else { </span><span style="color:#f92672">- x = defaultThing </span><span style="color:#f92672">- } </span><span style="color:#f92672"></span><span style="color:#a6e22e">+ x = haveThing ? thing! : defaultThing </span></code></pre></div><p>Which, in that common &ldquo;sub in a default&rdquo; case, can be further simplified:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#f92672">- x = haveThing ? thing! : defaultThing </span><span style="color:#f92672"></span><span style="color:#a6e22e">+ x = thing ?? defaultThing </span></code></pre></div><p>And if <code>nil</code> is an A-OK default, becomes the wonderfully concise:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#f92672">- let defaultThing = nil </span><span style="color:#f92672">- x = thing ?? defaultThing </span><span style="color:#f92672"></span><span style="color:#a6e22e">+ x = thing </span></code></pre></div><p>There&rsquo;s a similar transform that eliminates <code>guard let</code> stacks by using optional-chaining, but that deserves a bit more of an example, I think.</p> The Internet Speaks: Testing FP Code https://jeremywsherman.com/blog/2016/09/20/the-internet-speaks-testing-fp-code/ Tue, 20 Sep 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/09/20/the-internet-speaks-testing-fp-code/ One problem I have writing Swift is that I&rsquo;m not really sure how to tackle testing FP-ish code using XCTest. I did some quick Internet research. If you read it on the Internet, it must be true. This is a distillation of those great Internet truths. The Context: Data Persistence But first, some context. Why did I care about this? I ran into this in the context of sorting out how to persist and restore some app data at specific &ldquo;app lifecycle&rdquo; hooks. <p>One problem I have writing Swift is that I&rsquo;m not really sure how to tackle testing FP-ish code using XCTest.</p> <p>I did some quick Internet research. If you read it on the Internet, it must be true. This is a distillation of those great Internet truths.</p> <h2 id="the-context-data-persistence">The Context: Data Persistence</h2> <p>But first, some context. Why did I care about this?</p> <p>I ran into this in the context of sorting out how to persist and restore some app data at specific &ldquo;app lifecycle&rdquo; hooks.</p> <p>Specifically:</p> <ul> <li>When the app backgrounds, start a background task, then serialize and write to disk, then end the task. <ul> <li>Inputs: data store, serialization strategy, where to write to</li> <li>Outputs: updated file on disk (side effect)</li> </ul> </li> <li>When the app launches, block the main thread till we&rsquo;ve loaded the data from disk and unpacked it. This should be fast enough. Anything else will lead to folks seeing a not-yet-ready UI. <ul> <li>Inputs: serialization strategy, where we wrote to</li> <li>Outputs: We can see the restored DataStore (side effect)</li> </ul> </li> </ul> <p>This is very much &ldquo;app lifecycle&rdquo; stuff, so we want the App Delegate to do it.</p> <p>What&rsquo;s the cleanest code we could imagine?</p> <pre><code>bracket startBackgroundTask endBackgroundTask $ dataStore |&gt; serialize |&gt; write location deserialize(location) |&gt; fromJust seedDataStore |&gt; set dataStoreOwner .dataStore </code></pre><p>I think my big ??? is that I don&rsquo;t get how to test a functional pipeline. It seems to not having any of the seams you&rsquo;d usually rely on.</p> <h2 id="testing-fp-code">Testing FP Code</h2> <p>Summarizing:</p> <ul> <li>Separate out pure code from impure.</li> <li>Use PBT for the pure code.</li> <li>Use typeclasses or protocols or similar dynamic binding methods to swizzle impure actions.</li> </ul> <p>I guess, use acceptance testing to check that you got the wiring to impure stuff correct? That issue seems mostly ignored in favor of the much happier &ldquo;pure functions are easy to test&rdquo; story.</p> <p>In practice, I think I&rsquo;m now foundering on the mess that is object-functional blending. You&rsquo;d hope that the Scala folks might have something good to stay on that, but that&rsquo;ll have to be a later round of The Internet Speaks.</p> <h3 id="static-methods-are-death-to-testability">Static Methods Are Death to Testability</h3> <p><a href="http://misko.hevery.com/2008/12/15/static-methods-are-death-to-testability/">http://misko.hevery.com/2008/12/15/static-methods-are-death-to-testability/</a></p> <p>Recapitulates the problem I identified:</p> <blockquote> <p>Unit-testing needs seams, seams is where we prevent the execution of normal code path and is how we achieve isolation of the class under test. seams work through polymorphism, we override/implement class/interface and than wire the class under test differently in order to take control of the execution flow. With static methods there is nothing to override.</p> </blockquote> <p>Recommends converting static methods to instance methods:</p> <blockquote> <p>If your application has no global state than all of the input for your static method must come from its arguments. Chances are very good that you can move the method as an instance method to one of the method’s arguments. (As in method(a,b) becomes a.method(b).) Once you move it you realized that that is where the method should have been to begin with.</p> </blockquote> <p>Says not to even consider leaf methods as OK as static, because they tend not to remain leaves for long.</p> <h3 id="unit-testing-and-programming-paradigms">Unit Testing and Programming Paradigms</h3> <p><a href="http://www.giorgiosironi.com/2009/11/unit-testing-and-programming-paradigms.html">http://www.giorgiosironi.com/2009/11/unit-testing-and-programming-paradigms.html</a> Identifies the same problem as you move away from leaf functions in the context of procedural programming:</p> <blockquote> <p>The problem manifests when we want to do the equivalent of injecting stubs and mocks in higher-level functions: there are no seams where we can substitute collaborator functions with stubbed ones, useful for testing. If my function calls printf(), I cannot stub that out specifying a different implementation (unless maybe I recompile everytime and play a lot with the preprocessor).</p> </blockquote> <p>Outlines, in theory, what they would do, but have not done, for FP code: Pass in functions to parameterize behavior:</p> <blockquote> <p>So instead of injecting collaborators in the constructor we could provide them as arguments, earning the ability to pass in fake functions in tests. The upper layers can thus be insulated without problems (with this sort of dependency injection) and there are no side effects that we have to take care of in the tear down phase</p> </blockquote> <p>Omits stack and logic paradigms. No surprise there.</p> <h3 id="recoverability-and-testing-oo-vs-fp">Recoverability and Testing: OO vs FP</h3> <p><a href="https://www.infoq.com/news/2008/03/revoerability-and-testing-oo-fp">https://www.infoq.com/news/2008/03/revoerability-and-testing-oo-fp</a></p> <p><em>Sums up a conversation that happens across several blogs. Weirdly omits any links to primary sources. Yuck.</em></p> <p>OO is rife with seams that are easy to exploit, so Feathers likes it. Where you need a seam is a design issue:</p> <blockquote> <p>Another blogger, Andrew, highlights that if “code isn&rsquo;t factored into methods that align with the needs of your tests”, the implementation will need to be changed to accommodate the test. Hence, he argues as well that “thoughts about &ldquo;seams&rdquo; are really just getting at the underlying issue of design for testability”, i.e. the proper placement of seams.</p> </blockquote> <p>But not all systems are always so designed (putting it nicely), so &ldquo;recoverability&rdquo; matters: being able to make something testable in spite of itself.</p> <blockquote> <p>According to Feathers, even though there are alternative modules to link against in functional languages, “it’s clunky”, with exception of Haskel where “most of the code that you&rsquo;d ever want to avoid in a test can be sequestered in a monad”</p> </blockquote> <p>Then there&rsquo;s an argument that pushing the impurity to the edges makes things testable. No-one addresses validating correct composition of verified components, though. :(</p> <h3 id="so-testing-in-functional-programming">SO: Testing in Functional Programming</h3> <p><a href="https://stackoverflow.com/questions/28594186/testing-in-functional-programming">https://stackoverflow.com/questions/28594186/testing-in-functional-programming</a></p> <p>Answers point out:</p> <ul> <li>Function composition builds units, in that you can test them quickly.</li> <li>QuickCheck/SmallCheck dodge the combinatorial explosion of codepaths that you get by composing functions.</li> <li>Coding against a typeclass that you can swizzle out for a test one lets you stub out IO-like functions. (Or just manually pass in a dictionary type.)</li> </ul> Why I'm Meh About JSON API https://jeremywsherman.com/blog/2016/07/23/why-im-meh-about-json-api/ Sat, 23 Jul 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/07/23/why-im-meh-about-json-api/ <p><a href="http://jsonapi.org/">JSON API</a> has been pretty successful at providing a framework for APIs that lets you focus on roughly the entity–relationship diagram of your data.</p> <p>But I find it frustrating at some turns (too flexible!) and peculiar at others (why is it bound to just one content-type?).</p> <p>My frustrations with JSON API are ultimately because it doesn&rsquo;t solve the problems I have as an API consumer, and its aim of preserving flexibility results in API consumers paying the price of that in needing to deal with the foibles of a specific implementation and in manually tuning their API queries.</p> <p>I find the approach taken by <a href="http://graphql.org/">GraphQL</a> more directly and usefully addresses my needs as a client developer while also necessarily, by design, minimizing requests made and data transmitted.</p> <p>JSON API makes it possible to accomplish that, but it leaves the responsibility for doing so up to the client developer; GraphQL makes it possible to accomplish that, but it takes the perftuning responsibility upon itself, which makes my life as a client dev easier.</p> <p><a href="http://jsonapi.org/">JSON API</a> has been pretty successful at providing a framework for APIs that lets you focus on roughly the entity–relationship diagram of your data.</p> <p>But I find it frustrating at some turns (too flexible!) and peculiar at others (why is it bound to just one content-type?).</p> <p>My frustrations with JSON API are ultimately because it doesn&rsquo;t solve the problems I have as an API consumer, and its aim of preserving flexibility results in API consumers paying the price of that in needing to deal with the foibles of a specific implementation and in manually tuning their API queries.</p> <p>I find the approach taken by <a href="http://graphql.org/">GraphQL</a> more directly and usefully addresses my needs as a client developer while also necessarily, by design, minimizing requests made and data transmitted.</p> <p>JSON API makes it possible to accomplish that, but it leaves the responsibility for doing so up to the client developer; GraphQL makes it possible to accomplish that, but it takes the perftuning responsibility upon itself, which makes my life as a client dev easier.</p> <h2 id="in-this-article">In This Article</h2> <ul> <li><a href="#intro">Introduction</a></li> <li><a href="#json-api">JSON API: Bytecount Golfing with -ility Handicaps</a> <ul> <li><a href="#readability">Readability: Not Too Shabby</a></li> <li><a href="#discoverability">Discoverability: Meh</a></li> <li><a href="#flexibility">Flexibility: Hurts Discoverability and Limits Utility of Having a Spec</a></li> <li><a href="#peculiar">Peculiar: Why Only JSON?</a></li> </ul> </li> <li><a href="#not-its-job">Maybe That&rsquo;s Not JSON API&rsquo;s Job?</a></li> <li><a href="#how-better">How Is GraphQL Better?</a> <ul> <li><a href="#graphql-in-a-nutshell">GraphQL in a Nutshell</a></li> <li><a href="#no-intrinsic-content-type">No Intrinsic Content-Type</a></li> <li><a href="#no-manual-perftuning">No Manual Perftuning</a></li> </ul> </li> <li><a href="#conclusion">Conclusion</a></li> </ul> <p><a id="intro"></a></p> <h2 id="introduction">Introduction</h2> <p>I spent the last couple months working on an Ember app. The backend was running the <a href="https://github.com/cerebris/jsonapi-resources/">cerebris/jsonapi-resources</a> flavor of JSON API implementation. The frontend was using Ember Data&rsquo;s JSON API adapter.</p> <p>It worked, but I also kept running across ugly data requests like:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#66d9ef">this</span>.<span style="color:#a6e22e">store</span>.<span style="color:#a6e22e">findAll</span>( <span style="color:#e6db74">&#39;work-order&#39;</span>, { <span style="color:#a6e22e">include</span><span style="color:#f92672">:</span> [ <span style="color:#e6db74">&#39;location&#39;</span> , <span style="color:#e6db74">&#39;shipping-address&#39;</span> , <span style="color:#e6db74">&#39;credit-card&#39;</span> , <span style="color:#e6db74">&#39;user&#39;</span> , <span style="color:#e6db74">&#39;shipments.shipment-items.order-item.inventory-item.part&#39;</span> , <span style="color:#e6db74">&#39;order-items.inventory-item&#39;</span> , <span style="color:#e6db74">&#39;inventory-items.part.part-kind&#39;</span> ].<span style="color:#a6e22e">join</span>(<span style="color:#e6db74">&#39;,&#39;</span>) }); </code></pre></div><p>When I see something like that, all I can think is, <em>Why am I listing all this out for the computer? It should figure it out!</em> Maybe in a year or so, Ember Data will indeed do that, but you need to do that sort of thing today, unless you want template rendering to lead to this conversation between HTMLBars and Ember Data: &ldquo;render, oh crud we need some data, fetching… rerender, oh crud more‽ ok, fetching… rerender… what, more! fetching…&rdquo;</p> <p>But if you&rsquo;re hitting the API by hand – be it manual XMLHTTPRequest preparation or <code>curl</code> – that leads to a bear of a URL. And parsing out the data once it arrives is also not so fun. I hope you enjoy writing <code>JOIN</code> logic client-side!</p> <p>And how do you even find out what you can toss in that <code>include</code> bit? I just popped over to the backend source and nosed around. That&rsquo;s fine when you have access to the backend source code, but what if you don&rsquo;t? What&rsquo;s JSON API got to say to that?</p> <p>Well. I&rsquo;m not terribly happy with JSON API&rsquo;s answers – and we&rsquo;ll come to those in a bit – but let&rsquo;s see if we can understand where JSON API is coming from: How did JSON API end up like this, and to what end?</p> <p><a id="json-api"></a></p> <h2 id="json-api-bytecount-golfing-with--ility-handicaps">JSON API: Bytecount Golfing with -ility Handicaps</h2> <p><a href="http://jsonapi.org/format/1.0/#introduction">JSON API&rsquo;s primary purpose</a> is to minimize request count and data transmitted. It attempts to balance this against concerns for readability, flexibility, and discoverability.</p> <p><a id="readability"></a></p> <h3 id="readability-not-too-shabby">Readability: Not Too Shabby</h3> <p>JSON API is pretty readable. Hit a site using it (most anything Ember), and check out the API requests and responses in your browser debugging tools, and you can work out pretty quickly what&rsquo;s going on.</p> <p>The side-car style for included objects, where you have to bounce from an ID reference in the main response to a lookup table that got sent along with it, hurts a bit here for humans: you have to do manual joins client-side. But inlining them wouldn&rsquo;t play nice with the &ldquo;minimize transfer&rdquo; focus, so it makes sense.</p> <p>The URLs asking for those included objects get pretty gnarly, though.</p> <p><a id="discoverability"></a></p> <h3 id="discoverability-meh">Discoverability: Meh</h3> <p>I’d say its discoverability is pretty darn poor; this is partly a result of its flexibility, but mainly a result of its not providing much in the way of standardized introspection facilities.</p> <p>The most frustrating lack for me when I hear &ldquo;backend is using JSON API&rdquo; is not being able to hit the root of the API and crawl from there to work out the whole of the API and what it supports. This is one of the most important attributes for the usability of a RESTful API from where I stand, but JSON API drops the ball, or heck, doesn&rsquo;t even pick it up in the first place: Hypertext through-and-through it simply ain&rsquo;t.</p> <p>Where this often comes to a head is with <code>include</code>; there&rsquo;s no standard way to signal that this is supported by a backend. You can give it a go and see if it yells at you, though. But if it does support it, it&rsquo;s not clear what you&rsquo;re not/allowed to include with something until you try.</p> <p>And if the backend doesn&rsquo;t support <code>include</code>, then it&rsquo;s free to unilaterally include whatever alongside the data you asked for. If the backend API is sanely versioned – and JSON API does not specify how to manage that – you&rsquo;re probably fine, but if it&rsquo;s not, and your JSON API client library prefers to fail eagerly rather than being liberal in what it accepts, your backend can break your frontend pretty readily. Versioning aside, that&rsquo;s more an implementation issue than a spec issue, though, so we can let that slide.</p> <p>So we have our answer to the question from the intro: How do you even find out what you can toss in that <code>include</code> bit? You don&rsquo;t, or you guess, or you look up the docs or source code for the backend, or you email support. Mmm, emailing support: Definitely something I like to include smack in the middle of my development cycle.</p> <p><a id="flexibility"></a></p> <h3 id="flexibility-hurts-discoverability-and-limits-utility-of-having-a-spec">Flexibility: Hurts Discoverability and Limits Utility of Having a Spec</h3> <p>There are a lot of “servers may do this, or that, or maybe that…” bits in there too, which make finding out a server uses JSON API less of a “now I know everything about it” than it could be. (Search for &ldquo;MAY&rdquo; and &ldquo;SHOULD&rdquo; in the document.)</p> <p>We saw this with <code>include</code>, but it also comes into play with requesting only certain bits of a record (sparse fieldsets), sorting, pagination, and filtering, the latter of which is specified in its entirety as: &ldquo;The <a href="http://jsonapi.org/format/#fetching-filtering">filter query parameter</a> is reserved for filtering data. Servers and clients SHOULD use this key for filtering operations.&rdquo; The limited specification of filtering and sparse fieldsets seems suprising in the face of a focus on reducing the amount of data transferred: This seems very much fair game for a spec with that aim in mind, but it handwaves and throws it in the flexibility bin, instead.</p> <p>This really smarts for two reasons as a client dev:</p> <ul> <li>There&rsquo;s no standard way to communicate what implementation-defined choices a JSON API backend has made.</li> <li>There&rsquo;s no requirement to make those choices uniformly across all APIs.</li> </ul> <p>This again means that learning an API is using the JSON API spec doesn&rsquo;t buy you as much as it could; you still have to ask a lot of questions to sort out what that means in practice.</p> <p>It also means that any client-side de/serializer for JSON API is limited in the support it can provide to you. The spec is very open to customization, which means that you will have to learn those customizations in force for your backend and teach your JSON API parser about them.</p> <p>This reminds me a bit of how OAuth 2 moved from being a spec to a meta-&ldquo;spec&rdquo;, flexible to a fault, as described by the one-time lead author and editor of that spec:</p> <blockquote> <p>One of the compromises was to rename it from a protocol to a framework, and another to add a disclaimer that warns that the specification is unlike to produce interoperable implementations. (<a href="https://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-hell/">&ldquo;OAuth 2.0 and the Road to Hell&rdquo;</a>)</p> </blockquote> <p><a id="peculiar"></a></p> <h3 id="peculiar-why-only-json">Peculiar: Why Only JSON?</h3> <p>JSON API seems weirdly bound to the content-type (it’s an API! in JSON!), which is kind of funny to me in light of the &ldquo;A server MUST prepare responses, and a client MUST interpret responses, in accordance with HTTP semantics” language. This feels like following the letter rather than spirit of the law: There&rsquo;s no notion of a resource that might go by various possible representations. Content transferred under JSON API&rsquo;s auspices goes by a JSON-API–specific content-type.</p> <p>JSON is not a terribly expressive data format; you&rsquo;ve got the rudiments needed to cobble together more specific data types atop it. That also means there&rsquo;s little reason you couldn&rsquo;t translate the data in a JSON API response into another content type, be it BSON, XML, S-expressions, or something even more unique.</p> <p><a id="not-its-job"></a></p> <h2 id="maybe-thats-not-json-apis-job">Maybe That&rsquo;s Not JSON API&rsquo;s Job?</h2> <p>Perhaps the <a href="http://jsonapi.org/">JSON API homepage,</a> rather than the spec, is more honest in its aims:</p> <blockquote> <p>If you’ve ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool.</p> <p>By following shared conventions, you can increase productivity, take advantage of generalized tooling, and focus on what matters: your application.</p> <p>Clients built around JSON API are able to take advantage of its features around efficiently caching responses, sometimes eliminating network requests entirely.</p> </blockquote> <p>It takes for granted you&rsquo;re building an API, and it&rsquo;s only going to support JSON. Its pitch: Use JSON API so you don&rsquo;t have to quibble about how you encode your data, and you get this already thought-through support for caching data and minimizing the requests needed for free!</p> <p><strong>Perhaps JSON API&rsquo;s audience is specifically API producers, not consumers, and that&rsquo;s why I don&rsquo;t find it addressing my needs.</strong></p> <p><a id="how-better"></a></p> <h2 id="how-is-graphql-better">How Is GraphQL Better?</h2> <p>The more declarative approach of <a href="http://graphql.org/">GraphQL</a> (and, to a lesser degree, <a href="https://netflix.github.io/falcor/">Falcor</a>) fulfills the spec-stated goals better than JSON API does itself.</p> <p>Heck, it also satisfies those of the homepage better, too!</p> <p><a id="graphql-in-a-nutshell"></a></p> <h3 id="graphql-in-a-nutshell">GraphQL in a Nutshell</h3> <p>The rough idea is:</p> <ul> <li>There is a typed spec for what data is available and how it’s related.</li> <li>Components can request specific bits they need using a query language. Queries can be typechecked.</li> <li>A query builder can aggregate component requests into a more general request, coalesce them, and then hit the backend/respond from cache intelligently, without the components needing to worry about this.</li> <li>The system then vends back to the components precisely the info they requested, no more, no less.</li> </ul> <p>You can see where that’d be handy in a React world of little components asking for this or that nugget of info, which was the background against which GraphQL arose.</p> <p><a id="no-intrinsic-content-type"></a></p> <h3 id="no-intrinsic-content-type">No Intrinsic Content-Type</h3> <p>The actual wire protocol is kind of beside the point unless you’re the GraphQL engine implementor. Is it sent by JSON or BSON or MP3s modulated at 56kbps? Is it using HTTP over TCP? Audio blips over SCTP? Who cares! Here’s how the data is laid out when it reaches you, here&rsquo;s the types of that data; ask for what you need.</p> <p><a id="no-manual-perftuning"></a></p> <h3 id="no-manual-perftuning">No Manual Perftuning</h3> <p>The query optimization bit can get arbitrarily clever without impacting the components, which is excellent for future performance tuning: Upgrade your client-side GraphQL engine, and your app stands to suddenly get more performant, without any further work on your part.</p> <p>And then I go to an Ember app using JSON API where they have these insane URLs where they’ve got <code>&amp;include=this,that,theotherthing,ohandnowthisthing</code> and the diffs for that insanely long line are so fun to read, lemme tell you. (I prettied up the intro code snippet to use <code>[…].join()</code> so it&rsquo;d be readable at most widths without horizontal scrolling, but that was just one big ol&rsquo; string in the source. Ayup.)</p> <p>It’s this very manual, rough query optimization by hand that I think it’s silly they’re needing to worry about, when they&rsquo;re not even concerned about query optimization; they just need some data. And the optimization is limited by the size of the records, as well.</p> <p><a id="conclusion"></a></p> <h2 id="conclusion">Conclusion</h2> <p>JSON API is a rather limited spec that I find flexible to a fault as a client developer. It seems wide open to bikeshedding still on the API producer&rsquo;s side as well, due to that flexibility, so I&rsquo;m not sure how well it meets either its spec-stated or marketing-stated aims.</p> <p>GraphQL offers a declarative approach to directly expressing the data available – which addresses my desire to be able to pull that information without a lot of digging as a client dev – and the data requested – which addresses my desire to be able to pull the data I need without worrying about the details of how it&rsquo;s going to get to me.</p> <p>But JSON API slots neatly into an existing niche – an API! in JSON! Hey, I think I&rsquo;ve used a few of those! While GraphQL is a different bird entirely. Consequently, I&rsquo;ve yet to use GraphQL, while I have ended up working against a JSON API already, and expect I&rsquo;ll find myself doing so again in future as well.</p> <p>JSON API is an incremental improvement, serviceable and certainly no worse than even more thoroughly <em>ad hoc</em> API creations, and so I expect it to spread widely: I expect to run into a lot of JSON API backends, and may never have the chance to consume a GraphQL backend. I&rsquo;m glad for what sanity JSON API does bring to the wild west of APIs out there.</p> <hr> <p>Thanks to <a href="http://www.chriskrycho.com/">Chris Krycho</a> for feedback on a draft of this article. He tripped over all the awkward transitions so you don&rsquo;t have to. ;)</p> Father's Day: Happy Hurricane https://jeremywsherman.com/blog/2016/06/17/fathers-day-happy-hurricane/ Fri, 17 Jun 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/06/17/fathers-day-happy-hurricane/ <p>This Sunday marks my second Father&rsquo;s Day as a father. If you&rsquo;re not yourself a parent, that won&rsquo;t mean much to you. It certainly didn&rsquo;t to me. If you&rsquo;re en route to fatherhood, read on to learn what &ldquo;fatherhood&rdquo; actually means.</p> <p>This Sunday marks my second Father&rsquo;s Day as a father. If you&rsquo;re not yourself a parent, that won&rsquo;t mean much to you. It certainly didn&rsquo;t to me. If you&rsquo;re en route to fatherhood, read on to learn what &ldquo;fatherhood&rdquo; actually means.</p> <p>My experience was that preparation for new parents focused heavily on the birth experience. What I knew of what would follow focused primarily on early childhood development and dangers with a side helping of lactation and baby-wearing. These are good things to know, but they don&rsquo;t do jack for helping you cope with what having a newborn in your house means for you.</p> <h2 id="birth-as-loss">Birth as Loss</h2> <p>As a newborn, your kid is entirely dependent on its parents for everything. You will be eating, breathing, and sleeping baby. Your schedule is its schedule.</p> <p>Eventually, stuff might get a bit saner. You&rsquo;ll get nursing sorted out, you&rsquo;ll find a sleeping arrangement that works for your family, you&rsquo;ll find some ways to care for your kid.</p> <p>But there&rsquo;s an especially strong and demanding pairbond between mother and child, and you might very well feel left out, or more strongly, crowded out: You might feel like you&rsquo;ve lost your wife to this child. And babies are not terribly relatable creatures, but they are very demanding, and they know no patience. It can feel like a raw deal.</p> <p>I turned a corner once my son was able to laugh. I could do something, and he could respond to it, and I could relate to that. I think that&rsquo;s when my baby went from &ldquo;it&rdquo; to &ldquo;he&rdquo; for me.</p> <h2 id="say-goodbye-to-life-as-you-knew-it">Say Goodbye to Life as You Knew It</h2> <p>When they&rsquo;re a baby, their schedule is yours.</p> <p>Turns out, that doesn&rsquo;t really change as they age into toddlerhood.</p> <h3 id="youve-yielded-autonomy">You&rsquo;ve Yielded Autonomy</h3> <p>You&rsquo;ve lost a lot of autonomy by assuming stewardship of an amateur human. Sure, you can stay up late; but if your kid wakes up at 6 am, someone has to be up with them. That someone is likely you. So either you go to bed, or you spend a day tired and cranky, and no good to nobody. When you wake up is no longer your choice, and if you know what&rsquo;s best for you, then neither is when you go to sleep, either.</p> <p>More than that: What you do during the day is restricted to what you can do while watching over your child. Maybe you&rsquo;ll have a quiet kid who is fine sitting and playing with whatever for a while. This won&rsquo;t be much of a burden to bear. Maybe you&rsquo;ll have a very interactive and active child who very much wants to do something right now thank you very much and are you watching this because we&rsquo;re going to do this together. You&rsquo;ll find your options in that scenario are rather limited.</p> <p>Young children don&rsquo;t suffer fools, or delays, gladly. If doing something involves waiting around, especially quietly, you can probably cut it out. Like dinners out with a 45-minute wait spent standing around and ordering drinks at the bar and gabbing to pass the time? Yeah - that&rsquo;s incompatible with wee ones. Movies out don&rsquo;t really work either. Picking up donuts to go with a kid in tow can be touch and go if there&rsquo;s a line. A lot of stuff you took for granted that you could do, you can&rsquo;t, at least without a sonic tax, possibly with tears attached.</p> <p>Everyday stuff you take for granted that needs to happen can also become a challenge. Traveling by car means a lot more prep work. Shopping requires half a mind on what exactly your kid is doing with that produce you thought you safely tucked in the cart. And is vacuuming worth a tussle over who gets to control the vacuum? How badly do you need something cleaned, and how clean is clean enough?</p> <h3 id="youve-also-lost-environmental-control">You&rsquo;ve Also Lost Environmental Control</h3> <p>That&rsquo;s a good segue from loss of personal autonomy into loss of control. As an adult, you have a lot of control over your environment. If you&rsquo;ve got your own place, you can pretty much stick something wherever, and expect to find it still there. You can demolish a staircase and landing, cut a hole for a new door, and take your time fitting a new door and rebuilding the staircase. You know well enough not to try to leap a storey down onto unforgiving cement. Even your cat&rsquo;s curiosity isn&rsquo;t enough to overcome their caution in that case.</p> <p>Your kid is another matter. Even if they knew well enough that falling down that far would be a bad idea, they&rsquo;re just not terribly good at moving around and keeping track of the environment in their head. They can easily accidentally walk too far, or lose their balance near an edge, or forget to watch where they&rsquo;re going because there&rsquo;s a housefly or a patch of sunlight. So you&rsquo;ll find yourself reshaping your entire environment to fit their needs and behaviors. And you&rsquo;ll weigh convenience against how big of a mess they can make if you&rsquo;re looking elsewhere for thirty seconds. (A salt cellar makes a great mini-sandbox in a pinch, don&rsquo;t you know?)</p> <p>It&rsquo;s a big step from &ldquo;master of my tiny pocket universe&rdquo; to &ldquo;adult graciously allowed to exist as my caregiver and diviner of my needs and desires&rdquo;.</p> <h2 id="a-change-in-the-weather">A Change in the Weather</h2> <p>I found this stifling and isolating. It&rsquo;s a very peculiar experience to find you&rsquo;ve more autonomy in your work life than in your home life.</p> <p>But it has its upsides. And I&rsquo;d do it again.</p> <h3 id="recentering">Recentering</h3> <p>All those losses are losses from a point of view where you&rsquo;re at the center of your own universe.</p> <p>In practice, they&rsquo;re just side effects of shifting the center from yourself to your family. It&rsquo;s no longer all about you. There&rsquo;s not time for you to maintain that illusion any more. Welcome to adulthood.</p> <h3 id="learning-patience--humility">Learning Patience &amp; Humility</h3> <p>Children are elemental forces. You can&rsquo;t reason with them for the several years. You can empathize. You can distract with counter-proposals. But you can&rsquo;t negotiate.</p> <p>You mostly won&rsquo;t get your way. You&rsquo;ll find that it doesn&rsquo;t even matter that you don&rsquo;t get your way. You just wanted things to go your way because that&rsquo;s what you were comfortable with.</p> <p>You&rsquo;re going to have to relax control and work with the situation as it presents itself. You&rsquo;ll leave getting your way for when it matters – when there&rsquo;s risk to health or safety, or there&rsquo;s something important enough that it&rsquo;s worth possibly distressing your kid, stressing everyone around you out, and maybe dealing with some shrieking and crying.</p> <p>One concrete way this shows up is in learning to be patient. Yeah, I get it, you want to go right now. But your kid doesn&rsquo;t. And you don&rsquo;t really need to go right now. You just want to. Suck it up and wait a while. Set the expectation that you will be leaving soon. When the time comes, then you can leave.</p> <h3 id="facing-humanity-head-on">Facing Humanity Head-On</h3> <p>Stuff will get broken. Things will go wrong. A lot of things will go wrong. Kids are clumsy, curious, and not bridled by concern for cleanliness, hygiene, or common sense. This is OK.</p> <p>If you&rsquo;re a perfectionist, you&rsquo;re probably accustomed to everything going to plan, and ensuring that you have a plan and execute on it such that everything goes to plan. You won&rsquo;t be able to exert that level of control, that clockwork execution, when it comes to large parts of your life any more.</p> <p>You might have spent a couple decades driving out the human. It&rsquo;s back now in spades, and you&rsquo;ve no choice but to confront it head on.</p> <p>This also teaches patience. It teaches you to expect people to stumble, to make mistakes, to err. You&rsquo;ve probably kind of known that was the case in theory, but it wasn&rsquo;t your experience before, and it was hard to cut someone a break because you&rsquo;d worked out how to run things so you didn&rsquo;t need anyone to cut you a break. Now it is your experience, and theory is practice, and boy, will you be getting a lot of practice. And you&rsquo;ll probably be really glad when people cut you a break for acting in weird ways, running out of line to snatch up a kid about to get in trouble, walking all throughout the restaurant courtyard following an imp climbing up and down and around things and maybe bumping into people before bouncing off and away onto the next thing.</p> <h3 id="having-fun">Having Fun</h3> <p>With kids, the lows can be low, but the highs can be so high. You have a license to be silly and a new set of eyes to experience the world through. You get to look at such commonplaces as trees, birds, and squirrels with fresh awareness and naked joy at their existence and activity. And if you&rsquo;ve forgotten to play, you&rsquo;ll learn that anew, too.</p> <h2 id="the-traumatic-hurricane-of-fatherhood">The Traumatic Hurricane of Fatherhood</h2> <p>So, Father&rsquo;s Day. When you were born, you destroyed someone&rsquo;s world and remade it around you. As a new father, you have to come to terms with the dramatic difference in responsibility, relationships, and rituals that come with this hurricane of a change. It&rsquo;s sudden and total, but you can build a new and better life in its aftermath.</p> <p>Here&rsquo;s to hoping having a second kid is less hurricane and more tropical storm!</p> Types Complement Tests Complement Types https://jeremywsherman.com/blog/2016/05/05/types-complement-tests-complement-types/ Thu, 05 May 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/05/05/types-complement-tests-complement-types/ Types and tests are complementary. They might even be synergistic: The two together can accomplish what neither can alone. They definitely are not rivalrous goods, and if you&rsquo;re picking only one, you&rsquo;re doing yourself a disservice. If You Have To Pick One, Though There&rsquo;s a ceiling to how far we can get with types. Most languages developers work in have rather limited type systems. Most developers lack the skill, practice, and simple exposure to past examples to make use of more powerful type systems. <p>Types and tests are complementary. They might even be synergistic: The two together can accomplish what neither can alone. They definitely are not rivalrous goods, and if you&rsquo;re picking only one, you&rsquo;re doing yourself a disservice.</p> <h2 id="if-you-have-to-pick-one-though">If You Have To Pick One, Though</h2> <p>There&rsquo;s a ceiling to how far we can get with types. Most languages developers work in have rather limited type systems. Most developers lack the skill, practice, and simple exposure to past examples to make use of more powerful type systems. That&rsquo;s not a slight: Generating those examples today can be a good way to get yourself at least a Masters if not a PhD.</p> <p>We can push automated testing really far regardless of type system. There&rsquo;s an abundance of popular literature on the subject. If you want to get better, you don&rsquo;t have to look far, and you can put what you learn to practice immediately.</p> <p>If you had to pick between either building 100% TDD&rsquo;d code in a unityped language or building code with no automated tests in a conventionally typed language, you&rsquo;d be a fool not to pick the TDD&rsquo;d codebase.</p> <h2 id="but-you-dont-so-use-both">But You Don&rsquo;t, So Use Both</h2> <p>You don&rsquo;t have to choose one or the other. Reject the false dichotomy, chase off its acolytes on their hobby horses, and make the most of all the technologies available to you today to produce better software.</p> Beyond Type Wars: Types Can Be Tests Too https://jeremywsherman.com/blog/2016/05/05/beyond-type-wars-types-can-be-tests-too/ Thu, 05 May 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/05/05/beyond-type-wars-types-can-be-tests-too/ <p>Types and tests are not at war. <a href="https://jeremywsherman.com/blog/2016/05/05/types-complement-tests-complement-types/">Choose both.</a></p> <p>In fact, if we tilt our heads a bit, types are just another flavor of test.</p> <p>Don&rsquo;t use just one flavor of testing; use all the tools you have at your disposal to make the best software you can.</p> <h2 id="type-wars">Type Wars</h2> <p>Robert C. Martin believes code TDD&rsquo;d into existence, and so having 100% test coverage by construction, nullifies the value of types:</p> <p>Types and tests are not at war. <a href="https://jeremywsherman.com/blog/2016/05/05/types-complement-tests-complement-types/">Choose both.</a></p> <p>In fact, if we tilt our heads a bit, types are just another flavor of test.</p> <p>Don&rsquo;t use just one flavor of testing; use all the tools you have at your disposal to make the best software you can.</p> <h2 id="type-wars">Type Wars</h2> <p>Robert C. Martin believes code TDD&rsquo;d into existence, and so having 100% test coverage by construction, nullifies the value of types:</p> <blockquote> <p>My own prediction is that TDD is the deciding factor. You don&rsquo;t need static type checking if you have 100% unit test coverage. And, as we have repeatedly seen, unit test coverage close to 100% can, and is, being achieved. What&rsquo;s more, the benefits of that achievement are enormous.</p> <p>Therefore, I predict, that as TDD becomes ever more accepted as a necessary professional discipline, dynamic languages will become the preferred languages. The Smalltalkers will, eventually, win. (<a href="http://blog.cleancoder.com/uncle-bob/2016/05/01/TypeWars.html" title="Type Wars">Robert C. Martin, &ldquo;Type Wars&rdquo;, 2016</a>)</p> </blockquote> <p>The further your own development practice is from TDD, the more ludicrous this will seem to you.</p> <p>If you ignore Martin&rsquo;s emphasis on TDD, and focus instead on the &ldquo;100% unit test coverage&rdquo; bit, you&rsquo;re likely to reject it out of hand: <a href="http://blog.cgiffard.com/on-the-importance-of-manually-maintained-isolation-for-unit-tests-written-in-impure-languages/#quantifyingprogress">Coverage measures are a very fraught and limited measure.</a> Even if you go &ldquo;but that&rsquo;s just line coverage!&quot;, well, <a href="http://blog.regehr.org/archives/872">not even 100% branch coverage</a> suffices to demonstrate freedom from fairly mechanical bugs, never mind more abstract errors in how you&rsquo;ve implemented whatever half-imagined, unspecified system you&rsquo;re aiming at.</p> <h2 id="the-compilation-test">The Compilation Test</h2> <p>I think he&rsquo;s leaving a tool on the table, though. Not even any tool: A robust bevy of tests. And a tool that slots neatly into test-driven development.</p> <p>The more powerful your type system, the more oomph you can get out of simply, &ldquo;Does it compile?&rdquo;</p> <p>Even with Java, though, you can get rather far:</p> <blockquote> <p>Defining types is very much like writing tests—the compiler continuously checks the types for consistency while we loop back and fix errors. Step 0[, define all the types,] is exactly like normal TDD, except we are making formal statements about the system that the compiler maintains. Could step 0 take a long time? Sure. Maybe with a sufficiently-advanced type system we never even leave step 0. With Java I’m going to hit a wall pretty fast, but not before avoiding many of the worst problems with the Money design. (<a href="https://spin.atomicobject.com/2014/12/10/typed-language-tdd-part2/">Ken Fox, &ldquo;More Typing, Less Testing: TDD with Static Types, Part 2&rdquo;, 2014</a>)</p> </blockquote> <p>As that demonstrates, you can usefully incorporate types into test-driven development with thoroughly salutary effects.</p> <h2 id="it-cramps-my-style">It Cramps My Style</h2> <p>It&rsquo;s true that, once you&rsquo;ve got a type system, you&rsquo;re constrained to writing code that fits within its constraints. Often you can ram through something that doesn&rsquo;t, but it&rsquo;s uncomfortable and tends to come with some at least syntactic overhead that makes it not nice to do.</p> <p>TDD puts you under similar constraints, though: In order to achieve test isolation, you have to structure your software differently. You&rsquo;ve narrowed your collection of possible programs from all those that can be represented in your language to only those that can be test-driven into existence and so all those that yield readily to automated testing.</p> <p>Both typing and testing constrain what we can do with our code; we accept both limits because they free us to build with more confidence than we&rsquo;d have without either.</p> <h2 id="types-and-tests-or-types-are-tests">Types AND Tests, Or Types ARE Tests</h2> <p>Whichever way you look at it, use &lsquo;em both.</p> <p>Your software will be better for it, and you&rsquo;ll grow to be a better software engineer for the practice.</p> Beyond Our Ken https://jeremywsherman.com/blog/2016/05/05/beyond-our-ken/ Thu, 05 May 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/05/05/beyond-our-ken/ <p>The more I poke around, the more convinced I become that actually knowing what a piece of software is supposed to do is truly rather rare and generally beyond mortal ken. Making it do what you think it should do is nearly beyond our grasp.</p> <p>If we&rsquo;re honest with ourselves, we need every tool we can get just to wrangle software into behaving. That means types, that means tests, and that means, yes, even: proofs.</p> <p>And that also means that proofs need tests, too.</p> <p>What drove this home was reading a couple papers related to combining proving and testing.</p> <p>The more I poke around, the more convinced I become that actually knowing what a piece of software is supposed to do is truly rather rare and generally beyond mortal ken. Making it do what you think it should do is nearly beyond our grasp.</p> <p>If we&rsquo;re honest with ourselves, we need every tool we can get just to wrangle software into behaving. That means types, that means tests, and that means, yes, even: proofs.</p> <p>And that also means that proofs need tests, too.</p> <p>What drove this home was reading a couple papers related to combining proving and testing.</p> <h2 id="types-and-tests">Types and Tests</h2> <p>I&rsquo;m on record for arguing in favor of using both types and tests to their utmost in both <a href="https://jeremywsherman.com/blog/2016/05/05/types-complement-tests-complement-types/">Types Complement Tests Complement Types</a> and <a href="http://127.0.0.1:4000/blog/2016/05/05/beyond-type-wars/">Beyond Type Wars.</a></p> <p>It&rsquo;ll come as no surprise what I&rsquo;m going to recommend here: Use proofs and tests. And also types.</p> <p>(It&rsquo;s even less surprising if you&rsquo;ve run across the Curry-Howard isomorphism, which relates logical proofs and propositions to exhibiting an instance of a type – <a href="http://cacm.acm.org/magazines/2015/12/194626-propositions-as-types/fulltext">Propositions as Types</a> – or, more broadly, the notion of <a href="https://existentialtype.wordpress.com/2011/03/27/the-holy-trinity/">computational Trinitarianism.</a> There are some deep connections here, and we should wring them for every last ounce of help they can give us in crafting correct and elegant software.)</p> <h2 id="use-proofs-and-tests">Use Proofs AND Tests</h2> <p>This time, it&rsquo;s not gonna be me saying it, though.</p> <p>Really, you should use both tests and proofs, not just one or just the other:</p> <blockquote> <p>This also reinforces the general idea that testing and proving are synergistic activities, and gives us hope that a virtuous cycle between testing and proving can be achieved in a theorem prover. (<a href="http://prosecco.gforge.inria.fr/personal/hritcu/publications/foundational-pbt.pdf" title="Foundational Property-Based Testing">Zoe Paraskevopoulou et al., &ldquo;Foundational Property-Based Testing&rdquo;, 2015</a>)</p> </blockquote> <p>If you don&rsquo;t, you&rsquo;re going to screw up. In small ways often, basically just tripping over your feet, but sometimes in big ways, where no-one can see how to bail you out:</p> <blockquote> <p>Second, tests complement proofs. We encountered five papers in which explicitly claimed theorems are false as stated. […] In every case, though, rudimentary testing discovered errors missed with pencil-and-paper proofs.</p> <p>Indeed, we claim that tests complement even machine-checked proofs. As one example, two of the POPLmark solutions that contain proofs of type soundness use call-by-name beta in violation of the specification (Crary and Gacek, personal communication). We believe unit testing would quickly reveal this error.</p> <p>Even better, one can sometimes test propositions that cannot be validated via proof. […] Testing also removes another obstacle to proof, the requirement that we first state the proposition of interest. Due to its exploratory nature, testing can inadvertently falsify unstated but desired propositions, e.g., that threads block without busy waiting (section 4.4). This is especially true for system-level and randomized testing. To some degree, the same is true of proving, but testing seems to be more effective at covering a broad space of system behaviors. (<a href="http://eecs.northwestern.edu/~robby/lightweight-metatheory/">Casey Klein et al., &ldquo;Run Your Research: On the Effectiveness of Lightweight Mechanization&rdquo;, 2012</a>)</p> </blockquote> <h2 id="use-all-the-tools">Use ALL The Tools</h2> <p>We don&rsquo;t have to choose just tests.</p> <p>We don&rsquo;t have to choose just types.</p> <p>We don&rsquo;t have to choose just proofs.</p> <p>We have an abundance of tools waiting for us to take them up and apply them to our problems. It&rsquo;s simple and reassuring to reject a whole class of them out of hand; if we pick just one, perhaps we can convince ourselves of our expertise. And you can indeed get quite far with just one. But if you can stomach your own ignorance, you might find you can get even farther by striving to master all these many disciplines.</p> <h2 id="that-said-tests-are-a-really-mature-technology">That Said, Tests Are a Really Mature Technology</h2> <p>Automated testing stopped being rocket science at least a decade ago. If you do nothing else, at least write some automated tests.</p> <p>Humans suck at repeating mechanical tasks, we&rsquo;re bad at documenting them, bad at following them, we get bored really easily, and we&rsquo;re really slow. Be virtuously lazy and sic a computer on your testing, for everyone&rsquo;s sake.</p> <p>If you&rsquo;re going to pick just one of these, pick automated testing, and work it for all it&rsquo;s worth. (Quite a lot, honestly!)</p> Here's to iOS apps in F# https://jeremywsherman.com/blog/2016/04/06/heres-to-ios-apps-in-f/ Wed, 06 Apr 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/04/06/heres-to-ios-apps-in-f/ <p>At Build 2016, Microsoft announced that <a href="http://arstechnica.com/information-technology/2016/03/xamarin-now-free-in-visual-studio/">Xamarin is free with all versions of Visual Studio, and the Xamarin SDK will be open-sourced.</a></p> <p>My first thought was: iOS apps in F#? Lemme at it!</p> <p>At Build 2016, Microsoft announced that <a href="http://arstechnica.com/information-technology/2016/03/xamarin-now-free-in-visual-studio/">Xamarin is free with all versions of Visual Studio, and the Xamarin SDK will be open-sourced.</a></p> <p>My first thought was: iOS apps in F#? Lemme at it!</p> <h2 id="why-f">Why F#?</h2> <p>Swift is going through growing pains, and it&rsquo;s still substantially a statement-oriented language. It&rsquo;s supposed to be very comfortable if you come from a blocks-and-braces background, with seamless interop with C and Obj-C, and it&rsquo;s executed on that wonderfully. If you were hoping for a more truly functional language, though, it&rsquo;s kind of a downer; its gig at the bleeding edge seems more generic programming than functional programming (&ldquo;C++ done right&rdquo;).</p> <p>F# benefits more directly from the long evolution of ML languages. It&rsquo;s been public longer, and it&rsquo;s got a good pedigree: Microsoft have done interesting things with all their languages over the past decade, and F# is no exception.</p> <p>It might be &ldquo;grass is greener&rdquo;, but I&rsquo;d like to take that for a spin and kick its tires, without having to up and move to a completely different target platform.</p> <h3 id="interested">Interested?</h3> <p><a href="http://fsharpforfunandprofit.com/">F# for Fun and Profit</a> is really great for learning about F# and why it’s good stuff. It’s organized less like a blog and more like a collection of series of instructional content.</p> <p><a href="http://fsharpforfunandprofit.com/why-use-fsharp/">Here’s their one-page summary of “why use F#”.</a> Most of the bullets apply as well for Swift as for F#, but the core difference of expression-orientation rather than statement-orientation – not called out there – matters quite a bit in how easy it is to compose expressions and extract expressions as independent functions. (The <a href="https://en.wikibooks.org/wiki/F_Sharp_Programming/Computation_Expressions#Syntax_Sugar">workflow/computation expression sugar</a> is quite nice, as well, and of course <a href="http://fsharpforfunandprofit.com/series/computation-expressions.html">F# for Fun and Profit has a series teaching it in detail.</a>)</p> <p>If you prefer to listen rather than read, then check out:</p> <ul> <li><a href="https://www.functionalgeekery.com/functional-geekery-episode-21-andrea-magnorsky/">An interview with Andrea Magnorsky</a></li> <li><a href="https://www.functionalgeekery.com/episode-35-rachel-reese/">An interview with Rachel Reese</a></li> </ul> <p>Both of these also get into <a href="http://fsharpforfunandprofit.com/posts/low-risk-ways-to-use-fsharp-at-work">introducing F# in the workplace,</a> if that&rsquo;s something you&rsquo;re motivated to tackle.</p> <h2 id="why-xamarin">Why Xamarin?</h2> <p>Xamarin was $$$ before, but this drops the price of adoption for me (and anyone inheriting my codebase) significantly. Adoption costs matter!</p> <p>As a bonus, if I can be like, &ldquo;Hey, you&rsquo;ll get an iOS app, and you&rsquo;ll get a pile of platform-independent code you can point at Windows, Mac, or Android afterwards,&rdquo; that seems like a win all around.</p> <p>But really, I want a shot to use an expression-oriented language today as my main language.</p> <h2 id="warning-untested-speculation">Warning: Untested Speculation</h2> <p>I haven&rsquo;t actually tried to do this yet. It might go down in flames in practice when I try to get everything lined up and working together; lots of things sound good in outline that fail in implementation.</p> <p>If I throw an afternoon to it some time in future, I&rsquo;ll check back in with an experience report then.</p> XCTestExpectation Gotchas https://jeremywsherman.com/blog/2016/03/19/xctestexpectation-gotchas/ Sat, 19 Mar 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/03/19/xctestexpectation-gotchas/ <p><code>XCTestExpectation</code> simplifies testing callback-style code, but some of its design choices make tests using it fragile unless they&rsquo;re mitigated:</p> <ul> <li>It explodes if everything works right but later than you expected.</li> <li>It explodes if everything works right more than once.</li> </ul> <p>This article presents two concrete mitigations:</p> <ul> <li>Use weak references to ensure the expectation dies before it can cause you trouble.</li> <li>Use a different promise API to do your waiting.</li> </ul> <p><code>XCTestExpectation</code> simplifies testing callback-style code, but some of its design choices make tests using it fragile unless they&rsquo;re mitigated:</p> <ul> <li>It explodes if everything works right but later than you expected.</li> <li>It explodes if everything works right more than once.</li> </ul> <p>This article presents two concrete mitigations:</p> <ul> <li>Use weak references to ensure the expectation dies before it can cause you trouble.</li> <li>Use a different promise API to do your waiting.</li> </ul> <p><strong>Contents:</strong></p> <!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> <ul> <li><a href="#a-quick-review">A Quick Review</a></li> <li><a href="#example-well-call-you">Example: We&rsquo;ll Call You</a> <ul> <li><a href="#kaboom-missing-the-window">Kaboom: Missing the Window</a></li> <li><a href="#kaboom-calling-twice">Kaboom: Calling Twice</a></li> </ul> </li> <li><a href="#whats-wrong">What&rsquo;s Wrong?</a></li> <li><a href="#avoiding-these-assertions">Avoiding These Assertions</a> <ul> <li><a href="#let-the-expectation-die-with-the-test">Let the Expectation Die With the Test</a></li> <li><a href="#kill-the-expectation-proactively">Kill the Expectation Proactively</a></li> <li><a href="#use-a-different-promise-api">Use a Different Promise API</a></li> </ul> </li> <li><a href="#summary">Summary</a></li> </ul> <!-- END doctoc generated TOC please keep comment here to allow auto update --> <h2 id="a-quick-reviewa-ida-quick-reviewa">A Quick Review<a id="a-quick-review"></a></h2> <p><code>XCTestExpectation</code> is the tool Apple&rsquo;s unit testing framework XCTest provides for coping with asynchronous APIs.</p> <p>It&rsquo;s a promise/future with one purpose: to answer the question, &ldquo;did it get filled in time?&rdquo;</p> <p>To use it, you ask the test case to create one or more:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">let</span> promise = expectationWithDescription(<span style="color:#e6db74">&#34;it&#39;ll happen, trust me&#34;</span>) </code></pre></div><p>wait a configurable amount of time for every outstanding expectation to get filled:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift">waitForExpectationsWithTimeout(maxWaitSeconds, handler: <span style="color:#66d9ef">nil</span>) </code></pre></div><p>and log a test failure if time runs out before that happens:</p> <blockquote> <p>Asynchronous wait failed: Exceeded timeout of 1 seconds, with unfulfilled expectations: &ldquo;it&rsquo;ll happen, trust me&rdquo;.</p> </blockquote> <p>It would have succeeded if it had been filled in time:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift">promise?.fulfill() </code></pre></div><h2 id="example-well-call-youa-idexample-well-call-youa">Example: We&rsquo;ll Call You<a id="example-well-call-you"></a></h2> <p>You can&rsquo;t use the XCTest framework from a Playground (<a href="rdar://problem/17839045">rdar://problem/17839045</a>), so you&rsquo;ll need to throw this in a full-blown project:</p> <p><strong><a href="https://github.com/jeremy-w/XCTestExpectationGotchas">Get the code from GitHub</a></strong></p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">LateCallback</span>: XCTestCase { <span style="color:#66d9ef">let</span> callBackDelay: NSTimeInterval = <span style="color:#ae81ff">2</span> <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testNotWaitingLongEnough</span>() { <span style="color:#66d9ef">let</span> promiseToCallBack = expectationWithDescription(<span style="color:#e6db74">&#34;calls back&#34;</span>) after(seconds: callBackDelay) { () -&gt; Void <span style="color:#66d9ef">in</span> print(<span style="color:#e6db74">&#34;I knew you&#39;d call!&#34;</span>) promiseToCallBack.fulfill() } waitForExpectationsWithTimeout(callBackDelay <span style="color:#f92672">/</span> <span style="color:#ae81ff">2</span>) { error <span style="color:#66d9ef">in</span> print(<span style="color:#e6db74">&#34;Aww, we timed out: </span><span style="color:#e6db74">\(</span>error<span style="color:#e6db74">)</span><span style="color:#e6db74">&#34;</span>) } } } </code></pre></div><p>Go ahead and run this. Everything works fine – for now:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Test Suite &#39;All tests&#39; started at 2016-03-19 21:56:49.223 Test Suite &#39;Tests.xctest&#39; started at 2016-03-19 21:56:49.225 Test Suite &#39;LateCallback&#39; started at 2016-03-19 21:56:49.225 Test Case &#39;-[Tests.LateCallback testNotWaitingLongEnough]&#39; started. Aww, we timed out: Optional(Error Domain=com.apple.XCTestErrorDomain Code=0 &#34;The operation couldn’t be completed. (com.apple.XCTestErrorDomain error 0.)&#34;) /Users/jeremy/Github/XCTestExpectationGotchas/Tests/LateCallback.swift:26: error: -[Tests.LateCallback testNotWaitingLongEnough] : Asynchronous wait failed: Exceeded timeout of 1 seconds, with unfulfilled expectations: &#34;calls back&#34;. Test Case &#39;-[Tests.LateCallback testNotWaitingLongEnough]&#39; failed (2.247 seconds). Test Suite &#39;LateCallback&#39; failed at 2016-03-19 21:56:51.473. Executed 1 test, with 1 failure (0 unexpected) in 2.247 (2.248) seconds Test Suite &#39;Tests.xctest&#39; failed at 2016-03-19 21:56:51.474. Executed 1 test, with 1 failure (0 unexpected) in 2.247 (2.249) seconds Test Suite &#39;All tests&#39; failed at 2016-03-19 21:56:51.474. Executed 1 test, with 1 failure (0 unexpected) in 2.247 (2.251) seconds Test session log: /var/folders/63/np5g0d5j54x1s0z12rf41wxm0000gp/T/com.apple.dt.XCTest-status/Session-2016-03-19_21:56:45-vfvzhb.log Program ended with exit code: 1 </code></pre></div><p>Test suite kicks off, everything runs, the test fails due to a timeout while waiting for the expectation to be met, and the process exits. This is how <code>XCTestExpectation</code> is supposed to work.</p> <h3 id="kaboom-missing-the-windowa-idkaboom-missing-the-windowa">Kaboom: Missing the Window<a id="kaboom-missing-the-window"></a></h3> <p>We only ran the one test, though. Let&rsquo;s say you have more tests to run after this one.</p> <p>We can fake this out by adding a new test method whose name sorts alphabetically after our <code>testNotWaitingLongEnough</code> test that runs the runloop for a bit before exiting.</p> <p>Conveniently enough, XCTest happens to run tests in alphabetical order, so the test runner will run our first test, then run this second one, then exit.</p> <p>Here&rsquo;s our new test method:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"> <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testZzz</span>() { print(<span style="color:#e6db74">&#34;Let&#39;s just wait a while…&#34;</span>) <span style="color:#66d9ef">let</span> tillAfterCallBack = callBackDelay spin(forSeconds: tillAfterCallBack) print(<span style="color:#e6db74">&#34;Yawn, that was boring.&#34;</span>) } </code></pre></div><p>Let&rsquo;s see what happens (or <a href="#late-summary">you can skip to the summary</a>):</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Test Suite &#39;All tests&#39; started at 2016-03-19 22:19:31.796 Test Suite &#39;Tests.xctest&#39; started at 2016-03-19 22:19:31.798 Test Suite &#39;LateCallback&#39; started at 2016-03-19 22:19:31.798 Test Case &#39;-[Tests.LateCallback testNotWaitingLongEnough]&#39; started. Aww, we timed out: Optional(Error Domain=com.apple.XCTestErrorDomain Code=0 &#34;The operation couldn’t be completed. (com.apple.XCTestErrorDomain error 0.)&#34;) /Users/jeremy/Github/XCTestExpectationGotchas/Tests/LateCallback.swift:16: error: -[Tests.LateCallback testNotWaitingLongEnough] : Asynchronous wait failed: Exceeded timeout of 1 seconds, with unfulfilled expectations: &#34;calls back&#34;. Test Case &#39;-[Tests.LateCallback testNotWaitingLongEnough]&#39; failed (2.202 seconds). Test Case &#39;-[Tests.LateCallback testZzz]&#39; started. Let&#39;s just wait a while… 2.0: finished waiting I knew you&#39;d call! 2016-03-19 22:19:34.001 xctest[92369:96447173] *** Assertion failure in -[XCTestExpectation fulfill], /Library/Caches/com.apple.xbs/Sources/XCTest/XCTest-9530/XCTestFramework/Classes/XCTestCase+AsynchronousTesting.m:451 2016-03-19 22:19:34.002 xctest[92369:96447173] *** Terminating app due to uncaught exception &#39;NSInternalInconsistencyException&#39;, reason: &#39;API violation - called -[XCTestExpectation fulfill] after the wait context has ended for calls back.&#39; *** First throw call stack: ( 0 CoreFoundation 0x00007fff897ec03c __exceptionPreprocess + 172 1 libobjc.A.dylib 0x00007fff8674276e objc_exception_throw + 43 2 CoreFoundation 0x00007fff897ebe1a +[NSException raise:format:arguments:] + 106 3 Foundation 0x00007fff8b98b99b -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 195 4 XCTest 0x000000010006f149 -[XCTestExpectation fulfill] + 302 5 Tests 0x00000001006858ab _TFFC5Tests12LateCallback24testNotWaitingLongEnoughFS0_FT_T_U_FT_T_ + 203 6 Tests 0x0000000100685c4f _TFF5Tests5afterFT7secondsSd4callFT_T__T_U_FT_T_ + 367 7 Tests 0x0000000100685de7 _TTRXFo__dT__XFdCb__dT__ + 39 8 libdispatch.dylib 0x00007fff8301f700 _dispatch_call_block_and_release + 12 9 libdispatch.dylib 0x00007fff8301be73 _dispatch_client_callout + 8 10 libdispatch.dylib 0x00007fff8302d6a0 _dispatch_after_timer_callback + 77 11 libdispatch.dylib 0x00007fff8301be73 _dispatch_client_callout + 8 12 libdispatch.dylib 0x00007fff830284e6 _dispatch_source_latch_and_call + 721 13 libdispatch.dylib 0x00007fff8302093b _dispatch_source_invoke + 412 14 libdispatch.dylib 0x00007fff8302c5aa _dispatch_main_queue_callback_4CF + 416 15 CoreFoundation 0x00007fff8973f3f9 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9 16 CoreFoundation 0x00007fff896fa68f __CFRunLoopRun + 2159 17 CoreFoundation 0x00007fff896f9bd8 CFRunLoopRunSpecific + 296 18 Foundation 0x00007fff8b953b29 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 278 19 Foundation 0x00007fff8b971d9e -[NSRunLoop(NSRunLoop) runUntilDate:] + 108 20 Tests 0x0000000100685262 _TF5Tests4spinFT10forSecondsSd_T_ + 162 21 Tests 0x000000010068510f _TFC5Tests12LateCallback7testZzzfS0_FT_T_ + 207 22 Tests 0x00000001006852a2 _TToFC5Tests12LateCallback7testZzzfS0_FT_T_ + 34 23 CoreFoundation 0x00007fff896c37bc __invoking___ + 140 24 CoreFoundation 0x00007fff896c3612 -[NSInvocation invoke] + 290 25 XCTest 0x0000000100022598 __24-[XCTestCase invokeTest]_block_invoke_2 + 159 26 XCTest 0x000000010005602e -[XCTestContext performInScope:] + 184 27 XCTest 0x00000001000224e8 -[XCTestCase invokeTest] + 169 28 XCTest 0x0000000100022983 -[XCTestCase performTest:] + 443 29 XCTest 0x0000000100020654 -[XCTestSuite performTest:] + 377 30 XCTest 0x0000000100020654 -[XCTestSuite performTest:] + 377 31 XCTest 0x0000000100020654 -[XCTestSuite performTest:] + 377 32 XCTest 0x000000010000e892 __25-[XCTestDriver _runSuite]_block_invoke + 51 33 XCTest 0x0000000100033a1b -[XCTestObservationCenter _observeTestExecutionForBlock:] + 611 34 XCTest 0x000000010000e7db -[XCTestDriver _runSuite] + 408 35 XCTest 0x000000010000f38a -[XCTestDriver _checkForTestManager] + 696 36 XCTest 0x000000010005729f _XCTestMain + 628 37 xctest 0x0000000100001dca xctest + 7626 38 libdyld.dylib 0x00007fff8b25f5c9 start + 1 ) libc++abi.dylib: terminating with uncaught exception of type NSException (lldb) </code></pre></div><p><a id="late-summary"></a> And now we&rsquo;re sitting at the debugger. Oof, that smarts.</p> <p>Take a look at what&rsquo;s going on in that backtrace:</p> <ul> <li> <p>Our <code>Zzz</code> test is hanging out running the runloop.</p> </li> <li> <p>The <code>after(seconds:call:)</code> finishes waiting and calls its callback.</p> </li> <li> <p>The callback fulfills an expectation belonging to the (already finished, already failed) first test</p> </li> <li> <p>This trips a &ldquo;you&rsquo;re holding it wrong&rdquo; assertion in the test framework:</p> <blockquote> <p>Terminating app due to uncaught exception &lsquo;NSInternalInconsistencyException&rsquo;, reason: &lsquo;API violation - called <code>-[XCTestExpectation fulfill]</code> after the wait context has ended for calls back.&rsquo;</p> </blockquote> </li> </ul> <p>You might run up against this in practice when writing integration tests against a live, but not always quick to respond, backend service.</p> <h3 id="kaboom-calling-twicea-idkaboom-calling-twicea">Kaboom: Calling Twice<a id="kaboom-calling-twice"></a></h3> <p>That&rsquo;s not the only way things can go wrong.</p> <p>What happens if our callback has at-least-once rather than exactly-once behavior, and happens to call back twice?</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">DoubleCallback</span>: XCTestCase { <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testDoubleTheFulfillment</span>() { <span style="color:#66d9ef">let</span> promiseToCallBack = expectationWithDescription(<span style="color:#e6db74">&#34;calls back&#34;</span>) <span style="color:#66d9ef">let</span> callBackDelay: NSTimeInterval = <span style="color:#ae81ff">1</span> twiceAfter(seconds: callBackDelay) { print(<span style="color:#e6db74">&#34;i hear you calling me&#34;</span>) promiseToCallBack.fulfill() } <span style="color:#66d9ef">let</span> afterCallBack = <span style="color:#ae81ff">2</span> <span style="color:#f92672">*</span> callBackDelay waitForExpectationsWithTimeout(afterCallBack, handler: <span style="color:#66d9ef">nil</span>) } } </code></pre></div><p>This is what happens (or <a href="#twice-summary">skip to the summary</a>)</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Test Suite &#39;Selected tests&#39; started at 2016-03-19 22:38:09.451 Test Suite &#39;DoubleCallback&#39; started at 2016-03-19 22:38:09.452 Test Case &#39;-[Tests.DoubleCallback testDoubleTheFulfillment]&#39; started. 1.0: finished waiting now once i hear you calling me now twice i hear you calling me 2016-03-19 22:38:10.567 xctest[93147:96490281] *** Assertion failure in -[XCTestExpectation fulfill], /Library/Caches/com.apple.xbs/Sources/XCTest/XCTest-9530/XCTestFramework/Classes/XCTestCase+AsynchronousTesting.m:450 2016-03-19 22:38:10.568 xctest[93147:96490281] *** Terminating app due to uncaught exception &#39;NSInternalInconsistencyException&#39;, reason: &#39;API violation - multiple calls made to -[XCTestExpectation fulfill] for calls back.&#39; *** First throw call stack: ( 0 CoreFoundation 0x00007fff897ec03c __exceptionPreprocess + 172 1 libobjc.A.dylib 0x00007fff8674276e objc_exception_throw + 43 2 CoreFoundation 0x00007fff897ebe1a +[NSException raise:format:arguments:] + 106 3 Foundation 0x00007fff8b98b99b -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 195 4 XCTest 0x000000010006f0bb -[XCTestExpectation fulfill] + 160 5 Tests 0x0000000100795c6b _TFFC5Tests14DoubleCallback24testDoubleTheFulfillmentFS0_FT_T_U_FT_T_ + 203 6 Tests 0x0000000100795e05 _TFF5Tests10twiceAfterFT7secondsSd4callFT_T__T_U_FT_T_ + 389 7 Tests 0x0000000100794eff _TFF5Tests5afterFT7secondsSd4callFT_T__T_U_FT_T_ + 367 8 Tests 0x0000000100795097 _TTRXFo__dT__XFdCb__dT__ + 39 9 libdispatch.dylib 0x00007fff8301f700 _dispatch_call_block_and_release + 12 10 libdispatch.dylib 0x00007fff8301be73 _dispatch_client_callout + 8 11 libdispatch.dylib 0x00007fff8302d6a0 _dispatch_after_timer_callback + 77 12 libdispatch.dylib 0x00007fff8301be73 _dispatch_client_callout + 8 13 libdispatch.dylib 0x00007fff830284e6 _dispatch_source_latch_and_call + 721 14 libdispatch.dylib 0x00007fff8302093b _dispatch_source_invoke + 412 15 libdispatch.dylib 0x00007fff8302c5aa _dispatch_main_queue_callback_4CF + 416 16 CoreFoundation 0x00007fff8973f3f9 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9 17 CoreFoundation 0x00007fff896fa68f __CFRunLoopRun + 2159 18 CoreFoundation 0x00007fff896f9bd8 CFRunLoopRunSpecific + 296 19 Foundation 0x00007fff8b953b29 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 278 20 XCTest 0x000000010006e6e8 -[XCTestCase(AsynchronousTesting) waitForExpectationsWithTimeout:handler:] + 1083 21 Tests 0x00000001007954d6 _TFC5Tests14DoubleCallback24testDoubleTheFulfillmentfS0_FT_T_ + 614 22 Tests 0x0000000100795722 _TToFC5Tests14DoubleCallback24testDoubleTheFulfillmentfS0_FT_T_ + 34 23 CoreFoundation 0x00007fff896c37bc __invoking___ + 140 24 CoreFoundation 0x00007fff896c3612 -[NSInvocation invoke] + 290 25 XCTest 0x0000000100022598 __24-[XCTestCase invokeTest]_block_invoke_2 + 159 26 XCTest 0x000000010005602e -[XCTestContext performInScope:] + 184 27 XCTest 0x00000001000224e8 -[XCTestCase invokeTest] + 169 28 XCTest 0x0000000100022983 -[XCTestCase performTest:] + 443 29 XCTest 0x0000000100020654 -[XCTestSuite performTest:] + 377 30 XCTest 0x0000000100020654 -[XCTestSuite performTest:] + 377 31 XCTest 0x000000010000e892 __25-[XCTestDriver _runSuite]_block_invoke + 51 32 XCTest 0x0000000100033a1b -[XCTestObservationCenter _observeTestExecutionForBlock:] + 611 33 XCTest 0x000000010000e7db -[XCTestDriver _runSuite] + 408 34 XCTest 0x000000010000f38a -[XCTestDriver _checkForTestManager] + 696 35 XCTest 0x000000010005729f _XCTestMain + 628 36 xctest 0x0000000100001dca xctest + 7626 37 libdyld.dylib 0x00007fff8b25f5c9 start + 1 ) libc++abi.dylib: terminating with uncaught exception of type NSException (lldb) </code></pre></div><p><a id="twice-summary"></a> We trip yet another assertion in <code>XCTest</code>:</p> <blockquote> <p>Terminating app due to uncaught exception &lsquo;NSInternalInconsistencyException&rsquo;, reason: &lsquo;API violation - multiple calls made to <code>-[XCTestExpectation fulfill]</code> for calls back.&rsquo;</p> </blockquote> <p>This probably does indicate an actual error in the code calling the callback much of the time, but if it doesn&rsquo;t, you&rsquo;ll want to know about and be able to dodge this assertion, too.</p> <h2 id="whats-wronga-idwhats-wronga">What&rsquo;s Wrong?<a id="whats-wrong"></a></h2> <p>This double-callback scenario calls back twice in succession. But if there were a delay between the first and second call back, and the test runner happened to exit during that delay, you&rsquo;d get a successful test run rather than crashing every time.</p> <p>With a delay between callbacks, you&rsquo;d only trip the assertion when other tests kept the test runner process running long enough.</p> <p>This situation parallels that of the too-late callback: no problems till appear till something else runs out the clock.</p> <p>This is tricky:</p> <ul> <li>You won&rsquo;t ever trip them when you&rsquo;re banging away at whatever the latest test you&rsquo;re working on is, because a test runner running just that async test will exit as soon as the wait-timer runs out, before the too-late/second callback can occur.</li> <li>You might not even trip them when you run your whole test suite at first, because they might be the last test in the run or the tests that follow don&rsquo;t run for long enough.</li> </ul> <p>This is also obnoxious to run into: When an assertion trips, it bombs the entire test process. (Unwrapping an implicitly unwrapped optional to find a <code>nil</code> has the same effect.)</p> <p>These assertions aren&rsquo;t test failures that would allow testing to continue; instead, XCTest treats as programmer error:</p> <ul> <li>Fulfilling a promise after its test has already finished</li> <li>Filling an already-filled promise</li> </ul> <p>To be fair, these cases are called out in the documentation for <code>XCTestExpectation.fulfill()</code>:</p> <blockquote> <p>Call <code>-fulfill</code> to mark an expectation as having been met. It&rsquo;s an error to call <code>-fulfill</code> on an expectation that has already been fulfilled or when the test case that vended the expectation has already completed.</p> </blockquote> <p>but the documentation isn&rsquo;t explicit that &ldquo;it&rsquo;s an error&rdquo; translates to &ldquo;and it will bomb your whole test process&rdquo;.</p> <h2 id="avoiding-these-assertionsa-idavoiding-these-assertionsa">Avoiding These Assertions<a id="avoiding-these-assertions"></a></h2> <p>In both cases, the problem is that we&rsquo;re calling <code>fulfill</code> when we shouldn&rsquo;t. So let&rsquo;s not do that.</p> <h3 id="let-the-expectation-die-with-the-testa-idlet-the-expectation-die-with-the-testa">Let the Expectation Die With the Test<a id="let-the-expectation-die-with-the-test"></a></h3> <p><code>XCTest</code> actually hangs on to the expectations it creates so it can collect them during the wait call.</p> <p>Our test method doesn&rsquo;t need yet another strong reference to the expectation; if we instead work with a weak reference in our callback closure, the expectation will die with our test, rather than lingering for us to trip over after the test has completed, and we&rsquo;ll have turned our callback into a no-op.</p> <p>First, neuter the time-bombed <code>testNotWaitingLongEnough</code> by prefixing its name with an <code>x</code> so it won&rsquo;t get picked up by the test runner any more:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"> class LateCallback: XCTestCase { let callBackDelay: NSTimeInterval = 2 <span style="color:#f92672">- func testNotWaitingLongEnough() { </span><span style="color:#f92672"></span><span style="color:#a6e22e">+ func xtestNotWaitingLongEnough() { </span><span style="color:#a6e22e"></span> let promiseToCallBack = expectationWithDescription(&#34;calls back&#34;) after(seconds: callBackDelay) { () -&gt; Void in </code></pre></div><p>Now clone it, but this time, use a weak reference to the expectation:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"> <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testPreparedForNotWaitingLongEnough</span>() { <span style="color:#66d9ef">weak</span> <span style="color:#66d9ef">var</span> promiseToCallBack = expectationWithDescription(<span style="color:#e6db74">&#34;calls back&#34;</span>) after(seconds: callBackDelay) { () -&gt; Void <span style="color:#66d9ef">in</span> <span style="color:#66d9ef">guard</span> <span style="color:#66d9ef">let</span> promise = promiseToCallBack <span style="color:#66d9ef">else</span> { print(<span style="color:#e6db74">&#34;too late, buckaroo&#34;</span>) <span style="color:#66d9ef">return</span> } print(<span style="color:#e6db74">&#34;I knew you&#39;d call!&#34;</span>) promise.fulfill() } waitForExpectationsWithTimeout(callBackDelay <span style="color:#f92672">/</span> <span style="color:#ae81ff">2</span>) { error <span style="color:#66d9ef">in</span> print(<span style="color:#e6db74">&#34;Aww, we timed out: </span><span style="color:#e6db74">\(</span>error<span style="color:#e6db74">)</span><span style="color:#e6db74">&#34;</span>) } } </code></pre></div><p>Run the <code>LateCallback</code> suite again, and the logs now look like (or <a href="#summary-dodged">skip to the summary</a>):</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Test Suite &#39;Selected tests&#39; started at 2016-03-19 23:19:19.980 Test Suite &#39;LateCallback&#39; started at 2016-03-19 23:19:19.981 Test Case &#39;-[Tests.LateCallback testPreparedForNotWaitingLongEnough]&#39; started. Aww, we timed out: Optional(Error Domain=com.apple.XCTestErrorDomain Code=0 &#34;The operation couldn’t be completed. (com.apple.XCTestErrorDomain error 0.)&#34;) /Users/jeremy/Github/XCTestExpectationGotchas/Tests/LateCallback.swift:34: error: -[Tests.LateCallback testPreparedForNotWaitingLongEnough] : Asynchronous wait failed: Exceeded timeout of 1 seconds, with unfulfilled expectations: &#34;calls back&#34;. Test Case &#39;-[Tests.LateCallback testPreparedForNotWaitingLongEnough]&#39; failed (1.945 seconds). Test Case &#39;-[Tests.LateCallback testZzz]&#39; started. Let&#39;s just wait a while… 2.0: finished waiting too late, buckaroo 2.0: all done here Yawn, that was boring. Test Case &#39;-[Tests.LateCallback testZzz]&#39; passed (2.004 seconds). Test Suite &#39;LateCallback&#39; failed at 2016-03-19 23:19:23.932. Executed 2 tests, with 1 failure (0 unexpected) in 3.950 (3.951) seconds Test session log: /var/folders/63/np5g0d5j54x1s0z12rf41wxm0000gp/T/com.apple.dt.XCTest-status/Session-2016-03-19_23:19:16-QZf0lq.log Test Suite &#39;Selected tests&#39; failed at 2016-03-19 23:19:23.933. Executed 2 tests, with 1 failure (0 unexpected) in 3.950 (3.953) seconds Program ended with exit code: 1 </code></pre></div><p><a id="summary-dodged"></a> Our <code>testZzz</code> runs to completion and passes, and the test process exits on its own terms reporting the one failure.</p> <p>The late callback still happened, but by that time, <code>promiseToCallBack</code> had been zeroed, so we never called <code>fulfill()</code>.</p> <p>Assertion: Dodged!</p> <h3 id="kill-the-expectation-proactivelya-idkill-the-expectation-proactivelya">Kill the Expectation Proactively<a id="kill-the-expectation-proactively"></a></h3> <p>What about the double-callback case? We can use the same trick, only this time, we&rsquo;ll want to proactively annihilate the expectation:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"> <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testSafelyDoubleTheFulfillment</span>() { <span style="color:#66d9ef">weak</span> <span style="color:#66d9ef">var</span> promiseToCallBack = expectationWithDescription(<span style="color:#e6db74">&#34;calls back&#34;</span>) <span style="color:#66d9ef">let</span> callBackDelay: NSTimeInterval = <span style="color:#ae81ff">1</span> twiceAfter(seconds: callBackDelay) { <span style="color:#66d9ef">guard</span> <span style="color:#66d9ef">let</span> promise = promiseToCallBack <span style="color:#66d9ef">else</span> { print(<span style="color:#e6db74">&#34;once was enough, thanks!&#34;</span>) <span style="color:#66d9ef">return</span> } promise.fulfill() promiseToCallBack = <span style="color:#66d9ef">nil</span> } <span style="color:#66d9ef">let</span> afterCallBack = <span style="color:#ae81ff">2</span> <span style="color:#f92672">*</span> callBackDelay waitForExpectationsWithTimeout(afterCallBack, handler: <span style="color:#66d9ef">nil</span>) } </code></pre></div><p>With the unsafe test neutered via the prefix-<code>x</code> trick, running the test class gives (or <a href="#summary-dead-jim">skip to the summary</a>):</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Test Suite &#39;Selected tests&#39; started at 2016-03-19 23:22:56.356 Test Suite &#39;DoubleCallback&#39; started at 2016-03-19 23:22:56.357 Test Case &#39;-[Tests.DoubleCallback testSafelyDoubleTheFulfillment]&#39; started. 1.0: finished waiting Test session log: /var/folders/63/np5g0d5j54x1s0z12rf41wxm0000gp/T/com.apple.dt.XCTest-status/Session-2016-03-19_23:22:51-14ywpS.log now once i hear you calling me now twice once was enough, thanks! wasn&#39;t that nice? 1.0: all done here Test Case &#39;-[Tests.DoubleCallback testSafelyDoubleTheFulfillment]&#39; passed (1.099 seconds). Test Suite &#39;DoubleCallback&#39; passed at 2016-03-19 23:22:57.457. Executed 1 test, with 0 failures (0 unexpected) in 1.099 (1.100) seconds Test Suite &#39;Selected tests&#39; passed at 2016-03-19 23:22:57.458. Executed 1 test, with 0 failures (0 unexpected) in 1.099 (1.102) seconds Program ended with exit code: 0 </code></pre></div><p><a id="summary-dead-jim"></a> Since we explicitly set the promise to <code>nil</code>, we only end up fulfilling it once. No harm, no foul.</p> <h3 id="use-a-different-promise-apia-iduse-a-different-promise-apia">Use a Different Promise API<a id="use-a-different-promise-api"></a></h3> <p>If you&rsquo;ve got an API written in terms of a promise/future library already, such as <a href="https://github.com/bignerdranch/Deferred/">Deferred</a>, then there&rsquo;s no need to use XCTest&rsquo;s promises:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">BringYourOwnPromises</span>: XCTestCase { <span style="color:#66d9ef">let</span> anyDelay: NSTimeInterval = <span style="color:#ae81ff">1</span> <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testGettingAPony</span>() { <span style="color:#66d9ef">let</span> futurePony = giveMeAPony(after: anyDelay) <span style="color:#66d9ef">let</span> longEnough = anyDelay <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span> <span style="color:#66d9ef">guard</span> <span style="color:#66d9ef">let</span> pony = futurePony.wait(.Interval(longEnough)) <span style="color:#66d9ef">else</span> { XCTFail(<span style="color:#e6db74">&#34;no pony ;_;&#34;</span>) <span style="color:#66d9ef">return</span> } print(<span style="color:#e6db74">&#34;we got a pony! </span><span style="color:#e6db74">\(</span>pony<span style="color:#e6db74">)</span><span style="color:#e6db74">&#34;</span>) } <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testWhenImpatientNoPonyForYou</span>() { <span style="color:#66d9ef">let</span> futurePony = giveMeAPony(after: anyDelay) <span style="color:#66d9ef">guard</span> <span style="color:#66d9ef">let</span> pony = futurePony.wait(.Now) <span style="color:#66d9ef">else</span> { print(<span style="color:#e6db74">&#34;no patience, no pony&#34;</span>) <span style="color:#66d9ef">return</span> } XCTFail(<span style="color:#e6db74">&#34;we got a pony???! </span><span style="color:#e6db74">\(</span>pony<span style="color:#e6db74">)</span><span style="color:#e6db74">&#34;</span>) } <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testZzzDoesNotCrash</span>() { spin(forSeconds: <span style="color:#ae81ff">2</span> <span style="color:#f92672">*</span> anyDelay) } } </code></pre></div><h2 id="summarya-idsummarya">Summary<a id="summary"></a></h2> <ul> <li>Always assign your expectations to a weak reference, and then bail in your callback if it&rsquo;s nil.</li> <li>In the rare case where you expect your callback to be triggered more than once, you can avoid fulfilling by annihilating your weak reference after fulfilling it and then ignoring future calls. <ul> <li>More likely, you know how many times you should be called, and you&rsquo;ll want to fulfill the promise only on the last call. But the workaround is there if you need it.</li> </ul> </li> <li>If you&rsquo;re already working with a promise-based API, you can skip <code>XCTestExpectation</code> and use whatever wait-and-see API is provided by that promise instead of XCTest&rsquo;s own. <ul> <li>This has the added advantage of linearizing your test code by eliminating the need to handle the delivered value in the closure (or manually shuttle it out to assert against after the XCTest wait has finished).</li> </ul> </li> </ul> Embedded Content Contains Swift https://jeremywsherman.com/blog/2016/03/06/embedded-content-contains-swift/ Sun, 06 Mar 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/03/06/embedded-content-contains-swift/ <p>If you&rsquo;re developing a QuickLook plugin using Swift, make sure you flip on the <code>EMBEDDED_CONTENT_CONTAINS_SWIFT</code> build setting for the target, otherwise bundle loading will fail in a spectacularly unhelpful way.</p> <p>If you&rsquo;re developing a QuickLook plugin using Swift, make sure you flip on the <code>EMBEDDED_CONTENT_CONTAINS_SWIFT</code> build setting for the target, otherwise bundle loading will fail in a spectacularly unhelpful way.</p> <h2 id="creating-a-mixed-language-quicklook-plugin">Creating a Mixed-Language QuickLook Plugin</h2> <p>Recently I decided to add a QuickLook plugin to my <a href="https://github.com/jeremy-w/ImageSlicer">ImageSlicer</a> utility app.</p> <p>The default QuickLook plugin template stamps out an entirely C plugin. Changing the thumbnail/preview template files to have a <code>.m</code> suffix put us back in Obj-C land, but getting to Swift land takes a couple more steps.</p> <p>Not to worry: Add a new Swift file to the target, and Xcode will offer to make bridging easy-peasy for you. Give it the go-ahead, and you should be good to go, right?</p> <p>I add the main model and view classes from my app project to the QuickLook target, wire stuff up to load the document and render the view, and everything compiles and links all happy-like. Let&rsquo;s test this thing!</p> <h2 id="gatekeeper">Gatekeeper?</h2> <p>I fire up qlmanage, point it at my generator and a <code>.slicedimage</code> document, and I see That Error:</p> <blockquote> <p>The bundle “QuickLookSlicedImage” couldn’t be loaded because it is damaged or missing necessary resources.</p> </blockquote> <p>I&rsquo;ve seen this error way too many times when I grab an older app bundle off the Internet. Every time before, &ldquo;damaged or missing necessary resources&rdquo; has been code for &ldquo;no-one signed this app bundle&rdquo;.</p> <p>I&rsquo;m asking the system to execute code, so, sure, that kind of makes sense?</p> <p>I hare off looking at using <code>spctl</code> to whitelist my bundle, successfully whitelist it with <code>spctl --add --label JWSDev path/to/QuickLookSlicedImage.qlgenerator</code>, and <code>spctl --assess</code> is OK with it.</p> <p>Let&rsquo;s try again.</p> <h2 id="not-gatekeeper">Not Gatekeeper</h2> <p>I see the same error. Hrm. What if it really is missing something? Now I want to see the smoking gun.</p> <p>After sufficient rooting around, I eventually work through to where it loads the bundle, then the plugin, then finally to where the real business happens: <code>dlopen</code>.</p> <p>After the call to <code>dlopen</code>, the <code>CFBundle</code> machinery checked for success with <code>dlerror</code>, and that gave me an actually informative error message (which I&rsquo;ve abbreviated and hard-wrapped for readability):</p> <pre><code>(lldb) x/s $rax 0x100576819: &quot;dlopen(LONG_PATH/QuickLookSlicedImage, 262): Library not loaded: @rpath/libswiftAppKit.dylib\n Referenced from: LONG_PATH/QuickLookSlicedImage\n Reason: image not found&quot; </code></pre><p>Yup, missing Swift dylibs.</p> <h2 id="embedded_content_contains_swift">EMBEDDED_CONTENT_CONTAINS_SWIFT</h2> <p>The fix is to tell Xcode to copy all the Swift dylibs the built product needs into its bundle using the build setting <code>EMBEDDED_CONTENT_CONTAINS_SWIFT=YES</code>.</p> <p>(The other fix is to ensure <code>qlmanage</code> is actually running the generator you&rsquo;re building now, not the generator embedded in the copy of your app you built an hour or two ago that still has the missing-dylib issue. Oops.)</p> <h2 id="take-away">Take-Away</h2> <p>The take-away is this:</p> <ul> <li>When Xcode offers to add a Swift–Obj-C bridging header for you,</li> <li>Then that means the target was not previously configured for Swift,</li> <li>And you should probably ensure that <code>EMBEDDED_CONTENT_CONTAINS_SWIFT=YES</code> gets set for the target.</li> </ul> <p>The &ldquo;probably&rdquo; is there because, if you&rsquo;re baking it into an app bundle that&rsquo;s already embedding the Swift dylibs, you could probably mess with the rpath to get it to share those rather than having Yet Another Copy of the Swift support dylibs in your app bundle.</p> <p>But that&rsquo;ll be a pain, and disk space is cheap, so you&rsquo;ll probably still want to just flip on <code>EMBEDDED_CONTENT_CONTAINS_SWIFT=YES</code>.</p> Review: SE-0026: Abstract classes and methods https://jeremywsherman.com/blog/2016/02/29/review-se-0026-abstract-classes-and-methods/ Mon, 29 Feb 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/02/29/review-se-0026-abstract-classes-and-methods/ <p>This is a review of <a href="https://github.com/apple/swift-evolution/blob/master/proposals/0026-abstract-classes-and-methods.md">SE-0026 &ldquo;Abstract classes and methods&rdquo;.</a></p> <p>I am <strong>against</strong> the acceptance of this proposal:</p> <ul> <li>It lacks a clear problem.</li> <li>The leap from a nebulous problem to abstract classes as the solution is a <em>non sequitur.</em></li> <li>Its arguments are insufficient to justify the complication it would add to Swift, which is contrary the simplification and clarification aims of the Swift community.</li> </ul> <p>The contrast is sharpened by comparison to the Python Enhancement Proposal that accompanied the introduction of abstract base classes into Python. The present proposal fails to provide a correspondingly thoughtful rationale.</p> <p>This is a review of <a href="https://github.com/apple/swift-evolution/blob/master/proposals/0026-abstract-classes-and-methods.md">SE-0026 &ldquo;Abstract classes and methods&rdquo;.</a></p> <p>I am <strong>against</strong> the acceptance of this proposal:</p> <ul> <li>It lacks a clear problem.</li> <li>The leap from a nebulous problem to abstract classes as the solution is a <em>non sequitur.</em></li> <li>Its arguments are insufficient to justify the complication it would add to Swift, which is contrary the simplification and clarification aims of the Swift community.</li> </ul> <p>The contrast is sharpened by comparison to the Python Enhancement Proposal that accompanied the introduction of abstract base classes into Python. The present proposal fails to provide a correspondingly thoughtful rationale.</p> <h2 id="no-clear-problem">No Clear Problem</h2> <p>The proposal itself does little to define a practical problem, and less to explain how abstract classes solve this problem better than alternatives. It feels like a solution in want of a problem, which is the opposite of a considered addition to the language.</p> <p>As best I can determine, the primary problem introduced is that of wanting to have abstract properties. The example given is better resolved by providing the <code>url</code> as a constructor argument, as <a href="https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011207.html">noted by Stephen Celis.</a> Further, the immediate solution appears to be to argue in favor of <a href="https://en.wikipedia.org/wiki/Uniform_access_principle">uniform access</a> as found in Self and Eiffel, not abstract base classes, which compound non-uniform access by a further serving of complexity.</p> <p>Another problem mentioned is lack of easy delegation of implementation in the context of protocols; providing a simple way to proxy calls to another object would present a promising and useful avenue for resolving this problem that would also compose more generally with the rest of the language. <code>NSProxy</code> has always been somewhat awkward in this regard; perhaps we can do better in Swift?</p> <h2 id="no-clear-significance">No Clear Significance</h2> <p>Without a clear problem to address, it becomes difficult to evaluate the significance of the problem.</p> <p>Ultimately, it&rsquo;s unclear precisely what the problem under consideration is, unless the problem is stated simply as, &ldquo;Swift doesn&rsquo;t have abstract base classes.&rdquo; If that is the true problem considered to address, then it seems especially insignificant; Swift also lacks good support for relational programming à la mini-kanren, but a difference does not a problem make.</p> <p>If we focus on &ldquo;no abstract classes&rdquo; as the problem, then the problem appears insignificant: Smalltalk and Objective-C have both made do without formal support for abstract classes. Objective-C went so far as to remove <code>subclassResponsibility</code> from the common language vocabulary, which eliminated all inbuilt support for abstract classes. Never have I heard either Smalltalker or Obj-C hacker end up despondent and cursing over the lack of built-in abstract class support in these languages.</p> <h2 id="compared-to-pythons-rationale-for-adding-abstract-classes">Compared to Python&rsquo;s Rationale for Adding Abstract Classes</h2> <p>It is interesting to consider the motivation for adding abstract base class support to Python as explained in <a href="https://www.python.org/dev/peps/pep-3119/">PEP 3119</a>.</p> <p>In Python&rsquo;s case, the decision was motivated by the desire for a reliable means to test particularly for some shared quality of a group of objects - basically, a reliable <code>respondsToSelector:</code> or <code>isKindOfClass:</code> that allows detecting this quality without incidental risk of false positives or negatives (<a href="https://www.python.org/dev/peps/pep-3119/#rationale">&ldquo;Rationale&rdquo;</a>).</p> <p>As a result, Python adopted abstract base classes as an alternative to interfaces (<a href="https://www.python.org/dev/peps/pep-3119/#abcs-vs-interfaces">&ldquo;ABCs vs. Interfaces&rdquo;</a>). But Swift already has interfaces in the form of protocols; this answers the need that motivated the addition of abstract base classes to Swift.</p> <p>Because we cannot borrow the rationale used for adding abstract base classes to Python, and the document before us spends its effort explaining abstract base classes rather than the problem they would solve, it remains for those arguing for the added formal complexity of abstract base classes to motivate their addition in the context of Swift. The current proposal is manifestly lacking in this regard.</p> <h2 id="out-of-alignment-with-swift">Out of Alignment with Swift</h2> <p>Adding abstract class support to Swift seems unprincipled. I cannot see what problem would be solved, and Swift is working towards considered language growth, and even better, language contraction, at this point in time. Adding abstract base classes would feel like nodding to feature agglutination by cargo cult, not the careful evolution we aspire to.</p> <h2 id="effort">Effort</h2> <p>I read the article and then looked at the arguments in favor of supporting abstract base classes in Python for comparison. I would love to see a rationale as tailored to Swift and to real problems as PEP 3119 was to Python and its programmers&rsquo; problems! In Python&rsquo;s case, &ldquo;[m]uch of the thinking that went into the proposal [was] not about the specific mechanism of ABCs, as contrasted with Interfaces or Generic Functions (GFs), but about clarifying philosophical issues[…].&rdquo; This sort of laborious semantic work is a necessary accompaniment to any significant proposed changes to an object system, and that thought is unfortunately not apparent in this proposal.</p> <p><em>This article was originally <a href="https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011383.html">posted to swift-evolution</a> on 28 February 2016.</em></p> Go Versions and the Open-Closed Principle https://jeremywsherman.com/blog/2016/02/24/go-versions-and-the-open-closed-principle/ Wed, 24 Feb 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/02/24/go-versions-and-the-open-closed-principle/ People aren&rsquo;t happy about Go&rsquo;s approach to managing software versions: aren&rsquo;t different API versions supposed to live at different import paths in Go? This works great if you have a proprietary codebase, are using a monorepo, and don&rsquo;t support the sharing culture of open source. And, it doesn&rsquo;t address the issue of minor or patch versions. Hello, Open-Closed Principle The funny thing is that Go&rsquo;s official version management approach is effectively a strict reading of the open-closed principle as applied to libraries rather than classes. <p>People <a href="http://engineeredweb.com/blog/2016/path-go-pkg-mgmt/">aren&rsquo;t happy</a> about Go&rsquo;s approach to managing software versions:</p> <blockquote> <p>aren&rsquo;t different API versions supposed to live at different import paths in Go? This works great if you have a proprietary codebase, are using a monorepo, and don&rsquo;t support the sharing culture of open source. And, it doesn&rsquo;t address the issue of minor or patch versions.</p> </blockquote> <h2 id="hello-open-closed-principle">Hello, Open-Closed Principle</h2> <p>The funny thing is that Go&rsquo;s official version management approach is effectively a strict reading of the open-closed principle as applied to libraries rather than classes.</p> <p>The &ldquo;fork it and rename it&rdquo; approach was actually the way the principle was originally introduced for classes.</p> <p>You want to change how a class works?</p> <p>Fine, subclass it and make your changes.</p> <p>Dependents can adopt <code>MyVeryOwnFooV35</code> at their convenience, rather than you just stomping on the one and only <code>MyVeryOwnFoo</code> class in the project.</p> <h2 id="but-thats-crazy-talk">But That&rsquo;s Crazy Talk!</h2> <p>Yeah, it didn&rsquo;t much catch on in object-oriented programming, either, in spite of being enshrined in the SOLID acronym.</p> <p>Apparently Gophers think it&rsquo;s equally crazy for libraries (ibid):</p> <blockquote> <p>Can you imagine that every time a library needs to increment a major version it needs to create a new repo on GitHub? Yeah, no one does that. The path for major API version is a Go thing. It&rsquo;s not intuitive. Someone had to tell me. And, many Go developers just don&rsquo;t do it. If they did there would be no reason for gopkg.in.</p> </blockquote> <h2 id="people-actually-do-that">People Actually Do That</h2> <p>I can imagine it, and people actually do it. Check out the <a href="http://rfc.zeromq.org/spec:22#toc8">Creating Stable Releases section</a> of the <a href="http://rfc.zeromq.org/spec:22">Collective Code Construction Contract.</a> This is the social contract that governs development of ZeroMQ, amongst a few other projects.</p> <p>Every time they want to make a stable version, they shard off a new repo for that version, with its own steward.</p> <p>Thus, every time ZeroMQ needs to increment a major, or minor, or patch version, they need to fork a new repo. Mainline development continues on the main repo, and the stable release repo gets its own repo, its own maintenance patches, and its own name in the form of repo URL.</p> <h2 id="why-dont-we-do-that-in-oop">Why Don&rsquo;t We Do That in OOP?</h2> <p>I think we don&rsquo;t do this for OOP precisely because we find ourselves in the monorepo scenario that let Google avoid introducing package management into go. Most object-oriented projects live in one repo, so we can readily coordinate changes across the codebase: we don&rsquo;t need to fork a new subclass, because we can just update all callers to play ball with the new version.</p> Housekeeping https://jeremywsherman.com/blog/2016/01/07/housekeeping/ Thu, 07 Jan 2016 00:00:00 +0000 https://jeremywsherman.com/blog/2016/01/07/housekeeping/ This post automagically appeared on the site thanks to a post-receive hook. Every prior post was written, compiled, and rsynced from my laptop. No more! Now: I can post from my phone using Working Copy. Later: I&rsquo;ll work out handling for microposts, so I can send those here and sync to ADN after. Later still: Figuring out a good workflow for link blogging from my phone. For ADN, I&rsquo;ve got a very slick workflow using @dasdom&lsquo;s wonderful Jupp sharing extension, and I&rsquo;ve seen how any hiccups in that workflow significantly reduce how much I share my reading with ADN. <p>This post automagically appeared on the site thanks to a post-receive hook. Every prior post was written, compiled, and rsynced from my laptop. No more!</p> <p>Now: I can post from my phone using <a href="http://workingcopyapp.com/">Working Copy</a>.</p> <p>Later: I&rsquo;ll work out handling for microposts, so I can send those here and sync to <a href="https://alpha.app.net/jws">ADN</a> after.</p> <p>Later still: Figuring out a good workflow for link blogging from my phone. For ADN, I&rsquo;ve got a very slick workflow using <a href="https://alpha.app.net/dasdom">@dasdom</a>&lsquo;s wonderful <a href="https://itunes.apple.com/us/app/jupp-share-extension-for-app.net/id909926740?mt=8">Jupp</a> sharing extension, and I&rsquo;ve seen how any hiccups in that workflow significantly reduce how much I share my reading with ADN.</p> Do more of less https://jeremywsherman.com/blog/2015/11/21/do-more-of-less/ Sat, 21 Nov 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/11/21/do-more-of-less/ <p>The most valuable lesson of Kanban is to <em>limit work in progress.</em> At the personal level, this jives with studies showing that humans suck at multitasking.</p> <p>This is a hard lesson for me: My life is littered with the detritus of works begun, works planned, resources squirreled away against a future that rarely comes back to them.</p> <p>The most valuable lesson of Kanban is to <em>limit work in progress.</em> At the personal level, this jives with studies showing that humans suck at multitasking.</p> <p>This is a hard lesson for me: My life is littered with the detritus of works begun, works planned, resources squirreled away against a future that rarely comes back to them.</p> <p>A messy desk or hard-drive becomes an oppressive labyrinth: one sits down for a purpose, only to have all one&rsquo;s energies dispersed for nothing down the forking hallways of might-have-beens.</p> <p>Facing this honestly is terrifying: It means admitting one might never pursue that avenue, never chase that morning&rsquo;s dream. It means confronting the brief spark that is human life; no, that&rsquo;s not what frightens: rather, the dark that follows, as persistence of vision gives way to vanishing memory, and one&rsquo;s name and deeds fade forgotten.</p> <center>***</center> <p>Please forgive my messy desk; <a href="https://youtu.be/rEf5ygPIesE?t=1m44s">the dark is waiting,</a> and I would but close my eyes a while longer.</p> Agile https://jeremywsherman.com/blog/2015/11/13/agile/ Fri, 13 Nov 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/11/13/agile/ <p>I take agile as rejecting the notion of estimation as having value. In the event you have a deadline, the best you can hope for is to deliver as much working software as you can before that deadline. Dithering over what’s going to fall on which side of the deadline is time better spent delivering a feature and winnowing out low-value crap that came along with the high-value bits of your original ideas so you don’t waste time implementing the dross.</p> <p>I take agile as rejecting the notion of estimation as having value. In the event you have a deadline, the best you can hope for is to deliver as much working software as you can before that deadline. Dithering over what’s going to fall on which side of the deadline is time better spent delivering a feature and winnowing out low-value crap that came along with the high-value bits of your original ideas so you don’t waste time implementing the dross.</p> <p>If I do need to estimate, I reject the silly notion of giving a ludicrously precise single value, and instead give a more honest pair of (estimate, complexity), where complexity is rated on a scale from “I’ve done this a hundred times” to “nobody in the world has ever done this” (<a href="http://lizkeogh.com/2013/07/21/estimating-complexity/)">http://lizkeogh.com/2013/07/21/estimating-complexity/)</a>. And potentially give a range instead of a single value, though some sort of highly-skewed normal distribution might be better.</p> <p>I similarly reject the notion of “backlog” as wasting time counting your chickens before they’ve hatched (<a href="http://ronjeffries.com/articles/015-10/the-backlog/article.html)">http://ronjeffries.com/articles/015-10/the-backlog/article.html)</a>. The only project artifact that matters is the running code you have at the end of a sprint. Everything else is BS; use whatever support tools you need, but don’t confuse your list of dreams with what you have in hand now. If you can’t ship what you have now? You’re probably setting yourself up for serious pain and suffering when the budget suddenly runs out, or your main dev gets moved to another project, or quits, or…</p> <p>I joke that the product champion should keep a stack of might-wants in a Trello board with “a pony” at the bottom. No-one ever gets everything they want implemented in software; most of those inchoate wants are a mix of some good and valuable ideas and a bunch of lossy cruft that would be a waste of time to do anyway. (Also: diminishing returns.) We forget human finity at our, and our projects&rsquo;, peril.</p> <p>Many project management artifacts and behaviors seem smoke and mirrors rituals attempted in the vain hope of preventing the dread manifestation of Learning and consequent Change. Unfortunately, no matter how many ways we invent to scream, &ldquo;COME NOT IN THAT FORM!&rdquo; into the unknown, we remain saddled with imperfect knowledge, or alternatively, blessed with the joy of learning ever more and new things about our domain of interest. Dispensing with these distractors – from the primitive state of both today&rsquo;s tools and the discipline of software development as a whole, and more generally from our own cloud of unknowing – is terrifying, but addressing reality head-on frees you to make the best use of the precious time and limited tools you have.</p> Updating Plex on Synology NAS https://jeremywsherman.com/blog/2015/11/08/updating-plex-on-synology-nas/ Sun, 08 Nov 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/11/08/updating-plex-on-synology-nas/ <p>My family has been using ChromeCast to send YouTube videos to the TV. While flipping through the ChromeCast app on my phone, I noticed Plex integrates with ChromeCast. Funny enough, Synology also ships a Plex server package. How hard could this be?</p> <p>My family has been using ChromeCast to send YouTube videos to the TV. While flipping through the ChromeCast app on my phone, I noticed Plex integrates with ChromeCast. Funny enough, Synology also ships a Plex server package. How hard could this be?</p> <h2 id="chromecast-easy-come-easy-stow">ChromeCast: Easy Come, Easy Stow</h2> <p>I have a very curious toddler.</p> <p>The ChromeCast is easy to hook up and break down as needed when you want to use it, and there&rsquo;s not much to break. It was a simple and immediate solution to make the TV usable again without needing to run any cables or install a shelf outside toddler reach.</p> <p>Before this, we spent several months with the TV completely unplugged after we dismantled the entertainment center and mounted the TV to the wall as part of making our living room child-resistant.</p> <p>(Child-proof vs child-resistant is like waterproof/water-resistant: Nature finds a way, and all we can do is try to hold out – in this case, till an adult notices a curious and ill-omened silence.)</p> <h2 id="plex-client-is-picky-synology-plex-is-old">Plex Client Is Picky; Synology Plex Is Old</h2> <p>Setting up the app on my phone was fairly easy. I needed to create a new login, which, yawn, but 1Password is with me.</p> <p>Installing the package on my NAS was also one-click.</p> <p>Getting them talking to each other was a bloody mess. Plex client is very aggressive about not working with older versions of Plex server, which meant that right now, it didn&rsquo;t work at all with the version packaged by Synology.</p> <h2 id="installing-a-manual-package">Installing a Manual Package</h2> <p>Luckily, Plex packages Plex Server for Synology (and several other flavors of NAS) themselves.</p> <p><a href="https://support.plex.tv/hc/en-us/articles/205165858">Their instructions</a> only cover a small part of the install process, though. What papered over the gap for me is <a href="http://www.codeexplained.net/update-plex-media-server-for-synology/">this article</a>. (That article has pictures, unlike this one.)</p> <p>Here are the steps I followed:</p> <ul> <li>Check your processor type in the Synology Control Panel</li> <li>Download the package for that processor type from <a href="https://plex.tv/downloads">Plex Downloads</a></li> <li>Download the Plex package signing key linked from <a href="https://support.plex.tv/hc/en-us/articles/205165858">here</a></li> <li>Verify the md5sum they give you for what comfort that might give. (md5sum? Really?)</li> <li>Open the Synology package center and hit the Settings button: <ul> <li>On the General pane, widen your trust from just Synology to Synology plus trusted publishers.</li> <li>On the Certificate pane, upload the PlexSign.key you just grabbed. <ul> <li>Now Plex is a trusted publisher.</li> </ul> </li> </ul> </li> <li>Exit the Settings modal and hit the Manual Install button. <ul> <li>Select the Plex package you downloaded.</li> <li>Wait for it to upload, then OK the install.</li> </ul> </li> </ul> <p>Unfortunately, Plex doesn&rsquo;t seem to publish a stream of updates, just individual packages, so when the client yells at you again about the server being too old, you get to repeat most of this dance.</p> <h2 id="gotchas">Gotchas</h2> <h3 id="potential-synology-vs-plex-package-differences">Potential: Synology vs Plex package differences</h3> <p>I read some tales of issues with swiching between the two packages. I know that installing the Synology version first and then the Plex version worked fine for me. Your mileage may vary. If it breaks, you might need to pop the hood and ssh in to see what&rsquo;s gone wrong.</p> <p>I didn&rsquo;t encounter any issues myself, so I wouldn&rsquo;t worry about this unless you run into it.</p> <h3 id="derp-manual-packages-must-be-uploaded-from-the-client">Derp: Manual packages must be uploaded from the client</h3> <p>If you&rsquo;re like me, you might think like this:</p> <ul> <li>I will need to install this package to the NAS.</li> <li>The package file needs to end up on the NAS eventually.</li> <li>Downloading the package file directly to the NAS using Download Station will save transfer time.</li> </ul> <p>You&rsquo;re right in theory, and wrong in practice, because the manual install flow only lets you select a local file to upload. Let me say this again: <strong>There is no way to point the manual install wizard at a package that&rsquo;s already downloaded to the NAS.</strong> You have to upload it from your local machine directly to the manual package installer.</p> <p>The fun end result of this is that, if you downloaded the file to the NAS to begin with, you now get to download the file from your NAS so you can upload it back for the manual package install flow.</p> <p>The steps I listed above skip this time-wasting cleverness.</p> <h2 id="conclusion">Conclusion</h2> <p>Plex would be a lot easier to use if they&rsquo;d do a better job of preserving client–server compatibility across versions.</p> <p>If you&rsquo;ve been looking for an excuse to wander into manual Synology package installation, though, you&rsquo;ve come to the right product.</p> An Idris metaprogramming "hello world" https://jeremywsherman.com/blog/2015/10/10/an-idris-metaprogramming-hello-world/ Sat, 10 Oct 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/10/10/an-idris-metaprogramming-hello-world/ <p><a href="http://idris-lang.org/">Idris</a> is a programming language with dependent types. Like any civilized language, it has metaprogramming support; its REPL even supports interactively crafting metaprograms by employing tactic functions.</p> <p>Let&rsquo;s walk through a small example that covers the bases of interactive metaprogramming.</p> <p><small>*This article is an elaboration of <a href="https://gist.github.com/jeremy-w/80953ebe9f68064e78eb">a gist I posted a few days ago.</a>*</small></p> <p><a href="http://idris-lang.org/">Idris</a> is a programming language with dependent types. Like any civilized language, it has metaprogramming support; its REPL even supports interactively crafting metaprograms by employing tactic functions.</p> <p>Let&rsquo;s walk through a small example that covers the bases of interactive metaprogramming.</p> <p><small>*This article is an elaboration of <a href="https://gist.github.com/jeremy-w/80953ebe9f68064e78eb">a gist I posted a few days ago.</a>*</small></p> <h2 id="a-simple-function">A Simple Function</h2> <p>Here&rsquo;s a simple program that take a list of strings and returns either the first string in the list or the empty string when the list is empty:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-idris" data-lang="idris"><span style="color:#66d9ef">module</span> Head <span style="color:#a6e22e">firstStringOrEmpty</span> <span style="color:#f92672">:</span> <span style="color:#66d9ef">List</span> <span style="color:#66d9ef">String</span> <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">String</span> firstStringOrEmpty strings <span style="color:#f92672">=</span> <span style="color:#66d9ef">case</span> strings <span style="color:#66d9ef">of</span> <span style="color:#66d9ef">Nil</span> <span style="color:#f92672">=&gt;</span> <span style="color:#e6db74">&#34;&#34;</span> <span style="color:#a6e22e">string</span> <span style="color:#f92672">::</span> <span style="color:#66d9ef">_</span> <span style="color:#f92672">=&gt;</span> string </code></pre></div><p>This function pattern matches on the <code>strings</code> argument to see if it is the empty list <code>Nil</code> or a list with at least one item:</p> <ul> <li>In the <code>Nil</code> case, it evaluates to the empty string <code>&quot;&quot;</code>.</li> <li>In the non-<code>Nil</code> case, the first item is bound to the new name <code>string</code>. It ignores the rest of the list by binding it to the nameless name <code>_</code>. It then evaluates to that first item, <code>string</code>.</li> </ul> <p>Here&rsquo;s what it looks like in practice by firing up the REPL:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">&gt; idris Head.idr ____ __ _ / _/___/ /____(_)____ / // __ / ___/ / ___/ Version 0.9.19 _/ // /_/ / / / (__ ) http://www.idris-lang.org/ /___/\__,_/_/ /_/____/ Type :? for help Idris is free software with ABSOLUTELY NO WARRANTY. For details type :warranty. *Head&gt; firstStringOrEmpty [] &#34;&#34; : String *Head&gt; firstStringOrEmpty [&#34;a&#34;] &#34;a&#34; : String *Head&gt; firstStringOrEmpty [&#34;a&#34;, &#34;b&#34;, &#34;c&#34;] &#34;a&#34; : String </code></pre></div><p>(In future examples, I&rsquo;ll be omitting the banner with <code>--nobanner</code>, but I figure it&rsquo;s useful to see what version I was working with.)</p> <h2 id="poking-a-hole-in-it">Poking a Hole in It</h2> <p>Now that we know what we want the function to look like, let&rsquo;s poke a hole in it!</p> <p>Idris represents holes as names with a leading question mark, like <code>?hole</code>. Let&rsquo;s replace the <code>Nil</code> case with a hole:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-idris" data-lang="idris"><span style="color:#66d9ef">module</span> HoleyHead <span style="color:#a6e22e">firstStringOrEmpty</span> <span style="color:#f92672">:</span> <span style="color:#66d9ef">List</span> <span style="color:#66d9ef">String</span> <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">String</span> firstStringOrEmpty strings <span style="color:#f92672">=</span> <span style="color:#66d9ef">case</span> strings <span style="color:#66d9ef">of</span> <span style="color:#66d9ef">Nil</span> <span style="color:#f92672">=&gt;</span> <span style="color:#f92672">?</span>holeThatWasTheEmptyString <span style="color:#a6e22e">string</span> <span style="color:#f92672">::</span> <span style="color:#66d9ef">_</span> <span style="color:#f92672">=&gt;</span> string </code></pre></div><p>Now let&rsquo;s see what the REPL has to say:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">&gt; idris --nobanner HoleyHead.idr Type checking ./HoleyHead.idr Holes: HoleyHead.holeThatWasTheEmptyString </code></pre></div><p>Would you look at that! It found our hole.</p> <h2 id="filling-the-hole-with-elaborator-reflection">Filling the Hole with Elaborator Reflection</h2> <p>We know what we want this to look like, since we started with a working program and then poked a hole in it, but let&rsquo;s humor this example and pretend we need Idris&rsquo;s help to figure out how to fill it in.</p> <h3 id="import-elaboration-scripts">Import Elaboration Scripts</h3> <p>Start by pulling in the elaboration scripts provided by the system in the module <code>Language.Reflection.Elab</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">*HoleyHead&gt; :module Language.Reflection.Elab Holes: HoleyHead.holeThatWasTheEmptyString *HoleyHead *Language/Reflection/Elab&gt; </code></pre></div><p>Notice how the prompt changed to reflect the added module.</p> <h3 id="survey-your-tools">Survey Your Tools</h3> <p>The good stuff in <code>Language.Reflection.Elab</code> lives in the <code>Tactics</code> module. Take a look at the fun stuff in there:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">*HoleyHead *Language/Reflection/Elab&gt; :browse Language.Reflection.Elab.Tactics Namespaces: Names: addInstance : TTName -&gt; TTName -&gt; Elab () apply : Raw -&gt; List (Bool, Int) -&gt; Elab (List (TTName, TTName)) attack : Elab () check : Raw -&gt; Elab (TT, TT) claim : TTName -&gt; Raw -&gt; Elab () … [list abbreviated to save you scrolling] … solve : Elab () sourceLocation : Elab () unfocus : TTName -&gt; Elab () whnf : TT -&gt; Elab TT </code></pre></div><p>These are the functions we&rsquo;ll be using to fill the hole. Each of these has a docstring with more info about it. For example, here&rsquo;s one we&rsquo;ll use in a bit:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">*HoleyHead *Language/Reflection/Elab&gt; :doc intro&#39; Language.Reflection.Elab.Tactics.intro&#39; : Elab () Introduce a lambda binding around the current hole and focus on the body, using the name provided by the type of the hole. The function is Total Holes: HoleyHead.holeThatWasTheEmptyString </code></pre></div><p>Translated into plainer English, this says that <code>intro'</code> transforms a hole like <code>?hole</code> into a function <code>_ =&gt; ?newHole</code> and points you at filling in the new hole.</p> <p>(That line at the end of the docstring about &ldquo;The function is Total&rdquo; represents the judgment of Idris&rsquo;s totality checker. A total function is one that terminates for all its inputs, which means it eventually returns a value for each and every one and cannot run forever.)</p> <blockquote> <h3 id="gotcha">Gotcha</h3> <p>We&rsquo;re taking a peek now, because unfortunately, we won&rsquo;t be able to run <code>:browse</code> while we&rsquo;re working on filling the hole, short of firing up a new REPL. We will be able to run <code>:doc</code> still, so we can just scroll back to this list as needed.</p> </blockquote> <h3 id="enter-the-elaborator-shell">Enter the Elaborator Shell</h3> <p>Enter the interactive elaborator shell by issuing the <code>:elab &lt;HOLE&gt;</code> command:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">*HoleyHead *Language/Reflection/Elab&gt; :elab holeThatWasTheEmptyString ---------- Goal: ---------- {hole0} : List String -&gt; String -HoleyHead.holeThatWasTheEmptyString&gt; </code></pre></div><p>When you first enter the elaboration shell, it prints the current elaboration state, with the goal – the term you&rsquo;re trying to fill using elaboration scripts – shown below the line, and any other remaining goals shown above the line. You can review this state at any time by issuing the <code>:state</code> command. To get a list of available commands, run <code>:help</code>.</p> <p>The elaborator shell has its own prompt, which tells you the hole you&rsquo;re working on filling.</p> <h3 id="reach-your-goals">Reach Your Goals</h3> <p>Our current goal is <code>hole0</code>, and it has type <code>List String -&gt; String</code>. This reflects that we currently have in scope – and available to our hole-filling efforts – a <code>List String</code>, in the form of the <code>strings</code> argument to our function, and that the type of whatever we compute using this context must end up as <code>String</code>.</p> <p>In light of this <code>List String -&gt; String</code> phrasing, what we had filling this space before was a constant function <code>\strings =&gt; &quot;&quot;</code> that ignored its arguments and always returned the empty string.</p> <p>Since we don&rsquo;t actually care about the name of the function argument, let&rsquo;s use that <code>intro'</code> tactic function we saw earlier:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :state ---------- Goal: ---------- {hole0} : List String -&gt; String -HoleyHead.holeThatWasTheEmptyString&gt; intro&#39; ---------- Assumptions: ---------- ---------- Goal: ---------- {hole0} : String -HoleyHead.holeThatWasTheEmptyString&gt; </code></pre></div><h3 id="filling-a-hole">Filling a Hole</h3> <p>Cool, now all we need to do is provide a <code>String</code> to fill the hole with. What could we use to do that? Well…</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :doc fill Language.Reflection.Elab.Tactics.fill : Raw -&gt; Elab () Place a term into a hole, unifying its type The function is Total </code></pre></div><p>That <code>fill</code> function looks like exactly what we need if we&rsquo;re to put <code>&quot;&quot;</code> back in that hole. Maybe we can just <code>fill &quot;&quot;</code> directly?</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; fill &#34;&#34; (input):1:6:When checking an application of function Language.Reflection.Elab.Tactics.fill: Type mismatch between String (Type of &#34;&#34;) and Raw (Expected type) </code></pre></div><p>Welp, that didn&rsquo;t quite do it.</p> <h3 id="chasing-down-constructors">Chasing Down Constructors</h3> <p>We need a way to turn the string <code>&quot;&quot;</code> into a <code>Raw</code> value. What&rsquo;s a <code>Raw</code>?</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :type Raw FFI_C.Raw : Type -&gt; Type Language.Reflection.Raw : Type </code></pre></div><p>Looks like there are a couple kinds of <code>Raw</code> around, but one is a <code>Raw someType</code> and the other is just a <code>Raw</code>. Plus, we&rsquo;re not doing FFI, so let&rsquo;s bet on the <code>Language.Reflection.Raw</code>. Let&rsquo;s pull the docs on that:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :doc Language.Reflection.Raw Data type Language.Reflection.Raw : Type Raw terms without types Constructors: Var : TTName -&gt; Raw Variables, global or local RBind : TTName -&gt; Binder Raw -&gt; Raw -&gt; Raw Bind a variable RApp : Raw -&gt; Raw -&gt; Raw Application RType : Raw The type of types RUType : Universe -&gt; Raw RForce : Raw -&gt; Raw RConstant : Const -&gt; Raw Embed a constant </code></pre></div><p><code>&quot;&quot;</code> is a string constant, so <code>RConstant</code> sounds like what we want.</p> <p>In order to create an <code>RConstant</code>, we need to pass it a <code>Const</code>. What&rsquo;s a <code>Const</code>?</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :doc Const Data type Language.Reflection.Const : Type Primitive constants Constructors: I : Int -&gt; Const BI : Integer -&gt; Const Fl : Double -&gt; Const Ch : Char -&gt; Const Str : String -&gt; Const B8 : Bits8 -&gt; Const B16 : Bits16 -&gt; Const B32 : Bits32 -&gt; Const B64 : Bits64 -&gt; Const AType : ArithTy -&gt; Const StrType : Const VoidType : Const Forgot : Const WorldType : Const TheWorld : Const </code></pre></div><p>Bingo! <code>Str &quot;&quot;</code> will give us a <code>Const</code>, and then we can pass that to <code>RConstant</code> to get a <code>Raw</code>, and then we can pass that to <code>fill</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; fill (RConstant (Str &#34;&#34;)) ---------- Assumptions: ---------- ---------- Goal: ---------- {hole0} : String =?= &#34;&#34; </code></pre></div><h3 id="solve-et-coagulum">Solve et Coagulum</h3> <p>We&rsquo;re sitting with one last goal, to show that our guess <code>&quot;&quot;</code> really is a <code>String</code>. If you take a peek at the docs, you&rsquo;d see that <code>solve</code> does just this:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :doc solve Language.Reflection.Elab.Tactics.solve : Elab () Substitute a guess into a hole. The function is Total </code></pre></div><p>(You could also find <code>solve</code> by running <code>:search Elab ()</code> – all our tactics return a value in <code>Elab</code>, we have no inputs to provide, and we just want to trigger the side effect of finishing this up – and looking through the few lines of output to find a likely candidate. But peeping at the docs is faster.)</p> <p>So let&rsquo;s use it:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; solve holeThatWasTheEmptyString: No more goals. </code></pre></div><p>And victoriously exit the elaborator shell:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :qed Proof completed! HoleyHead.holeThatWasTheEmptyString = %runElab (do intro&#39; fill (RConstant (Str &#34;&#34;)) solve) *HoleyHead *Language/Reflection/Elab&gt; </code></pre></div><p>Let&rsquo;s give it a go:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">*HoleyHead *Language/Reflection/Elab&gt; firstStringOrEmpty [] &#34;&#34; : String </code></pre></div><p>It worked!</p> <h2 id="paste-in-the-script">Paste In the Script</h2> <p>When we left the shell with <code>:qed</code>, it wrote out a function definition:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">-HoleyHead.holeThatWasTheEmptyString&gt; :qed Proof completed! HoleyHead.holeThatWasTheEmptyString = %runElab (do intro&#39; fill (RConstant (Str &#34;&#34;)) solve) </code></pre></div><p>Copy and paste this into the code, and it will fill the hole when the code is compiled, just like we did interactively at the REPL:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-idris" data-lang="idris"><span style="color:#66d9ef">module</span> HoleyHead <span style="color:#a6e22e">firstStringOrEmpty</span> <span style="color:#f92672">:</span> <span style="color:#66d9ef">List</span> <span style="color:#66d9ef">String</span> <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">String</span> firstStringOrEmpty strings <span style="color:#f92672">=</span> <span style="color:#66d9ef">case</span> strings <span style="color:#66d9ef">of</span> <span style="color:#66d9ef">Nil</span> <span style="color:#f92672">=&gt;</span> <span style="color:#f92672">?</span>holeThatWasTheEmptyString <span style="color:#a6e22e">string</span> <span style="color:#f92672">::</span> <span style="color:#66d9ef">_</span> <span style="color:#f92672">=&gt;</span> string <span style="color:#66d9ef">HoleyHead</span><span style="color:#f92672">.</span>holeThatWasTheEmptyString <span style="color:#f92672">=</span> <span style="color:#f92672">%</span>runElab <span style="color:#f92672">(</span><span style="color:#66d9ef">do</span> intro&#39; fill <span style="color:#f92672">(</span><span style="color:#66d9ef">RConstant</span> <span style="color:#f92672">(</span><span style="color:#66d9ef">Str</span> <span style="color:#e6db74">&#34;&#34;</span><span style="color:#f92672">))</span> solve<span style="color:#f92672">)</span> </code></pre></div><p>Fire this up in the REPL, and notice that there aren&rsquo;t any unfilled holes:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">&gt; idris --nobanner HoleyHead.idr Type checking ./HoleyHead.idr HoleyHead.idr:12:37: When checking right hand side of HoleyHead.holeThatWasTheEmptyString: No such variable Language.Reflection.Elab.Elab Holes: HoleyHead.holeThatWasTheEmptyString *HoleyHead&gt; </code></pre></div><p>Oh, snap. Remember how we ran <code>:module Language.Reflection.Elab</code> before entering the elaboration shell? We also need to bring those names into scope in our file, too:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-idris" data-lang="idris"><span style="color:#66d9ef">module</span> HoleyHead <span style="color:#66d9ef">import</span> Language.Reflection.Elab <span style="color:#a6e22e">firstStringOrEmpty</span> <span style="color:#f92672">:</span> <span style="color:#66d9ef">List</span> <span style="color:#66d9ef">String</span> <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">String</span> firstStringOrEmpty strings <span style="color:#f92672">=</span> <span style="color:#66d9ef">case</span> strings <span style="color:#66d9ef">of</span> <span style="color:#66d9ef">Nil</span> <span style="color:#f92672">=&gt;</span> <span style="color:#f92672">?</span>holeThatWasTheEmptyString <span style="color:#a6e22e">string</span> <span style="color:#f92672">::</span> <span style="color:#66d9ef">_</span> <span style="color:#f92672">=&gt;</span> string <span style="color:#66d9ef">HoleyHead</span><span style="color:#f92672">.</span>holeThatWasTheEmptyString <span style="color:#f92672">=</span> <span style="color:#f92672">%</span>runElab <span style="color:#f92672">(</span><span style="color:#66d9ef">do</span> intro&#39; fill <span style="color:#f92672">(</span><span style="color:#66d9ef">RConstant</span> <span style="color:#f92672">(</span><span style="color:#66d9ef">Str</span> <span style="color:#e6db74">&#34;&#34;</span><span style="color:#f92672">))</span> solve<span style="color:#f92672">)</span> </code></pre></div><p>You can either run the <code>:edit</code> command at the REPL and add the <code>import</code> line, or edit it in your editor and ask the REPL to <code>:reload</code> your current file, or just exit, edit, and then run <code>idris --nobanner &lt;FILE&gt;</code> once more.</p> <p>Either way, you&rsquo;ve just completed your hello world!</p> <h2 id="summary">Summary</h2> <p>Idris allows you to interactively build a script that generates source code at compile time by solving for a goal represented by a hole in your source code.</p> <p>You&rsquo;ve seen how to:</p> <ul> <li>Represent a hole in source code using a question-mark–prefixed name like <code>?hole</code>.</li> <li>Import <code>Language.Reflection.Elab</code> to make the system-provide tactics available to you.</li> <li>Enter the elaborator shell using <code>:elab &lt;HOLE&gt;</code>.</li> <li>Get documentation on tactics using <code>:doc &lt;NAME&gt;</code>.</li> <li>Use <code>intro'</code> to focus on the body of a function without naming the function&rsquo;s arugment.</li> <li>Use <code>fill</code> to provide a guess for a hole&rsquo;s value.</li> <li>Use <code>solve</code> to instantiate the guess.</li> <li>Conclude the session with <code>:qed</code> to get a proof script you can paste into your file to fill the hole in future.</li> </ul> <h2 id="challenges">Challenges</h2> <p>Some challenges, in order of increasing difficulty:</p> <ul> <li>Fill the hole with the string &ldquo;bananas&rdquo; instead of &ldquo;&quot;.</li> <li>Name the function argument (check out <code>intro</code> rather than <code>intro'</code>).</li> <li>Replace the <code>string</code> result in the other case with a hole <code>?holeThatWasString</code>, and interactively create an elaborator script to fill that hole.</li> </ul> <h2 id="for-the-more-curious-quotation">For the More Curious: Quotation</h2> <p>Figuring out that whole <code>RConstant (Str &quot;&quot;))</code> bit was a pain. If you felt like Idris should have been able to do that work for you, you were right! Idris supports quoting syntax, which will seem awfully familiar if you&rsquo;ve written any Lisp macros:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">&gt; :doc quote Language.Reflection.quote : Quotable a t =&gt; a -&gt; t Quote a particular element of a. Each equation should look something like quote (Foo x y) = `(Foo ~(quote x) ~(quote y)) The function is Total </code></pre></div><p>Instead of laboriously hunting down the various type constructors and then writing out <code>fill (RConstant (Str &quot;&quot;))</code>, we can use quoting to do the hard work for us with:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-idris" data-lang="idris">fill <span style="color:#f92672">(</span>quote <span style="color:#e6db74">&#34;&#34;</span><span style="color:#f92672">)</span> </code></pre></div><p>Even sweeter, we can use Lisp-like quotation syntax to write this with even less typing as:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-idris" data-lang="idris">fill `<span style="color:#f92672">(</span><span style="color:#e6db74">&#34;&#34;</span><span style="color:#f92672">)</span> </code></pre></div> When is a proof not a proof? https://jeremywsherman.com/blog/2015/10/02/when-is-a-proof-not-a-proof/ Fri, 02 Oct 2015 15:19:59 +0000 https://jeremywsherman.com/blog/2015/10/02/when-is-a-proof-not-a-proof/ <blockquote> <p>When is a proof not a proof? When you think you&rsquo;ve proven one thing, but actually, you&rsquo;ve proven something else.</p> </blockquote> <p>In <a href="https://jeremywsherman.com/blog/2015/10/02/what-if-you-get-your-dependent-type-backwards/">the last post</a>, I addressed a good question raised in the first half of David Owens II&rsquo;s article <a href="http://owensd.io/2015/09/03/dependent-types.html" title="Dependent Types: I'm Missing Something">&ldquo;Dependent Types: I&rsquo;m Missing Something&rdquo;</a>, where he addresses my discussion of dependent types.</p> <p>This post looks at the latter half, which talks about an example drawn from early in the chapter <a href="http://adam.chlipala.net/cpdt/html/InductiveTypes.html">&ldquo;Introducing Inductive Types&rdquo;</a> of Adam Chlipala&rsquo;s <em><a href="http://adam.chlipala.net/cpdt/" title="Certified Programming with Dependent Types">Certified Programming with Dependent Types</a>.</em></p> <blockquote> <p>When is a proof not a proof? When you think you&rsquo;ve proven one thing, but actually, you&rsquo;ve proven something else.</p> </blockquote> <p>In <a href="https://jeremywsherman.com/blog/2015/10/02/what-if-you-get-your-dependent-type-backwards/">the last post</a>, I addressed a good question raised in the first half of David Owens II&rsquo;s article <a href="http://owensd.io/2015/09/03/dependent-types.html" title="Dependent Types: I'm Missing Something">&ldquo;Dependent Types: I&rsquo;m Missing Something&rdquo;</a>, where he addresses my discussion of dependent types.</p> <p>This post looks at the latter half, which talks about an example drawn from early in the chapter <a href="http://adam.chlipala.net/cpdt/html/InductiveTypes.html">&ldquo;Introducing Inductive Types&rdquo;</a> of Adam Chlipala&rsquo;s <em><a href="http://adam.chlipala.net/cpdt/" title="Certified Programming with Dependent Types">Certified Programming with Dependent Types</a>.</em></p> <p>This example is interesting in part because it is actually not an example of a dependent type, so it&rsquo;s something we can talk about almost entirely in terms of Swift, absent the computer-assisted theorem proving that is Coq&rsquo;s special sauce. Tackling theorems in Swift is a perfect fit for a generative testing framework.</p> <h2 id="putting-the-example-in-context">Putting the example in context</h2> <p>The book has introduced a <code>bool</code> type as an example of defining an enumeration type. It then defines a function <code>negb</code> that performs Boolean negation to demonstrate function definition using pattern matching. The book then proceeds to prove a couple theorems about the <code>negb</code> function as part of teaching the reader how to do proofs using Coq.</p> <h2 id="rewriting-the-example-in-swift">Rewriting the example in Swift</h2> <blockquote> <p><code>bool</code> is not a dependent type, so Swift and SwiftCheck are up to the task.</p> </blockquote> <p>The book doesn&rsquo;t get to dependent types till 5 chapters later. <code>bool</code> is an enumeration type, and <code>negb</code> relies on non-dependent pattern matching, all of which Swift can do. This means we can pretty directly translate the example definitions and theorem from Gallina into Swift:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">Bool_</span> { <span style="color:#66d9ef">case</span> True <span style="color:#66d9ef">case</span> False } <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">negb</span>(b: Bool_) -&gt; Bool_ { <span style="color:#66d9ef">switch</span> b { <span style="color:#66d9ef">case</span> .True: <span style="color:#66d9ef">return</span> .False <span style="color:#66d9ef">case</span> .False: <span style="color:#66d9ef">return</span> .True } } <span style="color:#75715e">/// Generative testing of the `negb_inverse` theorem&#39;s claim.</span> <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">testNegbInverse</span>() { property(<span style="color:#e6db74">&#34;negb is its own inverse&#34;</span>) <span style="color:#f92672">&lt;-</span> forAll { (b: Bool_) <span style="color:#66d9ef">in</span> <span style="color:#66d9ef">return</span> negb(negb(b)) == b } } <span style="color:#66d9ef">extension</span> <span style="color:#a6e22e">Bool_</span>: Arbitrary { <span style="color:#66d9ef">static</span> <span style="color:#66d9ef">var</span> arbitrary : Gen&lt;Bool_&gt; { <span style="color:#66d9ef">return</span> Gen.oneOf([Gen.pure(.True), Gen.pure(.False)]) } } </code></pre></div><p>I&rsquo;ve translated the <code>negb_inverse</code> theorem into a <a href="https://github.com/typelift/SwiftCheck">SwiftCheck</a> test and provided a generator of arbitrary <code>Bool_</code> values for the test generator to use.</p> <p>Running the test as-is gives:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Test Case &#39;-[NegbTests.NegbTests testTheoremNegbInverse]&#39; started. *** Passed 100 tests . </code></pre></div><p>You can run this and see for yourself by cloning <a href="https://github.com/jeremy-w/Negb">my Negb project repository</a>.</p> <h2 id="mistaken-proof">Mistaken proof</h2> <p>David&rsquo;s concern is that, &ldquo;It’s up to the programmer to realize that we have actually not created all of the proofs required to prove correctness.&rdquo; This concern arises from his thinking that the proof for <code>negb_inverse</code> goes through even if you alter <code>negb</code> so it always returns <code>true</code>.</p> <p>The version of <code>negb</code> he defines that always returns <code>true</code> actually already fails at the <code>negb_inverse</code> theorem, without need to proceed to trying to prove <code>negb_ineq</code>.</p> <p>You can see this in practice by pasting the bogus <code>negb</code> definition into <code>coqtop</code> and then trying to complete the proof of the theorem, but you can probably convince yourself without needing to do that simply by considering the <code>false</code> case.</p> <p>The theorem&rsquo;s claim is that <code>negb(negb(b)) == b</code>. In the false case, with the always-true <code>negb</code>, we get:</p> <pre><code>negb(negb(b)) == b // the negb_inverse claim --&gt; consider the case where b = false negb(negb(false)) == false --&gt; evaluate the inner negb application negb(true) == false --&gt; evaluate the remaining negb application true == false --&gt; contradiction! &lt;-- </code></pre><p>If you flip the <code>case .True: return .False</code> to <code>return .True</code> in the Swift version of <code>negb</code>, you&rsquo;ll see it quickly find this failing case using the test, as well:</p> <pre><code>Test Case '-[NegbTests.NegbTests testTheoremNegbInverse]' started. *** Failed! Proposition: negb is its own inverse Falsifiable (after 1 test): False *** Passed 0 tests . </code></pre><p>If you go the <code>coqtop</code> route, you&rsquo;ll be able to discharge the <code>true</code> case with <code>simpl. trivial.</code>, but using the <code>simpl.</code> tactic in the false case leaves you at the end step of our reasoning above, trying to prove that <code>true = false</code>.</p> <p><small>If you&rsquo;re in <code>coqtop</code>, you&rsquo;ll also quickly notice that the proof David copied in for <code>negb_inverse</code> was incomplete. <code>destruct.</code> leaves you with two subgoals left to prove, corresponding to a version of the theorem with <code>b</code> instantiated at each case of the enum. You can complete the proof via <code>reflexitivity. Qed.</code>, which the book gets to a half-page later after some discussion.</small></p> <h2 id="proving-the-right-thing-and-human-misapprehension">Proving the right thing and human misapprehension</h2> <blockquote> <p>A sound system won&rsquo;t draw bogus conclusions, but that doesn&rsquo;t mean we can&rsquo;t misinterpret our own claims, whether they&rsquo;ve been proven true or not.</p> </blockquote> <p>David&rsquo;s core concern about knowing you&rsquo;ve proved what you want to prove is a real problem.</p> <p>The problem doesn&rsquo;t appear where he thought it did, in a theorem that he knew shouldn&rsquo;t be provable but he thought still was.</p> <p>The problem shows up in the same place that it did with <a href="https://jeremywsherman.com/blog/2015/10/02/what-if-you-get-your-dependent-type-backwards/">the dependently typed flipped Boolean discussion</a>, where the human is giving names and external meaning to the mathematical expression. In a sound system, the proofs we provide will be correct, but we can still misunderstand what the theorem we have just proved really means.</p> <p>If we&rsquo;re building on top of that proof, that misunderstanding will undoubtedly come to light as we try to use it to prove new claims. We just won&rsquo;t be able to make a chain of reason from what we want to prove to what we did prove, because our new goal will depend on what we thought we proved, which was something different.</p> <p>If what we thought we proved was the final achievement of our development, though, then we might not run into trouble till we try to actually use it to do something. Then we&rsquo;ll see it breakdown at runtime, when it does what it in fact tells us on its face it does, which just so happens not to be what we read it as doing.</p> What if you get your dependent type backwards? https://jeremywsherman.com/blog/2015/10/02/what-if-you-get-your-dependent-type-backwards/ Fri, 02 Oct 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/10/02/what-if-you-get-your-dependent-type-backwards/ <p>David Owens II read <a href="https://jeremywsherman.com/blog/2015/08/26/read-why-dependent-types-matter/" title="Read: Why dependent types matter">my notes on &ldquo;Why Dependent Types Matter&rdquo;</a> and <a href="http://owensd.io/2015/09/03/dependent-types.html" title="Dependent Types: I'm Missing Something">asked a good question</a>:</p> <blockquote> <p>This is the part I don’t get: we allow for incorrect code to be written in the non-dependent-typed case, but we assume that we can’t do the same with dependently-typed code? Why? What’s preventing me from swapping <code>left</code> and <code>right</code> in the <code>Order</code> type that is returned?</p> </blockquote> <p>David Owens II read <a href="https://jeremywsherman.com/blog/2015/08/26/read-why-dependent-types-matter/" title="Read: Why dependent types matter">my notes on &ldquo;Why Dependent Types Matter&rdquo;</a> and <a href="http://owensd.io/2015/09/03/dependent-types.html" title="Dependent Types: I'm Missing Something">asked a good question</a>:</p> <blockquote> <p>This is the part I don’t get: we allow for incorrect code to be written in the non-dependent-typed case, but we assume that we can’t do the same with dependently-typed code? Why? What’s preventing me from swapping <code>left</code> and <code>right</code> in the <code>Order</code> type that is returned?</p> </blockquote> <h2 id="review-backwards-booleans">Review: Backwards Booleans</h2> <p>If we&rsquo;re not using dependent types, it&rsquo;s not hard to flip the sense of a test and end up with a result that means the opposite of what we intended due to a hiccup in mentally simulating our code.</p> <p>The running example is something like <code>is(left: UInt, lessThanOrEqualTo right: UInt) -&gt; Bool</code>.</p> <p>A <code>Bool</code> is just a bit that can be manufactured any which way, so this simple test fails to provide evidence – by which I mean constructive proof – for its claim.</p> <p>If you switch to <code>compare(left: UInt, to right: UInt) -&gt; NSComparisonResult</code>, you get more than a bit of info back, but you still can mess up. (If anything, it&rsquo;s easier, since you now have to get 2 tests right to discrimate between the 3 cases rather than the 1 needed to decide between <code>true</code>/<code>false</code> in the <code>Bool</code> version.)</p> <h2 id="how-do-dependent-types-prevent-backwards-booleans">How do dependent types prevent backwards Booleans?</h2> <p>Now let&rsquo;s say we&rsquo;re using the dependent type <code>Order(left: UInt, right: UInt)</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">Order</span>(<span style="color:#66d9ef">left</span>: UInt, <span style="color:#66d9ef">right</span>: UInt) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/* #1 */</span> <span style="color:#66d9ef">case</span> lessThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">left</span>, y: <span style="color:#66d9ef">right</span>)): <span style="color:#75715e">/* #2 */</span> Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) <span style="color:#75715e">/* #3 */</span> <span style="color:#66d9ef">case</span> greaterThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">right</span>, y: <span style="color:#66d9ef">left</span>)): <span style="color:#75715e">/* #4 */</span> Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) <span style="color:#75715e">/* #5 */</span> } </code></pre></div><p>We have 5 instances of using left/right together, labeled in comments at the end of each line.</p> <blockquote> <p>What’s preventing me from swapping <code>left</code> and <code>right</code> in the <code>Order</code> type that is returned?</p> </blockquote> <p>Well, let&rsquo;s see! David mentions swapping the left/right in the <code>Order</code> type that&rsquo;s returned, which corresponds to #3 and #5, so we&rsquo;ll start there.</p> <h3 id="dropping-the-evidence">Dropping the Evidence</h3> <p>Let&rsquo;s drop the evidence (the <code>because</code> bits) and swap left/right #3 and #5:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">OrderNoEvidence</span>(<span style="color:#66d9ef">left</span>: UInt, <span style="color:#66d9ef">right</span>: UInt) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/* #1 */</span> <span style="color:#66d9ef">case</span> lessThanOrEqual: <span style="color:#75715e">/* #2 - eliminated */</span> OrderNoEvidence(<span style="color:#66d9ef">right</span>, <span style="color:#66d9ef">left</span>) <span style="color:#75715e">/* #3 - flipped */</span> <span style="color:#66d9ef">case</span> greaterThanOrEqual: <span style="color:#75715e">/* #4 - eliminated */</span> OrderNoEvidence(<span style="color:#66d9ef">right</span>, <span style="color:#66d9ef">left</span>) <span style="color:#75715e">/* #5 - flipped */</span> } </code></pre></div><p>In this case, nothing would catch the inversion. <code>left</code> and <code>right</code> are both <code>UInt</code>, so there&rsquo;s no harm done as far as the compiler is concerned.</p> <p>There&rsquo;s also no actual distinction in shape between the two cases; we can freely substitute one for the other anywhere in our code without the type system griping, just as with <code>Bool</code> or <code>NSComparisonResult</code>.</p> <p>Unlike the non-dependent type, an instance of this type carries around information at the type level about the values whose comparison it represents, so you can recover at least which values were supposedly compared through pattern matching.<span id="wiygydtf1back"><a href="#wiygydtf1">*</a></span></p> <p>But this doesn&rsquo;t solve our flipped-Boolean problem.</p> <h3 id="restoring-the-evidence">Restoring the Evidence</h3> <p>Now let&rsquo;s restore our evidence in its un-flipped form, while leaving the resulting type still flipped in #3 and #5:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">OrderFlippedInstances</span>(<span style="color:#66d9ef">left</span>: UInt, <span style="color:#66d9ef">right</span>: UInt) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/* #1 */</span> <span style="color:#66d9ef">case</span> lessThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">left</span>, y: <span style="color:#66d9ef">right</span>)): <span style="color:#75715e">/* #2 */</span> OrderFlippedInstances(<span style="color:#66d9ef">right</span>, <span style="color:#66d9ef">left</span>) <span style="color:#75715e">/* #3 - flipped */</span> <span style="color:#66d9ef">case</span> greaterThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">right</span>, y: <span style="color:#66d9ef">left</span>)): <span style="color:#75715e">/* #4 */</span> OrderFlippedInstances(<span style="color:#66d9ef">right</span>, <span style="color:#66d9ef">left</span>) <span style="color:#75715e">/* #5 - flipped */</span> } </code></pre></div><p>We&rsquo;ve flipped #3 and #5 still, but the rest are back and not flipped.</p> <p>Compared to the evidenceless version of the type, you can no longer gin up an instance of <code>Order</code> out of thin air. You once again have to provide an instance of <code>LEQ</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">LEQ</span>(x: UInt, y: UInt): <span style="color:#66d9ef">Type</span> { <span style="color:#66d9ef">case</span> zeroLEQEverything: LEQ(<span style="color:#ae81ff">0</span>, y) <span style="color:#66d9ef">case</span> stepLEQ(LEQ(x, y)): LEQ(x <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>, y <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>) } </code></pre></div><p>Alas, even with <code>LEQ</code> completely correct, we can still define this version of <code>Order</code>. It will compile just fine.</p> <p>This might confuse the user of this API, especially if the documentation says that <code>Order(left, right)</code> tells how <code>left</code> compares to <code>right</code> rather than how <code>right</code> compares to <code>left</code>.</p> <p>Unlike in the evidence-less case, though, consumers of instances of this type can work out that it&rsquo;s the wrong way around based on the <code>because</code> evidence: an instance like <code>lessThanOrEqual(zeroLEQEverything: LEQ(0, 1)): OrderFlippedInstances(1, 0)</code> hands the consumer a proof that <code>LEQ(0, 1)</code>, and if they pattern-match that out and use it – as they likely would while producing evidence for the correctness of whatever they&rsquo;re building atop this data – it&rsquo;s merely frustrating that our documentation is backwards.</p> <p>This &ldquo;solves&rdquo; the flipped Boolean problem, but no tool can solve the problem of misleading names. Misleading names provide bad input into our informal reasoning processes, and we&rsquo;re likely to write bogus code as a result. If we&rsquo;re programming with evidence, as dependent types let us do, we&rsquo;ll catch this while interacting with the compiler; if we&rsquo;re trusting the names, and ignoring the evidence, as dependent types also let us do (and non-dependent types force us to do), we likely won&rsquo;t, absent testing.</p> <h3 id="flipping-all-pairs">Flipping All Pairs</h3> <p>If we flip all pairs, we get something identical to our original <code>Order(left: UInt, right: UInt)</code> type:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">OrderMirrored</span>(<span style="color:#66d9ef">right</span>: UInt, <span style="color:#66d9ef">left</span>: UInt) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/* #1 */</span> <span style="color:#66d9ef">case</span> lessThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">right</span>, y: <span style="color:#66d9ef">left</span>)): <span style="color:#75715e">/* #2 */</span> OrderMirrored(<span style="color:#66d9ef">right</span>, <span style="color:#66d9ef">left</span>) <span style="color:#75715e">/* #3 */</span> <span style="color:#66d9ef">case</span> greaterThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">left</span>, y: <span style="color:#66d9ef">right</span>)): <span style="color:#75715e">/* #4 */</span> OrderMirrored(<span style="color:#66d9ef">right</span>, <span style="color:#66d9ef">left</span>) <span style="color:#75715e">/* #5 */</span> } </code></pre></div><p>Aside from the backwards parameter names if you look at the definition or documentation, this works the same as the original <code>Order</code>. Even the case names are the right way around.</p> <h3 id="providing-the-wrong-evidence">Providing the Wrong Evidence</h3> <p>If we accept the wrong evidence, by flipping around just #4, then we finally get real breakage:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">OrderWrongEvidence</span>(<span style="color:#66d9ef">left</span>: UInt, <span style="color:#66d9ef">right</span>: UInt) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/* #1 */</span> <span style="color:#66d9ef">case</span> lessThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">left</span>, y: <span style="color:#66d9ef">right</span>)): <span style="color:#75715e">/* #2 */</span> Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) <span style="color:#75715e">/* #3 */</span> <span style="color:#66d9ef">case</span> greaterThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">left</span>, y: <span style="color:#66d9ef">right</span>)): <span style="color:#75715e">/* #4 - flipped */</span> Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) <span style="color:#75715e">/* #5 */</span> } </code></pre></div><p>By flipping #4, we&rsquo;ve ended up with two cases with different names that just say the same thing. This is clear because they have the same shape: each takes one argument of the same type, and each constructs an instance of the same type.</p> <p>Again, if you&rsquo;re leaning on that evidence, this will work itself out while you&rsquo;re writing code, because you&rsquo;ll reach for the expected <code>y &lt;= x</code> proof when handling the <code>greaterThanOrEqual</code> case only to find it&rsquo;s not there.</p> <p>If the client code is more trusting and ignores the evidence, then you&rsquo;ll get broken behavior, and you won&rsquo;t find it till you test the code by running it.</p> <h2 id="dependent-types-allow-us-to-provide-and-demand-evidence">Dependent types allow us to provide and demand evidence</h2> <p>This is the trade-off you&rsquo;re able to make when using dependent types:</p> <ul> <li>trust and test,</li> <li>or demand proof.</li> </ul> <p>Without dependent types, you don&rsquo;t have any proof – you&rsquo;re left to trusting and testing.</p> <!-- difference: transparency even absent source code due to types --> <p><span id="wiygydtf1"><a href="#wiygydtf1back">*</a></span><small>Dependent pattern matching can bind values not just out of the instance, as normal pattern matching does with <code>case let</code>, but also out of the type signature.</small></p> Read: Why dependent types matter https://jeremywsherman.com/blog/2015/08/26/read-why-dependent-types-matter/ Wed, 26 Aug 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/08/26/read-why-dependent-types-matter/ Altenkirch, Thorsten, Conor McBride, and James McKinna. Why Dependent Types Matter. 2005. 19 pages (+2 more of references). Accessed 2015-08-26. What is a dependent type? A dependent type is a type that depends, not just on other types, but on terms. This means you can have values – plain old data – in your types. A common example is that of length-indexed lists: /// A list parameterized by `ElementType` and indexed over its length. <p>Altenkirch, Thorsten, Conor McBride, and James McKinna. <a href="http://www.cs.nott.ac.uk/~txa/publ/ydtm.pdf" title="Why Dependent Types Matter (PDF)">Why Dependent Types Matter</a>. 2005. 19 pages (+2 more of references). Accessed 2015-08-26.</p> <h2 id="what-is-a-dependent-type">What is a dependent type?</h2> <p>A dependent type is a type that depends, not just on other types, but on terms. This means you can have values – plain old data – in your types. A common example is that of <strong>length-indexed lists</strong>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#75715e">/// A list parameterized by `ElementType` and indexed over its length.</span> <span style="color:#75715e">/// This means that `Vector(length: 3, ElementType: Int)` is a different</span> <span style="color:#75715e">/// *type* than Vector(length: 4, ElementType: Int)`.</span> <span style="color:#75715e">///</span> <span style="color:#75715e">/// You can read a type `Vector(n, T)` as &#34;the type of vectors containing</span> <span style="color:#75715e">/// precisely *n* Ts&#34;.</span> <span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">Vector</span>(length: UInt, ElementType: <span style="color:#66d9ef">Type</span>) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/// Returns an empty list of inferred type `type`.</span> <span style="color:#66d9ef">case</span> vnil(): Vector(length: <span style="color:#ae81ff">0</span>, ElementType: type) <span style="color:#75715e">/// Prefixes an `element` whose type matches `vector`&#39;s ElementType.</span> <span style="color:#75715e">///</span> <span style="color:#75715e">/// NOTE: The type of the result vector differs from the type of the input</span> <span style="color:#75715e">/// vector by being one greater than the input.</span> <span style="color:#66d9ef">case</span> vcons(element, vector: Vector(length: n, ElementType: type)) <span style="color:#66d9ef">where</span> element <span style="color:#66d9ef">is</span> type: Vector(length: n <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>, ElementType: type) } </code></pre></div><p>You likely caught on that that code is Swift-ish but not actually Swift. That&rsquo;s <span id="ydtmback0"><a href="#ydtmfn0"><strong>DTSwift</strong>*</a></span>, where I pretend that Swift is dependently typed.</p> <h3 id="types-are-terms-are-types">Types are terms are types</h3> <p>Notice that I&rsquo;m writing the type constructor using normal function application with parentheses <code>()</code> rather than &ldquo;generic type application&rdquo; with angle brackets <code>&lt;&gt;</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">Vector</span>(length: UInt, ElementType: <span style="color:#66d9ef">Type</span>) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">// NOT: enum Vector&lt;length, ElementType&gt; {</span> <span style="color:#75715e">// The &lt;&gt; assume that everything in them is a type, which just ain&#39;t so</span> <span style="color:#75715e">// in DTSwift.</span> </code></pre></div><p>As far as DTSwift is concerned, a type constructor is just a function from some terms/types to a type. We no longer need separate, parallel languages for working with types (like <code>Int</code> or <code>String</code>) vs working with terms in those types (like <code>3</code> or <code>&quot;perish the thought&quot;</code>).</p> <h2 id="why-do-we-care">Why do we care?</h2> <h3 id="provably-correct-code-where-you-want-it-loosey-goosey-where-you-dont">Provably correct code where you want it, loosey-goosey where you don&rsquo;t</h3> <p>Dependent types make it possible to be very specific about the terms in a type. We don&rsquo;t have to be, but now we have the option of cranking up the specificity from &ldquo;eh, whatever, gimme a list&rdquo; to &ldquo;this list has to be non-empty&rdquo; to &ldquo;this list has to have a prime number of elements&rdquo;.</p> <p>This means that dependent types give us pay-as-you-go provable correctness. You can require proof of the most critical properties in your program in the form of very specific types that ensure those invariants, while handwaving about components that are either less important or just more readily shown correct by test or inspection.</p> <h3 id="we-can-treat-different-data-in-different-ways-at-compile-time">We can treat different data in different ways at compile-time</h3> <p>No need to wait to crash at runtime or during a test; you can push &ldquo;make invalid states unrepresentable&rdquo; to the max if you want.</p> <h4 id="where-swifts-types-let-you-down">Where Swift&rsquo;s types let you down</h4> <p>Have you ever noticed how, if you write <code>let order = x &lt;= y ? .OrderedAscending : .OrderedDescending</code>, you could swap <code>x</code> and <code>y</code> and still end up with a valid program? It&rsquo;s like we learned nothing from that <code>test</code> we performed!</p> <p>As far as normal Swift&rsquo;s type system is concerned, any <code>T</code> is as good as any other; there&rsquo;s &ldquo;no means to express the way that different data mean different things, and should be treated accordingly in different ways&rdquo; (p. 11). We can always just swap around the result of an if/else expression <code>?:</code>, and our compiler will happily accept our backwards logic.</p> <p>While we can write:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">compare</span>(x: UInt, y: UInt) -&gt; NSComparisonResult { <span style="color:#66d9ef">return</span> (x <span style="color:#f92672">&lt;=</span> y) ? .OrderedAscending : .OrderedDescending } </code></pre></div><p>This would be just as correct as far as the compiler is concerned as either of these definitions:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">compare</span>(x: UInt, y: UInt) -&gt; NSComparisonResult { <span style="color:#75715e">// BACKWARDS! IT LIES!</span> <span style="color:#75715e">// (And how many times have you accidentally flipped these?)</span> <span style="color:#66d9ef">return</span> (x <span style="color:#f92672">&lt;=</span> y) ? .OrderedDescending : .OrderedAscending } <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">compare</span>(x: UInt, y: UInt) -&gt; UInt { <span style="color:#75715e">// I hope you weren&#39;t actually planning on relying on this function.</span> <span style="color:#66d9ef">return</span> .OrderedAscending } </code></pre></div><hr> <h5 id="aside-yes-this-is-a-toy-example">Aside: Yes, this is a toy example</h5> <p>This is a toy example. I&rsquo;m sure you can quickly figure out when an if-then-else has the branches swapped. Refining your code to use more specific types grows more valuable as the complexity of its behavior grows, and this is not a terribly complex example!</p> <p>The effort/reward calculus changes when it comes to library code, though. Specific types in library code can save a lot of collective confusion and wasted time. Think of some Apple API you&rsquo;ve looked at and gone, &ldquo;Gee, I wonder what this method does when it runs into this corner case?&rdquo; If the types clearly said what the method did, you wouldn&rsquo;t be left scratching your head.</p> <p>As a side benefit, specific types constrain your code to where the compiler can provide non-trivial help in writing the code. There&rsquo;s a section on this in the paper, or just <a href="http://www.tryidris.org/console" title="Try Idris in your browser!">give a dependently-typed language a spin</a>.</p> <p>If you need to see some less toy examples to be convinced, check out Oury and Swierstra&rsquo;s <a href="http://www.staff.science.uu.nl/~swier004/Publications/ThePowerOfPi.pdf" title="The Power of Pi (2008) [PDF]">The Power of Pi (2008)</a> and Fowler and Brady&rsquo;s <a href="http://eb.host.cs.st-andrews.ac.uk/drafts/ifl2013.pdf" title="Dependent Types for Safe and Secure Web Programming (2013) [PDF]">Dependent Types for Safe and Secure Web Programming (2013)</a>.</p> <hr> <h4 id="how-dependent-types-fix-this">How dependent types fix this</h4> <p>Dependent types let you replace a Boolean that throws away the knowledge gained from a test by a type that represents the outcome of that test.</p> <p>In DTSwift, we have the option of shifting from <code>operator &lt;=(left: UInt, _ right: UInt) -&gt; Bool</code> to a <code>LEQ</code> type that must provide evidence for how <code>left</code> and <code>right</code> in particular are related:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#75715e">/// A type expressing that `(x &lt;= y)` via proof by induction.</span> <span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">LEQ</span>(x: UInt, y: UInt): <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/// Base case: 0 &lt;= any other UInt.</span> <span style="color:#66d9ef">case</span> zeroLEQEverything: LEQ(<span style="color:#ae81ff">0</span>, y) <span style="color:#75715e">/// Induction: If we know x &lt;= y, then we also know x+1 &lt;= y+1.</span> <span style="color:#66d9ef">case</span> stepLEQ(LEQ(x, y)): LEQ(x <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>, y <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>) } </code></pre></div><p>We can then move from a &ldquo;blind&rdquo; <code>NSComparisonResult</code> to an <code>Order</code> type that expresses how two values are related and includes proof that this is indeed the case:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#75715e">/// A type expressing the relationship between `left` and `right`.</span> <span style="color:#75715e">/// Each case is associated with a term witnessing to the relationship</span> <span style="color:#75715e">/// between `left` and `right`.</span> <span style="color:#75715e">///</span> <span style="color:#75715e">/// See: `LEQ(left, right)` defined just after this.</span> <span style="color:#66d9ef">enum</span> <span style="color:#a6e22e">Order</span>(<span style="color:#66d9ef">left</span>: UInt, <span style="color:#66d9ef">right</span>: UInt) -&gt; <span style="color:#66d9ef">Type</span> { <span style="color:#75715e">/// Represents that `(left &lt;= right)`.</span> <span style="color:#75715e">/// Witnessed by a proof that `left &lt;= right`.</span> <span style="color:#66d9ef">case</span> lessThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">left</span>, y: <span style="color:#66d9ef">right</span>)): Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) <span style="color:#75715e">/// Represents that `(left &gt;= right)`.</span> <span style="color:#75715e">/// Witnessed by a proof that `right &lt;= left`.</span> <span style="color:#66d9ef">case</span> greaterThanOrEqual(because: LEQ(x: <span style="color:#66d9ef">right</span>, y: <span style="color:#66d9ef">left</span>)): Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) } </code></pre></div><p>This burden of proof would rule out both of our bogus definitions of <code>compare</code> above:</p> <ul> <li>In the flipped case, we would find we couldn&rsquo;t prove an untruth.</li> <li>In the constant case, we couldn&rsquo;t generate any proof by flat-out ignoring our inputs. We have to examine them and work with them to arrive at a proof of their relationship, and the type ensures it is indeed the relationship between those specific values.</li> </ul> <p>We can now write a type-correct version with these new types, one whose return type captures the relationship between the two input terms:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">compare</span>(<span style="color:#66d9ef">left</span>: UInt, <span style="color:#66d9ef">_</span> <span style="color:#66d9ef">right</span>: UInt) -&gt; Order(<span style="color:#66d9ef">left</span>, <span style="color:#66d9ef">right</span>) { <span style="color:#66d9ef">switch</span> <span style="color:#66d9ef">left</span> { <span style="color:#66d9ef">case</span> <span style="color:#ae81ff">0</span>: <span style="color:#66d9ef">return</span> lessThanOrEqual(because: zeroLEQEverything) <span style="color:#66d9ef">case</span> <span style="color:#66d9ef">let</span> xMinus1 = <span style="color:#66d9ef">left</span> <span style="color:#f92672">-</span> <span style="color:#ae81ff">1</span>: <span style="color:#66d9ef">switch</span> <span style="color:#66d9ef">right</span> { <span style="color:#66d9ef">case</span> <span style="color:#ae81ff">0</span>: <span style="color:#66d9ef">return</span> greaterThanOrEqual(because: zeroLEQEverything) <span style="color:#66d9ef">case</span> <span style="color:#66d9ef">let</span> yMinus1 = <span style="color:#66d9ef">right</span> <span style="color:#f92672">-</span> <span style="color:#ae81ff">1</span>: <span style="color:#66d9ef">switch</span> compare(<span style="color:#66d9ef">left</span>: xMinus1, <span style="color:#66d9ef">right</span>: yMinus1) { <span style="color:#66d9ef">case</span> lessThanOrEqual(because: evidence): <span style="color:#75715e">// We know now that (x - 1) &lt;= (y - 1), so we can prove</span> <span style="color:#75715e">// by adding 1 to each side that x &lt;= y.</span> <span style="color:#66d9ef">return</span> lessThanOrEqual(because: stepLEQ(evidence)) <span style="color:#66d9ef">case</span> greaterThanOrEqual(because: evidence): <span style="color:#75715e">// Ditto, but the evidence will show that (y - 1) &lt;= (x - 1)</span> <span style="color:#75715e">// so we can prove y &lt;= x AKA x &gt;= y.</span> <span style="color:#66d9ef">return</span> greaterThanOrEqual(because: stepLEQ(evidence)) } <span style="color:#75715e">// switch on recursive call</span> } <span style="color:#75715e">// switch on right</span> } <span style="color:#75715e">// switch on left</span> } </code></pre></div><h4 id="summary-of-the-example">Summary of the Example</h4> <p>We started with a <code>compare</code> function that did the right thing, but it didn&rsquo;t say so – it just said &ldquo;I&rsquo;ll give you back a <code>UInt</code>, one&rsquo;s as good as another, right?&rdquo; (paraphrasing p. 12).</p> <p>Because our starting version of <code>compare</code> didn&rsquo;t say what it did via its types, it could also maybe not do the right thing; you could test it, but for all you know, some perverse developer decided to return the wrong answer for exactly one combination of unsigned integers.</p> <p>We ended up with a <code>compare</code> that had no choice but to do the right thing, because if it didn&rsquo;t, it would no longer typecheck.</p> <h2 id="so-much-more">So Much More</h2> <p>I encourage you to read this paper. It&rsquo;s a great introduction to programming with dependent types, and it covers a number of other topics while demonstrating the topic by refining the implementation of merge-sort throughout.</p> <p>You&rsquo;ll meet:</p> <ul> <li>Totality, and why you should care</li> <li>How you can do real work without general recursion, or, how Turing completeness has caused you generally avoidable pain <ul> <li>I&rsquo;d write about these now, but they&rsquo;ll come up in more detail when I write up my notes on Turner&rsquo;s &ldquo;Total Functional Programming&rdquo;.</li> </ul> </li> <li>Interactive type-directed editing, where you cooperate with the compiler to write your program, since very specific types lead to very specific shapes of programs working with those types <ul> <li>Faking up a DTSwift example of this would be a blog post in itself.</li> </ul> </li> <li>Approximations to dependent types that have shown up already in some not quite (not yet?) dependently-typed languages <ul> <li>Spoiler: Swift isn&rsquo;t one of them.</li> </ul> </li> </ul> <blockquote> <p>…what we have tried to demonstrate here is that the distinctions term/type, dynamic/static, explicit/inferred are no longer naturally aligned to each other in a type system which recognizes the relationships between values. We have decoupled these dichotomies and found a language which enables us to explore the continuum of pragmatism and precision and find new sweet spots within it. Of course this continuum also contains opportunities for remarkable ugliness and convolution – one can never legislate against bad design – but that is no reason to toss away its opportunities. Often, by bringing out the ideas which lie behind good designs, by expressing the things which matter, dependent types make data and programs fit better. (p. 21)</p> </blockquote> <hr> <p><span id="ydtmfn0"><a href="#ydtmback0">*</a></span><small>Since I don&rsquo;t have a formal spec or implementation for this DTSwift fantasy language, it risks confusing things more than using an actual dependently-typed language, but I&rsquo;m willing to take that risk to make this topic more approachable for people coming from Swiftland (population: 1 Gulliver).</small></p> Read: A tutorial on the universality and expressiveness of fold https://jeremywsherman.com/blog/2015/08/18/read-a-tutorial-on-the-universality-and-expressiveness-of-fold/ Tue, 18 Aug 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/08/18/read-a-tutorial-on-the-universality-and-expressiveness-of-fold/ Hutton, Graham. A tutorial on the universality and expressiveness of fold. 1999. 16 pages (+ 2 more of references). Accessed 2015-08-18. Fold (reduce, inject, cata) lets you take a list and swap out its constructors for a combining function and base value of your own devising, like swizzling in + for cons/: and 0 for nil/[]: [1 , 2 , 3] == EXPAND THE SUGAR ==&gt; (1 : (2 : (3 : []))) == FOLD (+) 0 ==&gt; (1 + (2 + (3 + 0 ))) Notice how all the : were rewritten into the provided function ((+)), and [] was rewritten into the provided value (0). <p>Hutton, Graham. <a href="http://www.cs.nott.ac.uk/~gmh/fold.pdf">A tutorial on the universality and expressiveness of fold</a>. 1999. 16 pages (+ 2 more of references). Accessed 2015-08-18.</p> <p>Fold (reduce, inject, cata) lets you take a list and swap out its constructors for a combining function and base value of your own devising, like swizzling in <code>+</code> for <code>cons</code>/<code>:</code> and <code>0</code> for <code>nil</code>/<code>[]</code>:</p> <pre><code>[1 , 2 , 3] == EXPAND THE SUGAR ==&gt; (1 : (2 : (3 : []))) == FOLD (+) 0 ==&gt; (1 + (2 + (3 + 0 ))) </code></pre><p>Notice how all the <code>:</code> were rewritten into the provided function (<code>(+)</code>), and <code>[]</code> was rewritten into the provided value (<code>0</code>). Also notice how everything groups to the right; this <code>fold</code> operator is consequently sometimes called <code>foldr</code>, since it <strong>fold</strong>s to the <strong>r</strong>ight.</p> <h2 id="universality">Universality</h2> <p>You can write <code>fold</code> as a recursive function:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-haskell" data-lang="haskell"><span style="color:#a6e22e">fold</span> f v <span style="color:#66d9ef">[]</span> <span style="color:#f92672">=</span> v <span style="color:#a6e22e">fold</span> f v (x <span style="color:#66d9ef">:</span> xs) <span style="color:#f92672">=</span> x `f` (fold f v xs) </code></pre></div><p>or:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-haskell" data-lang="haskell"><span style="color:#66d9ef">let</span> g <span style="color:#f92672">=</span> fold f v <span style="color:#66d9ef">in</span> (<span style="color:#f92672">*</span> <span style="color:#66d9ef">Equation</span> <span style="color:#ae81ff">1</span><span style="color:#66d9ef">:</span> <span style="color:#f92672">*</span>) g <span style="color:#66d9ef">[]</span> <span style="color:#f92672">=</span> v (<span style="color:#f92672">*</span> <span style="color:#66d9ef">Equation</span> <span style="color:#ae81ff">2</span><span style="color:#66d9ef">:</span> <span style="color:#f92672">*</span>) g (x <span style="color:#66d9ef">:</span> xs) <span style="color:#f92672">=</span> f x (g xs) </code></pre></div><p>If you can mash a function <code>g</code> into being defined by those two equations, then you&rsquo;ve solved for <code>f</code> and <code>v</code> such that <code>g = fold f v</code>, and you can write <code>g</code> without needing to explicitly recurse.</p> <h3 id="the-universal-property-of-fold">The Universal Property of Fold</h3> <p>The <strong>universal property of <code>fold</code></strong> is that you get a bi-implication between the two equations and <code>g = fold f v</code>:</p> <ul> <li>substitute in <code>fold f v</code> for <code>g</code>, and you&rsquo;ve defined <code>fold</code>;</li> <li>prove the two equations, and you&rsquo;ve got the makings of a proof by induction that your function is simply a fold.</li> </ul> <p>&ldquo;Taken as a whole, the universal property states that for finite lists the function <code>fold f v</code> is not just a solution to its defining equations, but in fact the <em>unique</em> solution&rdquo; (p. 358 in the journal, which is p. 4 in the PDF).</p> <h4 id="substitution-an-aide-mémoire">Substitution: An aide-mémoire</h4> <p>The substitution bit makes it easy to remember the universal property: Write the obvious recursive definition of fold, then substitute away the <code>fold f v</code>.</p> <h4 id="proofs-and-definitions">Proofs and Definitions</h4> <p>The other bit is the part you&rsquo;ll actually use. It lets you prove that a function is equivalent to a fold. To do so, you check that your function gives the same result in both the base case (equation 1) and the step case (equation 2). (For examples, check the section &ldquo;Universality as a proof principle&rdquo;.)</p> <p>One case where this is handy is when it&rsquo;s obvious what the fold does and less obvious what the other function does.</p> <p>An arguably more useful case is when you have 2 different functions you want to prove equal. Rather than writing out an induction proof longhand, you can rewrite them both as folds; if the two folds are identical, then they&rsquo;re equal.</p> <p>It&rsquo;s also useful when you want to rewrite a function you&rsquo;ve written using explicit recursion to instead be a fold. In this case, you end up setting up an equality between some function you hope is a fold and those equations, then solving for <code>v</code> and then <code>f</code> using equational reasoning. (For examples, check the section &ldquo;Universality as a definition principle&rdquo;.)</p> <h3 id="fusion-pushing-function-composition-inside-the-fold">Fusion: Pushing function composition inside the fold</h3> <p>You can also use the universal property to derive the <strong>fusion property</strong> of fold, which gives the requirements to be met to push a function applied to a fold into the fold itself: <code>h . fold g w =&gt; fold fusedFunction fusedValue</code>. (There&rsquo;s a subsection on that, too.)</p> <h4 id="fusion-as-optimization">Fusion as optimization</h4> <p>Fusion plays a small role in this paper, but it&rsquo;s very useful for practical reasons of program optimization. There&rsquo;s a good chunk of machinery in some functional language compilers dedicated to automatically fusing (&ldquo;deforesting&rdquo;) functions, since this reduces the overhead of communicating between parts of a program via intermediate values: with fusion, the values get consumed as they&rsquo;re produced, rather than accumulated into a data structure and then handed off for the next step of processing.</p> <h2 id="expressiveness">Expressiveness</h2> <h3 id="tuples-give-you-primitive-recursion">Tuples give you primitive recursion</h3> <p>Generating tuples lets you pull a fun trick where you rebuild the entire input in the second part of a 2-tuple and then throw it away. This is useful because it makes the original input available to the function you&rsquo;re folding.</p> <ul> <li>This lets you define <code>dropWhile p xs</code> as a fold.</li> <li>This lets you define a generic primitive recursive function as a fold. <ul> <li>Primitive recursion: <code>fold</code> is as powerful as any program you can write that has a bounded number of repetitions in any loop.</li> </ul> </li> </ul> <h3 id="first-class-functions-give-you-general-recursion">First-class functions give you general recursion</h3> <p>Generating functions pushes the expressiveness from primitive recursion to general recursion, as demonstrated by writing the Ackermann function (evidently the standard example of a non-primitive recursive function) as a <code>fold</code> (using lists of length <em>n</em> to stand in for the natural number <em>n</em>).</p> <ul> <li>General recursion: <code>fold</code> is as powerful as any program you can write. It can perform unbounded iteration. Enjoy your halting problem.</li> </ul> <h3 id="first-class-functions-give-you-foldl-via-foldr">First-class functions give you foldl via foldr</h3> <p>Generating functions also lets you write <code>foldl</code> – the function that turns <code>[1, 2, 3]</code> into <code>(((0 + 1) + 2) + 3)</code>, i.e., which folds everything down to the left – using <code>fold</code>. Turns out the order <code>fold</code> processes the list in comes down to the function it&rsquo;s provided.</p> <h4 id="foldr-is-more-powerful-than-foldl-because-strictness">Foldr is more powerful than foldl because strictness</h4> <p>You can&rsquo;t write <code>fold</code> in terms of <code>foldl</code> because <code>foldl</code> is strict in the tail of its list. What that means in practice is that <code>foldr (\a b -&gt; a) 0 [0..]</code> evaluates to 0 lickety-split, but <code>foldl (\b a -&gt; a) 0 [0..]</code> runs forever (&ldquo;diverges&rdquo;).</p> <h2 id="questions-i-hesitate-to-call-frequently-asked">Questions I hesitate to call &ldquo;frequently asked&rdquo;</h2> <p>I don&rsquo;t know that these are frequently asked, but they&rsquo;re definitely questions I asked myself at some point in the past.</p> <h3 id="why-is-it-called-fold">Why is it called &ldquo;fold&rdquo;?</h3> <p>I explain calling it a &ldquo;fold&rdquo; or &ldquo;reduction&rdquo; to myself by thinking of the example that started this post, which collapses (folds up) a list of values to a single value. In that case, the list <code>[1, 2, 3]</code> collapses down to the single value <code>6</code>.</p> <p>This explanation makes less sense when you&rsquo;re building a structured value, like say reversing a list, or doubling each element, so that <code>[1, 2, 3]</code> goes to <code>[1, 1, 2, 2, 3, 3]</code>, but let&rsquo;s just ignore that unfortunate fact.</p> <h3 id="whats-up-with-the-argument-order-for-f-in-foldr-vs-foldl">What&rsquo;s up with the argument order for f in foldr vs foldl?</h3> <p>The arguments to the function provided to a fold reflect the fold&rsquo;s left/right bias.</p> <p>Say you want to fold a list of strings down to the sum of each string&rsquo;s length.</p> <p>If you use <code>foldr</code>, then it has a right-to-left bias, so you need the function <code>\ string sum -&gt; (length string) + sum</code>. With <code>foldr</code>, the base case value appears on the right, and the iteration step value on the left.</p> <p><code>foldl</code>'s bias is for left-to-right, so its function has the base case value appearing on the left and the iteration step value on the right. With <code>foldl</code>, you&rsquo;d need the function <code>\ sum string -&gt; (length string) + sum</code>.</p> <p>This has the fun result that the order of arguments to the function is flipped depending on if you&rsquo;re dealing with a <code>foldl</code> or a <code>foldr</code>, as you can see with these side-by-side:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-haskell" data-lang="haskell"><span style="color:#a6e22e">sumLengthsRTL</span> <span style="color:#f92672">::</span> [<span style="color:#66d9ef">String</span>] <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">Int</span> <span style="color:#a6e22e">sumLengthsRTL</span> <span style="color:#f92672">=</span> foldr (<span style="color:#a6e22e">\</span> string sum <span style="color:#f92672">-&gt;</span> (length string) <span style="color:#f92672">+</span> sum) <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">sumLengthsLTR</span> <span style="color:#f92672">::</span> [<span style="color:#66d9ef">String</span>] <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">Int</span> <span style="color:#a6e22e">sumLengthsLTR</span> <span style="color:#f92672">=</span> foldl (<span style="color:#a6e22e">\</span> sum string <span style="color:#f92672">-&gt;</span> (length string) <span style="color:#f92672">+</span> sum) <span style="color:#ae81ff">0</span> </code></pre></div> Read: Towards native higher-order remote procedure calls https://jeremywsherman.com/blog/2015/08/10/read-towards-native-higher-order-remote-procedure-calls/ Mon, 10 Aug 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/08/10/read-towards-native-higher-order-remote-procedure-calls/ I do a lot of reading on my iPhone 6+. It&rsquo;s amazing what you can accomplish reading only a couple pages a day. I write notes in either Day One or Logsit, depending on how long the work is. They don&rsquo;t do anyone much good buried in there, so I&rsquo;m sharing my notes. Fredrikson, Olle, Dan R. Ghica, and Bertram Wheen. &ldquo;Towards native higher-order remote procedure calls.&rdquo; IFL 2014. 12 pages. <p><em>I do a lot of reading on my iPhone 6+. It&rsquo;s amazing what you can accomplish reading only a couple pages a day. I write notes in either Day One or Logsit, depending on how long the work is. They don&rsquo;t do anyone much good buried in there, so I&rsquo;m sharing my notes.</em></p> <p>Fredrikson, Olle, Dan R. Ghica, and Bertram Wheen. &ldquo;Towards native higher-order remote procedure calls.&rdquo; IFL 2014. 12 pages.</p> <p>Introduces Floskel, a Haskell-like call-by-value language with ADTs and location-aware expressions (<code>expr @ nodeN</code>). Compiles down to abstract machine byte code. Authors start with a CES machine (SECD without the dump) and then step gradually through adding a heap (CESH) then adding distributed communication (DCESH), first in a trivial single-node case and then in the general multiple-node, actually distributed case. They finish up with an async single-threaded network of machines with fail-stop fault tolerance via backups.</p> <p>Each step from one machine to the next advances by a bisimulation proof formalized in Agda. It&rsquo;s interesting to note that the details of the compilation from the surface syntax down to the bytecode are not formalized; in fact, they&rsquo;re considered so well-known and commonplace that they&rsquo;re entirely glossed over.</p> <p>The compilation assumes a fixed number of nodes. All code is also known in advance. Since all nodes have the same code, each can call a remote function on another node just by sending a pointer to the function. A node also knows its node number. (They implemented this by compiling to C and using MPI.)</p> <p>Benchmarks show it does well next to OCaml for the single-node case, and RPC overhead doesn&rsquo;t hurt much.</p> <p>New things I learned:</p> <ul> <li> <p>The two-level operational semantics for the network was new to me, though apparently it&rsquo;s bog standard in the literature:</p> <blockquote> <p>Both kinds of networks [synchronous and asynchronous] are modelled by two-level transition systems, which is common in operational semantics for concurrent and parallel languages. A <em>global level</em> describes the transitions of the system as a whole, and a <em>local level</em> the local transitions of the nodes in the system. (p. 6, &ldquo;3.3 Network models&rdquo;)</p> </blockquote> <p>This makes sense - it lets you glue together a bunch of independent machines with some over-arching system behavior, and that&rsquo;s roughly how you&rsquo;d expect to naïvely approach describing how a network of independent machines acts. But that&rsquo;s not something I&rsquo;d ever tried to do formally, so, new to me!</p> </li> <li> <p>They modeled the sync case via rendezvous, where both nodes must be ready to send/receive before the communication happens.</p> </li> <li> <p>They modeled the async case as having a pool of in-flight messages; receiving a message corresponded to fishing one out of the &ldquo;message soup&rdquo;.</p> </li> <li> <p>Lambda lifting has some concrete meaning to me now.</p> <blockquote> <p>For compilation, we require that the terms <code>t</code> in all location specification sub-terms <code>t @ i</code> are closed. Terms where this does not hold are transformed automatically using lambda lifting [25] (transform every sub-term <code>t @ i</code> to <code>tʹ = ((λ fv. t) @ i) (fv t)</code>). (p. 7 &ldquo;3.5 DCESH: The distributed CESH machine&rdquo;)</p> </blockquote> <p>Put into words, if you have an expression with free variables, lambda lifting is when you convert the expression to a function where all the formerly free variables are now arguments to the function. The function no longer has free variables, and implicit binding has been replaced by explicit binding by way of function application.</p> </li> <li> <p>I also learned what a bisimulation is: think &ldquo;isomorphism for labelled transition systems&rdquo;.</p> <ul> <li> <p>A subtle point made on Wikipedia is that a bisimulation is different than just being able to establish different simulations in both directions (x can sim y and y can sim x).</p> </li> <li> <p>I cribbed from Wikipedia for an intro to transition systems, too. A labelled transition system itself is a generalization of a finite state machine where the states aren&rsquo;t necessarily countable, the transitions aren&rsquo;t necessarily countable, and there&rsquo;s no notion of initial or accepting states.</p> <p>You can view it as an abstract rewriting system (like how you&rsquo;d graph out reduction of a lambda calculus expression), only the focus is on the transition labels (interpreted as actions/events) rather than the objects at either end of the transitions (like the classic &ldquo;do we reach a normal form&rdquo; question).</p> </li> </ul> </li> </ul> <p><em>Unfortunately, I&rsquo;d forgotten what some of those things referred to. iBooks&rsquo; delightful sync erased this paper from my phone&rsquo;s library, along with my bookmarks, at some point. Lucky thing this is such a short paper that I was able to locate the context easily, but I take it I should be quicker to write up my notes in future!</em></p> Listened: SE Radio: Joe Armstrong https://jeremywsherman.com/blog/2015/08/02/listened-se-radio-joe-armstrong/ Sun, 02 Aug 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/08/02/listened-se-radio-joe-armstrong/ <p>I listened today to an interview with Joe Armstrong recorded in late 2007. Joe Armstrong created Erlang. This interview took place right as interest in Erlang was rising due to the growth in generally available multicore machines.</p> <p>In case you&rsquo;d like to give it a listen as well, this was <a href="http://www.se-radio.net/2008/03/episode-89-joe-armstrong-on-erlang/">Software Engineering Radio episode 89.</a></p> <p>My takeaways:</p> <ul> <li>Concurrency and fault tolerance are the heart of Erlang; the two forces together drove it to the actor model. Its functional nature is an accident of its birth as a modified Prolog.</li> <li>OTP provides a wealth of tools and ready-to-go patterns for building fault-tolerant systems.</li> <li>Erlang happened, not while looking to gin up a new programming language, but while trying to solve a concrete problem with interesting constraints: How can we make it easier to write telephone switching software that can run for ages and never go down?</li> </ul> <p>From memory, here are my more comprehensive (and correspondingly more rambling) notes:</p> <p>I listened today to an interview with Joe Armstrong recorded in late 2007. Joe Armstrong created Erlang. This interview took place right as interest in Erlang was rising due to the growth in generally available multicore machines.</p> <p>In case you&rsquo;d like to give it a listen as well, this was <a href="http://www.se-radio.net/2008/03/episode-89-joe-armstrong-on-erlang/">Software Engineering Radio episode 89.</a></p> <p>My takeaways:</p> <ul> <li>Concurrency and fault tolerance are the heart of Erlang; the two forces together drove it to the actor model. Its functional nature is an accident of its birth as a modified Prolog.</li> <li>OTP provides a wealth of tools and ready-to-go patterns for building fault-tolerant systems.</li> <li>Erlang happened, not while looking to gin up a new programming language, but while trying to solve a concrete problem with interesting constraints: How can we make it easier to write telephone switching software that can run for ages and never go down?</li> </ul> <p>From memory, here are my more comprehensive (and correspondingly more rambling) notes:</p> <ul> <li> <p>Reddit gave the impression at the time that Haskell and Erlang were fighting for the future. Eight years later, and the future still isn&rsquo;t quite here, or Reddit was as poor a predictor of general behavior then as now. ;)</p> </li> <li> <p>Erlang arose almost accidentally out of solving problems inherent in plain old telephone service (POTS) switching:</p> <ul> <li>You have a ton of different people connected at once, all doing their own thing. Having lightweight processes makes this easy to model.</li> <li>You can&rsquo;t take the system down. Ever. So you need hot-swapping.</li> <li>You can&rsquo;t let the system go down. Ever. So you need fault-tolerance.</li> </ul> </li> <li> <p>Erlang came out of modifying Prolog to have lightweight processes. This was alongside a number of other researchers implementing POTS switching logic in whatever languages they could get their hands on to run on the random VAX UNIX machine they had in the office: Ada, Concurrent Euclid, Smalltalk, ML, C, and Armstrong with Prolog.</p> </li> <li> <p>The driving forces behind Erlang were concurrency and fault tolerance. Its actor approach to concurrency was intentional; the functional nature of sequential Erlang, though, is an accident of growing up inside a Prolog (where you can&rsquo;t unify a term first to one thing and then to something unequal) rather than inside C. Nothing in the actor model would prohibit an actor performing local mutation of bindings.</p> <ul> <li>Armstrong points out that assign-once variables are very handy for debugging, though: If you find a bogus value in a variable, then there&rsquo;s exactly one place in your program where it was bound to that bogus value, vs one of several places where it might have been updated if multiple assignments to the same name were allowed.</li> </ul> </li> <li> <p>The &ldquo;let it crash&rdquo; ethos is a response to specifications&rsquo; refusal to spec anything but the happy path. Instead of ad-libbing some rubbish error handling that dwarfs the code you need to write, and likely introduces some errors of its own, you just don&rsquo;t even bother handling junk input. The worker process dies, its supervisor respawns it, and a report of its abend is written to an error log.</p> <ul> <li>You can later review the error log to decide if you want to introduce logic to handle an error case that keeps cropping up, or if it&rsquo;s so rare that abending and being reborn is still the right approach.</li> </ul> </li> <li> <p>The ability to actually distribute Erlang across machines came very late. Everyone involved had the sense it would be pretty easy based on how the rest of the system had been implemented, so no-one actually bothered getting around to doing it for a long time.</p> </li> <li> <p>Erlang uses its host OS to manage files and sockets and fork over some memory for the Erlang runtime to manage.</p> <ul> <li>Erlang&rsquo;s runtime system is basically an application container or operating system in and of itself.</li> <li>OTP is a framework (like Rails), but rather than being a framework for writing web sites, it&rsquo;s a framework for writing fault-tolerant systems. <ul> <li>Notice how &ldquo;fault tolerance&rdquo; keeps coming to the fore!</li> </ul> </li> </ul> </li> <li> <p>Armstrong contrasts shared memory concurrency (a headache) to message passing concurrency (not so bad). He sees Erlang&rsquo;s greatest selling point as being easy, painless concurrency.</p> <ul> <li>Software transactional memory doesn&rsquo;t enter into this discussion. Was that even a thing in late 2007? Yeah, in Haskell-land, Harris et al. introduced &ldquo;Composable Memory Transactions&rdquo; in PPoPP&rsquo;05, with a couple other papers following in 2006 per <a href="https://wiki.haskell.org/Research_papers/Parallelism_and_concurrency#Lock_free_data_structures_and_transactional_memory">the Haskell wiki</a>.)</li> <li>The message passing approach also arises out of the fault tolerance requirements. If machine A crashes, machine B has to have a copy of all needed data to continue, so A needs to send over a copy rather than just a pointer to its own memory.</li> </ul> </li> <li> <p>Armstrong reckons it unlikely that Akka can get as good of performance as Erlang, as it is unable to alter its abstract machine language to make the core actor model implementation primitives actually performant primitives, vs emulating these using an ill-suited abstract machine:</p> <ul> <li>Switch context from one actor to another</li> <li>Send a message</li> <li>Spawn a new process</li> </ul> </li> <li> <p>Armstrong&rsquo;s last remarks were on <a href="http://www.erlang.org/doc/programming_examples/bit_syntax.html">Erlang&rsquo;s bit-matching DSL.</a> He also asks a good question: Why do regex always work at the byte, rather than bit, level? Why are there no good bit-level regex libraries?</p> <ul> <li>I suppose the answer is likely that, unless you are doing systems programming, you don&rsquo;t much need them, and systems programmers raised on C are unlikely to turn to a regex library to do bit-smashing.</li> </ul> </li> </ul> Using rbenv with fish https://jeremywsherman.com/blog/2015/07/28/using-rbenv-with-fish/ Tue, 28 Jul 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/07/28/using-rbenv-with-fish/ <p>I switched from zsh to <a href="http://fishshell.com/">fish shell</a> a month or so ago. I lost bang-history (no more <code>!?gi</code>) and gained a shell small enough to understand and write scripts for without fearing I&rsquo;m going to step into some gotcha from the 1970s. No more shell-as-quirks-mode!</p> <p>There&rsquo;s a downside to shifting to a non-POSIX shell, though: scripts intended to modify the shell environment itself no longer Just Work.</p> <p>This tripped me up in one case: <code>rbenv</code>, the Ruby environment and version manager.</p> <p><code>rbenv</code> expects you to run the output of <code>rbenv init</code> in your shell. This fixes up your <code>PATH</code>, rebuilds <code>rbenv</code>'s sense of the world, and lastly redefines rbenv as a dispatching function. <code>rbenv</code> provides a few different flavors of script, but none is for fish.</p> <p>No problem! Let&rsquo;s rewrite this script for fish.</p> <p>I switched from zsh to <a href="http://fishshell.com/">fish shell</a> a month or so ago. I lost bang-history (no more <code>!?gi</code>) and gained a shell small enough to understand and write scripts for without fearing I&rsquo;m going to step into some gotcha from the 1970s. No more shell-as-quirks-mode!</p> <p>There&rsquo;s a downside to shifting to a non-POSIX shell, though: scripts intended to modify the shell environment itself no longer Just Work.</p> <p>This tripped me up in one case: <code>rbenv</code>, the Ruby environment and version manager.</p> <p><code>rbenv</code> expects you to run the output of <code>rbenv init</code> in your shell. This fixes up your <code>PATH</code>, rebuilds <code>rbenv</code>'s sense of the world, and lastly redefines rbenv as a dispatching function. <code>rbenv</code> provides a few different flavors of script, but none is for fish.</p> <p>No problem! Let&rsquo;s rewrite this script for fish.</p> <p>When you run <code>rbenv init</code>, it dumps out a call to eval:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh"><span style="color:#75715e"># Load rbenv automatically by adding</span> <span style="color:#75715e"># the following to your profile:</span> eval <span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>rbenv init -<span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span> </code></pre></div><p>When you run <em>that</em> bit of code, you see something like:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">export PATH<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;/Users/jeremy/.rbenv/shims:</span><span style="color:#e6db74">${</span>PATH<span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span> rbenv rehash 2&gt;/dev/null rbenv<span style="color:#f92672">()</span> <span style="color:#f92672">{</span> typeset command command<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span>$1<span style="color:#e6db74">&#34;</span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">[</span> <span style="color:#e6db74">&#34;</span>$#<span style="color:#e6db74">&#34;</span> -gt <span style="color:#ae81ff">0</span> <span style="color:#f92672">]</span>; <span style="color:#66d9ef">then</span> shift <span style="color:#66d9ef">fi</span> <span style="color:#66d9ef">case</span> <span style="color:#e6db74">&#34;</span>$command<span style="color:#e6db74">&#34;</span> in rehash|shell<span style="color:#f92672">)</span> eval <span style="color:#e6db74">`</span>rbenv <span style="color:#e6db74">&#34;sh-</span>$command<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;</span>$@<span style="color:#e6db74">&#34;</span><span style="color:#e6db74">`</span>;; *<span style="color:#f92672">)</span> command rbenv <span style="color:#e6db74">&#34;</span>$command<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#34;</span>$@<span style="color:#e6db74">&#34;</span>;; <span style="color:#66d9ef">esac</span> <span style="color:#f92672">}</span> </code></pre></div><p>Translating this to fish is a good introduction to scripting fish. Pop open <code>fish help</code> in a browser tab, and lean on <code>functions</code> to look at how the functions provided with the shell are coded.</p> <p>With a bit of that, I ended up with:</p> <pre><code>set PATH &quot;$HOME/.rbenv/shims&quot; $PATH rbenv rehash ^/dev/null function rbenv set -l command $argv[1] if test (count $argv) -gt 1 set argv $argv[2..-1] end switch &quot;$command&quot; case rehash shell eval (rbenv &quot;sh-$command&quot; $argv) case '*' command rbenv &quot;$command&quot; $argv end end </code></pre><p>I bet there&rsquo;s a fishier way to do this, but it&rsquo;s working fine for me. If you&rsquo;ve been considering adopting fish as your shell but ran into <code>rbenv</code> as a blocker, this should get you past that. Enjoy!</p> Dodging State https://jeremywsherman.com/blog/2015/07/15/dodging-state/ Wed, 15 Jul 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/07/15/dodging-state/ Soroush Khanlou gives some concrete tips for reducing the amount of state in your code. I&rsquo;m going to rephrase his advice into terms that connect better with other reading I&rsquo;ve been doing: When you recognize cohesive groups of properties, bud off a new object. Prefer domain types to primitive types. Take advantage of computed properties to express derived state; reintroduce caching only when unavoidable. Budding Off Sometimes you notice you&rsquo;re manipulating the same bits of state together, say properties dollars and cents of a Product. <p><a href="http://khanlou.com/2015/07/state-negotiations/">Soroush Khanlou gives some concrete tips</a> for reducing the amount of state in your code. I&rsquo;m going to rephrase his advice into terms that connect better with other reading I&rsquo;ve been doing:</p> <ul> <li>When you recognize cohesive groups of properties, bud off a new object.</li> <li>Prefer domain types to primitive types.</li> <li>Take advantage of computed properties to express derived state; reintroduce caching only when unavoidable.</li> </ul> <h2 id="budding-off">Budding Off</h2> <p>Sometimes you notice you&rsquo;re manipulating the same bits of state together, say properties <code>dollars</code> and <code>cents</code> of a <code>Product</code>. Well, viewed another way, maybe you&rsquo;re actually just inlining the methods of a latent type; <code>dollars + cents</code> might be the answer to the <code>total</code> message of a not yet existent <code>Price</code> class.</p> <p>You don&rsquo;t have to end up here by an act of deep vision; you can work you way up to it by way of doing some extract method refactorings in extant classes, then noting that you have a coherent group of methods and state, and then just bundling those up and schlepping them off to a new class. Budding complete!</p> <h2 id="domain-types">Domain Types</h2> <p>Whatever your code is doing, it&rsquo;s very likely not natural to talk about at the level of bools and ints. You might instead have something that&rsquo;s on or off, or red or green; something that&rsquo;s big, medium, or small, or maybe you have some money rather than just a decimal value.</p> <p>Using types native to the domain of discourse that you&rsquo;re modeling and automating raises the level of abstraction of your code. It also raises the profile of these ideas in your language. If you fail to name these types, you risk ending up with bits of functionality relating to them scattered across the code that needs to work with it.</p> <p>Naming primitives – even if to start with it&rsquo;s little more than a wrapper around a single primitive value, like <code>struct Name { let fullName: String }</code> – makes the the meaning of the primitive value explicit and provides a foothold from which you can drag further functionality into existence in an appropriate context.</p> <h2 id="derived-state">Derived State</h2> <p>You can split program state into two classes:</p> <ul> <li>Essential state: Information that you have to have.</li> <li>Derived state: Information that can be computed from other information you have.</li> </ul> <p>In theory, you should be able to get by holding onto only the essential state and computing the rest as needed via functions/methods/computed properties. In practice, you might need to throw a caching layer (like a stored property) in there more often than you&rsquo;d like, but let your profiler be your guide.</p> <p>(What about the stuff in your program that&rsquo;s not state? For a further teardown, you should check out <a href="http://shaffner.us/cs/papers/tarpit.pdf">Mosely &amp; Marks&rsquo; &ldquo;Out of the Tar Pit&rdquo;</a>, or at least skim <a href="http://blog.acolyer.org/2015/03/20/out-of-the-tar-pit/">Adrian Colyer&rsquo;s survey of it</a>.)</p> <h2 id="conclusion">Conclusion</h2> <p>Soroush draws these conclusions as a result of approaching code from a functional programming standpoint, but that&rsquo;s an accidental path dependency. Object-oriented design principles, as elaborated for example in <a href="http://amzn.to/1DhLpRW">Practical Object-Oriented Design in Ruby</a>, provide plenty of motivation to reduce state and better name and organize what&rsquo;s left of it.</p> <p>Now, go check out <a href="http://khanlou.com/2015/07/state-negotiations/">Soroush&rsquo;s article</a>, particularly the cool bits about state machines – then wonder how you might use discriminated unions (AKA enums) in Swift to tackle the same problem.</p> Should DRY entail call-by-need evaluation? https://jeremywsherman.com/blog/2015/07/14/should-dry-entail-call-by-need-evaluation/ Tue, 14 Jul 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/07/14/should-dry-entail-call-by-need-evaluation/ Swift brings Cocoa devs a standard library that includes map, filter, and reduce. But in a strict language such as Swift, you&rsquo;ll likely find yourself hand-fusing a composition of these operations at some point as a result of profiling-directed performance optimization. This isn&rsquo;t new to Swift. Lennart Augustsson observed in 2011 in More points for lazy evaluation: Strict evaluation is fundamentally flawed for function reuse. That&rsquo;s because: With strict evaluation you can no longer with a straight face tell people: don&rsquo;t use recursion, reuse the recursion patterns in map, filter, foldr, etc. <p>Swift brings Cocoa devs a standard library that includes <code>map</code>, <code>filter</code>, and <code>reduce</code>. But in a strict language such as Swift, you&rsquo;ll likely find yourself hand-fusing a composition of these operations at some point as a result of profiling-directed performance optimization.</p> <p>This isn&rsquo;t new to Swift. Lennart Augustsson observed in 2011 in <a href="http://augustss.blogspot.co.uk/2011/05/more-points-for-lazy-evaluation-in.html?m=1">More points for lazy evaluation</a>:</p> <blockquote> <p><em>Strict evaluation is fundamentally flawed for function reuse.</em></p> </blockquote> <p>That&rsquo;s because:</p> <blockquote> <p>With strict evaluation you can no longer with a straight face tell people: don&rsquo;t use recursion, reuse the recursion patterns in map, filter, foldr, etc. It simply doesn&rsquo;t work (in general).</p> </blockquote> <p>The strict evaluation order means naïve combination of functions will do more work than needed to produce a value: for example, in the case of <code>any</code>, which should short-circuit, <code>reduce</code> must still traverse its entire input.</p> <blockquote> <p>Using macros doesn&rsquo;t really save us this time, because of the recursive definitions. I don&rsquo;t really know of any way to fix this problem short of making all (most?) functions lazy, because the problem is pervasive. I.e., in the example it would not be enough to fix <code>foldr</code> [AKA <code>reduce</code>]; all the functions involved need to be lazy to get the desired semantics.</p> </blockquote> <p>Strict evaluation gets us a simple space-usage model, but it leaves us holding the bag when it comes to function composition.</p> CRC Cards https://jeremywsherman.com/blog/2015/07/10/crc-cards/ Fri, 10 Jul 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/07/10/crc-cards/ File under “old technique, still useful”: CRC cards The core elements of an object system are an object, its responsibilities, and its collaborators. With just that info, you can start walking through scenarios. You&rsquo;ll note this doesn&rsquo;t say anything about inheritance or protoypal delegation or whatever; that&rsquo;s intentional: The cards are being used as props to aid the telling of a story of computation. The cards allow its telling without recourse to programming language syntax or idiom. <p>File under “old technique, still useful”: <a href="http://c2.com/doc/oopsla89/paper.html">CRC cards</a></p> <p>The core elements of an object system are an object, its responsibilities, and its collaborators. With just that info, you can start walking through scenarios.</p> <p>You&rsquo;ll note this doesn&rsquo;t say anything about inheritance or protoypal delegation or whatever; that&rsquo;s intentional:</p> <blockquote> <p>The cards are being used as props to aid the telling of a story of computation. The cards allow its telling without recourse to programming language syntax or idiom.</p> </blockquote> <p>Being able to pick up and hold a card, and thereby signal to yourself that you&rsquo;ve become that actor, helps with getting in the object thinking groove:</p> <blockquote> <p>We were surprised at the value of physically moving the cards around. When learners pick up an object they seem to more readily identify with it, and are prepared to deal with the remainder of the design from its perspective. It is the value of this physical interaction that has led us to resist a computerization of the cards.</p> </blockquote> <p>In the modern HeaderDocful world, it&rsquo;s useful to write the responsibilities and collaborators bits right into the class-level doc comment.</p> <p>The &ldquo;collaborators&rdquo; bit is especially valuable for hinting at dynamic runtime context that can be hard to infer from static source text. The same info is handy in method docs (&ldquo;called by X when Y happens&rdquo;).</p> Using Swift Throws with Completion Callbacks https://jeremywsherman.com/blog/2015/06/17/using-swift-throws-with-completion-callbacks/ Wed, 17 Jun 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/06/17/using-swift-throws-with-completion-callbacks/ Swift 2 introduced the notion of throwing and propagating NSError values. It works pretty well in a linear, synchronous workflow, but at first glance, it doesn&rsquo;t appear to address the common case of completion callbacks. Consider NSURLSession.dataTaskWithURL(_:completionHandler:). Swift 2 bridges this in like so: func dataTaskWithURL(url: NSURL, completionHandler: (NSData?, NSURLResponse?, NSError?) -&gt; Void) -&gt; NSURLSessionDataTask? Note how, in the completion handler closure, you still have to do Ye Olde Check Data Then Check Error dance. <p>Swift 2 introduced the notion of throwing and propagating <code>NSError</code> values.</p> <p>It works pretty well in a linear, synchronous workflow, but at first glance, it doesn&rsquo;t appear to address the common case of completion callbacks.</p> <p>Consider <code>NSURLSession.dataTaskWithURL(_:completionHandler:)</code>. Swift 2 bridges this in like so:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">dataTaskWithURL</span>(url: NSURL, completionHandler: (NSData?, NSURLResponse?, NSError?) -&gt; Void) -&gt; NSURLSessionDataTask? </code></pre></div><p>Note how, in the completion handler closure, you still have to do Ye Olde Check Data Then Check Error dance. Yawn.</p> <p>There&rsquo;s a straightforward way to transform this into <code>throws</code>-land, though. Just think: What sort of thing can throw? A function call.</p> <p>So, let&rsquo;s use our functions, and rewrite this to:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift"><span style="color:#66d9ef">typealias</span> DataTaskResult = () <span style="color:#66d9ef">throws</span> -&gt; (NSData, NSURLResponse) <span style="color:#66d9ef">func</span> <span style="color:#a6e22e">dataTaskWithURL</span>(url: NSURL, completionHandler: DataTaskResult -&gt; Void) -&gt; NSURLSessionDataTask? </code></pre></div><p>The completion handler is not marked as <code>@rethrows</code>, so it has to handle any error. Extracting the result or error is then done in the completion handler like so:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-swift" data-lang="swift">{ result: DataTaskResult <span style="color:#66d9ef">in</span> <span style="color:#66d9ef">do</span> { <span style="color:#66d9ef">let</span> data, response = <span style="color:#66d9ef">try</span> result() <span style="color:#75715e">/* work with data and response */</span> } <span style="color:#66d9ef">catch</span> { <span style="color:#75715e">/* you got yourself an error! */</span> } } </code></pre></div><p>This straightforward transformation preserves Swift 2&rsquo;s <a href="http://martinfowler.com/bliki/DirectingAttitude.html">directing attitude</a> towards error-handling, while freeing users from having to remember the protocol for working with <code>NSError</code>s.</p> <p>It&rsquo;s unfortunate we can&rsquo;t ourselves apply this to Apple&rsquo;s code. We&rsquo;ll just have to continue to type through the error-prone, manual procedure for working with <code>NSError</code>s when working with their APIs.</p> <p>We needn&rsquo;t continue to do so with our own, though: if you&rsquo;re going to adopt <code>throws</code>, go whole-hog, and <code>throw</code>ify your entire API, both synchronous and asynchronous.</p> Functor &amp; Friends: Protocol + Tests https://jeremywsherman.com/blog/2015/03/08/functor-amp-friends-protocol-tests/ Sun, 08 Mar 2015 00:00:00 +0000 https://jeremywsherman.com/blog/2015/03/08/functor-amp-friends-protocol-tests/ I&rsquo;ve read articles that try to reduce the academic flim-flammery of functors, monads, and similar to concrete syntax by just presenting them as a series of interfaces, or protocols, that must be implemented. This is reassuring: It turns something unfamiliar into something familiar, if not downright mundane. Unfortunately, reducing these abstractions to protocols alone oversimplifies them and reduces their practical utility tremendously. What gives functional programming abstractions their oomph is the properties satisfied by the abstraction, not the specific API. <!-- Inspired by http://www.javiersoto.me/post/106875422394 and https://alpha.app.net/jws/post/47188013#47163273. --> <p>I&rsquo;ve read articles that try to reduce the academic flim-flammery of functors, monads, <a href="http://typeclassopedia.bitbucket.org/#slide-14">and similar</a> to concrete syntax by just presenting them as a series of interfaces, or protocols, that must be implemented.</p> <p>This is reassuring: It turns something unfamiliar into something familiar, if not downright mundane. Unfortunately, reducing these abstractions to protocols alone oversimplifies them and reduces their practical utility tremendously.</p> <p>What gives functional programming abstractions their oomph is the properties satisfied by the abstraction, not the specific API. These properties provide essential guarantees about the implementation of that API. Using these properties lets us reason about our code without getting bogged down in details of the implementation: The dream of meaningful abstraction lives on in FP.</p> <p>Protocols alone are not powerful enough to specify this. Fortunately, there is a way for object-oriented programmers to return the foreign concept of &ldquo;mathematical abstraction&rdquo; to a comfortable familiarity without losing reasoning power. This is by recasting the abstraction in terms of protocols <strong>AND tests.</strong> The tests specify properties that implementations of those protocols must satisfy in order to truly conform to the protocol.</p> <p>Together, protocols and tests capture the essence of functional abstractions in a way that an OO programmer can immediately be productive with.</p> <p>(It&rsquo;s an interesting observation that, as in TDD, these tests have more value than the protocols themselves.)</p> <h2 id="example-functor">Example: Functor</h2> <p>You might see Functor explained like this:</p> <ul> <li>Functor is a parameterized type with a <code>map</code> function, where</li> <li>The map function takes an instance of the type and applies a function from its parameterized type to another instance with a potentially different type parameter.</li> </ul> <p>Or, more concisely, in pseudo-Swift:</p> <pre><code>map(container: Container&lt;Type&gt;, function: Type -&gt; PotentiallyOtherType) -&gt; Container&lt;PotentiallyOtherType&gt; </code></pre><h3 id="the-protocol-fails-us-permuting-map">The protocol fails us: permuting map</h3> <p>A protocol captures only the superficial elements of Functorhood. For example, this implementation of map on an array satisfies the protocol:</p> <pre><code>func map&lt;T, U&gt;(a: [T], f: T -&gt; U) -&gt; [U] { let indexes = allIndexes(a) let permutedIndexes = permute(indexes) var b: [U] = [] for i in permutedIndexes { let element = a[i] let mappedValue = f(element) b.append(mappedValue) } return b } </code></pre><p>But it fails to be a Functor.</p> <p>Try it for yourself: Mapping the identity function repeatedly (<code>map([1, 2, 3, 4], { $0 })</code>) can give results that differ from the input array. (For the identity function, those results will all be permutations of the input array.)</p> <p>That makes for one frustrating faux-functor!</p> <h3 id="tests-to-the-rescue">Tests to the rescue!</h3> <p>This is why it is not enough to have a function with a certain type signature. The heart of the beast is the set of sanity-preserving properties it&rsquo;s required to conform to.</p> <p>These properties go by the name of the functor laws; you&rsquo;ll find equivalent laws for monads and similar abstractions. It&rsquo;s these laws that make the abstraction meaningful and tractable. Preserving these laws makes the interface a true abstraction, rather than (in the case of functor) simply a generalization of a common imperative programming pattern.</p> <p>Specifically, a functor must preserve identity and composition:</p> <ul> <li><strong>Identity Preservation:</strong> <code>XCTAssertEqual(id(x), x.map(id))</code></li> <li><strong>Composition Preservation:</strong> <code>XCTAssertEqual(x.map(f).map(g), x.map({ g(f($0)) }))</code></li> </ul> <p>(Unlike the protocol above, here I&rsquo;m using method syntax, because that composes a bit more readably.)</p> <p>The XCTest macros aren&rsquo;t quite up to the task of proving general properties. With something like <a href="https://github.com/typelift/SwiftCheck">SwiftCheck</a>, we can get close, though.</p> <h2 id="conclusion">Conclusion</h2> <p>To reiterate:</p> <ul> <li>It&rsquo;s <strong>not the protocol</strong> that makes the functor;</li> <li>It&rsquo;s <strong>not &ldquo;you can write an [f]map function&rdquo;;</strong></li> <li>Instead, it is specific <strong>properties</strong> exhibited by the compound entity of type + functions.</li> <li>Concretely: It is <strong>protocol + tests.</strong></li> </ul> No Single Swift Style https://jeremywsherman.com/blog/2014/10/24/no-single-swift-style/ Fri, 24 Oct 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/10/24/no-single-swift-style/ <p>&ldquo;Swift is too young for us to say what good Swift style is, or to have developed a sense for idiomatic Swift.&rdquo;</p> <p>Talks, blogs, and books repeat this sentiment or variations.</p> <p>This statement assumes that there will eventually develop a single, canonical Swift style.</p> <p>This is a mistake.</p> <p>&ldquo;Swift is too young for us to say what good Swift style is, or to have developed a sense for idiomatic Swift.&rdquo;</p> <p>Talks, blogs, and books repeat this sentiment or variations.</p> <p>This statement assumes that there will eventually develop a single, canonical Swift style.</p> <p>This is a mistake.</p> <p>I expect no single style will develop; instead, several different styles will flourish. Perhaps most developers will be conversant with multiple styles. More likely, these styles will develop into several mutually unintelligible dialects.</p> <p>What might these styles be?</p> <ul> <li> <p><strong>Obj-C with different syntax:</strong> Existing frameworks exert a strong pull in this direction. Frankly, it feels awkward to me in Swift, but I expect a lot of code to be written this way.</p> </li> <li> <p><strong>OOP:</strong> A Swift-flavored OOP dialect will likely emerge from the ashes of Obj-C. Many are comfortable with OOP, and they&rsquo;ll stick with it, using Swift as their new, less brackety vehicle.</p> </li> <li> <p><strong>Generic Programming:</strong> Swift&rsquo;s small standard library exemplifies this, though it compromises in places by having both a generic function and an instance method, rather than picking a single one. I expect the latter is often there to assist discoverability.</p> </li> <li> <p><strong>Functional Programming:</strong> There has been a lot of interest in the dataflow aspects of functional programming across computing lately. Swift makes no exception.</p> <p>In addition, Swift&rsquo;s type system makes it possible to borrow at least some of the approaches pioneered by the ML family of languages, though its lack of support for higher-kinded types and incomplete support for variably-sized enum types can make this awkward to express. With a solid Prelude library to provide basic tools missing from Swift&rsquo;s standard library, FP in Swift will be a very real possibility. Higher-level FP programming will likely be no less foreign to the other dialects in a couple years than it is today; reliance on FP-specific libraries will not help.</p> </li> </ul> <p>Swift&rsquo;s styles are being built today by the snippets and libraries that will constitute their core vocabulary.</p> <p>Some idioms will be common to several styles; some will be unique to a style; some of those will be inapplicable, even inexpressible, in other styles.</p> <p><strong>Swift isn&rsquo;t too young to have a style: It&rsquo;s just too big to be confined to having a single style.</strong></p> Radar tip: Shell one-liners to dump configuration info https://jeremywsherman.com/blog/2014/07/25/radar-tip-shell-one-liners-to-dump-configuration-info/ Fri, 25 Jul 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/07/25/radar-tip-shell-one-liners-to-dump-configuration-info/ With the current Xcode and Swift betas, I find myself needing to paste in the versions of my OS, Xcode, and Swift every time I file a new Radar. So I use these handy aliases: alias rdrconf=&#39;{ xcodebuild -version; echo; sw_vers; }&#39; alias swfconf=&#39;{ xcodebuild -version; echo; sw_vers; echo; xcrun swift --version; }&#39; The echo bits are there to put a blank line between the different lists of versions. The output looks like this: <p>With the current Xcode and Swift betas, I find myself needing to paste in the versions of my OS, Xcode, and Swift every time I file a new Radar.</p> <p>So I use these handy aliases:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">alias rdrconf<span style="color:#f92672">=</span><span style="color:#e6db74">&#39;{ xcodebuild -version; echo; sw_vers; }&#39;</span> alias swfconf<span style="color:#f92672">=</span><span style="color:#e6db74">&#39;{ xcodebuild -version; echo; sw_vers; echo; xcrun swift --version; }&#39;</span> </code></pre></div><p>The <code>echo</code> bits are there to put a blank line between the different lists of versions. The output looks like this:</p> <pre><code>% swfconf Xcode 6.0 Build version 6A267n ProductName: Mac OS X ProductVersion: 10.9.4 BuildVersion: 13E28 Swift version 1.0 (swift-600.0.41.2.2) Target: x86_64-apple-darwin13.3.0 </code></pre><p>To copy that into my browser, I do:</p> <pre><code>% swfconf | pbcopy </code></pre><p>followed by a Cmd-v into the form field.</p> Pervasive use of Optional in Swift is penance for nil https://jeremywsherman.com/blog/2014/07/09/pervasive-use-of-optional-in-swift-is-penance-for-nil/ Wed, 09 Jul 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/07/09/pervasive-use-of-optional-in-swift-is-penance-for-nil/ If you&rsquo;ve looked to do anything significant with Swift, you&rsquo;ve likely had to fall back on our old friend, Foundation, and likely also some newer friends in the form of other core Apple frameworks. One thing you cannot miss with this legacy APIs is the pervasive use of optional types. Thanks to them, we still get to angst about the billion-dollar mistake of nil, only now we get to pay a steady syntax tax throughout our codebase. <p>If you&rsquo;ve looked to do anything significant with Swift, you&rsquo;ve likely had to fall back on our old friend, Foundation, and likely also some newer friends in the form of other core Apple frameworks.</p> <p>One thing you cannot miss with this legacy APIs is the pervasive use of optional types.</p> <p>Thanks to them, we still get to angst about the billion-dollar mistake of <code>nil</code>, only now we get to pay a steady syntax tax throughout our codebase.</p> <p>It is in light of this that I read a snatch of <a href="http://existentialtype.wordpress.com/2014/04/21/bellman-confirms-a-suspicion/#comment-1390">a comment by Dr. Robert Harper</a> with some interest:</p> <blockquote> <p>Your emphasis on whether the nth argument to that function/method is an option or not does not do justice to the real issue, which is what Yaron Minsky calls making undefined states unrepresentable (or words to that effect).</p> <p>[…]</p> <p>It is of course possible to do C-like or “pythonic” programming in ML using options instead of the “null pointer”, but that’s not the way to write good code […]. What you want is</p> <ul> <li>types to express the correlations between components, these are called sums, and</li> <li>pattern matching to explicitly match the cases that are legal and allow the exhaustiveness checker to warn you when you’ve missed a case, either by mistake or by design or as a consequence of evolution of the code.</li> </ul> </blockquote> <p>I am afraid we know for a fact that legacy Objective-C APIs are not up to this challenge. Surprise exceptions and undocumented behavior on <code>nil</code> input will be with us for some time.</p> <p>It is an open question whether Swift-native APIs will be written to support and lead us by example to that latter style of programming.</p> <p>The way <a href="https://itun.es/us/jEUH0.l">the Swift book</a> punts in the face of a data source protocol with two options for providing data is not encouraging to this end:</p> <blockquote> <p>NOTE: Strictly speaking, you can write a custom class that conforms to <code>CounterDataSource</code> without implementing <em>either</em> protocol requirement. They are both optional, after all. Although technically allowed, this wouldn’t make for a very good data source. (&ldquo;Optional Protocol Requirements&rdquo;)</p> </blockquote> <p>Such an aside would have been the perfect time, not to shrug out, &ldquo;oh, they are both optional, too sad, but what are we to do?&quot;, but instead to explain how an enumeration could be used to admit a delegate implementing precisely one or the other but not none and not both.</p> Intuition behind the Swift external/local parameter system https://jeremywsherman.com/blog/2014/06/05/intuition-behind-the-swift-external/local-parameter-system/ Thu, 05 Jun 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/06/05/intuition-behind-the-swift-external/local-parameter-system/ <p>David Bryant Copeland <a href="http://www.naildrivin5.com/blog/2014/06/04/what-swift-tells-use-about-programming-language-trends.html">picks out</a> Swift&rsquo;s external/local parameter system as something never before seen:</p> <blockquote> <p>The notion of giving named parameters different names for the caller than are used in the implementation is not something I’ve seen before, and it’s kinda genius.</p> </blockquote> <p>But further reflection convinced me that allowing different external and local parameter names is simply the Swift version of a common Objective-C practice.</p> <p>David Bryant Copeland <a href="http://www.naildrivin5.com/blog/2014/06/04/what-swift-tells-use-about-programming-language-trends.html">picks out</a> Swift&rsquo;s external/local parameter system as something never before seen:</p> <blockquote> <p>The notion of giving named parameters different names for the caller than are used in the implementation is not something I’ve seen before, and it’s kinda genius.</p> </blockquote> <p>But further reflection convinced me that allowing different external and local parameter names is simply the Swift version of a common Objective-C practice.</p> <h3 id="external-and-local-parameters-in-obj-c">External and Local Parameters in Obj-C</h3> <p>Consider these parallel Obj-C and Swift method declarations:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-objc" data-lang="objc">- (<span style="color:#66d9ef">void</span>)<span style="color:#a6e22e">insertPerson:</span>(Person <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span>)p <span style="color:#a6e22e">atIndex:</span>(<span style="color:#66d9ef">const</span> NSUInteger)i; </code></pre></div><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">insert</span>(<span style="color:#a6e22e">person</span> <span style="color:#a6e22e">p</span>: <span style="color:#a6e22e">Person</span>, <span style="color:#a6e22e">index</span> <span style="color:#a6e22e">i</span>: <span style="color:#a6e22e">Int</span>) </code></pre></div><p>The Obj-C version demonstrates &ldquo;external&rdquo; parameter names in the form of a verbose selector. In Swift, the selector components move into the parens as external parameter names.</p> <p>The Obj-C formal parameter names are analogous to the local parameter names in Swift. The Swift external-then-local declaration order perfectly follows the Obj-C selector-chunk-then-argument order: Swift <code>person p: Person</code> vs. Obj-C <code>Person:(Person *const)p</code>.</p> <p>Reaching back beyond even Obj-C, to C, this has been possible by exploiting the difference between a function prototype, commonly publicized in the header, and its implementation, commonly in a private implementation.</p> <h3 id="external-and-local-parameters-in-c">External and Local Parameters in C</h3> <p>In C, the only thing about arguments that the compiler cares about in a function prototype is the argument type; the names are purely documentary.</p> <p>In the function implementation, so long as the types don&rsquo;t change, you can name the formal parameters whatever you want.</p> <p>So the C equivalent of the above would be:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#75715e">/* prototype, in header */</span> <span style="color:#66d9ef">void</span> <span style="color:#a6e22e">Insert</span>(Person <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> person, <span style="color:#66d9ef">const</span> NSUInteger index); <span style="color:#75715e">/* implementation, in .c file */</span> <span style="color:#66d9ef">void</span> Insert(Person <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> p, <span style="color:#66d9ef">const</span> NSUInteger i) </code></pre></div><h3 id="about-that-const">About That Const</h3> <p>Take another look at the Obj-C and Swift versions of the function declaration:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-objc" data-lang="objc">- (<span style="color:#66d9ef">void</span>)<span style="color:#a6e22e">insertPerson:</span>(Person <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span>)p <span style="color:#a6e22e">atIndex:</span>(<span style="color:#66d9ef">const</span> NSUInteger)i; </code></pre></div><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">insert</span>(<span style="color:#a6e22e">person</span> <span style="color:#a6e22e">p</span>: <span style="color:#a6e22e">Person</span>, <span style="color:#a6e22e">index</span> <span style="color:#a6e22e">i</span>: <span style="color:#a6e22e">Int</span>) </code></pre></div><p>It&rsquo;s uncommon to see <code>const</code> qualifiers on arguments in Obj-C. In this case, I was trying to remain faithful to the Swift default of const formal arguments.</p> <p>You see, a Swift function declared like so:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">insert</span>(<span style="color:#a6e22e">person</span> <span style="color:#a6e22e">p</span>: <span style="color:#a6e22e">Person</span>, <span style="color:#a6e22e">index</span> <span style="color:#a6e22e">i</span>: <span style="color:#a6e22e">Int</span>) </code></pre></div><p>accepts an implicit <code>let</code> declaration of its parameters:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">insert</span>(<span style="color:#a6e22e">let</span> <span style="color:#a6e22e">person</span> <span style="color:#a6e22e">p</span>: <span style="color:#a6e22e">Person</span>, <span style="color:#a6e22e">let</span> <span style="color:#a6e22e">index</span> <span style="color:#a6e22e">i</span>: <span style="color:#a6e22e">Int</span>) </code></pre></div><p>The mutability-faithful version of the more common Obj-C declaration:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-objc" data-lang="objc">- (<span style="color:#66d9ef">void</span>)<span style="color:#a6e22e">insertPerson:</span>(Person <span style="color:#f92672">*</span>)p <span style="color:#a6e22e">atIndex:</span>(NSUInteger)i; </code></pre></div><p>would similarly have mutable parameters in Swift:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">insert</span>(<span style="color:#66d9ef">var</span> <span style="color:#a6e22e">person</span> <span style="color:#a6e22e">p</span>: <span style="color:#a6e22e">Person</span>, <span style="color:#66d9ef">var</span> <span style="color:#a6e22e">index</span> <span style="color:#a6e22e">i</span>: <span style="color:#a6e22e">Int</span>) </code></pre></div><p>Swift parameters are const by default, and it&rsquo;s great: it&rsquo;s high time that was doable without stuttering <code>const</code> all over your codebase.</p> <h3 id="summary">Summary</h3> <ul> <li>Swift&rsquo;s external-local parameter declarations are a continuation of Obj-C selector chunk then parameter declarations: Obj-C <code>insertPerson:(Person *const)p</code> becomes Swift <code>insert(person p: Person)</code>.</li> <li>Swift function parameters are <code>let</code>-declared (const) by default. Qualify a parameter with <code>var</code> if you absolutely must have it mutable: <code>insert(var person p: Person, var index i: Int)</code>.</li> </ul> VDM &amp; The “Agile Spec” Problem https://jeremywsherman.com/blog/2014/05/28/vdm-amp-the-agile-spec-problem/ Wed, 28 May 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/05/28/vdm-amp-the-agile-spec-problem/ At the end of Distributed Programming &amp; CALM, I mentioned how organically growing software often fails to produce a system with clear semantics. So I count myself lucky that Mark Fernandes recently mentioned the Vienna Development Method (VDM). It’s like if design by contract, abstract data types, first-order logic, and a small imperative/OO language with a collection library had a lovechild. Developing an application using VDM starts like this: State some overall semantics claims, introducing state/objects, types, and functions as needed to flesh things out. <p>At the end of <a href="https://jeremywsherman.com/blog/2014/05/25/distributed-programming-and-calm/">Distributed Programming &amp; CALM</a>, I mentioned how organically growing software often fails to produce a system with clear semantics.</p> <p>So I count myself lucky that <a href="https://alpha.app.net/0xmf">Mark Fernandes</a> recently <a href="https://alpha.app.net/0xmf/post/31144222">mentioned</a> the <a href="https://en.wikipedia.org/wiki/Vienna_Development_Method">Vienna Development Method (VDM)</a>. It’s like if design by contract, abstract data types, first-order logic, and a small imperative/OO language with a collection library had a lovechild.</p> <p>Developing an application using VDM starts like this:</p> <ul> <li>State some overall semantics claims, introducing state/objects, types, and functions as needed to flesh things out.</li> <li>Use preconditions and postconditions (but preferably not any implementation) to define the functions.</li> <li>Use invariants to clarify the meaning of your types.</li> <li>Get it all to preserve the desired semantics.</li> </ul> <h2 id="progressive-refinement">Progressive Refinement</h2> <p>At this point, you&rsquo;re several removes away from a runnable application.</p> <p>You&rsquo;ve got some types and some operations on those types that have been implicitly defined using assertions.</p> <p>Progressive refinement is how you work your way to more concrete data types and more explicitly (operationally) defined functions, step by step.</p> <p>In concrete terms, you actually start writing some code in between the bunch of NSParameterAssert() and NSAssert() statements that currently comprise the entirety of your program&rsquo;s functions.</p> <p>The first part of refinement is coming up with a lower-level (closer to the implementation language) representation of the same system.</p> <p>You might convert a set into an array with a &ldquo;no duplicates&rdquo; invariant, or use strings to represent a certain enumeration.</p> <p>You&rsquo;ll update functions or add new ones as needed, and rephrase pre/postconditions in terms of the new data model, and flesh out the actual implementation of the functions in terms of your new data types.</p> <p>Then we get to the real heart of the matter: <strong>Prove this new model is truly a <em>refinement</em> of the last one.</strong> To do so, you must prove that your new version of the system preserves all the properties of the original system.</p> <p>…so I’ve just been cribbing from the Wikipedia page I linked above, and it’s <a href="https://en.wikipedia.org/wiki/Vienna_Development_Method#Refinement">pretty cryptic</a> about that refinement step. (See the end of this article for my best guess at how to interpret that section of the Wikipedia VDM article.)</p> <p>Looking <a href="http://kurser.iha.dk/eit/tivdm1/materiale.html">elsewhere</a>, it appears that in addition to straight-up proof, model-checking and simulation are also used to validate the model. This make sense given that, even before we begin refining our system, we need to convince ourselves that we&rsquo;ve actually modeled what we intended to model. If we haven&rsquo;t succeeded at that, all we&rsquo;ll be refining is high-level garbage into lower-level garbage.</p> <h2 id="the-agile-spec-problem">The Agile Spec Problem</h2> <p>Key to the Vienna Development Method is starting from a known-good high-level model, before the first line of implementation code is written.</p> <p>After the first step of formally stating the specification of the application, you function as a manual compiler. At each step, you apply a program transformation, then prove it preserves your application&rsquo;s semantics.</p> <p>You start with a high-level spec, and you push it down the abstraction hierarchy till you finally end up with an actual, executable implementation in your target language.</p> <p>But that&rsquo;s not terribly agile.</p> <p>If you know of any work bridging the very different worlds of “specs are great, we can lower them down into working software” and “stakeholder-driven organic growth is necessary for software to create value” – or how to iteratively, agilely develop a spec, with non-expert stakeholder validation – shoot me an email or reach out to me on App.net, where I’m <a href="https://alpha.app.net/jws">@jws</a>.</p> <hr> <p><small><strong>Refinement in VDM:</strong> This bit is a mess in the Wikipedia article. Here’s my understanding of it:</p> <ul> <li>A mapping must exist from your new data representation to the old for all instances possible in the old representation. The mapping function is called the “retrieval” function, since it retrieves the original/old version of the state.</li> <li>Once you have such a mapping, you must show that, for all your new data, and for all your new functions, the old invariants hold: <ul> <li>Pre-conditions that hold on the retrieval should hold on the new model.</li> <li>Post-conditions from a valid starting point that hold in the new version should also hold with the retrieval of the input and output values in the old model. </small></li> </ul> </li> </ul> Distributed Programming &amp; CALM https://jeremywsherman.com/blog/2014/05/25/distributed-programming-amp-calm/ Sun, 25 May 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/05/25/distributed-programming-amp-calm/ <p>Distributed programming doesn’t get much talk in those terms in Cocoaland.</p> <p>If you’re writing an iPhone app with a server, guess what: You’re writing a distributed system. For your users’ sake, I hope it’s also an offlineable system.</p> <p>And we can view a multithreaded program as distributed programming, only with the distribution being far more local. Ordering issues rear their head when you start pushing data in chunks through concurrent queues, and the notion of producer-consumer punctuations (see below, Consistency without Borders) is practically useful if for no other reason than, “oh yeah, you can hide that activity spinner now, no more search results for ‘z*’ are coming”.</p> <h2 id="the-end-of-the-api">The End of the API</h2> <p>Some recent delvings on my part started when I read an article by <a href="http://cemerick.com/">cemerick</a>, <a href="http://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/">“Distributed Systems and the End of the API”</a>. He brought up CRDTs and CALM. I’d heard of CRDTs before (thanks <a href="https://github.com/vanstee">Patrick!</a>), but not CALM.</p> <p>Distributed programming doesn’t get much talk in those terms in Cocoaland.</p> <p>If you’re writing an iPhone app with a server, guess what: You’re writing a distributed system. For your users’ sake, I hope it’s also an offlineable system.</p> <p>And we can view a multithreaded program as distributed programming, only with the distribution being far more local. Ordering issues rear their head when you start pushing data in chunks through concurrent queues, and the notion of producer-consumer punctuations (see below, Consistency without Borders) is practically useful if for no other reason than, “oh yeah, you can hide that activity spinner now, no more search results for ‘z*’ are coming”.</p> <h2 id="the-end-of-the-api">The End of the API</h2> <p>Some recent delvings on my part started when I read an article by <a href="http://cemerick.com/">cemerick</a>, <a href="http://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/">“Distributed Systems and the End of the API”</a>. He brought up CRDTs and CALM. I’d heard of CRDTs before (thanks <a href="https://github.com/vanstee">Patrick!</a>), but not CALM.</p> <p>I looked up CALM and found a good summary on the Bloom lang page and its intro: <a href="http://www.bloom-lang.net/calm/">http://www.bloom-lang.net/calm/</a></p> <h2 id="blooming-calm">BLOOMing CALM</h2> <p>From there read a <a href="http://databeta.wordpress.com/2010/10/28/the-calm-conjecture-reasoning-about-consistency/">blog post</a> introducing the idea of CALM. I found it most useful for its many links. I started with the <a href="http://db.cs.berkeley.edu/jmh/talks/podskeynote10.pdf">keynote slides</a> mentioned therein. Those didn’t make more than 80% sense till I read the <a href="http://www.sigmod.org/publications/sigmod-record/1003/p05.article.hellerstein.pdf">companion paper</a>. After that, they’d got my attention good, so then I read the <a href="http://db.cs.berkeley.edu/papers/cidr11-bloom.pdf">CIDR 2010 paper</a> for more background on Bloom.</p> <p>Bloom is a language implemented in/over Ruby that boils down to the Dedalus flavor/extension of Datalog, which makes time explicit in each relation and avoids the mess you find in Prolog where there&rsquo;s this execution algorithm outside the system you need to worry about and play with. The entire thing is declarative, but the real point is that it&rsquo;s straightforward to visualize the dataflow and analyze a Bloom program for points where the code computes a non-monotonic result.</p> <p>&ldquo;Non-monotonic&rdquo; means it might need to change its mind about the output as new results arrive, which means you need some degree of coordination to ensure that non-monotonic computation actually got all the data needed to render a final judgment. And coordination has costs, especially if it&rsquo;s between datacenters, or with a non-responding peer whose hard-drive just ate it.</p> <h2 id="programs-are-made-of-memories-guesses-and-apologies">Programs are made of memories, guesses, and apologies</h2> <p>From there I chased a reference to <a href="http://arxiv.org/abs/0909.1788">Building on Quicksand</a>, which introduced the notion of programs being structured around:</p> <ul> <li>memories (of what has happened),</li> <li>guesses (about what might be true),</li> <li>and apologies (for when those guesses turn out wrong).</li> </ul> <p>This is even more obviously true in distributed programs, where you can’t keep every actor on the same page. Also points out that sometimes the right response is for a program to throw up its hands, email a human, and say, “Something is wrong. Figure it out, and apologize to user 1312347 for this weirdness.”</p> <h2 id="a-tale-of-n-consistencies">A Tale of <em>N</em> Consistencies</h2> <p>And then back to the Databeta blog, where I found <a href="http://databeta.wordpress.com/2013/09/16/consistency-without-borders/">“Consistency Without Borders”</a> and <a href="http://db.cs.berkeley.edu/papers/socc13-consistency.pdf">accompanying paper</a>.</p> <p>This is a call for more research into assisting developers to grapple with consistency between two extremes. The first extreme is “let’s establish consistency only at the database layer in terms of reads and writes”, which is generally too conservative and expensive and too hard to safely and faithfully “compile” your program’s operations into. The second is, “let’s just handle all the consistency in our app”, which is also easy to get wrong, expensive, and not at all reusable.</p> <p>Consistency without Borders looks at 3 different middle-grounds:</p> <ul> <li>object-level consistency: think CRDT; build app around known-good objects, developed once and reused; can be hard, and can lead to you ending up with structuring the entire app as a single CRDT, which is not reusable; fails to capture properties of composition of objects <ul> <li>Also, CRDTs can only converge to a deterministic state that’s invariant under duplication and reordering. No good if you have non-deterministic but well-defined behavior, like “purchase request returns OK if non-zero inv, FAIL otherwise”, which depends on order of message processing.</li> </ul> </li> <li>flow-level consistency: look at flow of data between modules, processes, and services; key ideas are confluence (insensitivity to message delivery order), which is convergence at the dataflow level basically, and a neat trick is “data sealing” via producer-consumer punctuations (“yes, you have seen all results as of now”) <ul> <li>Confluence analysis has the same problem with handling non-deterministic behavior.</li> </ul> </li> <li>language-level consistency: encode all knowledge needed for dependency analysis and finding non-monotonicity; very convenient, but also requires completely changing how you write code</li> </ul> <p>The paper also highlights the lack of data to assist in choosing between the many flavors of consistency. What’s the cost in making at trade-off? Can we afford more/less consistency in this case?</p> <p>(I have yet to read their intriguing reference to the LADIS ’08 write-up <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.299.7273&amp;rep=rep1&amp;type=pdf">“Towards a Cloud-Computing Research Agenda”</a> about the extreme expense and danger of full consistency in an industrial context.)</p> <p>Then moved to start looking at Peter Alvaro’s <a href="http://databeta.wordpress.com/2014/05/08/blazes/">Blazes</a> (<a href="http://www.slideshare.net/palvaro/icde2014-slideshare-33117763">slide deck</a>. This is the flow-level analyzer mentioned in Consistency Without Borders.</p> <p>Blazes looks for non-monotonic operations that aren’t protected by coordination based on annotations of code. This is a beginning towards assisting with debugging the issues encountered in distributed systems, vs. those you can readily debug with gdb or lldb. Once your code is all correct, there’s that small matter of, “Oh yeah, and that thing it’s correctly doing, is that semantically correct?”</p> <h2 id="where-it-all-goes-wrong">Where It All Goes Wrong</h2> <p>But you still have to have correctly annotated everything you&rsquo;re using. Good luck with the balls of closed-source mud you get to work with in GUI programs.</p> <p>A major issue in organically grown software projects is even stating semantic properties, never mind ensuring they’re preserved across the application. We get tied up in matters like “did I just create a strong reference cycle” and futz about with that, do some refactoring, extract some things, whatever, and can continue in this vein for a good while till we have a serious mess in light of what the actual purpose of the application is. Leastwise, that’s what I seem to see too often. More on that later.</p> Use jsonlint to debug bogus JSON data https://jeremywsherman.com/blog/2014/04/02/use-jsonlint-to-debug-bogus-json-data/ Wed, 02 Apr 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/04/02/use-jsonlint-to-debug-bogus-json-data/ <p>NSJSONSerialization delights in an opaque &ldquo;lolnope character 12341234&rdquo; error message that provides zero context. It doesn&rsquo;t even bother to take advantage of line numbers to help you out, even if the JSON data has linebreaks. This is a royal pain, especially if you&rsquo;re working with hand-written stub data for a web service.</p> <p>I got fed up with this, went &ldquo;there has to be a linter!&quot;, and landed on <a href="https://github.com/zaach/jsonlint">jsonlint</a> by <a href="https://github.com/zaach/">Zach Carter</a>.</p> <p>Install it via <code>npm install jsonlint -g</code>, and you&rsquo;ll be able to find and fix syntax errors in JSON far faster than you would puzzling over NSJSONSerialization&rsquo;s error message.</p> <p>NSJSONSerialization delights in an opaque &ldquo;lolnope character 12341234&rdquo; error message that provides zero context. It doesn&rsquo;t even bother to take advantage of line numbers to help you out, even if the JSON data has linebreaks. This is a royal pain, especially if you&rsquo;re working with hand-written stub data for a web service.</p> <p>I got fed up with this, went &ldquo;there has to be a linter!&quot;, and landed on <a href="https://github.com/zaach/jsonlint">jsonlint</a> by <a href="https://github.com/zaach/">Zach Carter</a>.</p> <p>Install it via <code>npm install jsonlint -g</code>, and you&rsquo;ll be able to find and fix syntax errors in JSON far faster than you would puzzling over NSJSONSerialization&rsquo;s error message.</p> <h2 id="comparing-parser-error-messages">Comparing Parser Error Messages</h2> <h3 id="sample-data">Sample Data</h3> <p>Bogus JSON:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{ <span style="color:#f92672">&#34;something&#34;</span>: <span style="color:#e6db74">&#34;bogus&#34;</span>, <span style="color:#f92672">&#34;this way&#34;</span>: [{ <span style="color:#f92672">&#34;comes&#34;</span> <span style="color:#960050;background-color:#1e0010">]</span>, <span style="color:#f92672">&#34;don&#39;t you&#34;</span>: <span style="color:#e6db74">&#34;know&#34;</span>, <span style="color:#f92672">&#34;ayup&#34;</span>: <span style="color:#e6db74">&#34;you do&#34;</span>, <span style="color:#f92672">&#34;or you will&#34;</span>: <span style="color:#e6db74">&#34;soon&#34;</span> } </code></pre></div><h3 id="nsjsonserialization">NSJSONSerialization</h3> <p>The wonderfully useless NSJSONSerialization error:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">% nush Nu Shell. % (NSJSONSerialization JSONObjectWithData: (NSData dataWithContentsOfFile:&#34;bogus.json&#34;) options:0 error:(set e (NuReference new))) () % ((e value) description) &#34;Error Domain=NSCocoaErrorDomain Code=3840 \&#34;The data couldn\u2019t be read because it isn\u2019t in the correct format.\&#34; (No value for key in object around character 67.) UserInfo=0x7fbf2860e8b0 {NSDebugDescription=No value for key in object around character 67.}&#34; % (exit) </code></pre></div><h3 id="python">Python</h3> <p>The equally useless Python json error:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">% python Python 2.7.6 (default, Dec 28 2013, 00:41:57) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type &#34;help&#34;, &#34;copyright&#34;, &#34;credits&#34; or &#34;license&#34; for more information. &gt;&gt;&gt; import json &gt;&gt;&gt; json.load(file(&#34;bogus.json&#34;)) Traceback (most recent call last): # 8&lt; snipped 8&lt; ValueError: Expecting : delimiter: line 5 column 5 (char 67) &gt;&gt;&gt; </code></pre></div><p>Hey, at least Python takes advantage of the fact the file has line numbers.</p> <h3 id="jsonlint">jsonlint</h3> <p>The jsonlint output:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">% jsonlint bogus.json [Error: Parse error on line 4: ... &#34;comes&#34; ], &#34;don&#39;t you&#34;: ----------------------^ Expecting &#39;:&#39;, got &#39;]&#39;] </code></pre></div><p>Now, isn&rsquo;t that nice? No need to cross-reference to the file, since there&rsquo;s context right there, and you know exactly what it&rsquo;s expecting, rather than just some opaque &ldquo;delimiter&rdquo;.</p> <p>(The Cocoa error in this example isn&rsquo;t entirely useless, since it does call out that a key is missing its value. I&rsquo;ve seen more useless ones where it faults the very end of the file for something that went wrong way earlier.)</p> <h2 id="installing-jsonlint">Installing jsonlint</h2> <p>I&rsquo;ve installed it locally using <code>npm install jsonlint -g</code>.</p> <p>Apparently, you could also use <a href="http://jsonlint.com/,">http://jsonlint.com/,</a> but:</p> <ul> <li>I&rsquo;m wary of posting potentially sensitive JSON data to a public web service</li> <li>I like being able to work even without network access</li> <li>command-line tools compose easily into pipe-lines far better than most websites</li> </ul> <p>So, I naturally recommend you just install it locally.</p> OptionBits and BOOL gonna bite you one day https://jeremywsherman.com/blog/2014/03/14/optionbits-and-bool-gonna-bite-you-one-day/ Fri, 14 Mar 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/03/14/optionbits-and-bool-gonna-bite-you-one-day/ <p>I got to talking with a coworker about some code that tested bitmasks:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">BOOL</span> isFoo <span style="color:#f92672">=</span> flags <span style="color:#f92672">&amp;</span> FLAG_FOO; </code></pre></div><p>Don&rsquo;t do this; you are inviting pain, suffering, and head-scratching debugging.</p> <p>I wrote about the wonderland of joy and kittens that is C arithmetic <a href="https://jeremywsherman.com/blog/2013/09/26/math-is-hard/">earlier</a>, but only in the abstract. <code>NSUInteger</code> and <code>BOOL</code> provide concrete examples that hit where it hurts.</p> <p>I got to talking with a coworker about some code that tested bitmasks:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">BOOL</span> isFoo <span style="color:#f92672">=</span> flags <span style="color:#f92672">&amp;</span> FLAG_FOO; </code></pre></div><p>Don&rsquo;t do this; you are inviting pain, suffering, and head-scratching debugging.</p> <p>I wrote about the wonderland of joy and kittens that is C arithmetic <a href="https://jeremywsherman.com/blog/2013/09/26/math-is-hard/">earlier</a>, but only in the abstract. <code>NSUInteger</code> and <code>BOOL</code> provide concrete examples that hit where it hurts.</p> <h2 id="the-trouble">The Trouble</h2> <p><a href="https://developer.apple.com/library/ios/releasenotes/ObjectiveC/ModernizationObjC/AdoptingModernObjective-C/AdoptingModernObjective-C.html#//apple_ref/doc/uid/TP40014150-CH1-SW6">Apple now recommend</a> you use <code>NSUInteger</code> for your option bits. But we keep holding on for dear life to <code>BOOL</code>, which is a <code>signed char</code>. That means our bitmasks and our booleans differ in both signedness and width.</p> <p>Have a look-see: Given this seemingly innocuous arrangement:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">typedef</span> <span style="color:#a6e22e">NS_OPTIONS</span>(NSUInteger, Flags) { FLAG_A_BIT_TOO_BIG_FOR_BOOL <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x100</span> }; NSUInteger flags <span style="color:#f92672">=</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL; </code></pre></div><p>This naïve flag test assigns zero:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#75715e">/* DON&#39;T DO THIS! */</span> <span style="color:#66d9ef">BOOL</span> is_flag_set <span style="color:#f92672">=</span> flags <span style="color:#f92672">&amp;</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL; </code></pre></div><p>while this works just fine, and gives a non-zero result, as expected:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">bool</span> is_flag_set <span style="color:#f92672">=</span> flags <span style="color:#f92672">&amp;</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL; </code></pre></div><h2 id="why-_bool-is-so-swell">Why <code>_Bool</code> Is So Swell</h2> <p>The reason is that assignments to <code>_Bool</code> (which <code>bool</code> expands to when you include <code>&lt;stdbool.h&gt;</code>) are effectively run through a double-bang, as if you&rsquo;d written this:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">bool</span> is_flag_set <span style="color:#f92672">=</span> <span style="color:#f92672">!!</span>(flags <span style="color:#f92672">&amp;</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL); </code></pre></div><h3 id="aside-arm64s-bool-is-_bool">Aside: Arm64&rsquo;s <code>BOOL</code> is <code>_Bool</code></h3> <p><em>Added after initial publication. Thanks, <a href="http://borkwarellc.wordpress.com/">Mark</a>!</em></p> <p>Running iOS on arm64? Is today ever your lucky day!</p> <p>Unlike every other Apple platform, arm64 iOS (and the 64-bit simulator) typedefs <code>BOOL</code> to be <code>bool</code>. You get sanity for free.</p> <p>Just don&rsquo;t forget to test on a non-arm64 platform if you plan to release to a non-arm64 platform, because it&rsquo;s still the wild west out there.</p> <h2 id="introducing-bang-bang">Introducing Bang-Bang</h2> <p>The double-bang trick coerces the value to be either 0 or 1:</p> <ul> <li>The first bang inverts its logical value, so if it was non-zero (true) it&rsquo;s now 0 (false), and if it was zero (false), it&rsquo;s now 1 (true).</li> <li>The second bang reverses that, and NOT NOT TRUE is just TRUE, so we&rsquo;re logically back where we started, only now with a tidy, known arithmetic value representing that logical value.</li> </ul> <p>If you apply this trick, then the assignment to <code>BOOL</code> plays out as you&rsquo;d hope:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">BOOL</span> is_flag_set <span style="color:#f92672">=</span> <span style="color:#f92672">!!</span>(flags <span style="color:#f92672">&amp;</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL); </code></pre></div><p>You might also see this written like so:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#66d9ef">BOOL</span> is_flag_set <span style="color:#f92672">=</span> ((flags <span style="color:#f92672">&amp;</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL) <span style="color:#f92672">==</span> FLAG_A_BIT_TOO_BIG_FOR_BOOL); </code></pre></div><p>Since the result of the bit-and is either <code>0</code> or <code>FLAG_A_BIT_TOO_BIG_FOR_BOOL</code>, the <code>==</code> test results in either <code>0</code> or <code>1</code>.</p> <h2 id="theres-always-a-moral">There&rsquo;s Always a Moral</h2> <p>The moral is:</p> <blockquote> <p>Use <code>bool</code> or use <code>!!</code>, and beware the wicked type conversions.</p> </blockquote> <h2 id="get-the-gist">Get the Gist</h2> <p>You&rsquo;ll find some ready-to-compile, comment-full sample code demonstrating these issues over in <a href="https://gist.github.com/jeremy-w/9558336">this gist</a>.</p> <p><small>*Note: While <code>BOOL</code> might be <code>signed char</code>, <code>YES</code> and <code>NO</code> themselves are a bit more than just that now, to support integer literals. That&rsquo;s neither here nor there.*</small></p> <p><small>EDIT: Fixed 0 for 1 typo <a href="https://twitter.com/mike3k/status/444624960112189440">graciously pointed out by Mike Cohen</a>.</small></p> TestFlight TOS &amp; Privacy Changes https://jeremywsherman.com/blog/2014/02/21/testflight-tos-amp-privacy-changes/ Fri, 21 Feb 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/02/21/testflight-tos-amp-privacy-changes/ <p>TestFlight changed their terms of service and privacy policy on 18 Feb 2014.</p> <p>As is depressingly routine in this scenario, they conveniently neglected to provide the changes in a readily reviewable format. You just get a link to the current policy; no past policy, no diff.</p> <p>Thanks to Archive.org and <code>opendiff</code>, I was able to review the changes side-by-side.</p> <p>TestFlight changed their terms of service and privacy policy on 18 Feb 2014.</p> <p>As is depressingly routine in this scenario, they conveniently neglected to provide the changes in a readily reviewable format. You just get a link to the current policy; no past policy, no diff.</p> <p>Thanks to Archive.org and <code>opendiff</code>, I was able to review the changes side-by-side.</p> <h2 id="terms-of-service">Terms of Service</h2> <p>Aside from some wording changes, the big changes are:</p> <ul> <li>New section with &ldquo;we can shut this thing down at any time, suckers&rdquo;.</li> <li>Shift in court venue from LA to Northern District of California.</li> <li>Removal of arbitration option, previously offered for amounts less than $10,000.</li> </ul> <p>Oh, and FlightPath goes bye-bye.</p> <h2 id="privacy-policy">Privacy Policy</h2> <p>Privacy changes are innocuous. Mostly rewording of existing stuff.</p> <ul> <li> <p>Merged contact URL from separate ones for testflight and flightpath to just <a href="mailto:legal@burstly.com">legal@burstly.com</a>.</p> </li> <li> <p>Addressed Do Not Track, which they ignore, because they don&rsquo;t have any 3rd-party trackers on their page.</p> </li> </ul> <p>Though they did manage to work &ldquo;national security&rdquo; in there, just because it&rsquo;s the legal buzzword of the year.</p> Installing and using Euterpea under OS X 10.9 https://jeremywsherman.com/blog/2014/02/09/installing-and-using-euterpea-under-os-x-10.9/ Sun, 09 Feb 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/02/09/installing-and-using-euterpea-under-os-x-10.9/ <p>The BNR Book Club (<a href="http://j.mp/bnr-bookclub">JOIN US!</a>) is working through Hudak&rsquo;s <a href="http://haskell.cs.yale.edu/?post_type=publication&amp;p=112">The Haskell School of Music: From Signals to Symphonies</a>.</p> <p>The tricky part of working with pre-release books and unreleased software is getting both to run on your system.</p> <p>The site itself warns you of the difficulties of using Euterpea under OS X, but if you&rsquo;re pig-stubborn and Mac-happy like me, you might benefit from my setup instructions.</p> <p>The BNR Book Club (<a href="http://j.mp/bnr-bookclub">JOIN US!</a>) is working through Hudak&rsquo;s <a href="http://haskell.cs.yale.edu/?post_type=publication&amp;p=112">The Haskell School of Music: From Signals to Symphonies</a>.</p> <p>The tricky part of working with pre-release books and unreleased software is getting both to run on your system.</p> <p>The site itself warns you of the difficulties of using Euterpea under OS X, but if you&rsquo;re pig-stubborn and Mac-happy like me, you might benefit from my setup instructions.</p> <h2 id="the-scene">The Scene</h2> <p>I figured Chapter 2 would start by explaining how to set up Euterpea so you can actually run the code we&rsquo;re encountering, but no dice.</p> <p>There are some lengthy installation instructions <a href="http://haskell.cs.yale.edu/euterpea/download/">available from the author</a> that attempt to cover Windows, Linux, and OS X.</p> <p>The short of it is, Windows is very well supported, Linux so-so, and Mac, you&rsquo;re on your own.</p> <p>Anecdotal reports indicate that if you have a 32-bit version of GHC installed, things will pretty much just work on your Mac. Otherwise, you&rsquo;ll hit problems.</p> <p>I have the 64-bit version and was able to get things mostly working, to the point that I can write notes to midi files on disk. The trick is knowing what <em>doesn&rsquo;t</em> work so you avoid it.</p> <h2 id="installing">Installing</h2> <h3 id="adjust-ghcs-c-compiler-command">Adjust GHC&rsquo;s C Compiler Command</h3> <p><em>This patch-up is necessary as of the 2013.2.0.0 Haskell Platform release. It likely won&rsquo;t be necessary after the next release.</em></p> <p>Before you do anything else, if you&rsquo;re on Mavericks, follow the instructions under &ldquo;Xcode 5 &amp; OS X 10.9 (Mavericks)&rdquo; from <a href="http://www.haskell.org/platform/mac.html">the Haskell Platform Mac page</a>. You will be adjusting the C compiler setting for GHC to go through a shim that fixes up some differences between how gcc and clang do things.</p> <h3 id="clone-and-cabal-install-euterpea">Clone and Cabal Install Euterpea</h3> <p>Then you can get on to Euterpea. Follow the basic checkout and build instructions:</p> <pre><code>git clone https://github.com/Euterpea/Euterpea cd Euterpea cabal update cabal install </code></pre> <h2 id="gotchas">Gotchas</h2> <p>Once you have it, here&rsquo;s what not to do:</p> <ul> <li>Don&rsquo;t try to pull the top-level Euterpea in from GHC or GHCI. You&rsquo;ll bomb out linking in GLFW due to symbol relocation issues. <ul> <li>Instead, pull in the specific submodules you need, like Euterpea.Music.Note.Music or whatever.</li> </ul> </li> <li><code>play</code> will not work. The FFI call into PortAudio hits some enum range issue I haven&rsquo;t spent time debugging. <code>test</code>, which writes a midi file test.mid, does work.</li> <li>I think even there, I would sometimes run into an issue with ghci (or when using runghc) that weren&rsquo;t an issue when just compiling and then running a program. It stumbles into the GLFW issue, even though you&rsquo;re intentionally trying to avoid it.</li> </ul> <h2 id="verifying-your-install">Verifying Your Install</h2> <p>Here is what you should do:</p> <ul> <li>Just write a demo.hs program <a href="https://gist.github.com/jeremy-w/8908684">like so</a> that calls the <code>test</code> function, then use <code>ghc demo.hs</code> to compile it, then run it as <code>./demo</code> to dump your midi file. Once that&rsquo;s done, you can play the midi file just fine.</li> </ul> <p>How should you play the midi file? I just used <a href="http://timidity.sourceforge.net/"><strong>timidity</strong></a>, per the install document. I cut out almost all the optional libraries to quicken build time, so my brew install line went like this:</p> <pre><code>brew install timidity --without-libogg --without-speex \ --without-flac --without-libvorbis </code></pre> <p>I&rsquo;ve attached a demo program you can use to check you have this all working. In the end, you should be able to do this and hear an F# Major chord:</p> <pre><code>ghc demo.hs &amp;&amp; ./demo &amp;&amp; timidity test.mid </code></pre> <h2 id="enjoy">Enjoy!</h2> <p>Now you should be able to hobble your way through the rest of the text. I hope. I&rsquo;m still working my way through, as well.</p> <p><em>This post first appeared as a <a href="https://groups.google.com/d/msg/bnr-bookclub/rj76FUQafzs/LMDdhuEpWc0J">somewhat less-structured post</a> to the <a href="http://j.mp/bnr-bookclub">BNR Book Club Google Group</a>.</em></p> Self-initiating into Smalltalk https://jeremywsherman.com/blog/2014/01/19/self-initiating-into-smalltalk/ Sun, 19 Jan 2014 00:00:00 +0000 https://jeremywsherman.com/blog/2014/01/19/self-initiating-into-smalltalk/ If you’re an Obj-C dev, you should consider picking up some Smalltalk. See, if Obj-C is Romanian – that quirky little Romance language – then Smalltalk is Latin. Sometimes it’s good to know your roots. Based on my far less linear path, I recommend you: First, read a few chapters of Squeak by Example. Then, work through the first couple sections of Stephan B. Wessels’ Laser Game example. Then, you should pitch in on Graham Lee’s ClassBrowser project, which aims to bring the Smalltalk style of immediate development to Obj-C. <p>If you’re an Obj-C dev, you should consider picking up some Smalltalk.</p> <p>See, if Obj-C is Romanian – that quirky little Romance language – then Smalltalk is Latin. Sometimes it’s good to know your roots.</p> <p>Based on my far less linear path, I recommend you:</p> <ul> <li>First, read a few chapters of <a href="http://squeakbyexample.org/">Squeak by Example</a>.</li> <li>Then, work through the first couple sections of Stephan B. Wessels’ <a href="http://squeak.preeminent.org/tut2007/html/index.html">Laser Game example</a>.</li> </ul> <p>Then, you should pitch in on Graham Lee’s <a href="https://bitbucket.org/iamleeg/ikbclassbrowser/">ClassBrowser</a> project, which aims to bring the Smalltalk style of immediate development to Obj-C.</p> Clarified CQRS - Reading Notes https://jeremywsherman.com/blog/2013/12/20/clarified-cqrs-reading-notes/ Fri, 20 Dec 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/12/20/clarified-cqrs-reading-notes/ <p>On 19 Dec 2013, I read the article <a href="http://www.udidahan.com/2009/12/09/clarified-cqrs/">Clarified CQRS</a> published by <a href="http://www.udidahan.com/">Udi Dahan</a> on 9 Dec 2009, so four years ago.</p> <p>(This reading was for the <a href="http://j.mp/bnr-bookclub">BNR Book Club</a>: It&rsquo;s open to all, and you should join the group!)</p> <p>In it, Dahan elaborates their interpretation of CQRS.</p> <p>Dahan&rsquo;s new ideas:</p> <ul> <li>Data inevitably stales: Exploit this instead of fighting it.</li> <li>Each command can, and should, be processed autonomously from the others.</li> <li>DRY to the max by jettisoning code and data store complexity wherever the command–query system allows it, which is far more than you might think at first blush.</li> </ul> <p>Section-by-section notes follow.</p> <p>Two appendices include notes from additional articles by other authors on CQRS, to provide context to the discussion:</p> <ul> <li><a href="#clarified-cqrs-martin-fowler">Martin Fowler</a> provides a cogent summary and links to references.</li> <li><a href="#clarified-cqrs-greg-young">Greg Young</a> originated CQRS and distills it to its essence in a single example.</li> </ul> <p>On 19 Dec 2013, I read the article <a href="http://www.udidahan.com/2009/12/09/clarified-cqrs/">Clarified CQRS</a> published by <a href="http://www.udidahan.com/">Udi Dahan</a> on 9 Dec 2009, so four years ago.</p> <p>(This reading was for the <a href="http://j.mp/bnr-bookclub">BNR Book Club</a>: It&rsquo;s open to all, and you should join the group!)</p> <p>In it, Dahan elaborates their interpretation of CQRS.</p> <p>Dahan&rsquo;s new ideas:</p> <ul> <li>Data inevitably stales: Exploit this instead of fighting it.</li> <li>Each command can, and should, be processed autonomously from the others.</li> <li>DRY to the max by jettisoning code and data store complexity wherever the command–query system allows it, which is far more than you might think at first blush.</li> </ul> <p>Section-by-section notes follow.</p> <p>Two appendices include notes from additional articles by other authors on CQRS, to provide context to the discussion:</p> <ul> <li><a href="#clarified-cqrs-martin-fowler">Martin Fowler</a> provides a cogent summary and links to references.</li> <li><a href="#clarified-cqrs-greg-young">Greg Young</a> originated CQRS and distills it to its essence in a single example.</li> </ul> <h2 id="clarified-cqrs">Clarified CQRS</h2> <h3 id="why-cqrs">Why CQRS</h3> <ul> <li>Driven by: <ul> <li>Collaboration: Mutable state shared by &gt;1 actor</li> <li>Staleness: Data read by an actor can be invalidated by a subsequent write by another actor. <ul> <li>Exacerbated by caching.</li> <li>Leads users inevitably to act based on obsolete data.</li> </ul> </li> </ul> </li> </ul> <h3 id="queries">Queries</h3> <ul> <li>Data is going to be stale. Give in and skip DB hits.</li> <li>Cache data in view-model format to avoid unnecessary marshaling. <ul> <li>No need for the cache to be a RDBMS - ViewModels don&rsquo;t require any joins, they&rsquo;re already denormalized.</li> </ul> </li> <li>Views hit the cache rather than the DB for their display data.</li> </ul> <h4 id="scaling">Scaling</h4> <ul> <li>Add multiple caches.</li> <li>Don&rsquo;t worry about keeping them in sync across each other. Users just encounter different vintages of stale data in different caches.</li> </ul> <h3 id="data-modification">Data Modification</h3> <ul> <li> <p>Optimistic concurrency conflicts.</p> </li> <li> <p>Validation is a pain.</p> </li> <li> <p>End up rejecting a whole chunk of modifications because 1 is off, then users must redo their work based on the new data.</p> </li> <li> <p>More users, bigger entities =&gt; more frequent and annoying conflicts.</p> </li> <li> <p>Solved by <strong>commands</strong>:</p> <blockquote> <p>If only there was some way for our users to provide us with the right level of granularity and intent when modifying data. That’s what commands are all about.</p> </blockquote> </li> </ul> <h3 id="commands">Commands</h3> <ul> <li>&ldquo;Using an Excel-like UI for data changes doesn’t capture intent, as we saw above.&rdquo; <ul> <li>Submit <strong>commands</strong> instead of a &ldquo;write these fields&rdquo; instead. (Task-based UI)</li> <li>Capture intention - can even process asynchronously, report progress and failures in UI, let user investigate why failed.</li> </ul> </li> <li>&ldquo;Note that the client sends commands to the server – it doesn’t publish them. Publishing is reserved for events which state a fact – that something has happened, and that the publisher has no concern about what receivers of that event do with it.&rdquo;</li> </ul> <h4 id="commands--validation">Commands &amp; Validation</h4> <blockquote> <p>Validation is different from business rules in that it states a context-independent fact about a command. Either a command is valid, or it isn’t. Business rules on the other hand are context dependent.</p> </blockquote> <p>Returns to example of delinquency update arriving before preferred status application causing the latter to be reject; reverse the order, and we would have accepted both changes.</p> <p>This is basically just pointing out that &ldquo;a valid command is one that has all necessary, valid data&rdquo; rather than &ldquo;a valid command is one that won&rsquo;t be rejected&rdquo;.</p> <h4 id="rethinking-uis-and-commands">Rethinking UIs and commands</h4> <p>Can use query store to speed up updates - autocomplete from query store, update sends the ID we already have for the selected value rather than text. Again, less marshaling.</p> <h4 id="reasons-valid-commands-fail">Reasons valid commands fail</h4> <p>The delinquent vs. preferred race is just bad design. Should have same business outcome regardless of which arrives first.</p> <p>Outcome: Notify the user (email).</p> <p>No rejection errors are ever returned to the agent submitting changes. They can do nothing but notify the user, anyway.</p> <p>No need even to show pending commands: Instead, notify users as needed asynchronously out of band.</p> <h4 id="commands-and-autonomy">Commands and Autonomy</h4> <ul> <li>Command processor should be autonomous.</li> <li>Queue commands for processing, process them at leisure, rollback and retry as needed (DB down frex).</li> <li>Serving commands and queries from separate stores prevents cache thrashing.</li> </ul> <h3 id="autonomous-components">Autonomous Components</h3> <p><strong>Acronym: AC = Autonomous Component</strong></p> <p>Command processor is an AC with its own queue.</p> <p>Can go even further than that: Can have each command processed by its own AC.</p> <p>This lets you get detailed queue and processing time metrics, and can scale up ACs on a per-command basis.</p> <h4 id="service-layers">Service Layers</h4> <p>Per-command AC means each processor is independent. This is a stark contrast to the rat&rsquo;s nest at each layer of many layered architectures.</p> <h4 id="domain-model">Domain Model</h4> <p>Domain model is no longer used to service queries.</p> <p>Not really necessary for commands either.</p> <p>Scarcely need relationships – just precompute (denormalize) for queries, and have commands sent with needed IDs.</p> <h4 id="persistence-for-command-processing">Persistence for Command Processing</h4> <p>No need for fancy DB queries.</p> <p>Commands come in with IDs anyway.</p> <p>So ORM not strictly necessary; can do key-value, optionally splitting out properties that benefit from uniqueness constraint into their own columns.</p> <p>Key point here: <strong>&ldquo;How you process the commands is an implementation detail of CQRS.&quot;</strong></p> <h3 id="keeping-the-query-store-in-sync">Keeping the Query Store in Sync</h3> <ul> <li>Apply command and broadcast event in transaction.</li> <li>Per-command events - DoBlah broadcasts DidBlah on success.</li> <li>AC does Event -&gt; Query Store (cache) updates. <ul> <li>Can readily do one AC per ViewModel (aka table).</li> </ul> </li> </ul> <h3 id="bounded-contexts">Bounded Contexts</h3> <p>&ldquo;CQRS if used is employed within a bounded context (DDD) or a business component (SOA) – a cohesive piece of the problem domain. The events published by one BC are subscribed to by other BCs, each updating their query and command data stores as needed.&rdquo;</p> <p>Mash-up into a single UI as needed.</p> <h3 id="summary">Summary</h3> <blockquote> <p>CQRS is about coming up with an appropriate architecture for multi-user collaborative applications. It explicitly takes into account factors like data staleness and volatility and exploits those characteristics for creating simpler and more scalable constructs.</p> <p>One cannot truly enjoy the benefits of CQRS without considering the user-interface, making it capture user intent explicitly. When taking into account client-side validation, command structures may be somewhat adjusted. Thinking through the order in which commands and events are processed can lead to notification patterns which make returning errors unnecessary.</p> </blockquote> <h2 id="appendix-a-martin-fowler-on-cqrs-a-idclarified-cqrs-martin-fowlera">Appendix A: Martin Fowler on CQRS <a id="clarified-cqrs-martin-fowler"></a></h2> <p>Never expanded anywhere in the article is the acronym &ldquo;CQRS&rdquo;:</p> <blockquote> <p>CQRS stands for <strong>Command Query Responsibility Segregation</strong>. It&rsquo;s a pattern that I first heard described by Greg Young. At its heart is a simple notion that you can use a different model to update information than the model you use to read information. This simple notion leads to some profound consequences for the design of information systems.</p> <p>[…]</p> <p>The change that CQRS introduces is to split that conceptual model [integrating various views of the underlying data] into separate models for update and display, which it refers to as Command and Query respectively following the vocabulary of CommandQuerySeparation. The rationale is that for many problems, particularly in more complicated domains, having the same conceptual model for commands and queries leads to a more complex model that does neither well. (<a href="http://martinfowler.com/bliki/CQRS.html">Martin Fowler</a>)</p> </blockquote> <p>Fowler introduces more terms, like <a href="http://martinfowler.com/bliki/ReportingDatabase.html">Reporting Database</a> and <a href="http://martinfowler.com/bliki/EagerReadDerivation.html">Eager Read Derivation</a>, which can be used independently of CQRS but feature in it as well.</p> <p>Points out that where CRUD fits, you should likely use it. CQRS should also be deployed on a per-&ldquo;bounded context&rdquo; basis - it&rsquo;s effectively a domain modeling decision.</p> <p>Not clear that commands and queries are often really separate enough that it&rsquo;s worth having two entirely separate models.</p> <p>CQRS is nice for high-load apps – you can scale reads and writes independently. But still can handle this in CRUD by splitting out the really high reads into a ReportingDatabase used to serve just those queries.</p> <h2 id="appendix-b-greg-young-on-cqrs-a-idclarified-cqrs-greg-younga">Appendix B: Greg Young on CQRS <a id="clarified-cqrs-greg-young"></a></h2> <p>Greg Young originated CQRS per Fowler.</p> <p>Fowler links to <a href="http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh/">this summary by Greg Young</a>:</p> <ul> <li> <p>Split CustomerService into CustomerReadService and CustomerWriteService. Boom: CQRS.</p> </li> <li> <p>&ldquo;[No biggie, eh? But!] This separation however enables us to do many interesting things architecturally, the largest is that it forces a break of the mental retardation that because the two use the same data they should also use the same <strong>data model.</strong>&rdquo;</p> </li> <li> <p>&ldquo;There is however one thing that does really require a task based UI… That is Domain Driven Design.&rdquo;</p> </li> <li> <p>&ldquo;The Application Service Layer in Domain Driven Design represents the tasks the system can perform. It does not just copy data to domain objects and save them… It should be dealing with behaviors on the objects&rdquo;</p> <ul> <li>Don&rsquo;t use DDD for areas where CRUD really is the &ldquo;ubiquitous language&rdquo;.</li> </ul> </li> <li> <p>Conclusion:</p> <blockquote> <p>Going through all of these we can see that CQRS itself is actually a fairly trivial pattern. What is interesting around CQRS is not CQRS itself but the architectural properties in the integration of the two services. In other words the interesting stuff is not really the CQRS pattern itself but in the architectural decisions that can be made around it. Don’t get me wrong there are a lot of interesting decisions that can be made around a system that has had CQRS applied … just don’t confuse all of those architectural decisions with CQRS itself.</p> </blockquote> </li> </ul> Arithmetic Will Bite You One Day https://jeremywsherman.com/blog/2013/09/26/arithmetic-will-bite-you-one-day/ Thu, 26 Sep 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/09/26/arithmetic-will-bite-you-one-day/ Int: A Young Love Early C has an innocent air. Take for example this bounds-checking function, which converts a file descriptor to a pointer, after checking that the file descriptor is a valid index: getf(f) /* Unix 6th edition: unix/fio.c:6619 */ { register *fp, rf; rf = f; if(rf&lt;0 || rf&gt;=NOFILE) goto bad; fp = u.u_ofile[rf]; if(fp != NULL) return(fp); bad: u.u_error = EBADF; return(NULL); } Want a value in a register? <h2 id="int-a-young-love">Int: A Young Love</h2> <p>Early C has an innocent air. Take for example this bounds-checking function, which converts a file descriptor to a pointer, after checking that the file descriptor is a valid index:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c">getf(f) <span style="color:#75715e">/* Unix 6th edition: unix/fio.c:6619 */</span> { <span style="color:#66d9ef">register</span> <span style="color:#f92672">*</span>fp, rf; rf <span style="color:#f92672">=</span> f; <span style="color:#66d9ef">if</span>(rf<span style="color:#f92672">&lt;</span><span style="color:#ae81ff">0</span> <span style="color:#f92672">||</span> rf<span style="color:#f92672">&gt;=</span>NOFILE) <span style="color:#66d9ef">goto</span> bad; fp <span style="color:#f92672">=</span> u.u_ofile[rf]; <span style="color:#66d9ef">if</span>(fp <span style="color:#f92672">!=</span> NULL) <span style="color:#66d9ef">return</span>(fp); bad: u.u_error <span style="color:#f92672">=</span> EBADF; <span style="color:#66d9ef">return</span>(NULL); } </code></pre></div><p>Want a value in a register? Use <code>register</code>. It does what it says on the tin. (At least it did then.)</p> <p>Types? Those are a syntactic convenience. Go with the flow and use the native word size: the default type is <code>int</code>, and that&rsquo;s nearly all you&rsquo;ll need. It&rsquo;s the (default, so not explicitly declared) type of both <code>fp</code> and <code>rf</code> in this function. And what do you think the function&rsquo;s return type is, eh?</p> <h2 id="the-love-that-wouldnt-die">The Love That Wouldn&rsquo;t Die</h2> <p>Modern C shows its continuing love for <code>int</code> in subtle ways that will one day corrupt your code.</p> <p>It&rsquo;s subtle, because most of the time, C arithmetic just works, to the point where you can remain unaware of what&rsquo;s actually going on when you perform some innocent-looking arithmetic.</p> <p>What&rsquo;s <code>(0xFFFF &lt;&lt; 24)</code>? Let&rsquo;s see:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c">uint64_t mask <span style="color:#f92672">=</span> (<span style="color:#ae81ff">0xCAFF</span> <span style="color:#f92672">&lt;&lt;</span> <span style="color:#ae81ff">24</span>); uint64_t expected <span style="color:#f92672">=</span> <span style="color:#ae81ff">0xCAFF000000</span>; printf(<span style="color:#e6db74">&#34;%&#34;</span> PRIx64 <span style="color:#e6db74">&#34; == %&#34;</span> PRIx64 <span style="color:#e6db74">&#34;? %d</span><span style="color:#ae81ff">\n</span><span style="color:#e6db74">&#34;</span>, mask, expected, mask <span style="color:#f92672">==</span> expected); <span style="color:#75715e">/* ffffffffff000000 == caff000000? 0 */</span> </code></pre></div><p>Well, that ain&rsquo;t right.</p> <p>And if you&rsquo;ve got warnings turned on, your compiler might even be so kind as to warn you that you&rsquo;re doing something boneheaded:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">mask.c:7:25: warning: signed shift result (0xCAFF000000) requires 41 bits to represent, but &#39;int&#39; only has 32 bits [-Wshift-overflow] uint64_t mask = (0xCAFF &lt;&lt; 24); ~~~~~~ ^ ~~ 1 warning generated. </code></pre></div><p>Int, what int? I ordered a <code>uint64_t</code>, thank you muchly.</p> <p>But &ldquo;&lsquo;int&rsquo; only has 32 bits&rdquo;. And the compiler ran out of bits. And then, once it got done shifting the int around, it widened it to take up a full 64 bits, and the sign bit came with it:</p> <ul> <li> <p>We wanted</p> <pre><code>0000_00ca_ff00_0000 </code></pre></li> <li> <p>But our int only had room for</p> <pre><code>ff00_0000 </code></pre></li> <li> <p>Which widens to the <code>uint64_t</code></p> <pre><code>ffff_ffff_ff00_0000 </code></pre></li> </ul> <h2 id="integer-promotions-and-arithmetic-conversions">Integer Promotions and Arithmetic Conversions</h2> <p>So you see, <code>int</code> is still very much the preferred type for integral literals and arithmetic.</p> <p>This preference is embedded in the core rules underlying C arithmetic:</p> <ul> <li>The <strong>integer promotions,</strong> which describe how integral types smaller than <code>int</code> get promoted to <code>int</code> or <code>unsigned int</code></li> <li>The <strong>usual arithmetic conversions,</strong> which describe how to pick a common type for the arguments to an arithmetic operation, and what the final result type should be.</li> </ul> <p>Both rules make extensive use of the <strong>integer conversion rank</strong> of the various integral types, which we can roughly summarize as:</p> <ul> <li>Bigger integral types have a higher rank.</li> <li>Unsigned and signed types of the same size have the same rank.</li> </ul> <p>Put all this together, and you can draw up a big spreadsheet of what value converts how with what other type of value, and how the arithmetic operation&rsquo;s result type gets picked.</p> <p>And that&rsquo;s not even to speak of the fun we can have with the limited precision provided by a fixed-width integral type, namely overflow (<code>INT_MAX + 1</code>) and underflow (<code>INT_MIN - 1</code>).</p> <h2 id="what-do-you-do">What do you do?</h2> <ul> <li>Crank up compiler warnings (<code>-Weverything -Wextra -Werror</code>)</li> <li>Be suspicious of arithmetic: <ul> <li>Are the types right, even after promotion?</li> <li>Can this overflow?</li> <li>Can this underflow?</li> </ul> </li> <li>Be doubly suspicious of external data, whether from files, the network, or even your own web service.</li> </ul> <h2 id="further-reading">Further Reading</h2> <p>Even CPU simulator writers can get it wrong, as shown by <a href="http://kqueue.org/blog/2013/09/17/cltq/">The cltq story</a>.</p> <p>Arithmetic problems regularly feature as security vulnerabilities; you can dig into this angle starting with <a href="https://www.securecoding.cert.org/confluence/display/seccode/INT02-C.+Understand+integer+conversion+rules">INT02-C. Understand integer conversion rules</a> from the CERT Secure Coding Standards.</p> <p>And you can dive deep into the details, and several example vulnerabilities, starting with <a href="https://www.informit.com/articles/article.aspx?p=686170&amp;seqNum=5">Type Conversions</a> from the &ldquo;C Language Issues for Application Security&rdquo; chapter of McDonald et. al.&lsquo;s <em>Art of Software Security Assessment.</em></p> Why a 64-bit iPhone? https://jeremywsherman.com/blog/2013/09/12/why-a-64-bit-iphone/ Thu, 12 Sep 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/09/12/why-a-64-bit-iphone/ Apple announced a couple days ago an iPhone with a 64-bit processor. Some people worried because their code was unclean. Those people should read Compatibility Basics and follow it up with a strong draught of Apple&rsquo;s 64-Bit Transition Guide from the last time Apple transitioned a platform from 32- to 64-bit. Some people wondered why bother. &ldquo;It&rsquo;s just about the RAM, innit? Who&rsquo;s gonna put that much blasted RAM in a mobile device? <p>Apple announced a couple days ago an iPhone with a 64-bit processor.</p> <p>Some people worried because their code was unclean. Those people should read <a href="https://jeremywsherman.com/blog/2013/08/26/compatibility-basics/">Compatibility Basics</a> and follow it up with a strong draught of Apple&rsquo;s <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/64bitPorting/MakingCode64-BitClean/MakingCode64-BitClean.html#//apple_ref/doc/uid/TP40001064-CH226-SW2">64-Bit Transition Guide</a> from the last time Apple transitioned a platform from 32- to 64-bit.</p> <p>Some people wondered why bother. &ldquo;It&rsquo;s just about the RAM, innit? Who&rsquo;s gonna put that much blasted RAM in a mobile device?&rdquo;</p> <p>Well, someone will, and sooner rather than later. But regardless of that, there are other reasons to go 64-bit.</p> <p>We hear and remember &ldquo;we&rsquo;re going 64-bit&rdquo;. But it&rsquo;s not just a word width change: it&rsquo;s a whole shift in processor, ABI, and, in general, a chance to make a clean break from the past.</p> <p>A prime motivation behind moving from x86 to x86_64 was how tremendously register-starved x86 was.</p> <p>ARM isn&rsquo;t exactly register-starved, but it&rsquo;s got some rough edges. Nowhere near as rough as the Intel lineage, but rough nonetheless.</p> <p>Transitioning to 64-bit gives ARM that same opportunity. Only they weren&rsquo;t forced into it so soon, so they got to watch how everyone else tried (and <a href="https://duckduckgo.com/?q=death+of+itanium">failed</a>, and retried) to do it.</p> <p>I&rsquo;m no great guru of ARM, so I will instead direct you to the August 2012 write-up by Mr.\ David Kanter over at Real World Technologies, <a href="http://www.realworldtech.com/arm64/">ARM Goes 64-bit</a> for details.</p> <p>The key bits for me are:</p> <ul> <li>More registers.</li> <li>Bigger registers.</li> <li>Yup, even the vector registers get bigger and more numerous.</li> <li>Simplifications that make it easier for those of us coming from Intel-land: <ul> <li>The end of conditional suffixes on all the instructions.</li> <li>Load/pop multiple replaced by load/store pair.</li> <li>An exception-level system that more closely resembles the ring-system and replaces the far more complicated eight (!) processor modes of ARMv7.</li> </ul> </li> <li>A memory model and corresponding instructions that jive well with the memory model and atomic operation support in the most recent revisions of C++ and C.</li> <li>And yes, we can now directly address more RAM with a simple index. Nothing like <a href="http://msdn.microsoft.com/en-us/library/windows/hardware/gg487503.aspx">PAE</a> or manual DOSseriffic segment-indirection needed.</li> </ul> <p>Though it&rsquo;s less RAM than you might naïvely think:</p> <blockquote> <p>Currently, AArch64 features two 48-bit virtual address spaces, one for the kernel and one for applications. Application addressing starts at 0 and grows upwards, while kernel space grows down from 2^64; any references to unmapped addresses in between will trigger a fault. Pointers are sign extended to 64-bits, and can optionally be configured to use the upper 8-bits for tagging pointers with additional information. (<a href="http://www.realworldtech.com/arm64/4/">&ldquo;Virtual Address Space&rdquo;</a>)</p> </blockquote> <p>(I&rsquo;d point you at the horse&rsquo;s mouth, but as of this time, almost all the ARMv8 docs are only available for those cosy with ARM, not us plebs. An exception is the <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0677b/BBADCJBF.html">ARMv8 64-bit architecture overview</a>, which provides a concise one-pager of a description of the changes from ARMv7.)</p> Compatibility Basics https://jeremywsherman.com/blog/2013/08/26/compatibility-basics/ Mon, 26 Aug 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/08/26/compatibility-basics/ Supporting multiple platforms – &ldquo;platform&rdquo; being the ramshackle combination of languages, libraries, OS, hardware, etc., that your code depends on to run – is a burden. But you&rsquo;ve no choice once you push data out of your application and off into the wide wide world. Any data you write to disk or network is potentially going to end up being read back on another platform. It could be something wildly different – data sent from an iPhone 5 to a little-endian MIPS machine on someone&rsquo;s desktop halfway across the world – or it could be something less drastic, like an old file written to disk by version 1 of your application and then read back several OS and application upgrades later. <p>Supporting multiple platforms – &ldquo;platform&rdquo; being the ramshackle combination of languages, libraries, OS, hardware, etc., that your code depends on to run – is a burden.</p> <p>But you&rsquo;ve no choice once you push data out of your application and off into the wide wide world. Any data you write to disk or network is potentially going to end up being read back on another platform. It could be something wildly different – data sent from an iPhone 5 to a little-endian MIPS machine on someone&rsquo;s desktop halfway across the world – or it could be something less drastic, like an old file written to disk by version 1 of your application and then read back several OS and application upgrades later.</p> <p>There are techniques you can adopt to mitigate problems, but many applications will founder on far simpler issues well before they reach that level of sophistication.</p> <p>You likely take for granted:</p> <ul> <li>How large an integer is.</li> <li>What happens when you add 1 to the largest representable signed integer on your platform.</li> <li>Whether the bytes in a word are stored big-end or little-end first.</li> <li>That C strings are encoded using UTF-8.</li> <li>Struct padding is always the same, right?</li> </ul> <p>But all these assumptions are a trap:</p> <ul> <li> <p>Integer size varies from machine to machine.</p> <p>A user ditches their 32-bit machine and restores all their data to a 64-bit machine. Can your app still load its saved files from before the migration?</p> <p>OS X developers of some years past are well-acquainted with this problem, and Apple wrote the <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/64bitPorting/MakingCode64-BitClean/MakingCode64-BitClean.html#//apple_ref/doc/uid/TP40001064-CH226-SW2">64-Bit Transition Guide</a> to aid them – and now you – in coping with this issue.</p> </li> <li> <p>Integer overflow is <a href="http://blog.regehr.org/archives/213">undefined behavior</a> in C. Unchecked overflow also often presents a security risk. When a C program is compiled for an architecture using <a href="http://mathforum.org/library/drmath/view/54344.html">two&rsquo;s-complement representation</a> for integers (read: pretty much everything you&rsquo;ll likely ever compile for, but not everything someone else might want to compile your code for), often integers will wrap around from their largest to their smallest representable value on overflow.</p> <p>But not always, and sometimes the behavior changes as you change compiler optimizations, so don&rsquo;t count on it!</p> </li> <li> <p>Endianness bites most people when they first do networking.</p> <p>They soon find <code>ntohs</code>/<code>ntohl</code> and <code>htons</code>/<code>htonl</code>, which are used to convert port numbers (s, short, 16-bit) and IPv4 addresses in numeric (l, long, 32-bit) format between net (i.e., big) and host-endian representations.</p> <p>There&rsquo;s no great magic hidden in these functions, except that they bake in knowledge of whether the host platform stores bytes in network byte order natively or requires byte swapping (or <a href="https://en.wikipedia.org/wiki/Endian#Middle-endian">more complicated hijinks</a>) to convert to and from network byte order.</p> <p>A simple byte-swap requires no great magic:</p> <ul> <li>Initialize an output value to all zeroes.</li> <li>Repeatedly: <ul> <li>Rotate the byte you&rsquo;re working on now to be rightmost.</li> <li>Mask it off using a bit-and.</li> <li>Shift it into the correct (mirrored about the middle) position.</li> </ul> </li> <li>Bit-or it into the output value.</li> </ul> </li> </ul> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#75715e">/** Swaps bytes, and so swaps big/little endianness. </span><span style="color:#75715e"> * (Hope your platform doesn&#39;t use packed binary-coded decimal!) */</span> uint16_t <span style="color:#a6e22e">byteswap16</span>(uint16_t in) { { uint16_t out <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; <span style="color:#66d9ef">for</span> (size_t i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>, e <span style="color:#f92672">=</span> <span style="color:#66d9ef">sizeof</span>(in); i <span style="color:#f92672">&lt;</span> e; <span style="color:#f92672">++</span>i) { uint16_t shifted_right <span style="color:#f92672">=</span> (in <span style="color:#f92672">&gt;&gt;</span> (i <span style="color:#f92672">*</span> CHAR_BIT)); uint16_t byte_i <span style="color:#f92672">=</span> shifted_right <span style="color:#f92672">&amp;</span> <span style="color:#ae81ff">0xFF</span>; uint16_t mirrored <span style="color:#f92672">=</span> (byte_i <span style="color:#f92672">&lt;&lt;</span> ((e <span style="color:#f92672">-</span> (i <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>)) <span style="color:#f92672">*</span> CHAR_BIT)); out <span style="color:#f92672">|=</span> mirrored; } <span style="color:#66d9ef">return</span> out; } </code></pre></div><p>But endianness is also an issue for binary data files, which includes many applications&rsquo; document formats. If you just spit raw bytes to disk, in whatever order you find them in-memory, then you&rsquo;ll run into trouble when an app in a different endian environment slurps that file in.</p> <ul> <li>C strings can use whatever encoding. If you&rsquo;re doing data interchange, you need to make sure the encoding is specified, and take the appropriate steps to convert strings to and fro.</li> </ul> <p>All of these traps are straightforward to avoid once you&rsquo;re aware of them, but are a royal pain to redress after you&rsquo;ve built a mountain of code atop platform-naïve foundations.</p> <p>Unless you&rsquo;ve got these right, don&rsquo;t even bother worrying about the more obvious (and <code>#ifdef</code>-multiplying) differences between platforms.</p> Retain Still Matters https://jeremywsherman.com/blog/2013/08/07/retain-still-matters/ Wed, 07 Aug 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/08/07/retain-still-matters/ <p>I recently received a question about this particular bit of code:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#75715e">//cc -Wall -Wextra -Weverything -g -framework Foundation -fobjc-arc \ </span><span style="color:#75715e">//nsstring-weak.m -o nsstring-weak </span><span style="color:#75715e"></span><span style="color:#75715e">#import &lt;Foundation/Foundation.h&gt; </span><span style="color:#75715e"></span> <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">main</span>(<span style="color:#66d9ef">void</span>) { <span style="color:#75715e">/* nsstring-weak.m:7:22: </span><span style="color:#75715e"> * warning: assigning retained object to weak variable; object will be released </span><span style="color:#75715e"> * after assignment [-Warc-unsafe-retained-assign] */</span> NSString <span style="color:#f92672">*</span><span style="color:#66d9ef">__weak</span> stringInit <span style="color:#f92672">=</span> [[NSString alloc] initWithFormat:<span style="color:#e6db74">@&#34;Lewis&#34;</span>]; NSLog(<span style="color:#e6db74">@&#34;stringInit: %@&#34;</span>, stringInit); <span style="color:#75715e">// stringInit logs as (null) </span><span style="color:#75715e"></span> NSString <span style="color:#f92672">*</span><span style="color:#66d9ef">__weak</span> stringLiteral <span style="color:#f92672">=</span> <span style="color:#e6db74">@&#34;String&#34;</span>; NSLog(<span style="color:#e6db74">@&#34;stringLiteral: %@&#34;</span>, stringLiteral); NSString <span style="color:#f92672">*</span><span style="color:#66d9ef">__weak</span> stringFormat <span style="color:#f92672">=</span> [NSString stringWithFormat:<span style="color:#e6db74">@&#34;SomeString&#34;</span>]; NSLog(<span style="color:#e6db74">@&#34;stringFormat: %@&#34;</span>, stringFormat); } </code></pre></div><p>As the comment points out, clang warns about the first assignment, the one to <code>stringInit</code>, but not to any of the others.</p> <p>You might read this as, &ldquo;Assigning to weak blah blah blah released after assignment.&rdquo; OK, well, now that you point it out clang, duh. I&rsquo;ll fix that.</p> <p>But then you look at the other two assignments: Why is clang OK with these?</p> <p>I recently received a question about this particular bit of code:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-obj-c" data-lang="obj-c"><span style="color:#75715e">//cc -Wall -Wextra -Weverything -g -framework Foundation -fobjc-arc \ </span><span style="color:#75715e">//nsstring-weak.m -o nsstring-weak </span><span style="color:#75715e"></span><span style="color:#75715e">#import &lt;Foundation/Foundation.h&gt; </span><span style="color:#75715e"></span> <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">main</span>(<span style="color:#66d9ef">void</span>) { <span style="color:#75715e">/* nsstring-weak.m:7:22: </span><span style="color:#75715e"> * warning: assigning retained object to weak variable; object will be released </span><span style="color:#75715e"> * after assignment [-Warc-unsafe-retained-assign] */</span> NSString <span style="color:#f92672">*</span><span style="color:#66d9ef">__weak</span> stringInit <span style="color:#f92672">=</span> [[NSString alloc] initWithFormat:<span style="color:#e6db74">@&#34;Lewis&#34;</span>]; NSLog(<span style="color:#e6db74">@&#34;stringInit: %@&#34;</span>, stringInit); <span style="color:#75715e">// stringInit logs as (null) </span><span style="color:#75715e"></span> NSString <span style="color:#f92672">*</span><span style="color:#66d9ef">__weak</span> stringLiteral <span style="color:#f92672">=</span> <span style="color:#e6db74">@&#34;String&#34;</span>; NSLog(<span style="color:#e6db74">@&#34;stringLiteral: %@&#34;</span>, stringLiteral); NSString <span style="color:#f92672">*</span><span style="color:#66d9ef">__weak</span> stringFormat <span style="color:#f92672">=</span> [NSString stringWithFormat:<span style="color:#e6db74">@&#34;SomeString&#34;</span>]; NSLog(<span style="color:#e6db74">@&#34;stringFormat: %@&#34;</span>, stringFormat); } </code></pre></div><p>As the comment points out, clang warns about the first assignment, the one to <code>stringInit</code>, but not to any of the others.</p> <p>You might read this as, &ldquo;Assigning to weak blah blah blah released after assignment.&rdquo; OK, well, now that you point it out clang, duh. I&rsquo;ll fix that.</p> <p>But then you look at the other two assignments: Why is clang OK with these?</p> <p>Here&rsquo;s the warning again:</p> <blockquote> <p>nsstring-weak.m:7:22: warning: assigning retained object to weak variable; object will be released after assignment [-Warc-unsafe-retained-assign]</p> </blockquote> <p>It turns out the real keyword in that message is the one you very well might have glossed over the first time: <strong>retained.</strong></p> <p>That&rsquo;s the difference between the three object references:</p> <ul> <li><code>stringInit</code> refers to an object returned by alloc–init. This was returned retained, and will be lovingly put to death ASAP by ARC.</li> <li><code>stringLiteral</code> refers to a string literal. This will outlast even the run of the program; it&rsquo;s chilling in static memory.</li> <li><code>stringFormat</code>'s referent was returned by a message parallel to that of <code>stringInit</code>'s, with one difference: the return value is autoreleased, not retained.</li> </ul> <p>So you can still run into differences between objects returned from an alloc–init pair and those returned autoreleased under ARC, even in a bit of wotsit-wotsit code as minimal as this.</p> <p>If you&rsquo;d like to learn more, the nearly definitive reference on ARC lives <a href="http://clang.llvm.org/docs/AutomaticReferenceCounting.html">in the Clang docs</a>.</p> <p>The actually definitive reference, of course, is <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/">clang&rsquo;s source code</a> coupled with the <a href="http://opensource.apple.com/source/objc4/objc4-532.2/">Obj-C runtime code</a>.</p> MissingM: Ansible and Salt: A detailed comparison https://jeremywsherman.com/blog/2013/08/06/missingm-ansible-and-salt-a-detailed-comparison/ Tue, 06 Aug 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/08/06/missingm-ansible-and-salt-a-detailed-comparison/ http://missingm.co/2013/06/ansible-and-salt-a-detailed-comparison/ If you haven’t heard of them before, Ansible and Salt are frameworks that let you automate various system tasks. The biggest advantage that they have relative to other solutions like Chef and Puppet is that they are capable of handling not only the initial setup and provisioning of a server, but also application deployment, and command execution. This means you don’t need to augment them with other tools like Capistrano, Fabric or Func. <p><a href="http://missingm.co/2013/06/ansible-and-salt-a-detailed-comparison/">http://missingm.co/2013/06/ansible-and-salt-a-detailed-comparison/</a></p> <blockquote> <p>If you haven’t heard of them before, Ansible and Salt are frameworks that let you automate various system tasks. The biggest advantage that they have relative to other solutions like Chef and Puppet is that they are capable of handling not only the initial setup and provisioning of a server, but also application deployment, and command execution. This means you don’t need to augment them with other tools like Capistrano, Fabric or Func.</p> <p>[…]</p> <p>As an experiment, I decided to write a collection of Ansible Roles and Salt States to perform the same set of tasks and configure a brand new Ubuntu 12.04.2 LTS server [to run a Sinatra webapp under Nginx + Passenger with a Falcon-patched Ruby].</p> </blockquote> <p>After my recent hosting adventures, I decided to investigate configuration management, and picked up O&rsquo;Reilly&rsquo;s <em>Test-Driven Infrastructure with Chef</em>.</p> <p>I haven&rsquo;t gotten to the good parts, but let me just gently caution that you should not try to work the exercises under OS X, and that running them in a VM is also advisable.</p> <p>I&rsquo;ve already seen enough, though, to hazard a guess that Ansible might be more my speed than Chef.</p> <p>The post above by <a href="https://github.com/jlund">Joshua Lund</a> provides a good flavor of Ansible and its spirit-brother, Salt. (This solution-space is getting pretty crowded, eh?)</p> Go Reflection Codex https://jeremywsherman.com/blog/2013/08/05/go-reflection-codex/ Mon, 05 Aug 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/08/05/go-reflection-codex/ http://jimmyfrasche.github.io/go-reflection-codex/ Each article contains some snippets of Go code followed by the equivalent code using the reflect package. If you were foggy about how to work with reflection in Go, this should provide more than enough examples and tips. Share and enjoy! <p><a href="http://jimmyfrasche.github.io/go-reflection-codex/">http://jimmyfrasche.github.io/go-reflection-codex/</a></p> <blockquote> <p>Each article contains some snippets of Go code followed by the equivalent code using the reflect package.</p> </blockquote> <p>If you were foggy about how to work with reflection in Go, this should provide more than enough examples and tips.</p> <p>Share and enjoy!</p> /var/log/bhs: golang impressions https://jeremywsherman.com/blog/2013/08/05/var/log/bhs-golang-impressions/ Mon, 05 Aug 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/08/05/var/log/bhs-golang-impressions/ http://blog.bensigelman.org/post/56158760736/golang-impressions At this point, Go became a viable contender. However, when I was doing my initial research about it, I had trouble finding high-level impressions from recent converts that (a) sounded like they were written by an author with substantial experiences building systems in other languages, and (b) didn’t seem zealous. This post is my attempt to provide the sort of overview I would have valued reading before I started my own investigation into the language. <p><a href="http://blog.bensigelman.org/post/56158760736/golang-impressions">http://blog.bensigelman.org/post/56158760736/golang-impressions</a></p> <blockquote> <p>At this point, Go became a viable contender. However, when I was doing my initial research about it, I had trouble finding high-level impressions from recent converts that (a) sounded like they were written by an author with substantial experiences building systems in other languages, and (b) didn’t seem zealous. This post is my attempt to provide the sort of overview I would have valued reading before I started my own investigation into the language.</p> </blockquote> Surfing with ctags https://jeremywsherman.com/blog/2013/07/25/surfing-with-ctags/ Thu, 25 Jul 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/07/25/surfing-with-ctags/ Big code, little time, and you&rsquo;ve work to do. Where&rsquo;s that function at? What&rsquo;s in that struct? There&rsquo;s an app for that: ctags. In Brief The short version: Install exuberant ctags. ctags -R . in the project&rsquo;s top-level directory to generate the tags file. vim $file from within the same directory as your tags file. Alternatively: vim -t $tagname to jump straight to that tag. CTRL-] while cursor in a tag (roughly fn/var/other name) pushes your current location then jumps to definition of the tag you were on. <p>Big code, little time, and you&rsquo;ve work to do.</p> <p>Where&rsquo;s that function at? What&rsquo;s in that struct?</p> <p>There&rsquo;s an app for that: <strong>ctags</strong>.</p> <h2 id="in-brief">In Brief</h2> <p>The short version:</p> <ul> <li>Install exuberant ctags.</li> <li><code>ctags -R .</code> in the project&rsquo;s top-level directory to generate the <strong>tags</strong> file.</li> <li><code>vim $file</code> from within the same directory as your <strong>tags</strong> file. <ul> <li>Alternatively: <code>vim -t $tagname</code> to jump straight to that tag.</li> </ul> </li> <li><code>CTRL-]</code> while cursor in a tag (roughly fn/var/other name) pushes your current location then jumps to definition of the tag you were on.</li> <li><code>CTRL-t</code> pops back up the tag stack, so you can continue where you left off.</li> <li><code>:help tags</code> for more, but that&rsquo;s about all you need.</li> </ul> <p>(The astute vimmer will now note that the ^]/^t pair is the same one you use to jump around in vim help files. Handy, that.)</p> <p>If you forget this, just <code>man ctags</code>, <code>/HOW TO USE</code>, and read that very brief section.</p> <h2 id="ctags">Ctags!</h2> <p>You don&rsquo;t want just any ctags. You want the exuberant kind. Don&rsquo;t ask why, just trust me: <code>brew install ctags</code> and get yourself some <a href="http://ctags.sourceforge.net/">exuberant ctags</a>.</p> <h2 id="caramel-apple-tarball">Caramel Apple Tarball</h2> <p>To demonstrate, nab yourself delicious Apple-flavored <a href="http://opensource.apple.com/tarballs/Libc/Libc-825.26.tar.gz">libc</a>. This particular variety is the one included with 10.8.3. <code>tar xzf Libc-825.26.tar.gz</code>, <code>cd Libc-825.26</code> into the resulting directory, and wow that&rsquo;s a good chunk of stuff.</p> <h2 id="surfing-ctags">Surfing Ctags</h2> <p>Now it&rsquo;s time for ctags. First, we generate the <strong>tags</strong> file by walking the entire directory hierarchy we&rsquo;re sitting at the root of. Don&rsquo;t worry, it&rsquo;s not so bad as all that, just a <code>-R</code>ecurse flag and a <code>.</code> directory to kick things off:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">Libc-825.26% ctags -R . Libc-825.26% wc -l tags <span style="color:#ae81ff">12832</span> tags </code></pre></div><p>Thassa lotta tags.</p> <p>Let&rsquo;s jump into the thick of it. Fire up <code>vim</code> at the definition of <code>pthread_cond_signal</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">Libc-825.26% vi -t pthread_cond_signal </code></pre></div><p>We find ourselves with cursor blinking right on the <code>p</code> at the start of <code>pthread_cond_signal</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#75715e">/* </span><span style="color:#75715e"> * Signal a condition variable, waking only one thread. </span><span style="color:#75715e"> */</span> <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">pthread_cond_signal</span>(pthread_cond_t <span style="color:#f92672">*</span>cond) { <span style="color:#66d9ef">return</span> pthread_cond_signal_thread_np(cond, NULL); } </code></pre></div><p>This is where things would start to suck if we didn&rsquo;t have ctags. The entire function is just a call to yet another function. The <code>_np</code> non-portable suffix makes me think maybe it will be somewhere else. Maybe they exiled all the non-portable functions to a different file? Who knows. Even better: we don&rsquo;t care.</p> <p><code>/_np</code>, <code>RET</code>, and hit <code>CTRL-]</code> to jump right to that function:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#75715e">/* </span><span style="color:#75715e"> * Signal a condition variable, waking a specified thread. </span><span style="color:#75715e"> */</span> <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">pthread_cond_signal_thread_np</span>(pthread_cond_t <span style="color:#f92672">*</span>ocond, pthread_t <span style="color:#66d9ef">thread</span>) { npthread_cond_t <span style="color:#f92672">*</span> cond <span style="color:#f92672">=</span> (npthread_cond_t <span style="color:#f92672">*</span>)ocond; <span style="color:#66d9ef">int</span> sig <span style="color:#f92672">=</span> cond<span style="color:#f92672">-&gt;</span>sig; </code></pre></div><p>Ooh, fun, we transition from an old-cond to a new-style struct. (Maybe that <code>_np</code> suffix was <code>new pthreads</code> this time.) What&rsquo;s the difference between old and new? Let&rsquo;s take a look-see and find out.</p> <p><code>/np</code>, <code>n</code>, <code>RET</code>, <code>CTRL-]</code>. Looks like the definition of <code>npthread_cond_t</code> lives right below the original <code>pthread_cond_t</code> struct definition, so comparing the two is easy. The new version looks to have swapped out most of the old guts:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#75715e">/* </span><span style="color:#75715e"> * Condition variables </span><span style="color:#75715e"> */</span> <span style="color:#75715e">#define _PTHREAD_COND_T </span><span style="color:#75715e"></span><span style="color:#66d9ef">typedef</span> <span style="color:#66d9ef">struct</span> _pthread_cond { <span style="color:#66d9ef">long</span> sig; <span style="color:#75715e">/* Unique signature for this structure */</span> pthread_lock_t lock; <span style="color:#75715e">/* Used for internal mutex on structure */</span> uint32_t waiters:<span style="color:#ae81ff">15</span>, <span style="color:#75715e">/* Number of threads waiting */</span> sigspending:<span style="color:#ae81ff">15</span>, <span style="color:#75715e">/* Number of outstanding signals */</span> pshared:<span style="color:#ae81ff">2</span>; <span style="color:#66d9ef">struct</span> _pthread_cond <span style="color:#f92672">*</span>next, <span style="color:#f92672">*</span>prev; <span style="color:#75715e">/* List of condition variables using mutex */</span> <span style="color:#66d9ef">struct</span> _pthread_mutex <span style="color:#f92672">*</span>busy; <span style="color:#75715e">/* mutex associated with variable */</span> semaphore_t sem; <span style="color:#75715e">/* Kernel semaphore */</span> } pthread_cond_t; </code></pre></div><p>for free space and a different approach to getting things done:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#66d9ef">typedef</span> <span style="color:#66d9ef">struct</span> _npthread_cond { <span style="color:#66d9ef">long</span> sig; <span style="color:#75715e">/* Unique signature for this structure */</span> pthread_lock_t lock; <span style="color:#75715e">/* Used for internal mutex on structure */</span> uint32_t rfu:<span style="color:#ae81ff">29</span>, <span style="color:#75715e">/* not in use*/</span> misalign: <span style="color:#ae81ff">1</span>, <span style="color:#75715e">/* structure is not aligned to 8 byte boundary */</span> pshared:<span style="color:#ae81ff">2</span>; <span style="color:#66d9ef">struct</span> _npthread_mutex <span style="color:#f92672">*</span>busy; <span style="color:#75715e">/* mutex associated with variable */</span> uint32_t c_seq[<span style="color:#ae81ff">3</span>]; <span style="color:#75715e">#if defined(__LP64__) </span><span style="color:#75715e"></span> uint32_t reserved[<span style="color:#ae81ff">3</span>]; <span style="color:#75715e">#endif </span><span style="color:#75715e">/* __LP64__ */</span><span style="color:#75715e"> </span><span style="color:#75715e"></span>} npthread_cond_t; </code></pre></div><p>There&rsquo;s also LP64 support and some alignment games. Fun times.</p> <p>Well, that&rsquo;s enough struct-gazing, we can pop back on up to where we were with <code>CTRL-T</code> and resume looking at the implementation of <code>pthread_cond_signal_thread_np</code>, right where we left off.</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#66d9ef">int</span> pthread_cond_signal_thread_np(pthread_cond_t <span style="color:#f92672">*</span>ocond, pthread_t <span style="color:#66d9ef">thread</span>) </code></pre></div><p>I bet it&rsquo;d be interesting to see what happens when that <code>thread</code> argument is <code>NULL</code>, as it is in a standard <code>pthread_cond_signal</code> call.</p> <p>And when might it be called with a non-<code>NULL</code> value?</p> <p>Curious and curiouser.</p> <p>I leave you to it, dear reader, with exuberant ctags frolicking and wagging tail at your side.</p> <p><strong>Added 2013-07-26T15:03:08Z-0400:</strong> This post is an introductory walkthrough. Once you&rsquo;ve got the hang of the basics, check out the discussion at <a href="http://j.mp/1aNjqfY">/r/vim</a> for further pointers.</p> Intruder Alert https://jeremywsherman.com/blog/2013/07/21/intruder-alert/ Sun, 21 Jul 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/07/21/intruder-alert/ The Apple developer center has been down since Thursday. The only information available from Apple was a &ldquo;we&rsquo;ll be back soon, and we&rsquo;re not yanking any apps&rdquo; reassurance. Thought maybe to check the dev forums? Those were down too, with roughly the same message. Finally, Sunday evening, Apple revealed, by email and by updating the holding page at the iOS Dev Center, that they yanked their entire system down after <p>The Apple developer center has been down since Thursday. The only information available from Apple was a &ldquo;we&rsquo;ll be back soon, and we&rsquo;re not yanking any apps&rdquo; reassurance. Thought maybe to check the dev forums? Those were down too, with roughly the same message.</p> <p>Finally, Sunday evening, Apple revealed, by email and by updating the holding page at the iOS Dev Center, that they yanked their entire system down after</p> <blockquote> <p>an intruder attempted to secure personal information of our registered developers from our developer website.</p> </blockquote> <p>Well, that doesn&rsquo;t sound good, does it? It gets better!</p> <blockquote> <p>Sensitive personal information was encrypted and cannot be accessed –</p> </blockquote> <p>Whoah there, Apple, I&rsquo;ma let you finish, but <em>cannot</em> be accessed? Then how were you accessing it? Plainly it can be.</p> <p>How hard it is for a cracker to access will depend on how it was encrypted. Apple have seen fit to reassure us that it was &ldquo;encrypted&rdquo;, as if all should be forgiven at the merest mention of &ldquo;encryption&rdquo;. Wave the magic encryption wand, and presto-changeo, insta-secure!</p> <p>So, don&rsquo;t worry, security fairies are at work. Just a slight slip-up:</p> <blockquote> <p>we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed.</p> </blockquote> <p>Wow, so a good chunk of what you&rsquo;d need to open conversations with people&rsquo;s banks, or credit card companies, or talk your way through most any phone tree. This is great info for social engineering and phishing. Someone could use this info to impersonate you and convince one of your friends or relatives to reveal some information they&rsquo;d rather not share with strangers.</p> <blockquote> <p>In the spirit of transparency, we want to inform you of the issue. We took the site down immediately on Thursday and have been working around the clock since then.</p> </blockquote> <p>Waiting three days to tell everyone is a real class move. Phishers and impersonators have a three-day head-start on you. Enjoy!</p> <p>Now, why would they need to stop the world, yank everything down, and beaver away for three days? It sounds like it took them about this long to figure out what even happened. Who knows how long the actual intrusion occurred and lasted before they <em>noticed</em> it happened on Thursday.</p> <blockquote> <p>In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database</p> </blockquote> <p>I&rsquo;m not clear how they can make a threat vanish by any means. Apple can make changes to better defend against a threat, but there will still be someone out there who wants developers&rsquo; credit card info, billing info, home address, phone, email – whatever personally identifiable information they can get their hands on, they&rsquo;ll take.</p> <p>So it&rsquo;s great they&rsquo;re fixing things. Somehow. Maybe. We still have no idea what actually happened, for all Apple rushed to shower us with transparency.</p> <p>But it sounds like they got caught with their pants down, and are now trying to fix years of neglecting the server-side of their developer-facing services. Technical debt bites hard.</p> <p>Keep an eye on your credit card and bank statements, and put your friends and family on guard for any funny business from someone claiming to be you. I hope we learn more details of this debacle soon, and I wish good luck to the poor folk at Apple who have been rousted from their beds and whip-cracked through the weekend.</p> <figure> <img src="https://jeremywsherman.com/images/posts/2013/apple-intrusion-maintenance.png"/> <figcaption> <h4>Screenshot of the intrusion maintenance message</h4> </figcaption> </figure> Moving House https://jeremywsherman.com/blog/2013/07/19/moving-house/ Fri, 19 Jul 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/07/19/moving-house/ Arch Linux logo For the last two years, this blog lived in shared hosting over at DreamHost. This worked great. I had never had my own domain or server-in-the-sky. There&rsquo;s a learning curve to this &ldquo;now your stuff lives out there&rdquo; experience. But I learned, and shared hosting grew constricting. I wanted to run ZNC to keep up with the office chitchat. This is rightly verboten under DreamHost&rsquo;s shared hosting plan. <figure> <img src="https://jeremywsherman.com/images/posts/2013/archlinux-logo-dark.png"/> <figcaption> <h4>Arch Linux logo</h4> </figcaption> </figure> <p>For the last two years, this blog lived in shared hosting over at DreamHost.</p> <p>This worked great. I had never had my own domain or server-in-the-sky. There&rsquo;s a learning curve to this &ldquo;now your stuff lives out there&rdquo; experience.</p> <p>But I learned, and shared hosting grew constricting.</p> <p>I wanted to run <a href="http://znc.in/">ZNC</a> to keep up with the office chitchat. This is rightly verboten under DreamHost&rsquo;s shared hosting plan.</p> <h2 id="vps">VPS</h2> <p>Why not move to a VPS? Because $$$. Every plan I found cost as much as my home Internet connection does per month, if not more. I can&rsquo;t justify that for hosting a mere blog.</p> <p>But a discussion on <a href="https://join.app.net/from/xmyyfkgtjz">App.Net</a> introduced me to <a href="https://www.digitalocean.com/?refcode=2346c12dbae4">Digital Ocean</a>, and boy was I happy to meet them.</p> <p>$5/mo VPS. A sweet little server all to my self, at half the price I was paying for shared hosting. My fingers itched to make the jump, and the first weekend after I learnt of it, I started moving in.</p> <h2 id="gateway-arch">Gateway Arch</h2> <p>I dove headfirst in. All options are Linux. I&rsquo;m not wedded to anything; I&rsquo;ve used some Ubuntu on and off for the last several years, because that seems to be everyone&rsquo;s favorite distro for VMs. But I have the teensiest host Digital Ocean offers, and Ubuntu is far from my notion of svelte. I want a change: Arch it is.</p> <p>If you&rsquo;re a Machead like me, you might not be acquainted with Arch. Here is <a href="https://wiki.archlinux.org/index.php/The_Arch_Way">Arch in a nutshell</a>:</p> <ul> <li>Simplicity: Provide a lightweight base which an individual user can shape to meet their needs.</li> <li>Correctness: Aim for compact, simple implementations of distro services.</li> <li>User-Centered: Hand over complete control and responsibility. Lightly opposed to &ldquo;user-friendly&rdquo;.</li> <li>Open: Select or build simple, open-source tools.</li> <li>Freedom: Allow users to choose everything about their system.</li> </ul> <p>As someone who has flirted on and off with <a href="http://tinycorelinux.net/downloads.html">TinyCore Linux</a> – seriously, why don&rsquo;t people base VMs they expect others to download off something that&rsquo;s only 12 MB, and only that much if you need a GUI? – this appealed to me. Hard.</p> <p>I got to skip the install process and jump straight into a VM with some networking tools already installed and some details of integration with the hosting platform already worked out.</p> <h2 id="migrating-from-dreamhost-to-digital-ocean">Migrating from DreamHost to Digital Ocean</h2> <p>So now I had a working machine to call my own, it was time to migrate everything over from shared hosting.</p> <p>I only had four services to move over:</p> <ul> <li>The blog, a simple blob of static pages produced by Octopress.</li> <li>Git, which I use to provide myself with private project hosting.</li> <li>Piwik, which I use as an alternative to Google Analytics.</li> <li>Tiny Tiny RSS, about which see <a href="https://jeremywsherman.com/blog/2013/06/19/farewell-google-reader/">Farewell, Google Reader</a></li> </ul> <p>These required a few other details:</p> <ul> <li>A web server.</li> <li>MySQL, which both Piwik and Tiny Tiny RSS rely on for external storage.</li> <li>PHP. (I&rsquo;m sorry. It makes me sad too.)</li> </ul> <p>The plan was to get everything up and running, then cut over DNS, then shut down the DreamHost shared hosting plan.</p> <p>Arch turned out to have packages for everything I wanted to run except Piwik. Installing that is little more than unzipping an archive in the right place and rigging up your web server.</p> <p>About that.</p> <h2 id="nginx">Nginx</h2> <p>Key to all this is the web server.</p> <p>Instead of Apache, I went with Nginx - it seems lighter, and I didn&rsquo;t want to mess with Apache&rsquo;s configuration files.</p> <p>I&rsquo;d never used Nginx before, but it turns out that enough web searching can vanquish most any ignorance these days.</p> <p>Most everything went fine once I found the <a href="http://nginx.org/en/docs/dirindex.html">alphabetical list of directives</a>, but debugging what stanza handles what can become a pain. Fortunately, it&rsquo;s a pain with a handy solution, as detailed by Justin Carmony in <a href="http://www.justincarmony.com/blog/2012/01/13/debugging-nginx-configuration-trick/">Debugging Nginx Configuration Trick</a>:</p> <ul> <li> <p>Want to see some output to verify you&rsquo;re tickling the place you thought you were when visiting a URL?</p> </li> <li> <p>Redirect to <code>google.com/?q=(your debug message here)</code>:</p> <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-nginx" data-lang="nginx"><span style="color:#66d9ef">redirect</span> <span style="color:#e6db74">^</span> <span style="color:#e6db74">http://www.google.com/?q=HELLO!</span> <span style="color:#e6db74">last</span>; <span style="color:#66d9ef">break</span>; </code></pre></div></li> </ul> <h2 id="to-be-continued">To Be Continued</h2> <p>I reckon that&rsquo;s enough for today. I plan to go over some more of the gory details in future, because they might help someone else the same way a smattering of random blog posts from all over the &lsquo;net helped me.</p> <p>Share and enjoy!</p> Farewell, Google Reader https://jeremywsherman.com/blog/2013/06/19/farewell-google-reader/ Wed, 19 Jun 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/06/19/farewell-google-reader/ <p>Google Reader is shutting down. This is an excellent opportunity to move your blog-reading from the company store over to some fresh property you control.</p> <p>If all you want is a Reader-alike, you have your pick of several options.</p> <p>But I had something else – something <em>better</em> – in mind.</p> <p>Google Reader is shutting down. This is an excellent opportunity to move your blog-reading from the company store over to some fresh property you control.</p> <p>If all you want is a Reader-alike, you have your pick of several options.</p> <p>But I had something else – something <em>better</em> – in mind.</p> <h2 id="reader-schmeader">Reader-Schmeader</h2> <p>First, let&rsquo;s get the Reader-alikes out of the way:</p> <ul> <li><a href="https://github.com/swanson/stringer">Stringer</a>, a Rails app that’s easy to set up running free on Heroku, for which see Eric Dejonckheere’s <a href="http://aya.io/blog/stringer-rss/">excellent and well-illustrated walkthrough</a> (in French).</li> <li><a href="http://tt-rss.org/">Tiny Tiny RSS</a>, a PHP conglomeration that’s easy to set up running wherever you have hosting, for which see Alan Henry’s <a href="http://lifehacker.com/5992404/how-to-build-your-own-syncing-rss-reader-with-tiny-tiny-rss-and-kick-google-reader-to-the-curb">Lifehacker article</a>.</li> </ul> <p>If that&rsquo;s all you&rsquo;re looking for, have at it.</p> <h2 id="a-river-runs-through-it">A River Runs Through It</h2> <p>What I had in mind was somewhat different.</p> <p>Past RSS reader avoidance had taught me to run away from the gotta-read-them-all whack-a-mole of mailbox-style RSS readers. I’d had a better experience with Twitter and similar social streams, even as my follow-list ballooned to be far busier than my RSS feeds ever were.</p> <p>It was Dave Winer’s <a href="">River2</a> that showed me I could have it all for RSS. The only problem: It runs under Windows. I don’t have a Windows web host, nor do I have any interest in maintaining a Windows box, so I had to look elsewhere.</p> <p>So what I wanted instead of The Next (Yet Another) Reader was a solution that would:</p> <ul> <li>run fine on a *nix box</li> <li>present feeds as a River of News (stream-style) rather than as a mailbox</li> <li>have a server component with API, so I could access my feeds however</li> <li>support multiple users so I could share with family or friends</li> </ul> <h2 id="nothing-doing">Nothing Doing</h2> <p>After some looking, the closest things I’d found to the River-spirit were the Planet aggregators, like <a href="http://intertwingly.net/code/venus/">Venus</a>. Planet feeds are really aggregate-feeds that repackage several feeds into one large feed. You can see an example of a planet-river at <a href="http://planet-if.com/">Planet IF</a>.</p> <p>A planet really only hits the ’nix and river bullet points, which is a real let-down as far as a personal solution goes. It won&rsquo;t track where I am in the stream, and it only supports multiple users inasmuch as it has no notion of users. The existing solutions also had a very Rube Goldberg sense to them; not something I could place much confidence in. And subscribing to more feeds would require editing config files.</p> <p>So I looked for a good long time. In the end, I came up with this list of not-quite solutions:</p> <ul> <li>Local only: <ul> <li><a href="http://newsbeuter.org/">Newsbeuter</a></li> <li><a href="http://codezen.org/canto/">Canto</a></li> </ul> </li> <li>Server-side: <ul> <li><a href="http://intertwingly.net/code/venus/">Venus</a></li> <li><a href="http://quick.newsriver.org/">River</a> (Windows only)</li> <li><a href="https://github.com/swanson/stringer">Stringer</a></li> <li><a href="http://www.allthingsrss.com/rss2email/">RSS to Email</a></li> </ul> </li> </ul> <p>Nothing was quite what I wanted. I thought of making my own solution. Then I thought better, and instead gave Newsbeuter a chance.</p> <h2 id="newsbeuter">Newsbeuter</h2> <p>Newsbeuter claims to be “the Mutt of RSS Readers”. Zed Shaw first <a href="http://zedshaw.com/essays/i_want_the_mutt_of_feed_readers.html">used this title</a>, and Newsbeuter has since embraced it.</p> <p>I don’t use mutt, so I can’t speak to the title’s appropriateness, but Newsbeuter works pretty well. It mostly stays out of the way. It’s less a mailbox and more a hierarchical menu. It’s actually surprisingly iPhone-flavored in how its view stacks work.</p> <p>So what of my requirements?</p> <ul> <li>You could scarce dream up a more ’nix reader if you tried. (OK, <a href="http://www.vanheusden.com/rsstail/">rsstail</a> might win Unix-philosophy-wise, but Newsbeuter nails the feel and is more practical.)</li> <li>As far as the River of News experience goes, Newsbeuter still tracks read/unread status per article, but it’s not obnoxious about something being <strong>OMG unread.</strong> It’s configurable enough that you could almost entirely hide read status if you so wished.</li> <li>Newsbeuter is local only. So much for the server-side with API dream. Well, I was about to give up on finding a reader solution entirely, so a half-measure was better than none.</li> <li>And supporting multiple users doesn&rsquo;t mean much without a server component.</li> </ul> <p>So I had something that would sort of work. It didn’t satisfy all my dreams, but it worked. That’s a good start, I figured.</p> <figure> <img src="https://jeremywsherman.com/images/posts/2013/newsbeuter-last-24-hours.png" alt="screenshot of Terminal showing Newsbeuter viewing my Last 24 Hours query feed"/> <figcaption> <h4>Newsbeuter viewing my Last 24 Hours query feed</h4> </figcaption> </figure> <h2 id="rtfm">RTFM</h2> <p>I read through <a href="http://newsbeuter.org/doc/newsbeuter.html">Newsbeuter’s manual</a>. The manual is more comprehensive than most.</p> <p>The manual provides! It turns out I can shape Newsbeuter into a river of news reader. You just need to set up a query feed that shows you recent articles, ignore everything else, and be generous with the “mark all as read” command.</p> <p>And provides again! It turns out Newsbeuter can sync with a server, so long as that server is Google Reader or Tiny Tiny RSS. Well, Google Reader was not an option, but Tiny Tiny RSS certainly was.</p> <p>But it doesn&rsquo;t answer all questions.</p> <p>Could I use query feeds, my solution for creating a semi-river, alongside tt-rss, my solution for keeping feeds subscribed to synced across several devices? I couldn&rsquo;t tell from the manual alone.</p> <p>So I looked up <a href="https://github.com/akrennmair/newsbeuter">Newsbeuter&rsquo;s source</a> on my phone. I expected to give up in frustration after a few taps and wait till I got back to my laptop to git clone and git grep. But I found <a href="https://github.com/akrennmair/newsbeuter/blob/03c0a304d92636df787d20146caef1d4bde010ed/src/ttrss_urlreader.cpp#L23-L45">the answer</a> in no time:</p> <ul> <li><code>ttrss_urlreader</code> first loads all query feeds from the <code>urls</code> file,</li> <li>then it hits the tt-rss API for the list of feed URLs</li> </ul> <p>So I can totally rig up all the query feeds I want alongside using tt-rss as the main feed URL source.</p> <p>The manual doesn&rsquo;t answer all questions, but it answers most of them. For the rest, there&rsquo;s the sourcee. Newsbeuter is written in clean, well-organized C++. It&rsquo;s one rare project that won&rsquo;t make you cry to read the source. Call me impressed.</p> <h2 id="tiny-tiny-rss">Tiny Tiny RSS</h2> <p><a href="http://tt-rss.org/">Tiny Tiny RSS</a>, you say? Well, that will let me sync my feeds across devices. I pull up the <a href="http://tt-rss.org/redmine/projects/tt-rss/wiki/InstallationNotes">tt-rss project’s docs</a> (oh the pain of slow slow Trac); I read the Lifehacker article I linked up above; it looks workable.</p> <p>I set Tiny Tiny RSS up in no time on Dreamhost, and soon I am viewing RSS feeds in my browser. I’m not too keen on the UI; the Actions menu and Preferences sections feel like a sin bin of options, and early 2000s console-style “Loading” screens aren’t really my thing; but, it works!</p> <figure> <img src="https://jeremywsherman.com/images/posts/2013/tt-rss-cargando.jpg" alt="a bar 50% filled with text &#34;Loading. Please wait…&#34; above"/> <figcaption> <h4>Loading. Please wait…</h4> </figcaption> </figure> <p>Once configured, the UI doesn’t matter anyway, because tt-rss is just a sync point, not my reader. Its whole purpose in life is editing the feeds I subscribe to and servicing API requests from my actual readers.</p> <p>But there was a hitch: I couldn’t get Newsbeuter to auth with my tt-rss install no matter what I did. So I turned to my trusty <a href="http://mitmproxy.org/">mitmproxy</a>.</p> <p>Reviewing the server&rsquo;s responses quickly revealed the problem: API access wasn’t turned on for my user! You have to go under settings for your user in Tiny Tiny RSS and turn on the “external API” support manually.</p> <p>Surprise! Without that, everything third-party will fail to work with Tiny Tiny RSS. And it’s disabled by default. Of course.</p> <figure> <img src="https://jeremywsherman.com/images/posts/2013/tt-rss-habilitar-api.png" alt="screenshot of the Preferences screen with the Enable API option highlighted"/> <figcaption> <h4>Enable API: Default is no!</h4> </figcaption> </figure> <h2 id="victory">Victory?</h2> <p>So in the end, I got what I wanted, after a fashion:</p> <ul> <li>I run Newsbeuter locally and tt-rss remotely. Both happily hum along atop ’nix.</li> <li>tt-rss doesn&rsquo;t present a river, but I only use it to edit my feed list. Instead, I use Newsbeuter, which I have transformed into a River-style reader through careful configuration.</li> <li>tt-rss rescues the server-side dream. A very PHP rescue in need of a UX and design team, but I’ll take “it works” over “it doesn’t even exist” any day.</li> <li>And tt-rss has multiple user support.</li> </ul> <p>If you’re looking for an RSS reader solution, and are interested in trying something off the beaten path, give <a href="http://newsbeuter.org/">Newsbeuter</a> and <a href="http://tt-rss.org/">Tiny Tiny RSS</a> a shot. They’re more than the sum of their parts.</p> Big Data https://jeremywsherman.com/blog/2013/06/16/big-data/ Sun, 16 Jun 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/06/16/big-data/ One Finger Moon-Pointed Everyone seems enamored of the accidental scaffolding we find ourselves forced to erect around our growing tower of data exhaust by our increasing cupidity for data data DATA and the treasure we hope we’ll one day find therein. We have to develop machinery to deal with data at the latest greatest scale, but the machines are just nuts and bolts. We needed a garage to park our data-Hummer in. <h2 id="one-finger-moon-pointed">One Finger Moon-Pointed</h2> <p>Everyone seems enamored of the accidental scaffolding we find ourselves forced to erect around our growing tower of data exhaust by our increasing cupidity for data data DATA and the treasure we hope we’ll one day find therein.</p> <p>We have to develop machinery to deal with data at the latest greatest scale, but the machines are just nuts and bolts. We needed a garage to park our data-Hummer in. We had a crew build one by hand using the poor Bronze Age tools available. The final result is eccentric and baroque; it requires constant maintenance just to avoid collapsing; but it solves the problem.</p> <p>The part no-one outside well-insulated research centers seems to be getting all moon-eyed over is the part that matters long-term: you have what was till recently an unthinkable amount of data. How do you extract meaning from it?</p> <p>This gets us into statistics, machine learning, curve fitting, segmentation, clustering, sentiment analysis, etc. That’s the part everyone should be getting excited about.</p> <p>I want to read breathless blog posts about how you have advanced what we currently know how to do statistically. Because I can’t write those posts; I can poke machines and set them on their feet and point them one way or another and dust them off when they fall down, but I know not nearly enough about statistics and numerical analysis.</p> <p>So much of today’s tech news is just oohing and ahhing and cooing at the latest gilded set of Napier’s bones. We’re an industry in love with its slide rule, polishing and primping and extolling its many virtues. We forget these Rube Goldberg contraption abstractions are tools for the real pursuit of knowledge.</p> <p>We’ll use Bayes’ law for the next several centuries. We&rsquo;ll use today&rsquo;s wonder-systems for the next several decades, and we&rsquo;ll curse them as legacy rubbish long before that time is through.</p> <p>See also: Bryan O’Sullivan’s <a href="http://www.serpentine.com/blog/2013/04/30/big-fucking-deal/">“Big Fucking Deal”</a></p> <h2 id="one-serf-disjointed">One Serf Disjointed</h2> <p>There’s another angle on this: Our continuous tech-wow is an obscene celebration of the enclosing of the commons. You’ve worked out how to pen up a bunch of users and soak them for all they’re worth. You’ve built the airiest friendliest most inescapable prison-hotel. To do so required innovations legal political and technical the like of which we could only have dreamt of a century ago. But this is not innovation in the service of anything noble; it’s the sad extraction of value from a benighted underclass of peon users.</p> <p>See also: Hammerbacher’s comments in <a href="http://www.businessweek.com/magazine/content/11_17/b4225060960537.htm">“This Tech Bubble Is Different”</a>; Schneier with <a href="https://www.schneier.com/blog/archives/2013/06/more_on_feudal.html">“More on Feudal Security”</a>.</p> Clean Code Is Not Enough https://jeremywsherman.com/blog/2013/06/09/clean-code-is-not-enough/ Sun, 09 Jun 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/06/09/clean-code-is-not-enough/ So much effort is expended on making code look fine at the micro-level: indentation, naming, method size, etc. So little attention is paid to the macro-level: How does everything come together to effect the desired result? Why is it there? Developers world-wide waste countless hours debugging, not because of anything they have done, but instead because of library authors&rsquo; failure to express preconditions, effects, and expected uses. You can&rsquo;t do anything to guard against this save choose libraries carefully. <p>So much effort is expended on making code look fine at the micro-level: indentation, naming, method size, etc. So little attention is paid to the macro-level: How does everything come together to effect the desired result? Why is it there?</p> <p>Developers world-wide waste countless hours debugging, not because of anything they have done, but instead because of library authors&rsquo; failure to express preconditions, effects, and expected uses. You can&rsquo;t do anything to guard against this save choose libraries carefully. And all too often you have no choice.</p> <p>Literate programming tried to address whole-program understanding. Its core idea: Construct a program via a linear narrative that proceeds by progressive refinement to elaborate the entire program. Code chunks can be incorporated by reference into other chunks. The source code is written for human consumption, not machine. The input to the compiler is ultimately produced by weaving chunks together. For a small example of the pretty-printed human-oriented product, see <a href="http://shanecelis.github.io/">Shane Celis’</a> <a href="http://gnufoo.org/emacsy/minimal-emacsy-example.pdf">Minimal Emacsy Example Program</a>.</p> <p>But literate programming fizzled. The highest level of documentation you&rsquo;ll find in many projects today is class-level. That&rsquo;s not enough to fully comprehend, never mind maintain, a system. Post-hoc analyzers that visualize the rat’s nest of dependencies serve only as adjuvants in sussing out the Escher’s rat’s nest we’ve crafted ourselves.</p> <p>In the hands of a capable author, a literate program explains the wherefores of the code to its recipients. As test-driven development guarantees all developed code can be tested, literate programming guarantees all developed code can be coherently explained from start to finish.</p> <p>But that doesn’t mean the code is correct. And that doesn’t mean the code has been coherently specified. Humans are very limited in their ability to cross-check the validity of a complex, interconnected web of logical statements.</p> <p>This is where specification languages, like <a href="http://staff.washington.edu/jon/z-book/what-is-z.html">Z</a> (pronounced “zed” in this context), come into play. <a href="http://alloy.mit.edu/">Alloy</a> applies model-checking in a limited universe to the specification. But bridging the gap between specification and implementation remains in the hands of human – all too human! – implementors.</p> <p>So we come to the notion of extracting programs from a verified proof. If you go at it right, you only arrive at a proof that everything does what you specified it does by way of producing an existence proof. The proof demonstrates by example that your claim, that your app does the umpteen things you say it does, must be true. Once proven, you can <a href="http://coq.inria.fr/distrib/current/refman/Reference-Manual025.html">extract this latent program</a>, and, ideally, run it.</p> <p>But let us set aside the opium pipe. We’ve come full circle: Unless all the libraries and all the frameworks you are using in developing your program have been developed in such a way that you can incorporate their specifications directly (and what world would that happen in?), or at least documented such that you can make meaningful and tractable claims you can adopt as axioms, you’ll still find yourself falling through the sandcastles that are all we have to use as the latest app’s foundation.</p> <p>We can’t, any solitary one of us, fix this alone. Until everyone starts taking responsibility for building their programs correct from the get-go and documenting them so they can be used in line with the assumptions inherent in the built product, we will all continue to suffer from our continued haze of confusion, no matter how sparkling clean and transcendently clear our own code might be.</p> Notes on QuickCheck https://jeremywsherman.com/blog/2013/05/27/notes-on-quickcheck/ Mon, 27 May 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/05/27/notes-on-quickcheck/ <p>QuickCheck is a nifty library embedded in Haskell for specification-based random testing of programs.</p> <p>Instead of hand-writing your tests as you would with one of the TestUnit frameworks, you specify computable properties about your program functions. The specification includes a description of how to generate inputs to the property equation. The test framework evaluates the property against a fixed number of inputs produced by the spec’s generator and reports whether the results were correct or not.</p> <p>This compresses the amount of test code you have to write. It effectively raises the level of discourse from hand-testing input/output pairs or execution traces to making general assertions you could use to reason about your functions.</p> <p>Random testing is uniquely suited to revealing mistakes in the programmer’s conception of the program, since it can produce inputs you wouldn’t dream of on your own. (Composing generators to produce a big mess of a data structure is pretty nifty, too; hand-writing those inputs is just about the most boring thing ever.)</p> <p>QuickCheck is a nifty library embedded in Haskell for specification-based random testing of programs.</p> <p>Instead of hand-writing your tests as you would with one of the TestUnit frameworks, you specify computable properties about your program functions. The specification includes a description of how to generate inputs to the property equation. The test framework evaluates the property against a fixed number of inputs produced by the spec’s generator and reports whether the results were correct or not.</p> <p>This compresses the amount of test code you have to write. It effectively raises the level of discourse from hand-testing input/output pairs or execution traces to making general assertions you could use to reason about your functions.</p> <p>Random testing is uniquely suited to revealing mistakes in the programmer’s conception of the program, since it can produce inputs you wouldn’t dream of on your own. (Composing generators to produce a big mess of a data structure is pretty nifty, too; hand-writing those inputs is just about the most boring thing ever.)</p> <h3 id="reference">Reference</h3> <p>Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In <em>Proceedings of the fifth ACM SIGPLAN international conference on Functional programming</em> (ICFP &lsquo;00). ACM, New York, NY, USA, 268-279. DOI=10.1145/351240.351266 <a href="http://doi.acm.org/10.1145/351240.351266">http://doi.acm.org/10.1145/351240.351266</a> </p> <h3 id="notes">Notes</h3> <p>“Random testing is especially suitable for functional programs because properties can be stated at a fine grain.”</p> <p>“domain-specific language of <em>testable specifications</em>”</p> <p>“<em>test data generation language</em>”</p> <p>“unless specifically stated otherwise, we always quantify over completely defined finite values.”</p> <p>“the programmer must specify a fixed type at which the law is to be tested”</p> <p>“+ is associative for the type <code>Int</code>, but not for <code>Double</code>!”</p> <p>“In some cases, we can use parametricity [17] to argue that a property holds polymorphically.” ([17] is Wadler’s “Theorems for free!” paper.)</p> <p>“implication combinator” for conditional laws</p> <p>“the result type of the property is changed from Bool to Property” - handles “failed precondition” state in addition to pass/fail</p> <p>“The <code>classify</code> combinator does not change the meaning of a law, but it classifies some of the test cases”</p> <p>“<code>collect</code> will gather all values that are passed to it, and print out a histogram”</p> <p>“risk of [bias towards trivial test cases] every time we use conditional laws, so it is always important to investigate the proportion of trivial cases among those actually tested.”</p> <p>solution: “replace the condition with a <strong>custom data generator</strong>” (emphasis added)</p> <p>“two infinite lists are equal if all finite initial segments are equal”</p> <p>“in general it is not clear how to formulate and execute properties about structures containing bottom”</p> <p>“we must rely on the user to provide instances [of the type class <code>Arbitrary</code>] for user-defined types”</p> <p>“the size bound is simply an extra, global parameter which every test data generator may access; every use of sized sees the same bound”</p> <p>“use [<code>promote</code>] to produce a generator for a function type, provided we can construct a generator for the result type which somehow depends on the argument value”</p> <p>“think of <code>coarbitrary</code> as producing a <em>generator transformer</em> from its first argument”</p> <p>“<code>variant n g</code> constructs a generator which transforms the random number seed it is passed in a way depending on <code>n</code> before passing it to <code>g</code>”</p> <p>“mapping each constructor to an independent transformer, and composing these with transformers from each component. Other recursive datatypes can be treated in the same way.”</p> <p>“it is more convenient to let test data be automatically generated using arbitrary so one is encouraged to make distinctions explicit in types”</p> <p>“using QuickCheck changes the balance of convenience in the question of introducing new types in programs”</p> <p>“The most serious pitfall we uncovered with this experiment was the false sense of security that can be engendered when one’s program passes a large number of tests in trivial cases.”</p> <p>“a typical development cycle is to write down the specification of the circuit first, then make an implementation, QuickCheck it for obvious bugs, and lastly call the external theorem prover for verifying the correctness.”</p> <p>“we can test <em>more</em> properties than we can formally verify!” because testing is not restricted to first-order logic, which the Lava system’s external theorem prover was restricted to</p> <p>“A drawback is that we have to fix the types of these circuits” This is the “we have to test with concrete data, y’all” requirement again.</p> <p>Section 5.4 Pretty Printing describes an interesting way of reaching a working Java implementation: write once functionally, write once using state + exceptions in a functional language, check equivalence, then port the state + exceptions model to an imperative language, generate test inputs, and check the final implementation.</p> <p>“since the specification refers to the implementation, then the specification module must import the implementation currently under test” which is annoying for different implementations of the same abstract interface (and solved by ML functors)</p> <p>“‘By taking 20% more points in a random test, any advantage a partition test might have had is wiped out.’” From [11]. Explains why we shouldn’t bother using random testing rather than carefully selected tests or automatically selected tests that meet some coverage criteria.</p> <p>“often one can check that a programs output is correct much more efficiently than one can compute the output” - see [4] on “result-checking”; QuickCheck is simultaneously more general (can check associative property rather than output correctness) and more directed (exclude testing bad/irrelevant inputs)</p> <p>grammar-directed testing - “context-free grammars cannot express all the - also one ugly thing to learn, and divorced from your primary language desired properties of test data” (frex def-before-use constraint)</p> <p>“the <code>Gen</code> monad which QuickCheck is based on is not a monad at all!” - seed use/order matters - “morally” the same wrt distributions, but even still, observable differences in the output - “There is some interesting semantic theory to be done here.”</p> <p>“What we cannot do is observe non-termination in a test result.” This is in part a limit of the compiler, which reports errors to humans that it won’t to programs.</p> <p>“one of the major advantages of using QuickCheck is that it encourages us to formulate formal specications, thus improving our understanding of our programs” And <strong>QuickCheck provides a practical reason to write specifications</strong> where there wasn’t one before.</p> <p>Errors in practice divide roughly evenly between:</p> <ul> <li>bad data generators (useless to find, but necessary for testing to proceed)</li> <li>bad program code (this is why we’re testing!)</li> <li>bad specifications (reveal misconceptions about the program)</li> </ul> <p>Spec writers need libs they wouldn’t for program code, frex finite set theory. Different requirements on the libraries: can use less performant but more concise functions in spec than in main program.</p> <p>“The <strong>major limitation of QuickCheck</strong> is that there is no measurement of test <em>coverage</em>: it is <strong>up to the user to investigate the distribution of test data</strong> and decide whether sufficiently many tests have been run.” (bold emphasis mine)</p> HTTP Library Survey https://jeremywsherman.com/blog/2013/05/25/http-library-survey/ Sat, 25 May 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/05/25/http-library-survey/ <p><figure> <img src="https://jeremywsherman.com/images/posts/2013/Holbein-Totentanz.png" alt="15th century print of skeletons dancing and playing musical instruments."/> <figcaption> <h4>Totentanz by Michael Wolgemut, 1493</h4> </figcaption> </figure> Working with the Web seems to be all apps do these days. The stock Foundation URL loading system requires endless ceremony and a lot of work to try to patch it up to anything resembling watch-batteries-included.</p> <p>It doesn’t have to be this way. Perhaps you’ve seen Kenneth Reitz’s quick and easy <a href="http://docs.python-requests.org/en/latest/index.html">Requests library</a> for Python.</p> <p>Well, you’re not the first person to think that it doesn’t have to be that way <em>in Cocoaland,</em> either.</p> <p><figure> <img src="https://jeremywsherman.com/images/posts/2013/Holbein-Totentanz.png" alt="15th century print of skeletons dancing and playing musical instruments."/> <figcaption> <h4>Totentanz by Michael Wolgemut, 1493</h4> </figcaption> </figure> Working with the Web seems to be all apps do these days. The stock Foundation URL loading system requires endless ceremony and a lot of work to try to patch it up to anything resembling watch-batteries-included.</p> <p>It doesn’t have to be this way. Perhaps you’ve seen Kenneth Reitz’s quick and easy <a href="http://docs.python-requests.org/en/latest/index.html">Requests library</a> for Python.</p> <p>Well, you’re not the first person to think that it doesn’t have to be that way <em>in Cocoaland,</em> either.</p> <h2 id="better-dead-than-red">Better Dead than Red?</h2> <p>A quick survey of search results for “http” over at Cocoapods yields:</p> <ul> <li> <p><a href="https://github.com/ognen/httpriot">HTTPRiot</a></p> <ul> <li>Inspired by Ruby&rsquo;s <a href="https://github.com/jnunemaker/httparty">httparty</a>.</li> <li>README&rsquo;s <a href="http://labratrevenge.com/httpriot">documentation URL</a> 404s.</li> <li>Last updated 2 years ago.</li> <li>“Get/post/delete with options” is the name of the game. See <a href="https://github.com/ognen/httpriot/blob/master/Documentation/Samples/Tweet.m">Tweet.m</a> for how it gets woven into a model object and <a href="https://github.com/ognen/httpriot/blob/master/Source/HTTPRiot/HRResponseDelegate.h">HRResponseDelegate</a> for the optional (last resort) delegate methods.</li> </ul> </li> <li> <p><a href="https://github.com/yfactorial/objectiveresource/tree/master">ObjectiveResource</a></p> <ul> <li>A 4-year dead port of Ruby’s ActiveResource to Cocoa.</li> <li>Possibly also a great name for an Objectivist weekly newsletter.</li> </ul> </li> <li> <p><a href="https://github.com/lukeredpath/LRResty">LRResty</a></p> <ul> <li>Inspired by the Ruby <a href="http://github.com/archiloque/rest-client">RestClient library</a>.</li> <li>README still treats ARC as new and iOS 4 as something to care about. No recent updates.</li> <li>The <a href="http://projects.lukeredpath.co.uk/resty/">website</a> is pretty though.</li> </ul> </li> </ul> <p>Perhaps you’re starting to see a pattern:</p> <ul> <li>Take Ruby library</li> <li>Make Obj-C library</li> </ul> <h2 id="tacking-for-the-horizon">Tacking For the Horizon</h2> <p>The next few take a different tack. They also seem to be actively maintained.</p> <ul> <li> <p><a href="https://github.com/nicklockwood/RequestUtils">RequestUtils</a></p> <ul> <li>Hews close to the standard Foundation conventions of separating URLs from strings, but makes it easier to build a more complex request without having to repeatedly mess with a mutable URL request.</li> <li>Simplifies form and URL encoding and decoding, with options for how to handle repeated specifications of the same query key.</li> <li>Recently updated.</li> </ul> </li> <li> <p><a href="https://github.com/brunodecarvalho/BBHTTP">BBHTTP</a></p> <ul> <li>ARC-only.</li> <li>Uses before/success/failure/finally-style continuation blocks.</li> <li>Avoids manual string-&gt;URL coercions.</li> <li>Convenient support for downloading and working with JSON, image, and file content.</li> </ul> </li> <li> <p><a href="https://github.com/AFNetworking/AFIncrementalStore">AFIncrementalStore</a></p> <ul> <li>Bridges Core Data to a REST backend.</li> <li>Works by your telling it how to un/pickle a resource and map objects and relationships onto paths and IDs.</li> <li>Convenient if you’re using Core Data.</li> <li>One of those Hell-cobblestones if you’re also hooking into iCloud. (And who doesn’t want to be cloud-high these days?)</li> </ul> </li> <li> <p><a href="https://github.com/afnetworking/afnetworking">AFNetworking</a></p> <ul> <li> <p>Continuation-based with a reactor.</p> </li> <li> <p>Minor class-splosion for handling various response Content-Types, but it’s a reasonable way to bake in support for common response types.</p> </li> <li> <p>By far the most popular option of all of these (judging by Github stars, as you do).</p> </li> <li> <p>Under active development.</p> </li> <li> <p>I mean, you knew we&rsquo;d end up here eventually.</p> </li> </ul> </li> </ul> <h2 id="conclusion">Conclusion</h2> <p>The road to the Web from an iOS app is littered with corpses. We&rsquo;re stuck with the Foundation mummy, but you might as well pick a fresher partner for your own Totentanz. <a href="https://github.com/afnetworking/afnetworking">AFNetworking</a> certainly has some life left.</p> <p>There&rsquo;s also room for a Cocoa Toolbox site along the lines of the <a href="http://www.clojure-toolbox.com/">Clojure Toolbox</a>. It&rsquo;s great we&rsquo;ve something of a dependency management system now, but without a well-curated guide, we&rsquo;ve more of a run-down museum than a lively workshop.</p> <p><em>The image above is a 15th Century print by Michael Wolgemut, found on <a href="https://en.wikipedia.org/wiki/The_Dance_of_Death">Wikipedia</a>. Dude&rsquo;s been dead long enough we can at least use his work freely. Unless someone holds a submarine patent on display of over-half-a-millenium–old prints <strong>on the Internet.</strong> I mean, shopping carts <strong>on the Internet</strong> were a major brainwave, right? No-one would ever have thought of that.</em></p> Key Reordering: MongoDB's Achilles' Heel https://jeremywsherman.com/blog/2013/04/23/key-reordering-mongodbs-achilles-heel/ Tue, 23 Apr 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/04/23/key-reordering-mongodbs-achilles-heel/ <p><em>MongoDB relies on key order but doesn&rsquo;t guarantee it will preserve document key order. Steer clear.</em></p> <p>MongoDB document keys are ordered. Subdocument queries fail to retrieve results whose keys are in a different order. Indexing only works with keys in the same order.</p> <p><em>MongoDB relies on key order but doesn&rsquo;t guarantee it will preserve document key order. Steer clear.</em></p> <p>MongoDB document keys are ordered. Subdocument queries fail to retrieve results whose keys are in a different order. Indexing only works with keys in the same order.</p> <p>This uglies up the client application interface. Languages regularly provide literal syntax for unordered associative arrays (aka hashes, maps, dictionaries). To guarantee all similar documents have the same key order, you either forgo the literal syntax provided by your language to use an order-preserving data structure, or you throw a sorting layer in between the application and MongoDB.</p> <p>That&rsquo;s bad, but you can work around it.</p> <p>What you can&rsquo;t work around is MongoDB rearranging keys behind your back:</p> <blockquote> <p>When performing update operations that increase the document size beyond the allocated space for that document, the update operation relocates the document on disk and may reorder the document fields depending on the type of update. (<a href="http://docs.mongodb.org/manual/core/update/">Update - MongoDB Manual 2.4.2</a>)</p> </blockquote> <p>Let&rsquo;s read that again: &ldquo;[T]he update operation […] may reorder the document fields.&rdquo;</p> <p>Just dandy.</p> <p>Using MongoDB requires a fixed key order for reliable operation, but MongoDB fails to preserve key order. As convenient and simple as MongoDB is, this is enough for me to advise anyone: Stay well away from MongoDB.</p> Clojure/West 2013 https://jeremywsherman.com/blog/2013/04/10/clojure/west-2013/ Wed, 10 Apr 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/04/10/clojure/west-2013/ I stumbled from the red-eye into the airport shortly after dawn. Bleary-eyed and back on the ground in Atlanta at last, Clojure/West, and its talk of Lisp and browser and JVMs, seemed already a distant memory. My laptop hadn’t left its TSA-friendly bag for the past week. My notebook was full of scribbles and mindmaps. What was a Cocoa programmer to make of it all? Continue reading at the Big Nerd Ranch Blog → <p>I stumbled from the red-eye into the airport shortly after dawn. Bleary-eyed and back on the ground in Atlanta at last, Clojure/West, and its talk of Lisp and browser and JVMs, seemed already a distant memory. My laptop hadn’t left its TSA-friendly bag for the past week. My notebook was full of scribbles and mindmaps. What was a Cocoa programmer to make of it all?</p> <p><a href="http://blog.bignerdranch.com/2434-clojurewest-2013/">Continue reading at the Big Nerd Ranch Blog →</a></p> Leak-Free Recursive Blocks https://jeremywsherman.com/blog/2013/02/27/leak-free-recursive-blocks/ Wed, 27 Feb 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/02/27/leak-free-recursive-blocks/ <p>Sometimes, you want a block to be able to call itself. That means it needs a reference to itself. And that means you have a wonderful opportunity to create a strong reference cycle that will endure till the end of time, or at least till your app exits.</p> <h2 id="the-solution">The Solution</h2> <pre><code>__weak __block block_t recurse; block_t block; recurse = block = ^(id val) { … recurse(subval); … } </code></pre> <h2 id="getting-there">Getting There</h2> <p>Sometimes, you want a block to be able to call itself. That means it needs a reference to itself. And that means you have a wonderful opportunity to create a strong reference cycle that will endure till the end of time, or at least till your app exits.</p> <h2 id="the-solution">The Solution</h2> <pre><code>__weak __block block_t recurse; block_t block; recurse = block = ^(id val) { … recurse(subval); … } </code></pre> <h2 id="getting-there">Getting There</h2> <p>The prototypical recursive function example is the factorial function:</p> <pre><code>uintmax_t factorial(uintmax_t n) { NSParameterAssert(n &gt;= 0); if (n == 0) return 1; return n * factorial(n); } </code></pre> <p>So that&rsquo;s a function. Now we want to make it a block.</p> <p>As a block, that <code>factorial</code> self-reference is tricky. A block only gets a name when you assign it to a variable. That variable assignment won&rsquo;t happen till after the block is created. So you need a <code>__block</code> variable:</p> <pre><code>uintmax_t (^__block factorial)(uintmax_t n) = ^uintmax_t (uintmax_t n) { NSParameterAssert(n &gt;= 0); if (n == 0) return 1; return n * factorial(n); } </code></pre> <p>Sadly, that&rsquo;s a <code>__strong __block</code> reference by default. Oops. If you don&rsquo;t want to leak, you either need to take care to break the strong reference manually by <code>NULL</code>ing out the <code>factorial</code> variable – good luck – or you need a weak reference.</p> <p>So you retag the block variable as weak:</p> <pre><code>uintmax_t (^__block __weak factorial)(uintmax_t n) = … </code></pre> <p>Only now there&rsquo;s never a strong reference to your block, so your block is eligible for deallocation right after it&rsquo;s returned.</p> <p>So you need <em>both</em> a strong <em>and</em> a weak reference to your block. And the block needs to be stored in the strong reference first, so you anchor it to this world. So maybe you do this:</p> <pre><code>uintmax_t (^__block __weak weakFactorial)(uintmax_t n); uintmax_t (^factorial)(uintmax_t n) = … weakFactorial = factorial; </code></pre> <p>That fixes it. It works. And it&rsquo;s nice that we could drop the <code>__block</code> from the main <code>factorial</code> reference, since that stays stable; it&rsquo;s only the <code>weakFactorial</code> self-reference that gets updated after the block first captures it.</p> <p>But having to do this follow-on assignment is kind of ugly, and it&rsquo;s pretty forgettable way down there at the end of your block, and a few revisions later they&rsquo;ll get separated, and then maybe you&rsquo;ll accidentally delete the weak assignment, and then you&rsquo;ll have to track down a bug. Ugh.</p> <p>So try this instead:</p> <pre><code>uintmax_t (^__block __weak weakFactorial)(uintmax_t n); uintmax_t (^factorial)(uintmax_t n); weakFactorial = factorial = … </code></pre> <p>You&rsquo;re not going to forget <em>that</em> assignment. It&rsquo;s still ugly, but that&rsquo;s what happens at the edges of ARC, and it&rsquo;s at least not too fragile. You could even snippetize it, if you&rsquo;re into that sort of thing.</p> <h2 id="caveat-asynchronous-recursion">Caveat: Asynchronous Recursion</h2> <p><a href="https://github.com/jspahrsummers/">Justin Spahr-Summers</a> rightly notes that this approach works only for synchronous recursion:</p> <blockquote> <p>Once your strong reference goes out of scope, wouldn&rsquo;t it just stop recursing because all that&rsquo;s left is a weak reference? (<a href="https://twitter.com/jspahrsummers/status/307017606206984192">Twitter</a>)</p> </blockquote> <p>It&rsquo;s actually worse than that: you&rsquo;ll get a segfault, because you try to invoke a NULL block as a function. If you have some sort of concurrent or asynchronous recursion – maybe invoking the block kicks off a gradual countdown – then you&rsquo;ll need to handle the case where the block is dead and has been zeroed but doesn&rsquo;t know it yet.</p> <p>Use the standard trick of trying to obtain a strong reference and then testing whether that reference is <code>nil</code> or not to decide how to proceed:</p> <pre><code>block_t recurse = weakRef; bool zeroed = !recurse; if (zeroed) /* bail out */; /* use |recurse| to reference yourself, so that you don't segfault */ </code></pre> <p>If you want the recursion to run to completion rather than fizzling out, I imagine you can work out the appropriate juggling act.</p> <p>This isn&rsquo;t a problem for the common case of synchronous block handlers, where you set the handler and don&rsquo;t change it, and the object owning the strong reference is guaranteed to outlive the handler block&rsquo;s recursion, but it is something to watch out for as you get cleverer or otherwise start destroying references while the block might be running.</p> Dollars, Not Donuts https://jeremywsherman.com/blog/2013/02/16/dollars-not-donuts/ Sat, 16 Feb 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/02/16/dollars-not-donuts/ <p>Internationalization is hard. If you&rsquo;ve always lived where you have to travel to another continent to encounter something other than your local monoculture (hello, many fellow Americans), it&rsquo;s even harder.</p> <p>Perhaps it should be no surprise then that everyone gets it wrong. Over and over, I see the same amateur mistakes. Two seconds using your app in some locale other than the one you developed in would make you go, &ldquo;Oh. Crap.&rdquo;</p> <p>Internationalization is hard. If you&rsquo;ve always lived where you have to travel to another continent to encounter something other than your local monoculture (hello, many fellow Americans), it&rsquo;s even harder.</p> <p>Perhaps it should be no surprise then that everyone gets it wrong. Over and over, I see the same amateur mistakes. Two seconds using your app in some locale other than the one you developed in would make you go, &ldquo;Oh. Crap.&rdquo;</p> <h2 id="home-depot-accepts-swedish-kronor-now">Home Depot accepts Swedish kronor now?</h2> <p>Today&rsquo;s mistake is &ldquo;localizing&rdquo; currency amounts in your application.</p> <p>Take a look at this prime shot from the Home Depot website this morning:</p> <figure> <img src="https://jeremywsherman.com/images/posts/2013/HomeDepotKronor.jpg" alt="Home Depot product listing shows price as Swedish kronor, even though the amount is clearly US dollars."/> <figcaption> <h4>Home Depot product - price listed as Swedish kronor</h4> </figcaption> </figure> <p>I&rsquo;m sending sv-SE as my preferred language, so I get back the site&rsquo;s closest approximation to Swedish in Sweden as possible. Oh, how cool, that includes converting currencies to Swedish kronor!</p> <p>Oh wait, no it doesn&rsquo;t. They just rigged up their currency formatter wrong. The formatter thinks the amount they specified – an amount actually in US dollars – must be Swedish kronor (SEK), because they didn&rsquo;t tell it anything more specific, and so the formatter printed the amount as SEK.</p> <p>I ran into this same thing when I went to renew my car insurance. A whole table of US dollar amounts formatted as Swedish kronor.</p> <figure> <img src="https://jeremywsherman.com/images/posts/2013/GeicoCoverageKronor.jpg"/> </figure> <p>This happens so often, across both websites and apps, that when I encounter something that actually does support displaying actual honest-to-God SEK amounts, I still assume it&rsquo;s USD misrepresented as SEK until I realize the prices are wildly different from what the USD amount would be. This is so rare that it&rsquo;s only happened once: thank you, Kayak, for getting this right!</p> <p>If the exchange rate weren&rsquo;t so far from 1 between USD and SEK (currently about 6 SEK to 1 USD), I don&rsquo;t know how I&rsquo;d be able to tell whether an app got it wrong or got it right.</p> <p>This amateur mistake, repeated across vendors, applications, and platforms, makes for a terrible user experience for anyone working in a different region, even if they speak the same language as you.</p> <h2 id="its-easy-to-get-it-wrong">It&rsquo;s easy to get it wrong</h2> <p>Let&rsquo;s see, how would you format a currency amount, say, $1,234.59 USD? How&rsquo;s about this:</p> <pre><code>id amount = [NSDecimalNumber decimalNumberWithMantissa:123459 exponent:-2 isNegative:NO]; NSString *display = [NSNumberFormatter localizedStringFromNumber:amount numberStyle:NSNumberFormatterCurrencyStyle]; NSLog(@&quot;%@&quot;, display); </code></pre> <p>Very easy, right? And very wrong. That prints, &ldquo;1 234,59 kr&rdquo;.</p> <p>Well, let&rsquo;s try rigging up the currency formatter ourselves, eh?</p> <pre><code>NSNumberFormatter *fmt = [NSNumberFormatter new]; [fmt setNumberStyle:NSNumberFormatterCurrencyStyle]; [fmt setGeneratesDecimalNumbers:YES]; NSString *formattedAmount = [fmt stringFromNumber:amount]; NSLog(@&quot;%@&quot;, formattedAmount); </code></pre> <p>Nope, still prints &ldquo;1 234,59 kr&rdquo;. This is likely exactly what the &ldquo;convenience&rdquo; converter did for us, minus the configuration setting to convert strings to decimal numbers. So that&rsquo;s a lot of hoopla for no gain.</p> <h2 id="but-not-hard-to-get-it-right">But not hard to get it right</h2> <p>But it just takes one more line to get it right. You need to tell the formatter what exactly it&rsquo;s formatting. Raw numbers aren&rsquo;t enough: it needs to know the currency you&rsquo;re giving it. Like so:</p> <pre><code>NSNumberFormatter *fmt = [NSNumberFormatter new]; [fmt setNumberStyle:NSNumberFormatterCurrencyStyle]; [fmt setGeneratesDecimalNumbers:YES]; [fmt setCurrencyCode:@&quot;USD&quot;]; NSString *formattedAmount = [fmt stringFromNumber:amount]; </code></pre> <p>Yup, it just takes a <code>setCurrencyCode:</code> message, and all is right with the world. This version here prints &ldquo;1 234,59 US$&rdquo; when using the Swedish region. The currency symbol follows the amount, as is usual in that region, and the separators for thousands and decimal are localized, as well.</p> <h2 id="testing">Testing</h2> <p>Testing this is dead easy. You don&rsquo;t even have to abandon your beloved mother tongue! Just change your region settings to something different. Swedish in Sweden has triggered enough localization mistakes for me, but I&rsquo;m sure many other locales get messed up just as readily.</p> <p>Changing languages is awkward – iOS does this semi-reboot thing, and you have to restart Mac apps to get them to pick up the change – but region settings can be changed in a jif and changed back just as easily, without forcing you to wander through menus in a language you might not understand. So switching regions is the list you can do to test how your localization – even inadvertent localization – might be wrong.</p> <p>Change your region setting like so:</p> <ul> <li>iOS: Settings &gt; General &gt; International &gt; Region Format &gt; Swedish &gt; Sweden</li> <li>Mac: Settings &gt; Language and Text &gt; Region &gt; Swedish &gt; Sweden</li> </ul> <p>On the Mac, you&rsquo;ll probably need to tic the &ldquo;Show all regions&rdquo; box before you can choose a region that doesn&rsquo;t use your top language choice.</p> <h2 id="sum-up">Sum-Up</h2> <ul> <li> <p>Distinguish between localizing presentation of an amount in a unit and both the presentation and the unit.</p> <ul> <li>Compare 1 mile versus 1.6 kilometers: does using &ldquo;1 km&rdquo; instead of &ldquo;1 mi&rdquo; in a metric-using region really represent what you meant to communicate?</li> </ul> </li> <li> <p>Declare the currency code to your currency-style number formatters.</p> <pre><code>[numberFormatter setCurrencyCode:@&quot;USD&quot;]; </code></pre> </li> <li> <p>Test your app in different locales.</p> <ul> <li>Just because you&rsquo;re not localizing doesn&rsquo;t mean system components you&rsquo;re using aren&rsquo;t doing it wrong for you.</li> </ul> </li> </ul> Toll-Free __bridging https://jeremywsherman.com/blog/2013/01/29/toll-free-__bridging/ Tue, 29 Jan 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/01/29/toll-free-__bridging/ <p>Starting with OS X 10.6, you can use the <code>__attribute__</code> keyword to specify that a Core Foundation property should be treated like an Objective-C object for memory management:</p> <pre><code>@property(retain) __attribute__((NSObject)) CTFrameRef frame; </code></pre> <p>This is an easy attribute to miss. It’s also one you can go a long time without finding, because it’s not hard to work around.</p> <p>(Do note that you can use the <code>NSObject</code> attribute with anything where you’d use <code>CFRetain</code>/<code>CFRelease</code>, not just actual toll-free bridged objects. The toll you’re dodging with <code>__attribute__((NSObject))</code> is purely syntactic.)</p> <p>Starting with OS X 10.6, you can use the <code>__attribute__</code> keyword to specify that a Core Foundation property should be treated like an Objective-C object for memory management:</p> <pre><code>@property(retain) __attribute__((NSObject)) CTFrameRef frame; </code></pre> <p>This is an easy attribute to miss. It’s also one you can go a long time without finding, because it’s not hard to work around.</p> <p>(Do note that you can use the <code>NSObject</code> attribute with anything where you’d use <code>CFRetain</code>/<code>CFRelease</code>, not just actual toll-free bridged objects. The toll you’re dodging with <code>__attribute__((NSObject))</code> is purely syntactic.)</p> <h2 id="work-arounds">Work Arounds</h2> <p>You can get surprisingly far by just pretending that a <code>CTFrameRef</code> is an <code>id</code>:</p> <pre><code>@interface MyClass @property(strong, nonatomic) id frame; /* CTFrameRef */ @end; </code></pre> <h3 id="__bridge">__bridge</h3> <p>You just have to sprinkle casts in the appropriate places:</p> <pre><code>CTFrameRef frame = CTFrameCreate… self.frame = (__bridge id)frame; CFRelease(frame); </code></pre> <h3 id="__bridge_transfer">__bridge_transfer</h3> <p>You can even use the casts to save you a line of code here and there:</p> <pre><code>CTFrameRef frame = CTFrameCreate… self.frame = (__bridge_transfer id)frame; /* ARC now has ownership of |frame|, so it is responsible for releasing it. */ </code></pre> <h3 id="cfbridgingrelease">CFBridgingRelease</h3> <p>Or perhaps use one of the less underscore-y Core Foundation wrappers:</p> <pre><code>CTFrameRef frame = CTFrameCreate… self.frame = CFBridgingRelease(frame); /* Now your Create-rule-trained brain can rest easy, because there’s a balancing Release. */ </code></pre> <h3 id="macros">Macros</h3> <p>But I find the casts clutter up my code, and CF memory management is not bad in small doses, so I used to use macros:</p> <pre><code>#define $ID (__bridge id) #define $CF (__bridge void *) </code></pre> <p>The <code>$CF</code> macro exploits C’s willingless to coerce <code>void *</code> the way <code>$ID</code> expresses <code>id</code>’s willingness to be coerced. That breaks down under Obj-C++, because C++ is not so willing to coerce, so you end up doing something like this instead:</p> <pre><code>#define $CF(var, obj) lval = ((__bridge __typeof__((var)))(obj)) </code></pre> <p>This ends up working OK, because you tend to assign to a variable of the Core Foundation type, make the cast once there, and then use that CF-typed var throughout the next bit of code:</p> <pre><code>CTFrameRef frame = $CF(frame, self.frame); /* do something with |frame| */ </code></pre> <h2 id="just-use-__attribute__nsobject-already">Just Use __attribute__((NSObject)) Already!</h2> <p>But I could have saved myself all that mess had I just used <code>__attribute__((NSObject))</code>. Aren&rsquo;t attributes a wonderful thing?</p> <pre><code>//cc -g -c -Weverything -Wno-objc-missing-property-synthesis attribute_nsobject.m /* @file attribute_nsobject.m * @author Jeremy W. Sherman * @date 2013-01-29 * * Demonstrates the wondrous simplicity of `__attribute__((NSObject))`. */ #import &lt;Foundation/Foundation.h&gt; #import &lt;CoreText/CoreText.h&gt; @interface MyClass : NSObject @property(strong, nonatomic) __attribute__((NSObject)) CTFrameRef frame; @end /* Look ma, no casts! */ @implementation MyClass - (void)storeFrame { CTFrameRef frame = NULL; self.frame = frame; } - (void)loadFrame { CTFrameRef frame __unused = self.frame; } @end </code></pre> <h3 id="eta-version-concerns">ETA: Version Concerns</h3> <p><a href="http://jspahrsummers.github.com/">Justin Spahr-Summers</a> points out <a href="https://twitter.com/jspahrsummers/status/296486907909648384">via Twitter</a> that this story used not to have such a happy ending. Some member of the clang/objc/ARC juggling act used to fail to retain nonatomic properties. The short tale is documented in a <a href="http://stackoverflow.com/questions/9684972/">Stack Overflow thread</a> and has been reported as <a href="rdar://problem/11040306">rdar://problem/11040306</a>.</p> <p><strong>Good news:</strong> As of Xcode 4.6 and OS X 10.8.2 (which are what I have on hand to test with), the issue seems to be fixed. The compiler generates a call to <code>objc_setProperty_nonatomic</code> which will <code>objc_retain</code> the new value as expected.</p> <p>The <code>_nonatomic</code> variant doesn&rsquo;t seem to exist in my copy of 10.7.1&rsquo;s objc4-493.9, so from where I&rsquo;m sitting, this looks to have been fixed in part by an SPI change.</p> <p>This appears to be an undocumented change affecting only Apple clang as of this time. It also seems that the fix will only work for this property-focused usage pattern; if you need a generic instruction to the compiler to use full ARC semantics for pointers of a certain type, you&rsquo;ll still have to create a typedef to attach the type info to.</p> <p>The ARC reference documentation continues to specify that only typedefs can be annotated to create a retainable object pointer type, and the open-source version of clang (as of r173899) still tests for this, and, per the implementation in lib/AST/Type.cpp of <code>Type::isObjCNSObjectType()</code>, this still seems to be the case:</p> <pre><code>bool Type::isObjCNSObjectType() const { if (const TypedefType *typedefType = dyn_cast&lt;TypedefType&gt;(this)) return typedefType-&gt;getDecl()-&gt;hasAttr&lt;ObjCNSObjectAttr&gt;(); return false; } bool Type::isObjCRetainableType() const { return isObjCObjectPointerType() || isBlockPointerType() || isObjCNSObjectType(); } </code></pre> The Otherwise Operator https://jeremywsherman.com/blog/2013/01/21/the-otherwise-operator/ Mon, 21 Jan 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/01/21/the-otherwise-operator/ <p>GNU C adds a binary operator <code>?:</code>. Use it to fall back to a default value when a nil check fails:</p> <pre><code>id target = [self.delegate target] ?: [self.class defaultTarget]; </code></pre> <p>GNU C adds a binary operator <code>?:</code>. Use it to fall back to a default value when a nil check fails:</p> <pre><code>id target = [self.delegate target] ?: [self.class defaultTarget]; </code></pre> <p>The <a href="http://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/Conditionals.html#Conditionals" title="Conditionals with Omitted Operands">GCC docs</a> present the binary <code>?:</code> operator as eliding a repeated first term when using the ternary conditional operator, so</p> <pre><code>x ? x : y </code></pre> <p>can now be written</p> <pre><code>x ? : y </code></pre> <p>and have the same effect as the full form, save that the first term, <code>x</code>, is only evaluated once.</p> <p>From this point of view, the binary <code>?:</code> exists to avoid unwanted side effects:</p> <pre><code>int z = (x++) ? (x++) : y; // bad news int w = (x++) ? : y; // OK! </code></pre> <p>But thinking of <code>?:</code> as a special-purpose variant of the ternary operator misses its true calling: cleaning up <code>nil</code> and <code>NULL</code> checks. It compacts several lines of code:</p> <pre><code>id target = [self.delegate target]; if (!target) { target = [self.class defaultTarget]; } </code></pre> <p>down to a one-liner:</p> <pre><code>id target = [self.delegate target] ?: [self.class defaultTarget]; </code></pre> <p>So: The ”otherwise” – or ”if nil then” – operator: <code>?:</code>. Use it.</p> Queue-Specific Data https://jeremywsherman.com/blog/2013/01/19/queue-specific-data/ Sat, 19 Jan 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/01/19/queue-specific-data/ <p>GCD has a queue-specific storage API accessed using <code>dispatch_queue_{set,get}_specific</code>. This replaces the thread-specific storage provided by <code>pthread_{set,get}_specific</code> that you cannot use with GCD blocks:</p> <p>GCD has a queue-specific storage API accessed using <code>dispatch_queue_{set,get}_specific</code>. This replaces the thread-specific storage provided by <code>pthread_{set,get}_specific</code> that you cannot use with GCD blocks:</p> <p>static void *sQueueKey_Client = &ldquo;client&rdquo;; struct my_client *client = calloc(1, sizeof(*client)); <em>client = (struct my_client){ .val = 1 }; /</em> use the unique static address as the key, * <em>not</em> the address of the string itself */ dispatch_queue_set_specific(q, &amp;sQueueKey_Client, client, free); dispatch_async(q, ^{ struct my_client *client = dispatch_queue_get_specific(q, &amp;sQueueKey_Client); DoStuffWith(client); });</p> <p>Only there’s one new addition to the family: <code>dispatch_get_specific</code> looks up the value in the current context defined by the current queue. This context is broader than the single queue that <code>dispatch_queue_get_specific</code> will search. If a key is not set on the current queue, it will check that queue’s target queue. If it’s not found on that queue, it will move down the line to <em>that</em> queue’s target queue:</p> <pre><code>dispatch_queue_t io_q = dispatch_queue_create(&quot;client_io_queue&quot;, 0); dispatch_set_target_queue(io_q, q); dispatch_async(io_q, ^{ /* This will check the current queue (io_q), fail to find * the key, then check the target queue (q) and find it. */ struct my_client *client = dispatch_get_specific(&amp;sQueueKey_Client); SendMessage(client); }); </code></pre> <p>Queue-specific value lookup sounds a lot like chasing the prototype chain in a prototypal object system like JavaScript. In Obj-C, it echoes how method implementation search runs up the inheritance chain to find an implementation for a given message.</p> <p>It turns out you can abuse this to transform dispatch queue value lookup into the heart of a prototypal object system embedded within Objective-C – where it’s not terribly useful, because Obj-C already has its own object system – or C, where it could be an improvement over hand-writing OOP in C.</p> <p>I wrote a small, ugly demo of this. It&rsquo;s available from GitHub as <a href="https://github.com/jeremy-w/demo-draft">jeremy-w/demo-draft</a>. As it stands, it’s certainly not an improvement over hand-written C OOP, but it did prove an interesting exercise.</p> Using debugDescription with GCD and XPC objects https://jeremywsherman.com/blog/2013/01/08/using-debugdescription-with-gcd-and-xpc-objects/ Tue, 08 Jan 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/01/08/using-debugdescription-with-gcd-and-xpc-objects/ <p><code>dispatch_debug</code> and <code>xpc_copy_description</code> are inconvenient, particularly during impromptu debugging.</p> <p>Mountain Lion&rsquo;s Obj-C-ification of GCD and XPC objects lets you use your comfortable Obj-C tools:</p> <ul> <li><code>NSLog</code> with <code>%@</code>,</li> <li>the <code>debugDescription</code> method, and</li> <li><code>po obj</code> while in the debugger.</li> </ul> <p><code>dispatch_debug</code> and <code>xpc_copy_description</code> are inconvenient, particularly during impromptu debugging.</p> <p>Mountain Lion&rsquo;s Obj-C-ification of GCD and XPC objects lets you use your comfortable Obj-C tools:</p> <ul> <li><code>NSLog</code> with <code>%@</code>,</li> <li>the <code>debugDescription</code> method, and</li> <li><code>po obj</code> while in the debugger.</li> </ul> <h2 id="dispatch_debug">dispatch_debug</h2> <p>If you first learned GCD back before Mountain Lion, you might have played around with the <code>dispatch_debug</code> function:</p> <pre><code>void dispatch_debug(dispatch_object_t object, const char *message, ...); </code></pre> <p>This function is the <code>NSLog</code> of Grand Central Dispatch land. If you need to pin down what exactly is going on with a complex network of dispatch objects, this can be a useful tool, especially since you can use the <a href="https://libdispatch.macosforge.org/trac/browser/trunk" title="trunk - libdispatch"><code>libdispatch</code> source code</a> to illuminate the more cryptic debug info.</p> <p>But it&rsquo;s also kind of annoying: unless you remembered to set <code>LIBDISPATCH_LOG=stderr</code> in the environment before starting your process, you&rsquo;ll have to watch the system log for your <code>dispatch_debug</code> output; it won&rsquo;t show up in Xcode&rsquo;s debug console.</p> <p>Changing the value of the environment variable after startup also doesn&rsquo;t seem to affect <code>dispatch_debug</code>'s behavior, so by the time you realize you&rsquo;ve forgotten to set this environment variable, it&rsquo;s already too late.</p> <h2 id="xpc_copy_description">xpc_copy_description</h2> <p>If you wanted to log information about an XPC object without leaking, you used to have to <code>xpc_copy_description</code>, log the string, then <code>free</code> the returned pointer when you&rsquo;re done with it:</p> <pre><code>char *desc = xpc_copy_description(obj); NSLog(@&quot;%s: xpc obj %p %s&quot;, __func__, obj, desc); free(desc); </code></pre> <h2 id="debugdescription">debugDescription</h2> <p>Well, good news: As of Mountain Lion, GCD and XPC objects are all also <code>NSObject</code>s, so you can use them as the target for the <code>%@</code> format specifier and as the target for the <code>-debugDescription</code> instance method. The latter dumps all the information you used to get from <code>dispatch_debug</code>.</p> <p>As an example:</p> <pre><code>2013-01-08 23:53:13.451 debug[80089:707] dispatch queue: description: &lt;OS_dispatch_queue: com.jeremywsherman.demo[0x7f8980c07f80]&gt; 2013-01-08 23:53:13.453 debug[80089:707] dispatch queue: debugDescription: &lt;OS_dispatch_queue: com.jeremywsherman.demo[0x7f8980c07f80] = { xrefcnt = 0x2, refcnt = 0x1, suspend_cnt = 0x0, locked = 0, target = com.apple.root.default-priority[0x7fff72c47d00], width = 0x7fffffff, running = 0x0, barrier = 0 }&gt; </code></pre> <p>XPC objects are pretty verbose even with <code>description</code>, but you get a bit – sometimes quite a bit – more info if you send <code>debugDescription</code>:</p> <pre><code>2013-01-08 23:53:13.453 debug[80089:707] xpc connection: description: &lt;OS_xpc_connection: &lt;connection: 0x7f8980e017f0&gt; { name = com.jeremywsherman.conn, listener = false, PID = 0, EUID = 4294967295, EGID = 4294967295, ASID = 4294967295 }&gt; 2013-01-08 23:53:13.454 debug[80089:707] xpc connection: debugDescription: &lt;OS_xpc_connection: connection[0x7f8980e017f0]: { refcnt = 1, xrefcnt = 2, name = com.jeremywsherman.conn, type = named, state = new, queue = 0x7f8980e00420-&gt;0x0, error = 0x0, mach = false, privileged = false, bssendp = 0x0, recvp = 0x0, sendp = 0x0, pid/euid/egid/asid = 0/4294967295/4294967295/4294967295 } &lt;connection: 0x7f8980e017f0&gt; { name = com.jeremywsherman.conn, listener = false, PID = 0, EUID = 4294967295, EGID = 4294967295, ASID = 4294967295 }&gt; 2013-01-08 23:53:13.454 debug[80089:707] xpc bool: description: &lt;OS_xpc_bool: &lt;bool: 0x7fff7244d320&gt;: true&gt; 2013-01-08 23:53:13.455 debug[80089:707] xpc bool: debugDescription: &lt;OS_xpc_bool: bool[0x7fff7244d320]: { refcnt = 80000000, xrefcnt = 80000000, value = true } &lt;bool: 0x7fff7244d320&gt;: true&gt; </code></pre> <p>The odd trailer to the XPC objects&rsquo; debug descriptions is not a typo – the XPC objects really do include their regular description as a component of their debug description.</p> <h2 id="print-object-po">print-object (po)</h2> <p><code>debugDescription</code> also happens to be what gets printed when you <code>print-object</code> (or <code>po</code> for short) an object while debugging.</p> <p>Treating a GCD/XPC object as a regular Objective-C object is particularly handy during impromptu debugging, since you no longer need to futz about with <code>dispatch_debug</code> and <code>xpc_copy_description</code>.</p> <p>Instead, just use <code>po obj</code> when debugging:</p> <pre><code>% lldb ./debug (lldb) Current executable set to './debug' (x86_64). b debug.m:34 breakpoint set --file 'debug.m' --line 34 Breakpoint created: 1: file ='debug.m', line = 34, locations = 1 (lldb) r Process 80337 launched: '/Users/jeremy/Documents/Blog/GCDTips/debug' (x86_64) Process 80337 stopped * thread #1: tid = 0x1c03, 0x0000000100000db7 debug`main + 135 at debug.m:34, stop reason = breakpoint 1.1 frame #0: 0x0000000100000db7 debug`main + 135 at debug.m:34 31 Log(@&quot;xpc connection&quot;, conn); 32 33 xpc_object_t pred = xpc_bool_create(true); -&gt; 34 Log(@&quot;xpc bool&quot;, pred); 35 } 36 return 0; 37 } (lldb) fr var (dispatch_queue_t) q = 0x0000000100107fa0 (xpc_connection_t) conn = 0x0000000100400830 (xpc_object_t) pred = 0x00007fff7244d320 (lldb) po q (dispatch_queue_t) $0 = 0x0000000100107fa0 &lt;OS_dispatch_queue: com.jeremywsherman.demo[0x100107fa0] = { xrefcnt = 0x1, refcnt = 0x2, suspend_cnt = 0x0, locked = 0, target = com.apple.root.default-priority[0x7fff72c47d00], width = 0x7fffffff, running = 0x0, barrier = 0 }&gt; (lldb) po conn (xpc_connection_t) $1 = 0x0000000100400830 &lt;OS_xpc_connection: connection[0x100400830]: { refcnt = 1, xrefcnt = 1, name = com.jeremywsherman.conn, type = named, state = new, queue = 0x100400530-&gt;0x0, error = 0x0, mach = false, privileged = false, bssendp = 0x0, recvp = 0x0, sendp = 0x0, pid/euid/egid/asid = 0/4294967295/4294967295/4294967295 } &lt;connection: 0x100400830&gt; { name = com.jeremywsherman.conn, listener = false, PID = 0, EUID = 4294967295, EGID = 4294967295, ASID = 4294967295 }&gt; (lldb) po pred (xpc_object_t) $2 = 0x00007fff7244d320 &lt;OS_xpc_bool: bool[0x7fff7244d320]: { refcnt = 80000000, xrefcnt = 80000000, value = true } &lt;bool: 0x7fff7244d320&gt;: true&gt; </code></pre> Introducing OS Object https://jeremywsherman.com/blog/2013/01/01/introducing-os-object/ Tue, 01 Jan 2013 00:00:00 +0000 https://jeremywsherman.com/blog/2013/01/01/introducing-os-object/ <p>GCD came along with 10.6 and made concurrent programming easy. ARC came along with 10.7 and let us mostly forget about this whole refcounting business.</p> <p>But 10.7’s GCD was left behind in manual retain-release land. (XPC was too, but GCD is our hero this time.) 10.8 fixed that oversight via a clever hack hidden away in <code>&lt;os/object.h&gt;</code>.</p> <h2 id="behold-i-bring-you-an-object">Behold, I bring you an object!</h2> <p>The magic happens in the interaction between two macros, <code>OS_OBJECT_DECL</code> and <code>OS_OBJECT_DECL_SUBCLASS</code>.</p> <p>GCD came along with 10.6 and made concurrent programming easy. ARC came along with 10.7 and let us mostly forget about this whole refcounting business.</p> <p>But 10.7’s GCD was left behind in manual retain-release land. (XPC was too, but GCD is our hero this time.) 10.8 fixed that oversight via a clever hack hidden away in <code>&lt;os/object.h&gt;</code>.</p> <h2 id="behold-i-bring-you-an-object">Behold, I bring you an object!</h2> <p>The magic happens in the interaction between two macros, <code>OS_OBJECT_DECL</code> and <code>OS_OBJECT_DECL_SUBCLASS</code>.</p> <p><code>OS_OBJECT_DECL</code> is used to declare the base object type of your refcounted C library. It conceptually creates a new root class:</p> <pre><code>OS_OBJECT_DECL(dispatch_object); </code></pre> <p>Once you’ve declared a root class using <code>OS_OBJECT_DECL</code>, you use <code>OS_OBJECT_DECL_SUBCLASS</code> to declare new subclasses:</p> <pre><code>OS_OBJECT_DECL_SUBCLASS(dispatch_queue, dispatch_object); OS_OBJECT_DECL_SUBCLASS(dispatch_source, dispatch_object); </code></pre> <p>And magically, you now have types <code>dispatch_object_t</code>, <code>dispatch_queue_t</code>, and <code>dispatch_source_t</code>.</p> <h2 id="presto-changeo">Presto-Changeo</h2> <p>As far as casts are concerned, these new types behave just like <code>NSObject</code>, <code>NSString</code>, and <code>NSNumber</code>. If you declare variables like so:</p> <pre><code>NSObject *o; NSNumber *n; NSString *s; </code></pre> <p>The compiler will allow you to implicitly upcast without complaint:</p> <pre><code>/* hunky dory */ o = n; o = s; </code></pre> <p>but not down or crosswise:</p> <pre><code>dispatch_cast.m:13:4: warning: incompatible pointer types assigning to 'NSNumber *__strong' from 'NSObject *__strong' [-Wincompatible-pointer-types] n = o; ^ ~ dispatch_cast.m:14:4: warning: incompatible pointer types assigning to 'NSNumber *__strong' from 'NSString *__strong' [-Wincompatible-pointer-types] n = s; ^ ~ </code></pre> <p>Similarly, with these declarations:</p> <pre><code>dispatch_object_t o; dispatch_queue_t q; dispatch_source_t s; </code></pre> <p>This is fine:</p> <pre><code>/* hunky dory */ o = q; o = s; </code></pre> <p>But this is not:</p> <pre><code>q = o; q = s; </code></pre> <p>The error messages hint at how this is implemented:</p> <pre><code>dispatch_cast.m:27:4: warning: incompatible pointer types assigning to '__strong dispatch_queue_t' (aka 'NSObject&lt;OS_dispatch_queue&gt; *__strong') from '__strong dispatch_object_t' (aka 'NSObject&lt;OS_dispatch_object&gt; *__strong') [-Wincompatible-pointer-types] q = o; ^ ~ dispatch_cast.m:28:4: warning: incompatible pointer types assigning to '__strong dispatch_queue_t' (aka 'NSObject&lt;OS_dispatch_queue&gt; *__strong') from '__strong dispatch_source_t' (aka 'NSObject&lt;OS_dispatch_source&gt; *__strong') [-Wincompatible-pointer-types] q = s; ^ ~ </code></pre> <h2 id="osobject-is-protocols">OSObject Is Protocols!</h2> <p>And that’s the trick, you see. There aren’t any classes, just protocols. Because protocols can be declared as conforming to other protocols, we have a protocol hierarchy parallel to our class hierarchy. By using a protocol-qualified type – <code>NSObject&lt;OS_dispatch_queue&gt; *</code> meaning, “Any <code>NSObject</code>, so long as it conforms to <code>OS_dispatch_queue</code>” – we can make our hierarchy concrete in terms of which OS objects can be pointed at by which pointers.</p> <p>Why <code>NSObject</code> and not <code>id</code>? Because ARC needs to be able to use retain/release/autorelease, and NSObject provides a convenient declaration of those and other methods.</p> <h2 id="or-is-it">Or Is It?</h2> <p>Of course, there would have to be more to this OSObject thing than just protocols for ARC to work: whatever type-level hackery you might perpetrate, the message send <code>[pointer retain]</code> is only going to work if the whole Objective-C message send machinery can use what’s at <code>*pointer</code> as an Obj-C object.</p> <p>Consequently, things look a lot different from inside libdispatch. There are covert class interfaces and corresponding implementations that go along with the public protocols.</p> <p>A shame you can’t just sprinkle a few macros over a C library that uses refcounting and have it work automagically with ARC. Now, there’s a thought…</p> Reconsidering +new https://jeremywsherman.com/blog/2012/09/08/reconsidering-new/ Sat, 08 Sep 2012 00:00:00 +0000 https://jeremywsherman.com/blog/2012/09/08/reconsidering-new/ <p>Received wisdom teaches that Objective-C&rsquo;s alloc-init two-step is important for both clarity and extensibility. And even if those two reasons don&rsquo;t sway you, it&rsquo;s both childish and déclassé to continue using <code>+new</code> past a certain programmer-age.</p> <p>Or is it?</p> <p>Received wisdom teaches that Objective-C&rsquo;s alloc-init two-step is important for both clarity and extensibility. And even if those two reasons don&rsquo;t sway you, it&rsquo;s both childish and déclassé to continue using <code>+new</code> past a certain programmer-age.</p> <p>Or is it?</p> <p>Yes, creating a new object requires allocating its storage and then initializing it. But they&rsquo;re not really distinct any more. It&rsquo;s not like we do:</p> <pre><code>id obj = [Foo alloc]; if (!obj) error(&quot;allocation failed&quot;); obj = [obj init]; if (!obj) error(&quot;init failed&quot;); </code></pre> <p>And zones are dead, so separating alloc and init so you can do:</p> <pre><code>id obj = [[Foo allocWithZone:fooZone] init]; </code></pre> <p>doesn&rsquo;t really matter any more, either.</p> <p>And it breaks down even further when you look at Core Foundation analogs. There is no <code>CFAlloc()</code> followed by <code>CFArrayInit()</code>. Core Foundation just has Create methods that take an allocator to handle the &ldquo;different zones&rdquo; concern. Normally, you just pass <code>NULL</code> or <code>kCFAllocatorDefault</code> for the allocator argument, but either allocators have better support than zones at this time, or Apple just doesn&rsquo;t care enough to write &ldquo;don&rsquo;t use allocators any more&rdquo; anywhere.</p> <p>Since these are equivalent:</p> <pre><code>CFMutableArrayRef array = CFArrayCreateMutable( NULL, 0, &amp;kCFTypeArrayCallbacks); NSMutableArray *array = [[NSMutableArray alloc] init]; </code></pre> <p>I see no reason not to just do a &ldquo;single call&rdquo; alloc-init in Foundation-land, too:</p> <pre><code>NSMutableArray *array = [NSMutableArray new]; </code></pre> <p>This for the common case. When you need to pass args in during construction, back to <code>alloc-initWith…</code> it is!</p> <p>Aside: As a practical motivation for <code>+new</code>, when I&rsquo;m throwing together a quick commandline program to see whether something behaves one way or another, <code>[Blah new]</code> types a lot faster than <code>[[Blah alloc] init]</code>, particularly if I forget the double-bracket at the start and have to back up and fix it.</p> <p>Aside 2: Many of the Foundation types let you get away with a compromise, like <code>[NSMutableArray array]</code>. In ARC-land, this is effectively no different than writing <code>[NSMutableArray new]</code> – what if you later need an <code>arrayWithObjects:</code>! what if you later need to allocate it in a different zone! –, but I never see anyone inveighing against <code>-array</code>, or <code>-string</code>, or <code>-dictionary</code>. So.</p> Migrating to Obj-C Literals https://jeremywsherman.com/blog/2012/08/27/migrating-to-obj-c-literals/ Mon, 27 Aug 2012 00:00:00 +0000 https://jeremywsherman.com/blog/2012/08/27/migrating-to-obj-c-literals/ <p>Obj-C literals make your code cleaner and more compact, but hand-updating a large codebase to take advantage of Obj-C literals would be a bore, and all too easy to mess up during a distracted moment.</p> <p>This is what automated refactoring tools were designed for. And Apple has provided us with an oft-overlooked arrow in our devtools quiver that&rsquo;s just what we need here: <code>tops</code>.</p> <p>Obj-C literals make your code cleaner and more compact, but hand-updating a large codebase to take advantage of Obj-C literals would be a bore, and all too easy to mess up during a distracted moment.</p> <p>This is what automated refactoring tools were designed for. And Apple has provided us with an oft-overlooked arrow in our devtools quiver that&rsquo;s just what we need here: <code>tops</code>.</p> <p>Check out <a href="(https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/tops.1.html)"><code>man tops</code></a>. The tool has a decent understanding of Obj-C syntax and accepts scripts that let you rewrite code to use new method calls, new functions, and what-have-you. The examples make it look like this tool was invented to ease the transition from NeXT-style Obj-C to Cocoa, like this gem:</p> <pre><code>replace &quot;NXGetNamedObject(&lt;b args&gt;)&quot; with same error &quot;ApplicationConversion: NXGetNamedObject() is obsolete. Replace with nib file outlets.&quot; </code></pre> <p>That should take some of you way back.</p> <p>Anyway, with this tool, modernizing your code can be as simple as:</p> <pre><code>tops -semiverbose -scriptfile literals.tops **/*.(h|m|hpp|mm) </code></pre> <p>Want to check that it will do the right thing? Throw <code>-dont</code> into the args.</p> <p>Want to watch over its shoulders as it rewrites your code? Replace <code>-semiverbose</code> with straight-up <code>-verbose</code>.</p> <p>Now, for that magical script file:</p> <script type="application/javascript" src="https://gist.github.com/jeremy-w/3493503.js?file=literals.tops"></script> <p>And here&rsquo;s an Obj-C file to test it against:</p> <script type="application/javascript" src="https://gist.github.com/jeremy-w/3493503.js?file=literals.m"></script> Prefix Notation https://jeremywsherman.com/blog/2012/08/04/prefix-notation/ Sat, 04 Aug 2012 00:00:00 +0000 https://jeremywsherman.com/blog/2012/08/04/prefix-notation/ I don&rsquo;t believe your spoken language syntax order has any bearing on whether you find Lisp prefix notation hard. Consider thesis 2 of Tim Bray&rsquo;s &ldquo;Eleven Theses on Clojure&rdquo;: In school, we all learn 3 + 4 = 7 and then sin(π/2) = 1 and then many of us speak languages with infix verbs. So Lisp is fighting uphill. I call bunkum. I don&rsquo;t want to single out Tim Bray here. <p>I don&rsquo;t believe your spoken language syntax order has any bearing on whether you find Lisp prefix notation hard.</p> <p>Consider thesis 2 of <a href="http://www.tbray.org/ongoing/When/200x/2009/12/01/Clojure-Theses">Tim Bray&rsquo;s &ldquo;Eleven Theses on Clojure&rdquo;</a>:</p> <blockquote> <p>In school, we all learn 3 + 4 = 7 and then sin(π/2) = 1 and then many of us speak languages with infix verbs. So Lisp is fighting uphill.</p> </blockquote> <p>I call bunkum.</p> <p>I don&rsquo;t want to single out Tim Bray here. I&rsquo;ve seen this other places before. It&rsquo;s a popular folk explanation. But his is the straw that broke the camel&rsquo;s back.</p> <p>Folks often reach for natural language or arithmetic notation to explain why Lisp prefix notation is golly gee so hard. The argument goes like this:</p> <ul> <li>I speak a language with syntactic order subject-verb-object, or SVO for short, which is like infix notation.</li> <li>I write arithmetic using infix notation.</li> <li>Lisp uses prefix notation, which is not infix notation.</li> <li>Therefore, Lisp is hard to read.</li> </ul> <p>The heart of the argument is a mismatch between spoken language sentence order and Lisp syntactic form order makes reading Lisp hard.</p> <p>But if we remove the mismatch, does Lisp get any easier? Let&rsquo;s see:</p> <ul> <li> <p>I speak a language with syntactic order verb-subject-object, which is like prefix notation.</p> </li> <li> <p>Lisp uses prefix notation, which is the same as my language&rsquo;s!</p> </li> <li> <p>Therefore, Lisp is:</p> <ul> <li>Hard?</li> <li>Easy?</li> <li>Error: Invalid Deduction?</li> </ul> </li> </ul> <p>Does speaking Gaelic condemn you to unassuageable puzzlement at infix notation?</p> <p>Does speaking German grant a supernatural facility with postfix notation?</p> <p>Are Finns left out in the cold, waiting for an appropriately agglutinative programming language?</p> <p>Are French speakers flocking to <a href="http://langagelinotte.free.fr/wordpress/" title="Linotte, l'algorithmique et la programmation facilement">Linotte</a>?</p> <p>I think not.</p> Visualizing Consequences https://jeremywsherman.com/blog/2012/07/29/visualizing-consequences/ Sun, 29 Jul 2012 00:00:00 +0000 https://jeremywsherman.com/blog/2012/07/29/visualizing-consequences/ The ability to visualize the consequences of the actions under consideration is crucial to becoming an expert programmer, just as it is in any synthetic, creative activity. (SICP 1.2) There&rsquo;s a larval Big Nerd Ranch reading group, and it has me reading through Ye Olde Wizard Book, Structure and Interpretation of Computer Programs. I&rsquo;m pretty early yet in the text, and just happened upon the quote you find up top there starting this post. <blockquote> <p>The ability to visualize the consequences of the actions under consideration is crucial to becoming an expert programmer, just as it is in any synthetic, creative activity. (<a href="http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-11.html">SICP 1.2</a>)</p> </blockquote> <p>There&rsquo;s a larval Big Nerd Ranch reading group, and it has me reading through Ye Olde Wizard Book, <a href="http://mitpress.mit.edu/sicp/"><em>Structure and Interpretation of Computer Programs.</em></a> I&rsquo;m pretty early yet in the text, and just happened upon the quote you find up top there starting this post.</p> <p>You could blow by this pretty fast on your way to some deep wizardry.</p> <p>Don&rsquo;t.</p> <p>This ability to stare deep into and through a line of code and watch the clockwork wheels spin is key to mastering the craft of programming. It&rsquo;s what separates those who understand what their code is doing from those who continue to view the operations of their compiler or interpreter as a mystery concealed behind a veil impenetrable by mortal eyne.</p> <p>Let me give it to you straight: <em>There ain&rsquo;t no deep black magic here.</em> There is in fact nothing more quotidian than the process that takes a line of code and translates through layer upon layer of lengthy and tedious documentation into something that ultimately can be executed by the materia technologica sitting there upon your desk. Or your lap. Or held in the palm of your hand. Form factor changes; number and names of layers change; ultimate lack of magic does not.</p> <p>Don&rsquo;t lie to yourself that all that happens between <code>make</code> and <code>./a.out</code> is impenetrable. It&rsquo;s all there, waiting for you. It&rsquo;s a long and well-trod path. Don&rsquo;t turn away from it: put one foot in front of the other, work your way down one more layer of abstraction, and start to see how the sausage is made.</p> Behavioral Programming https://jeremywsherman.com/blog/2012/07/23/behavioral-programming/ Mon, 23 Jul 2012 00:00:00 +0000 https://jeremywsherman.com/blog/2012/07/23/behavioral-programming/ I recently read an exciting article on by Harel et al. on behavioral programming. The basic idea of behavior programming is to compose a bunch of simultaneously executing state machines. Each machine represents a behavior. But you don&rsquo;t use the plain event in/event out state chart to define these machines. Instead, you add modal operators to specify what must/may/mustn&rsquo;t happen next: must: Please carry out this action. may: Please let me know when this happens. <p>I recently read an exciting <a href="http://cacm.acm.org/magazines/2012/7/151241-behavioral-programming/fulltext">article on by Harel et al. on behavioral programming.</a></p> <p>The basic idea of behavior programming is to compose a bunch of simultaneously executing state machines. Each machine represents a behavior.</p> <p>But you don&rsquo;t use the plain event in/event out state chart to define these machines. Instead, you add modal operators to specify what must/may/mustn&rsquo;t happen next:</p> <ul> <li><em>must</em>: Please carry out this action.</li> <li><em>may</em>: Please let me know when this happens.</li> <li><em>mustn&rsquo;t</em>: No matter who tells you to do this, don&rsquo;t; if it does happen, abend.</li> </ul> <p>These operators represent a synchronization point between behaviors. Once all behavioral threads, or &ldquo;bthreads&rdquo; for short, have blocked expressing a modal preference, the executive picks a must-event that&rsquo;s not blocked by a mustn&rsquo;t operator, carries out the event, and notifies any bthread that had that event listed as a must or may event. Execution then continues till every bthread blocks again by specifying a modal operator.</p> <p>I like this model because it&rsquo;s a thin but powerful layer over existing models. You can implement it with pthreads and synchronous pub/sub, which could be as simple as a group of pthread conditions.</p> <p>The development approach you end up with differs markedly from the model you build your bthread framework in.</p> <p>Here&rsquo;s the powerful part of the bthread layer: Decomposing your problem into bthreads frees you up to pick a set of coordinating events, then start coding scenarios around those events. Fire up your program, see how it does, and then fix any bugs that testing/simulation/model checking shows up and go again.</p> <p>The state machine ness also lets you react to entire event traces in order to handle things like a &ldquo;win rather than defend&rdquo; strategy in tic-tac-toe, which is one of the basic examples given in the article.</p> <p>There&rsquo;s room for plenty of cleverness in how the executive selects the next event and how to test and check bthread programs, but the core idea is elegant and exciting. The full article is worth a read.</p> <hr> <span style="font-size: 80%;"> David Harel, Assaf Marron, Gera Weiss. [Behavioral Programming.][bprog] *Communications of the ACM,* Vol. 55 No. 7, Pages 90-100. doi 10.1145/2209249.2209270. <http://cacm.acm.org/magazines/2012/7/151241-behavioral-programming/fulltext>. Retrieved 2012-07-23.</span> What's this about @import? https://jeremywsherman.com/2012/02/29/whats-this-about-at-import/ Wed, 29 Feb 2012 00:00:00 +0000 https://jeremywsherman.com/2012/02/29/whats-this-about-at-import/ <p>So today in the company IRC channel my illustrious colleague <a href="http://twitter.com/borkware">Mark Dalrymple</a> (of <em><a href="http://borkware.com/corebook/">Advanced Mac OS X Programming</a></em> fame) mentioned this new-fangled <code>@import</code> compiler directive. News of this compiler directive appears to be <a href="https://twitter.com/#!/search/%40import">spreading</a> through the Objective-C developer community mostly by way of Twitter-pigeon.</p> <p>As it happened, I had not heard of <code>@import</code>. But then the inimitable Mikey Ward (alias: <a href="http://twitter.com/wookiee">Wookiee</a>) asked me about it. Two persons independently inquiring? Now I had to look into it.</p> <p>So today in the company IRC channel my illustrious colleague <a href="http://twitter.com/borkware">Mark Dalrymple</a> (of <em><a href="http://borkware.com/corebook/">Advanced Mac OS X Programming</a></em> fame) mentioned this new-fangled <code>@import</code> compiler directive. News of this compiler directive appears to be <a href="https://twitter.com/#!/search/%40import">spreading</a> through the Objective-C developer community mostly by way of Twitter-pigeon.</p> <p>As it happened, I had not heard of <code>@import</code>. But then the inimitable Mikey Ward (alias: <a href="http://twitter.com/wookiee">Wookiee</a>) asked me about it. Two persons independently inquiring? Now I had to look into it.</p> <p>It appears modules are filtering into Objective-C by way of C++, the same way Objective-C is rumored to be inheriting you-pick-the-base-type enums from C++TNG. Only this time the feature isn&rsquo;t part of any standard.</p> <p>I get this idea from <a href="http://lists.cs.uiuc.edu/pipermail/cfe-dev/2011-December/019322.html">an exchange</a> on the cfe-dev mailing list in late December, which I have condensed into a single apocryphal message:</p> <blockquote> <p>If you check recent (the last 6 months or so) commits to clang by Doug Gregor, you&rsquo;ll find some work to implement C++ modules is already underway.</p> </blockquote> <blockquote> <p>I&rsquo;m not sure how much it&rsquo;s based on any specific proposal.</p> </blockquote> <blockquote> <p>To misquote Doug [Gregor] (can&rsquo;t find the email, I think it might&rsquo;ve been on IRC): &ldquo;The semantics are obvious enough, so I&rsquo;m implementing those. After that we can haggle over the syntax&rdquo;</p> </blockquote> <p>(In case it&rsquo;s driving you crazy, &ldquo;cfe&rdquo; is short for &ldquo;c/clang front-end&rdquo;, which is all the <code>clang</code> tool you use from the commandline is: a driver for a whole mess of surprisingly unmessy library code.)</p> <p>At this point, Doug was kind enough to <a href="http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-January/019334.html">chime in</a>:</p> <blockquote> <p>Most of the work I&rsquo;m doing is in three places. The Serialization module, which takes care of serializing/deserializing an already-parsed AST, is the hardest part: it&rsquo;s the infrastructure that allows one to compile a module on its own, storing the serialized AST to disk, and then load that module into another translation unit later on. This part is likely to be the same regardless of how modules behave. [Clang will produce and cache module AST files on the fly. Authors and build systems will remain ignorant of these AST files.]</p> </blockquote> <blockquote> <p>The module map part of the Lex module handles the mapping between headers and modules. It&rsquo;s mainly a transitional a little sub-language that allows one to describe the relationships between headers (which are used everywhere today) and modules.</p> </blockquote> <blockquote> <p>The easy part is the parsing of module imports, labeling what is exported/hidden, and name-lookup semantics. It&rsquo;s also the part that people will want to discuss endlessly, so for now the various keywords are uglified so that we don&rsquo;t commit to any one syntax.</p> </blockquote> <p>Once you chase on down through the code, you find yourself at the abstract syntax tree level staring at the <code>ImportDecl</code> class. What does it do? Well,</p> <blockquote> <p>[it] describes a module import declaration, which makes the contents of the named module visible in the current translation unit. An import declaration imports the named module (or submodule). For example: @import std.vector; Import declarations can also be implicitly generated from #include/#import directives.</p> </blockquote> <p>That&rsquo;s right: <code>#include</code>/<code>#import</code> are going to become legacy syntax for this nifty new modules system. Then, instead of playing &ldquo;find the header that includes the symbol you want to use&rdquo;, you will be able to basically just import the functionality directly.</p> <p>The actual mapping from module name to file is handled by the <code>ModuleLoader</code>. Judging by the <a href="http://llvm.org/svn/llvm-project/cfe/trunk/test/Modules/">tests</a>, there&rsquo;s going to be a way to explicitly manage this mapping using <a href="module-map">module stanzas</a> in a <a href="mod-map-h">ModuleMap</a>:</p> <pre><code>module category_left { header &quot;category_right.h&quot; export category_top } </code></pre> <p>You also get another way to manage symbol visibility via <code>export</code> directives in the module stanzas, as you can see there.</p> <p>Possibly the awesomest future visibility control is that over those pesky preprocessor macros. That&rsquo;s right: the <a href="prop-syn">proposed syntax</a> is</p> <pre><code>#define MODULE_H_MACRO 1 #define MODULE_H_PUBMACRO 2 #__private_macro MODULE_H_MACRO #__public_macro MODULE_H_PUBMACRO </code></pre> <p>The private/public macro preprocessor directives update the visibility of the named macro. If you have multiple macros, you have to issue multiple directives &ndash; there&rsquo;s no support for privatizing multiple macros something like:</p> <pre><code>#__private_macro XYZZY PLUGH PLOVER /* &lt;-- THIS DOES NOT WORK */ </code></pre> <p>I haven&rsquo;t the faintest notion when we&rsquo;ll see this live and slaving away under Xcode, but I am looking forward to the coming sleeker, faster import process.</p> <hr> <p>References galore, from top to bottom:</p> <ul> <li>Homepage of the fellow behind much of this: <a href="http://llvm.org/svn/llvm-project/cfe/trunk/test/Modules/Inputs/module.map">Doug Gregor</a>.</li> <li><a href="http://llvm.org/viewvc/llvm-project?view=rev&amp;revision=147452">Commit</a> switching from <code>__import_module__</code> to <code>@import</code> syntax.</li> <li><a href="http://llvm.org/svn/llvm-project/cfe/trunk/include/clang/Basic/LangOptions.def">Declaration</a> of the modules language option as a C (not C++) extension.</li> <li><a href="http://llvm.org/svn/llvm-project/cfe/trunk/lib/Lex/Preprocessor.cpp">Preprocessor</a> lexer support. Search for <code>LexAfterModuleImport</code>.</li> <li><a href="http://llvm.org/svn/llvm-project/cfe/trunk/lib/Parse/Parser.cpp">Parsing</a> support for the syntax. Search for <code>ParseModuleImport</code>.</li> <li><a href="http://llvm.org/svn/llvm-project/cfe/trunk/lib/Sema/SemaDecl.cpp">Semantic</a> implementation. Search for <code>ActOnModuleImport</code>.</li> <li><a href="http://llvm.org/svn/llvm-project/cfe/trunk/include/clang/Lex/ModuleLoader.h">ModuleLoader</a> declaration.</li> <li><a href="http://llvm.org/svn/llvm-project/cfe/trunk/test/Modules/">Tests</a> exercising the upcoming modules extension.</li> </ul> Memory Allocation in <tt>sam</tt> https://jeremywsherman.com/2012/02/28/memory-allocation-in-sam/ Tue, 28 Feb 2012 00:00:00 +0000 https://jeremywsherman.com/2012/02/28/memory-allocation-in-sam/ <p>Bad news, bucko. <code>Malloc</code> is merely &ldquo;adequate.&rdquo; And it&rsquo;s only adequate if you&rsquo;re writing <em>simple</em> programs. Real programmers write their own memory manager. It&rsquo;s the first thing they do after they ditch their shaving kit and start growing their Samson neckbeard.</p> <p>Don&rsquo;t believe me? Listen up:</p> <blockquote> <p>The <code>C</code> language has no memory allocation primitives, although a standard library routine, <code>malloc</code>, provides adequate service for simple programs. For specific uses, however, it can be better to write a custom allocator.</p> <footer> <strong>Rob Pike</strong><cite><a href="http://plan9.bell-labs.com/sys/doc/sam/sam.pdf">The Text Editor <tt>sam</tt> (1987)</a></cite></footer> </blockquote> <p>Bad news, bucko. <code>Malloc</code> is merely &ldquo;adequate.&rdquo; And it&rsquo;s only adequate if you&rsquo;re writing <em>simple</em> programs. Real programmers write their own memory manager. It&rsquo;s the first thing they do after they ditch their shaving kit and start growing their Samson neckbeard.</p> <p>Don&rsquo;t believe me? Listen up:</p> <blockquote> <p>The <code>C</code> language has no memory allocation primitives, although a standard library routine, <code>malloc</code>, provides adequate service for simple programs. For specific uses, however, it can be better to write a custom allocator.</p> <footer> <strong>Rob Pike</strong><cite><a href="http://plan9.bell-labs.com/sys/doc/sam/sam.pdf">The Text Editor <tt>sam</tt> (1987)</a></cite></footer> </blockquote> <p><code>Sam</code> was just some text editor from the 80s, and its memory management was way more rocking than your Twitter client&rsquo;s will ever be.</p> <p><code>Sam</code> memory management was so rocking that it filled <em>two</em> arenas. That&rsquo;s right: two arenas. Your memory management needs are insignificant, puny, and plebeian, serviced adequately by the C standard library. <code>Sam</code> got true rockstar treatment: two arenas; two custom allocators; high maintenance, premium memory management.</p> <p>The first arena holds staid structs of fixed length. It&rsquo;s filled first-fit. Nothing magic there.</p> <p>The second arena holds variably sized objects like strings. In an editor, strings are always changing, growing, splitting, combining. A regular bunch of problem children. So it&rsquo;s managed by a garbage-compacting allocator.</p> <p>These arenas are erected side-by-side in memory, with the second arena getting the higher addresses. When the first-fit arena needs more space, it just bumps the compacting arena up in memory.</p> <p>The real magic is how these two arenas are used together. Take for example a variable-length array. <code>Sam</code> handles this by creating a struct with a length and a pointer. The struct is allocated in the struct arena of course, but its pointer points into the compacting arena. The allocator knew to go back and rewrite the struct pointer whenever it moved its memory during compaction, and the programmer knew (or learned really fast the hard way) to always use the struct&rsquo;s pointer field directly each time rather than caching it away somewhere.</p> <p>Now that&rsquo;s some pretty boss hacking: elegant, but at what many today might consider an advanced, &ldquo;don&rsquo;t go there without a friend&rdquo; low level.</p> <hr> <p><em>P.S.:</em> I would encourage you to check out <code>sam</code>'s source to see how it&rsquo;s done, but yesterday&rsquo;s arenas are no more. The <a href="http://plan9.bell-labs.com/sources/plan9/sys/src/cmd/sam/" title="/sys/src/cmd/sam - Plan 9 from Bell Labs">current source</a> just calls <code>malloc</code> once for each allocation.</p> <tt>sam</tt>'s Structural Regular Expressions https://jeremywsherman.com/2012/02/27/sam-s-structural-regular-expressions/ Mon, 27 Feb 2012 00:00:00 +0000 https://jeremywsherman.com/2012/02/27/sam-s-structural-regular-expressions/ <p>This is the first post in a series of quotes from papers. These are the great turns of phrase, the intriguing idea I&rsquo;ve run into nowhere else, the start of something that could have been great but probably fizzled.</p> <p>First up: structural regular expressions, as introduced in the GUI text editor <code>sam</code></p> <p>This is the first post in a series of quotes from papers. These are the great turns of phrase, the intriguing idea I&rsquo;ve run into nowhere else, the start of something that could have been great but probably fizzled.</p> <p>First up: structural regular expressions, as introduced in the GUI text editor <code>sam</code></p> <p>:</p> <blockquote> <p>In other UNIX programs, regular expressions are used only for selection, as in the <tt>sam</tt> <tt>g</tt> command, never for extraction as in the <tt>x</tt> or <tt>y</tt> command. For example, patterns in <tt>awk</tt> are used to select lines to be operated on, but cannot be used to describe the format of the input text, or to handle newline-free text. The use of regular expressions to describe the structure of a piece of text rather than its contents, as in the <tt>x</tt> command, has been given a name: <em>structural regular expressions.</em> When they are composed, as in the above example, they are pleasantly expressive. Their use is discussed at greater length elsewhere.</p> <footer> <strong>Rob Pike</strong><cite><a href="http://plan9.bell-labs.com/sys/doc/sam/sam.pdf">The Text Editor <tt>sam</tt> (1987)</a></cite></footer> </blockquote> <p><code>x</code> extracts every chunk of text matching the regex provided to it. Each chunk has the rest of the editing pipeline run on it. Want to change every <code>n</code> in a hunk of text to an <code>m</code>? Select it all in the window with button 1, focus the <code>sam</code> command window with button 1, and type in:</p> <pre><code>x/n/ c/m/ </code></pre> <p>Hit return and this command pipeline runs on the (implicit range) <code>dot</code>, also known as &ldquo;the current selection.&rdquo; <code>x</code> grabs an <code>n</code>, <code>c</code> then changes it to an <code>m</code>. You can layer on more commands, including <code>g</code> (guard) as a by-the-way if statement. The command text stays in the command window in case you want to run it again.</p> <p>Boom! Instant macro, no memorization of registers required. Take that, <code>vim</code> <code>qX…q @X</code>.</p> <p>This search for symbiosis between mouse and keyboard is what led to <code>sam</code>. Most UNIX editors bolt mouse input onto an established keyboard-centric paradigm. <code>Sam</code> rethinks editing to make the mouse an integral part of it. (<code>Acme</code> would later take this mouse integration to new heights. We&rsquo;ll get to <code>acme</code> in time.)</p> <p>Back to structural regular expressions now. Pike has a whole paper on the topic that I will doubtless get to eventually. But just this little bit is tantalizing enough.</p> <p>I mean, think about <code>awk</code>, think about how you use regular expressions there, or how you use them in your editor <em>du jour</em>. Are these tools really making the most of regular expressions? <code>awk</code> and friends just perform record splitting on a set of separator characters. Imagine how limited your regexes would be if all you got to do was specify what to stick between two character class braces: <code>[</code><em>your characters here</em><code>]{1,}</code>. That&rsquo;s all you get with this simple record separator construction.</p> <p>And it&rsquo;s not like we&rsquo;ve made great strides: Search-and-replace in an IDE like Xcode or Eclipse gives you even less expressiveness.</p> <p>I look forward to reading more about structural regular expressions in future. For more on <code>sam</code>, see:</p> <ul> <li><a href="http://www.cs.bell-labs.com/sys/doc/sam/sam.html">Rob Pike&rsquo;s paper</a> introducing the editor. (Also available in <a href="http://plan9.bell-labs.com/sys/doc/sam/sam.pdf">PDF</a>.)</li> <li><a href="http://sam.cat-v.org/"><code>sam</code> at cat-v</a></li> </ul> Pasting HTML into Markdown https://jeremywsherman.com/2012/02/08/pasting-html-into-markdown/ Wed, 08 Feb 2012 00:00:00 +0000 https://jeremywsherman.com/2012/02/08/pasting-html-into-markdown/ <p>So I was writing a comment on Reddit today, and the easiest way to answer was to quote a list of search results, links and all. One problem: Reddit&rsquo;s post interface uses Markdown, not HTML. That&rsquo;s swell when you&rsquo;re writing your comment fresh, but if you want to paste something in from a webpage, it&rsquo;s no good.</p> <p>I&rsquo;d run into this once or twice before, but I always took the simple way out: just rewrite the one or two links in the text by hand. No big deal.</p> <p>But these search results were just a list of links. And as a programmer, I am vocationally virtuously lazy.</p> <p>That&rsquo;s when I remembered <a href="http://johnmacfarlane.net/pandoc/" title="Pandoc: A universal document converter">Pandoc</a>.</p> <p>So I was writing a comment on Reddit today, and the easiest way to answer was to quote a list of search results, links and all. One problem: Reddit&rsquo;s post interface uses Markdown, not HTML. That&rsquo;s swell when you&rsquo;re writing your comment fresh, but if you want to paste something in from a webpage, it&rsquo;s no good.</p> <p>I&rsquo;d run into this once or twice before, but I always took the simple way out: just rewrite the one or two links in the text by hand. No big deal.</p> <p>But these search results were just a list of links. And as a programmer, I am vocationally virtuously lazy.</p> <p>That&rsquo;s when I remembered <a href="http://johnmacfarlane.net/pandoc/" title="Pandoc: A universal document converter">Pandoc</a>.</p> <p>Pandoc is a tool for converting between markup languages. I grabbed it as a Swiss army knife alternative to the more questionable Markdown formatters out there. (Markdown&rsquo;s reference implementation is in Perl. I have trouble regarding any Perl as anything <em>but</em> a fragile hack.) I actually used it the first time as part of avoiding wordprocessors: instead of emailing a PDF/Word doc/Pages doc (in order of increasing uckiness), I just write up a Markdown doc, format it into a standalone webpage, slap in some CSS, and email it off.</p> <p>Veering back on course, I recalled it could be used not only from Markdown to HTML but from HTML to Markdown. And how to get pasted text from the browser into HTML? I didn&rsquo;t want to muck with View Source, so just opened up TextEdit and let its erstwhile annoying habit of preserving pasted formatting work to my benefit. Copy from Aurora, paste into TextEdit, save as HTML, then <code>pandoc -f html -t markdown foo.html | pbcopy</code>, back to Aurora, and paste, and beautiful Markdown appears.</p> <p>Long story short: Use <code>pandoc</code> to convert HTML into Markdown for your Reddit or Stackoverflow or blogging needs.</p> The Artful Edit https://jeremywsherman.com/2012/02/06/the-artful-edit/ Mon, 06 Feb 2012 00:00:00 +0000 https://jeremywsherman.com/2012/02/06/the-artful-edit/ <p>Susan Bell&rsquo;s <em><a href="http://www.amazon.com/gp/product/0393332179/ref=as_li_qf_sp_asin_tl?ie=UTF8&tag=jerwshe-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0393332179">The Artful Edit</a><img src="http://www.assoc-amazon.com/e/ir?t=jerwshe-20&l=as2&o=1&a=0393332179" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></em> offers a brief but thorough introduction to editing your own writing.</p> <p>Susan Bell&rsquo;s <em><a href="http://www.amazon.com/gp/product/0393332179/ref=as_li_qf_sp_asin_tl?ie=UTF8&tag=jerwshe-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0393332179">The Artful Edit</a><img src="http://www.assoc-amazon.com/e/ir?t=jerwshe-20&l=as2&o=1&a=0393332179" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></em> offers a brief but thorough introduction to editing your own writing.</p> <p>The book is structured around three steps: gaining perspective, macro-editing, and micro-editing. Macro-editing addresses the structure of the work. It requires elucidating then shaping that structure and the characters and themes that build it. Micro-editing examines word choice, continuity, and other concerns at the level of the individual paragraph, sentence, and even word.</p> <p>Each practical chapter ends with a bulleted summary and exercises. The summary frees you to focus on reading the book. Without it, you&rsquo;d regularly interrupt your reading to scratch down notes. The exercises give concrete practices to improve your editing.</p> <p>After each chapter comes an interlude wherein various authors reflect on writing and editing. These leaven the book&rsquo;s didactic tone, but all are forgettable save the last, Michael Ondaatje&rsquo;s &ldquo;One Doesn&rsquo;t Just Write a Book, One Makes a Book.&rdquo;</p> <p>The chapter on gaining perspective covers the usual approaches &ndash; bury the work to revisit later &ndash; and some unusual approaches &ndash; string your work across your study, step back till it&rsquo;s just squiggles, and examine its topography.</p> <p>The macro-editing chapter seamlessly blends literary criticism with instruction in the structural elements of writing. Before-and-after passages from <em>The Great Gatsby</em> demonstrate each element, while excerpts from letters between F. Scott Fitzgerald and his editor Max Perkins illustrate the editing process.</p> <p>The micro-editing chapter tries to maintain the style of the macro-editing chapter but fails. It drags, and I was glad to move on.</p> <p>The last two chapters turn from the mechanisms of editing to its variety and historical background.</p> <p>Second to last is a chapter of interviews with authors and artists about their editing process. The story of Walter Murch editing the film <em>The Conversation</em> stands out. The director demanded a refrain repeat exactly the same throughout the film. He threw out one take because the actor&rsquo;s accentuation differed. Murch decided against the director&rsquo;s instructions to cut this take back in over the last seconds of the film. The different word stress recontextualized the refrain and so the film. The other interviews reinforce the variety of approaches to writing and editing, but none stays with you the way Murch&rsquo;s will.</p> <p>The last chapter recaps the role of the editor since ancient Rome. It ends with Robin Robertson editing Adam Thorpe&rsquo;s <em>Ulverton.</em> The history entertained; the story behind <em>Ulverton</em> grabbed me. The men developed unconvential ways of editing this intricate work, including extensive color-coded diagrams tracking leitmotif, themes, and lineages across fictional centuries. Their cooperation parallels Fitzgerald and Perkins&rsquo;, bringing the book back to where it started: the necessary pleasure of editing.</p> Enjoy a simpler ifMUD experience with this tintin++ script https://jeremywsherman.com/2012/01/28/enjoy-a-simpler-ifmud-experience-with-this-tintin-plus-plus-script/ Sat, 28 Jan 2012 00:00:00 +0000 https://jeremywsherman.com/2012/01/28/enjoy-a-simpler-ifmud-experience-with-this-tintin-plus-plus-script/ I have posted a tintin++ script to simplify connecting to and interacting with ifMUD at https://gist.github.com/1692854. To use it, just chmod u+x tt-ifmud and execute the file from a terminal. Script features: Highlights dis/connects, exit directions, and start of item/player/exit list. Highlights AFK and new message announcements. Simplifies going AFK: just afk msg or away msg, and it will @away and zone you. Defines the standard i and inv abbreviations for inventory. <p>I have posted a <a href="http://tintin.sourceforge.net/">tintin++</a> script to simplify connecting to and interacting with <a href="http://ifmud.ziz.org/">ifMUD</a> at <a href="https://gist.github.com/1692854">https://gist.github.com/1692854</a>. To use it, just <code>chmod u+x tt-ifmud</code> and execute the file from a terminal.</p> <p>Script features:</p> <ul> <li>Highlights dis/connects, exit directions, and start of item/player/exit list.</li> <li>Highlights AFK and new message announcements.</li> <li>Simplifies going AFK: just <code>afk msg</code> or <code>away msg</code>, and it will @away and zone you.</li> <li>Defines the standard i and inv abbreviations for inventory.</li> </ul> <p>Why tintin++? Process of elimination!</p> <ul> <li><a href="http://tinyfugue.sourceforge.net/">tinyfugue&rsquo;s website</a> had no useful information and was even more outdated than tintin++'s.</li> <li><a href="http://www.heynow.com/Savitar/">Savitar</a> was shareware, and it&rsquo;s looking long in the tooth these days.</li> <li>The <a href="http://www.riverdark.net/atlantis/">Atlantis</a> interface was awkward, and I had trouble wrangling it&rsquo;s flexible but underdocumented scripting interface to do what I want.</li> <li><a href="http://tintin.sourceforge.net/">tintin++</a>&lsquo;s interface is simple enough that you can figure out how to do pretty much what you want by brute force.</li> </ul> <p>Getting tintin++:</p> <ul> <li>Mac OS X: <code>brew install tintin</code> (Fink and MacPorts users, you&rsquo;re on your own.)</li> <li>Other platforms: Likewise on your own.</li> </ul> <p>I hope this saves some other ifMUD newbie some time. Those of you who haven&rsquo;t checked out ifMUD, <a href="http://ifmud.port4000.com:4002/">check it out!</a></p> Amazon-enhancing your library https://jeremywsherman.com/2011/10/30/amazon-enhancing-your-library/ Sun, 30 Oct 2011 16:00:17 +0000 https://jeremywsherman.com/2011/10/30/amazon-enhancing-your-library/ <p>You have a great Amazon wishlist. But Amazon books cost money, and you can&rsquo;t read them till someone delivers them. Lame. I demand free and instant gratification!</p> <p>Enter the library! But if your library website is anything like mine, the online browsing experience is nowhere near as awesome as Amazon&rsquo;s. And who wants to laboriously search each and every title they&rsquo;ve wished for on Amazon? There&rsquo;s gotta be a better way to figure out which of your Amazon books you can get for free.</p> <p>Enter Ruby! With a little prep work, you can fetch all the titles from Amazon, run them through your library&rsquo;s search interface, and get some useful information:</p> <p>You have a great Amazon wishlist. But Amazon books cost money, and you can&rsquo;t read them till someone delivers them. Lame. I demand free and instant gratification!</p> <p>Enter the library! But if your library website is anything like mine, the online browsing experience is nowhere near as awesome as Amazon&rsquo;s. And who wants to laboriously search each and every title they&rsquo;ve wished for on Amazon? There&rsquo;s gotta be a better way to figure out which of your Amazon books you can get for free.</p> <p>Enter Ruby! With a little prep work, you can fetch all the titles from Amazon, run them through your library&rsquo;s search interface, and get some useful information:</p> <p>Fetching Amazon wishlist&hellip; 39 titles found. Searching library for 39 titles&hellip; Found 9 titles with results: - The Bicycling Guide to Complete Bicycle Maintenance &amp; Repair: For Road &amp; Mountain Bikes (2 results) - The Weed Tree (2 results) - Broken Bells (9 results) - Capacity (24 results) - Civilian (27 results) - Cults (38 results) - High Society (54 results) - A History of Reading (89 results) - Mirror (373 results)</p> <p>I started with a wishlist of 39 items and winnowed it down to just 9 that the library might have. The frustrating part has been done for me: checking book after book only to find the library has absolutely nothing for 30 of them. Sure, I still have to check whether Enon&rsquo;s <em>High Society</em> was actually among the 54 results for that search, but at least I know there are results. More hacking time could undoubtedly improve the false positive rate and add a &ldquo;request this for me&rdquo; feature, but as-is, this quick hack will pay off over and over.</p> <p><a href="https://github.com/jeremy-w/Dekazon">Grab the source</a> to &ldquo;dekazon&rdquo; (Dekalb county library district + Amazon) and use it as a starting point to hack something similar up for your library/college/home. Bringing datasets together is a beautiful thing.</p> Absolute and relative paths https://jeremywsherman.com/2011/09/26/absolute-and-relative-paths/ Mon, 26 Sep 2011 06:00:38 +0000 https://jeremywsherman.com/2011/09/26/absolute-and-relative-paths/ <p><a href="http://www.flickr.com/photos/2is3/72115389/" title="&quot;garden steps&quot; by 2is3"><img src="https://jeremywsherman.com/images/posts/2011/2is3-garden-steps-500.jpg" alt="&ldquo;garden steps&rdquo; by 2is3"></a></p> <p>Summary:</p> <ul> <li>A <strong>path</strong> is a list of components (directory or file names) separated by a slash (/).</li> <li>An <strong>absolute path</strong> starts from the root directory and works its way down: <code>/A/B/file</code>.</li> <li>A <strong>relative path</strong> starts from some contextually-determined parent directory and works its way down from there: <code>A/B/file</code>.</li> <li>In a terminal context, the default parent directory for a relative path is the <strong>present working directory</strong>, which you can print using the <strong><code>pwd</code></strong> command.</li> </ul> <p><a href="http://www.flickr.com/photos/2is3/72115389/" title="&quot;garden steps&quot; by 2is3"><img src="https://jeremywsherman.com/images/posts/2011/2is3-garden-steps-500.jpg" alt="&ldquo;garden steps&rdquo; by 2is3"></a></p> <p>Summary:</p> <ul> <li>A <strong>path</strong> is a list of components (directory or file names) separated by a slash (/).</li> <li>An <strong>absolute path</strong> starts from the root directory and works its way down: <code>/A/B/file</code>.</li> <li>A <strong>relative path</strong> starts from some contextually-determined parent directory and works its way down from there: <code>A/B/file</code>.</li> <li>In a terminal context, the default parent directory for a relative path is the <strong>present working directory</strong>, which you can print using the <strong><code>pwd</code></strong> command.</li> </ul> <p>A <strong>path</strong> is how you refer to a file or directory. A path like <code>/A/B/file</code> is a concise description of how to find a file:</p> <ul> <li>start at the system root directory <code>/</code>,</li> <li>move to the <code>A</code> directory,</li> <li>next move to the <code>B</code> directory,</li> <li>and end with <code>file</code>.</li> </ul> <p>As you can see, a path is just a list of components (<code>A</code>, <code>B</code>, <code>file</code>) separated by slashes (<code>/</code>).</p> <p><code>/A/B/file</code> is an <strong>absolute path</strong>. An absolute path is in truth relative to the filesystem root directory <code>/</code>. No matter the context, no matter what terminal you paste this path into, it will always specify the same path and so the same file.</p> <p><code>A/B/file</code> is a <strong>relative path</strong>. It tells you to move down two directories and end with the file named &ldquo;file&rdquo;. It doesn&rsquo;t tell you how to get to two directories above <code>file</code>: a relative path relies on context to provide its starting point. When you pair it with that starting point, you resolve it into an absolute path.</p> <p>The terminal understands relative paths as relative to its <strong>present working directory,</strong> just as a Finder window provides a context to situate the filenames displayed in it. When you double-click the <code>Downloads</code> folder in a window showing the contents of the <code>/Users/Me</code> folder, the Finder understands that you want to view the contents of the <code>/Users/Me/Downloads</code> folder. Likewise, if your working directory is <code>/Users/Me/</code>, then a command to <code>ls Downloads/</code> will be understood by starting with the absolute path to your present working directory <code>/Users/Me/</code> and appending the relative path <code>Downloads/</code> to end up with <code>/Users/Me/Downloads/</code>. The command is carried out as if you had typed <code>ls /Users/Me/Downloads/</code>, but you just saved yourself some typing by relying on the context provided by the present working directory.</p> <p>Use an absolute path when you want to refer to the same file or directory in the same location regardless of the current working directory. This is useful when you want to reference the same path from multiple contexts. The meaning of an absolute path does not change with the circumstances: it is fixed no matter which user is logged in, no matter what your present working directory is, no matter what computer you are working with.</p> <p>Use a relative path when you want to refer to a file relative to some folder. You&rsquo;ll see this a lot in step-by-step instructions that rely on the Terminal: step 1 will have you change to some directory whose absolute path is unknown to the author (&ldquo;After unzipping the file, change to the unpacked directory…&quot;), and step 2 will have you execute some command relative to that path (&quot;…and make the <code>awesome-script</code> file executable by running <code>chmod u+x awesome-script</code>&rdquo;).</p> Browsing files from the commandline https://jeremywsherman.com/2011/09/09/browsing-files-from-the-commandline/ Fri, 09 Sep 2011 22:01:18 +0000 https://jeremywsherman.com/2011/09/09/browsing-files-from-the-commandline/ <p>Open a folder in Finder. Double-click on it, and you get a window. Look at that window&rsquo;s title bar. In the middle of the title bar, it says maybe, &ldquo;Documents&rdquo;. Where&rsquo;s that? Maybe you switch to column view and arrow left for a while, or maybe you hit Cmd-Up to go up a few folders, or command-click the file icon in the title bar and look at the drop-down. After clicking and tapping and staring for a bit, you finally figure it out: I&rsquo;m in the <em>Documents</em> folder in my user directory on the hard drive named <em>My Mac</em>.</p> <p>That just took way too long.</p> <p>Open a folder in Finder. Double-click on it, and you get a window. Look at that window&rsquo;s title bar. In the middle of the title bar, it says maybe, &ldquo;Documents&rdquo;. Where&rsquo;s that? Maybe you switch to column view and arrow left for a while, or maybe you hit Cmd-Up to go up a few folders, or command-click the file icon in the title bar and look at the drop-down. After clicking and tapping and staring for a bit, you finally figure it out: I&rsquo;m in the <em>Documents</em> folder in my user directory on the hard drive named <em>My Mac</em>.</p> <p>That just took way too long.</p> <p>Go open Terminal.app. You&rsquo;ll see a mostly empty tab with some text and a cursor, something like:</p> <p><tt><strong>Documents</strong>$ <strong>_</strong></tt></p> <p>That text is called a prompt, because it&rsquo;s giving you a bit of information (where you&rsquo;re at) and then asking you, &ldquo;What next?&rdquo; Pretend you are that terminal, that cursor. You are standing in the middle of the filesystem. Where are you? You could do like you did in Finder and poke around for a while and figure out the answer. But you don&rsquo;t need to do that; this is the Terminal.</p> <p>Instead, you can just ask directly, &ldquo;Where am I?&rdquo; Type in <strong>pwd</strong> and hit return to send the command, and you&rsquo;ll see an answer come back with something like:</p> <p><code>/Users/Me/Documents</code></p> <p>Or, in words, the <em>Documents</em> folder in the <em>Me</em> folder in the <em>Users</em> folder on your main hard drive (which is not named, here). Or, in the usual words, so you can read left-to-right, &ldquo;slash Users slash Me slash Documents&rdquo;. You asked, it answered. &ldquo;pwd&rdquo; is actually short for &ldquo;present working directory&rdquo;. &ldquo;Present working directory&rdquo; is a more impersonal name for where you (via the terminal) are standing right now.</p> <p>What&rsquo;s in <em>/Users/Me/Documents</em>? Take a look around with <strong>ls</strong>, short for &ldquo;list&rdquo;. It will print out a list of files and folders (directories) in that directory. You can work with those files, or you can go into any folder you can see in there using cd (&ldquo;change directory&rdquo;).</p> <p>Two folders are always present in each folder, and so ls omits them from its list by default. The folders have short names: &ldquo;.&rdquo; (read &ldquo;dot&rdquo;) means &ldquo;the current directory (whatever it happens to be now)&quot;. &ldquo;..&rdquo; (read &ldquo;dot-dot&rdquo; or &ldquo;double-dot&rdquo;) means &ldquo;the parent of the current directory&rdquo;; if you&rsquo;re in <em>/Users/Me/Documents</em>, then &ldquo;.&rdquo; refers to <em>Documents</em>, and &ldquo;..&rdquo; refers to <em>/Users/Me</em>.</p> <p>Let&rsquo;s move to &ldquo;..&quot;. Type in <strong>cd ..</strong> (&ldquo;change directory to ..&quot;) and you will see the prompt that comes back now reports that you are in the parent of your old directory. Use pwd to confirm this. If you keep going up, eventually you&rsquo;ll hit the top, and <strong>cd ..</strong> will take you right back to that same place (or do nothing, depending on how you look at it). This place is called the root directory, and it&rsquo;s named <em>/</em> (forward slash; slash or solidus to its friends). From the root directory, cd . and cd .. mean the same thing, and neither actually takes you anywhere different from where you started in <em>/</em>.</p> <p>You can move back down the directory tree using <strong>cd name</strong>, where <em>name</em> is the name of a folder. From <em>/</em>, cd Users changes your current directory to <em>/Users</em>. From <em>/Users</em>, cd Me changes your current directory to <em>/Users/Me</em>. Move back up a level with cd .. and you&rsquo;re back at <em>/Users</em>. Got it? Good.</p> <p>Maybe you&rsquo;re wondering about that &ldquo;directory tree&rdquo; phrase that I just slipped in. Why &ldquo;tree&rdquo;? It&rsquo;s a tree because it has a root (&quot;/&quot;) that then branches out. Each branch is a folder or file. Files can&rsquo;t have branches; they are dead-end leaves. Folders can have branches, either more folders or files.</p> <p>You might also hear terminology taken from genealogical trees instead: a folder can be a parent to several children, and it has a parent directory of its own. And if you hear talk of &ldquo;moving up&rdquo;, that means towards the root directory (cd ..); and &ldquo;moving down&rdquo; means away from the root directory, by cding into another directory (one other than <em>.</em> or <em>..</em>).</p> <p>So now you can ask, &ldquo;Where am I?&rdquo; with <strong>pwd</strong>. You can take a look around your present working directory using <strong>ls</strong>, which lists the files and folders in that directory. And you can change directories to go both up and down the directory tree using <strong>cd</strong>. And now browsing your files is one less thing you&rsquo;ll need Finder for.</p> Get more done with Terminal.app https://jeremywsherman.com/2011/08/05/get-more-done-with-terminal-app/ Fri, 05 Aug 2011 03:03:17 +0000 https://jeremywsherman.com/2011/08/05/get-more-done-with-terminal-app/ <p><a href="https://jeremywsherman.com/images/posts/2011/Terminal-Intro.png" title="A terminal with a few tabs. The selected tab shows the text of this post in an Emacs buffer."><img src="https://jeremywsherman.com/images/posts/2011/Terminal-Intro.png" alt="A terminal with a few tabs. The selected tab shows the text of this post in an Emacs buffer."></a></p> <p>You use a Mac. You are comfortable with your Mac. But the terminal remains a foreign land. When forced into Terminal.app, you type what you&rsquo;re told to, hit enter, and cross your fingers that you didn&rsquo;t break anything.</p> <p>You once felt the same way about using the Finder. With exposure and guidance, you will become as comfortable typing commands into the terminal as clicking around the Finder.</p> <p><strong>Why go through the pain of learning the terminal when you&rsquo;re already comfortable with the Finder?</strong></p> <p><a href="https://jeremywsherman.com/images/posts/2011/Terminal-Intro.png" title="A terminal with a few tabs. The selected tab shows the text of this post in an Emacs buffer."><img src="https://jeremywsherman.com/images/posts/2011/Terminal-Intro.png" alt="A terminal with a few tabs. The selected tab shows the text of this post in an Emacs buffer."></a></p> <p>You use a Mac. You are comfortable with your Mac. But the terminal remains a foreign land. When forced into Terminal.app, you type what you&rsquo;re told to, hit enter, and cross your fingers that you didn&rsquo;t break anything.</p> <p>You once felt the same way about using the Finder. With exposure and guidance, you will become as comfortable typing commands into the terminal as clicking around the Finder.</p> <p><strong>Why go through the pain of learning the terminal when you&rsquo;re already comfortable with the Finder?</strong></p> <ul> <li>Do you want to <strong>be a power user</strong>? If you&rsquo;re not using the terminal, you&rsquo;re not making the most of your computer. Want to rename all your documents to start with the date they were created? Not hard from the terminal; slow as can be using the Finder. <ul> <li>Do you want to be able to <strong>choose the best tool</strong>? Without the terminal, what you can do is limited to what there are graphical tools to do. There are great tools that don&rsquo;t – or even can&rsquo;t – have a graphical interface. You want to be able to use those tools.</li> <li>Do you want to be able to <strong>use your computer from anywhere</strong>? You can use the terminal when you can&rsquo;t use the Finder. You can get terminal access to your Mac from your iPhone, from your office, from your parents&rsquo; ancient Win95 machine.</li> <li>Do you want to <strong>be comfortable using any computer</strong>? Once you can use a Mac terminal, you can use a Linux terminal. Even a Windows terminal won&rsquo;t be all that strange. You can bring your commandline knowledge to bear on any platform.</li> </ul> </li> </ul> <p><strong>Power, choice, availability, and portability.</strong> That&rsquo;s why you should learn the terminal.</p> <p>In the coming weeks, we&rsquo;re going to throw out Finder, Aqua, and the Apple Human Interface Guidelines. We&rsquo;ll hear tales of the days of tele-typewriters, time-sharing, and bearded wizards. We&rsquo;ll learn their secrets, we&rsquo;ll learn their language, we&rsquo;ll learn their tools: <strong>We&rsquo;ll learn to be wizards.</strong> (Beard optional.)</p> How to give Lion a new voice https://jeremywsherman.com/2011/08/02/how-to-give-lion-a-new-voice/ Tue, 02 Aug 2011 01:10:36 +0000 https://jeremywsherman.com/2011/08/02/how-to-give-lion-a-new-voice/ <p><a href="https://jeremywsherman.com/images/posts/2011/Captura-de-pantalla-2011-08-02-a-las-00.32.05.png" title="Changing the System Voice"><img src="https://jeremywsherman.com/images/posts/2011/Captura-de-pantalla-2011-08-02-a-las-00.32.05-300x230.png" alt="Lion Speech preference pane"></a></p> <p>Tired of your Mac&rsquo;s voice? Lion introduces some great new voices.</p> <p>My favorite is Virginie. It pronounces English well but gives your Mac an exotic feel. Each time I hit Build, my Xcode purrs, &ldquo;Build started,&rdquo; in a suave French accent.</p> <p><a href="https://jeremywsherman.com/images/posts/2011/Captura-de-pantalla-2011-08-02-a-las-00.32.05.png" title="Changing the System Voice"><img src="https://jeremywsherman.com/images/posts/2011/Captura-de-pantalla-2011-08-02-a-las-00.32.05-300x230.png" alt="Lion Speech preference pane"></a></p> <p>Tired of your Mac&rsquo;s voice? Lion introduces some great new voices.</p> <p>My favorite is Virginie. It pronounces English well but gives your Mac an exotic feel. Each time I hit Build, my Xcode purrs, &ldquo;Build started,&rdquo; in a suave French accent.</p> <p>To change you Mac&rsquo;s voice:</p> <ul> <li>Open System Preferences.</li> <li>Click on the Speech preference pane, the one with the microphone icon. It should be on the right-hand side of the System section.</li> <li>Click on the Text to Speech tab.</li> <li>Click the drop-down next to System Voice and pick &ldquo;Customize….&rdquo;</li> </ul> <p>Now, try out all the voices and pick your favorites. Once Software Update finishes downloading them, head back to the preference pane to mark your favorite as Lion&rsquo;s new voice.</p> <p>Once your new voices are installed, you face a tough decision: Which voice should be the Voice of Lion? To make your final decision, test each voice with a phrase like, &ldquo;Hello, how are you?&rdquo; Pick the voice from the drop-down, then switch to Terminal and use the say command to have your Mac speak the phrase.</p> <p>This is how I ended up with a Lion that speaks French. <em>Miaou.</em></p>