Recently, we learned a painful lesson about using SSL Certificates on a website that hosts a podcast.
Our website serves not only as the home of Grow The Dream, our digital marketing agency, but also as the home of the Grow The Dream Show, a weekly podcast we produce along with 2 colleagues — one a veteran journalist turned PR & messaging guru, and the other an entrepreneur and startup guy.
Not long ago, our commercial SSL certificate was expiring, and I made the decision to deploy a Let’s Encrypt certificate in its place.
The process was astonishingly simple and straightforward: since we use NGINX as our web server, I was able to connect to the server via SSH and run a few simple shell commands (more on that in a moment) and within about 60 seconds we had a shiny new certificate. Oh and thanks to the automation, that 60 seconds included the Domain Control Validation phase of the process.
But the best part? Let’s Encrypt isn’t just automated, it’s also open.
Oh… and the cert is free.
Immediately I tested a whole bunch of stuff to verify that it worked. Typical stuff: open the site with fresh browsers on various operating systems, including Android, iOS, various versions of Windows, Ubuntu, etc.
And yes… to be certain that our podcast was working properly, I opened the iTunes desktop client for Windows 64-bit operating systems (painful though it is to even launch that software) and it found our podcast’s RSS feed, updated, and downloaded the latest episode—the one published within a few minutes of the time that the cert was updated.
That is until the next day.
That’s when I discovered that our podcast’s official listing at iTunes.com was not updating. Even worse, when I used Apple’s new iTunes Connect interface for Podcast Providers (a/k/a “Podcast Connect”), I found an enormous error message staring me in the face:
“Can’t read your feed. sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
Even worse, the “Refresh Feed” link was gone, as you can see here:
Interestingly, although the main function that reads feeds for iTunes was apparently unable to process our feed, the “Mirror URL” was perfectly up to date.
Every Apple device (I have friends with Macs and iPhones) seemed to work. We checked with the other major podcast aggregators where our show is often featured (Stitcher, Player.FM, Pocket Casts, and so on), and they all showed our newest episode and were therefore updating just fine.
The Let’s Encrypt Community Responds
As you might expect with any good open project, there’s a community. And in this case, someone within the Let’s Encrypt community had already started a discussion thread about this very problem.
While there was a fair amount of speculation, ultimately my research yielded a simple set of facts. Apple’s main engine for iTunes runs on a version of Java that did not include the Let’s Encrypt Root Certificate from the Let’s Encrypt Certificate Authority (“CA”), therefore Let’s Encrypt is not in its list of trusted CAs. Thus, the software written on that version also does not trust the Let’s Encrypt CA.
Since iOS, Mac OSX, and other similar systems have their own list of trusted CAs, often the iTunes software or Podcasts App had no problem reading a feed encrypted with a Let’s Encrypt certificate (which is signed by the Let’s Encrypt Certificate Authority).
The builders of Let’s Encrypt anticipated scenarios where their Root Certificate would not be included, so they had it “cross-signed” by IdenTrust, which would create trust for their Certificates… at least in theory.
In the case of iTunes, for whatever reason, this isn’t sufficient.
The iTunes store, long considered the “official” directory of podcasts (probably because “podcast” includes part of the word “iPod” in the very name of the medium) ultimately simply doesn’t trust Let’s Encrypt certificates.
This means if you submit a new podcast to iTunes for inclusion in the directory, and your podcast’s official RSS feed is encrypted with a Let’s Encrypt Certificate, iTunes will reject your podcast.
On the other hand, if you have a podcast already listed in the official iTunes directory, and you change to a Let’s Encrypt Certificate from one that had previously been trusted by Apple’s platform, your feed will stop being updated.
iTunes Support Responds
As you might imagine, I didn’t just accept that the status quo had to remain. I contacted Apple via the iTunes Connect interface mentioned earlier, and thus began a support dialogue that went on for weeks.
When I began the dialogue, I did not understand what I do now about the fact that their system did not (and does not, as of this writing) support Let’s Encrypt Certificates. I was simply contacting them about the error message.
Let’s just say that their efforts at diagnosing the problem left a little to be desired. But I can hardly fault them for this, as I imagine that the ins and outs of encryption, SSL, Certificates, and whatnot are not for the faint of heart, and are hardly likely to be everyday fodder for their support mill.
In the end, an Apple support representative provided a list of “known working” Certificate Authorities and urged me to use one from their list.
I Give Up
What is the importance of the “official” podcast directory at iTunes?
It’s hard to say. Ultimately, I can say that our download numbers suffered during the time period that our RSS feed was encrypted with a Let’s Encrypt Certificate. By how much? That’s also hard to say. Podcast statistics are difficult enough, with a huge number of variables to factor in, to isolate the importance of any one factor.
However… a sneakier unknown loomed behind the knowledge that Apple’s system did not trust my certificate, and that is this: what other systems don’t trust my certificate?
Sadly… I relented.
I contacted the our commercial SSL Certificate [note: that’s an affiliate link*] provider of choice, and bought a new, low-cost certificate. For less than $30, I had a new commercial certificate, which I installed with a far greater level of difficulty than the super-simple Let’s Encrypt process.
As predicted by iTunes Support, within a few hours (they said up to 24), the error message at iTunes was resolved and our listing updated. By this time, several weeks had elapsed and thus several new episodes of our show suddenly appeared. Unfortunately, what little statistical data that iTunes provides (e.g. the relative “popularity” of a given episode) is permanently lost to the world for those new episodes, as well as for other episodes where interactions occurred on Apple’s devices and software platforms during that time period.
Not to be satisfied, I replied back to the Support thread with as much detail about the problem that I felt would be digestible, and asked why Let’s Encrypt Certificates weren’t being trusted. Apple responded with an acknowledgment of the fact that my new Certificate had resolved the error message, and that they have since updated their Podcast Connect FAQs with their list of trusted Certificate providers, saying:
iTunes requires an SSL certificate from one of the following providers:
…and of course, our preferred provider* isn’t on the list (although they are a reseller for some of the providers Apple identified).
We host the site on a Digital Ocean* droplet, which is a decision we made because it allows us to have the autonomy of a VPS along with the scalability of a hosting platform where we can readily access additional resources as needed.
That autonomy allows us full control over facets of our hosting such as:
- exactly how we deploy SSL (or TLS, to be more precise) — the answer is: sitewide, with HSTS to ensure that no resources are accessed without encryption
- exactly what web server software (we use NGINX) we use, and exactly how we configure it
- fine-tuning the entire configuration for speed, security, and so forth (I’m sure there’s lots of room for improvement here, but we enjoy the process).
Why Use SSL at All?
Some might argue that hosting a podcast and a relatively straightforward website for a digital marketing agency really doesn’t merit running encrypting the data between our web server and our site’s visitors.
And while I see that point, I disagree.
Certainly, Google has famously created an incentive for website owners to run full SSL, and while we’re grateful for any benefit we might enjoy where our site’s search engine rankings are concerned, that really isn’t the primary reason to encrypt.
We encrypt transfers between our site’s visitors and our web server for one major reason: to protect our site visitors.
Perhaps the single biggest protection this affords them is that of preventing man in the middle attacks.
While we don’t have any delusions that our content is life-alteringly important to the point that someone would want to tamper with it, per se, the truth is that any website whose content, downloads, and whatnot is delivered over the unencrypted http protocol is subject to having it monkeyed with by ISPs, cellular carriers, and others… not just a malicious hacker sniffing wi-fi traffic at the local coffee shop.
For more on the importance of encrypting the web, check out this report from the Electronic Frontier Foundation.
For now, Let’s Encrypt may not be best suited for everyone — especially those who publish content on the web for commercial purposes. If you use Let’s Encrypt, you accept some risk that a percentage of your audience (however small) may not be able to access, or even discover, your content.
For noncommercial purposes where that risk may not come with an economic cost, Let’s Encrypt is already a fantastic solution.
We look forward to the day when those few holdouts that don’t seem to trust it come around.
I’m looking at you, Apple.
*Disclosure: We have an affiliate relationship with some of the providers linked to in this post. If you click one of those links and subsequently make a purchase, we may make a small amount of money. Our policy is that we do not recommend any providers we do not also use and believe in, so the affiliate relationship does not change who we are linking to.