I think ... - url-shortenerhttps://blog.kmonsoor.com/2021-06-06T00:00:00+06:00Create a free go-link server “on edge” using Cloudflare Worker KV2021-06-06T00:00:00+06:002021-06-06T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-06-06:/golink-server-using-cloudflare-worker-kv/<p>Among quite a few ways to implement a go-link server (i.e. url-forwarder, short-url server etc.), here I&rsquo;m going to show you how to use free-tier Cloudflare Worker (&amp; <span class="caps">KV</span>) to create an in-house, on-edge, <strong>no-webserver</strong> go-link&nbsp;server.</p><p>Among quite a few ways to implement a go-link server (i.e. url-forwarder, short-url server etc.), here I&rsquo;m going to show you how to use free-tier Cloudflare Worker (&amp; <span class="caps">KV</span>) to create an in-house, on-edge, <strong>no-webserver</strong> go-link&nbsp;server.</p> <p>For example, the short-link for this article is <a href="https://go.kmonsoor.com/golink-kv">go.kmonsoor.com/golink-kv</a> </p> <p><img alt="overall structure" src="https://i.imgur.com/MjIS5gD.png"></p> <ul> <li><code>/latest</code> (by which I mean <code>go.yourdomain.co/latest</code>) may point to <code>https://www.yourcompany.com/about/news</code> which is a public&nbsp;page</li> <li><code>/hr-help</code> may point to <code>https://www.company-internal.com/long-link/hr/contact.html</code> which is company&rsquo;s internal human-resources help&nbsp;portal</li> <li><code>/cnypromo</code> may point to <code>https://shop.yourcompany.com/sales/promotions/?marketing-promo=2021-cny</code> which is a temporary sales promotions page targeting the shoppers during the Chinese new year of&nbsp;2021.</li> </ul> <p>Please note that using the setup and the code below, it&rsquo;ll be possible resolve shortlinks via a <strong>single</strong> sub-domain, e.g. <code>go.your-domain.co</code>. However, it&rsquo;s totally possible (with some modification of the code) to resolve/re-direct via <em>any number of domains</em> (your own, of course) towards any other public or private <span class="caps">URL</span>, and all sorts of novelties. However, for brevity&rsquo;s sake, I&rsquo;m going to discuss the first one, single sub-domain&nbsp;usecase.</p> <p>To setup a go-link server or short-<span class="caps">URL</span> resolver via a proper <span class="caps">KV</span>+Worker combination, we&rsquo;ll go through these&nbsp;steps:</p> <div class="toc"> <ul> <li><a href="#pre-requisites">Pre-requisites</a></li> <li><a href="#create-the-short-link-map-as-a-kv">Create the short-link map as a <span class="caps">KV</span></a></li> <li><a href="#mapping-a-kv-to-a-worker-variable">Mapping a <span class="caps">KV</span> to a Worker&nbsp;variable</a></li> <li><a href="#handling-a-route-with-webworker">Handling a route with&nbsp;webworker</a></li> <li><a href="#create-the-worker">Create the&nbsp;Worker</a></li> <li><a href="#pointing-a-dns-record-to-the-worker">Pointing a <span class="caps">DNS</span> record to the&nbsp;Worker</a></li> <li><a href="#next-step">Next&nbsp;step</a></li> <li><a href="#related">Related</a></li> </ul> </div> <h1 id="pre-requisites">Pre-requisites<a class="headerlink" href="#pre-requisites" title="Permanent link">&para;</a></h1> <ul> <li>The <span class="caps">DNS</span> resolver for the <strong>root</strong> domain (in the example below, <em><code>kmonsoor.com</code></em>) needs to be Cloudflare. Because the core of the solution, the &ldquo;worker&rdquo;, runs on the nearest (from the user) edge of Cloudflare using a common <span class="caps">KV</span> (&ldquo;key, value&rdquo;)&nbsp;list.</li> <li>Write permission to the <span class="caps">DNS</span> configuration as you&rsquo;d need to add a new <span class="caps">AAAA</span> <span class="caps">DNS</span>&nbsp;record.</li> <li>Some knowledge of Javascript(<span class="caps">ES6</span>), as we gonna write the worker in that&nbsp;language.</li> </ul> <h1 id="create-the-short-link-map-as-a-kv">Create the short-link map as a <span class="caps">KV</span><a class="headerlink" href="#create-the-short-link-map-as-a-kv" title="Permanent link">&para;</a></h1> <p>We&rsquo;ll start the setup by creating the short-link map, the list between the short-link segments that you (or, someone in your org) define and the actual URLs they needed to point&nbsp;to.</p> <p>Find the <span class="caps">KV</span> stuff in the <code>Workers</code> section. From the screenshot, please ignore the &ldquo;Route&rdquo; section for&nbsp;now. </p> <p><img alt="Find the KV stuff in the Workers section" src="https://i.imgur.com/b2Rk45u.png"></p> <ul> <li>you&rsquo;d need to create a Worker <span class="caps">KV</span> &ldquo;Namespace&rdquo;. Name the namespace as you seem fit. I named it <code>REDIRECTS</code> (in all caps just as convention, not&nbsp;required). </li> <li>List the short-links <span class="amp">&amp;</span> their respective target URLs. From the examples in the intro, the keys <code>latest</code>, <code>hr-help</code>, <code>cnypromo</code> etc. would be in as the &ldquo;key&rdquo;, and the target full links as the respective&nbsp;&ldquo;value&rdquo;.</li> <li>Remember <span class="caps">NOT</span> to start the short part with &lsquo;/&rsquo;. In the code, it&rsquo;ll be taken care&nbsp;of.</li> </ul> <p><img alt="Create the short-link map as a KV" src="https://i.imgur.com/jkC8bSr.png"></p> <p>Once you&rsquo;ve listed all your desired (short-link, target-link) combinations, now we have a <span class="caps">KV</span> on CLoudflare. However, it&rsquo;s not referencable from your Worker code, not yet. Hence the next&nbsp;step.</p> <h1 id="mapping-a-kv-to-a-worker-variable">Mapping a <span class="caps">KV</span> to a Worker variable<a class="headerlink" href="#mapping-a-kv-to-a-worker-variable" title="Permanent link">&para;</a></h1> <p>Now, we gonna map the previously created <span class="caps">KV</span> to a variable that can be referenced from our Workeer code. Please note that though I used diffrent names, but it can be same as well. Also note that a single <span class="caps">KV</span> can be accessed by multiple workers, and vice versa, a single Worker can reference multiple&nbsp;KVs.</p> <p><img alt="Mapping a KV to a Worker variable" src="https://i.imgur.com/lb7G9si.png"></p> <h1 id="handling-a-route-with-webworker">Handling a route with webworker<a class="headerlink" href="#handling-a-route-with-webworker" title="Permanent link">&para;</a></h1> <p><img alt="Handling a route with webworker" src="https://i.imgur.com/KohHRfR.png"></p> <h1 id="create-the-worker">Create the Worker<a class="headerlink" href="#create-the-worker" title="Permanent link">&para;</a></h1> <p>Now, we going to write Workercode that runs on V8-runtime on an nearest(from the requesting user) edge location of Cloudflare to complete its functionality and deliver it to the user. Tn this case, that would be to redirect user&rsquo;s requested address to the mapped one by you, in the <span class="caps">KV</span> namespace&nbsp;above.</p> <p><img alt="Creating a worker" src="https://i.imgur.com/eNfZNyN.png"></p> <p>The code editor looks like&nbsp;this: </p> <p><img alt="The code editor for Cloudflare worker" src="https://i.imgur.com/pb9AE9v.png"></p> <p>If you rather prefer to copy-paste, please feel free to do it from the below GitHub&nbsp;Gist.</p> <div class="gist"> <script src="https://gist.github.com/kmonsoor/dc9f96660423c96471f8574ba018d867.js"></script> </div> <p>Once done, it should look like &hellip; <img alt="created webworker" src="https://i.imgur.com/XSdKB56.png"></p> <h1 id="pointing-a-dns-record-to-the-worker">Pointing a <span class="caps">DNS</span> record to the Worker<a class="headerlink" href="#pointing-a-dns-record-to-the-worker" title="Permanent link">&para;</a></h1> <p>Finally, we need to point a <span class="caps">DNS</span> record that&rsquo;ll redirect all requests to your re-soutign sub-domain (e.g. <code>go.your-domain.com</code>) to the Cloudflare Worker that we just&nbsp;created.</p> <p>According to the Cloudflare docs, the <span class="caps">DNNS</span> record must be an <span class="caps">AAAA</span> record, pointing to the IPv6 address <code>100:</code>. The &ldquo;Name&rdquo; here is the &ldquo;sub-domain&rdquo; part of your choice which is better be short to serve our goal&nbsp;here. </p> <p><img alt="Pointing a DNS record to it" src="https://i.imgur.com/62bk7pe.png"></p> <p>Voila ! Now, test some of the short-urls that you&rsquo;ve mapped via the <span class="caps">KV</span>. Enjoy ! Watch out for the target usage though <a href="https://developers.cloudflare.com/workers/platform/limits#worker-limits">against the limit</a>. </p> <p>I think you&rsquo;ll be fine, unnless you&rsquo;re some kind of celebrity&nbsp;;)</p> <h1 id="next-step">Next step<a class="headerlink" href="#next-step" title="Permanent link">&para;</a></h1> <p>As the next next step, I&rsquo;m thinking to create a generic <code>Go/Link</code> resolver browser extension. Then, someone can set their own default domain or company domain of choice as short-domain host. In that case, entering just <code>go/hr-help</code> on the browser will take to <code>https://www.company-internal.com/.../hr/contact.html</code> that we have discussed at the beginning (remember the example case of an internal human resources help&nbsp;portal?).</p> <h1 id="related">Related<a class="headerlink" href="#related" title="Permanent link">&para;</a></h1> <ul> <li>If you want to do this url-direction <strong>on your own webserver, but only using webserver</strong>, try this: <a href="https://blog.kmonsoor.com/personal-shortlink-server-using-Caddy/">Personal short-link server using only&nbsp;Caddyserver</a></li> </ul>Deploying a short-link aka go-link server using only Caddyserver2021-04-16T00:00:00+06:002021-04-16T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-04-16:/deploying-golink-server-using-Caddy/<p>Yeah, there are tons of open-source, full-fledged link-shorteners. But, none were exactly what I wanted. Hence, the minimal approach only utilizing the amazing webserver, <code>Caddy</code>. Here we go&nbsp;&hellip;</p><p>Before I go into any details, please note that I&rsquo;ve used the term &ldquo;shortlink-server&rdquo; instead of &ldquo;url-shortener&rdquo; because the difference is significant for this post. A url-shortener takes a long url, and gives a short url, then redirects any requests for the shortened link to its long counterpart. On the contrary, shortlink-server takes both the long and short url, then only does the redirect-ion&nbsp;part.</p> <h2 id="backstory">Backstory<a class="headerlink" href="#backstory" title="Permanent link">&para;</a></h2> <p>If I wanted an amazing, personal url-shortener or &ldquo;go link&rdquo; server, there are many amazing solutions out there, which are not only free or open-source but full-fledged as well for personal or public usage. Rather, I wanted some &ldquo;service&rdquo; that will &ldquo;resolve&rdquo; my personal, short links. I have been using bit.ly for a long time for its customizable &ldquo;short-half&rdquo; part, but the problem with bit.ly is &ndash; for some God-forsaken reason &ndash; blocked by the Bangladeshi govt. So, I needed a replacement to be properly &ldquo;glocal&rdquo;. There are many free (unbranded) and commercial (branded) options as well, but I wanted something that will be resolved via my hosted service (and personal domain) and as cheap as possible. So, basically solution for a poor nerd&nbsp;:D</p> <p>Before jumped into this solution, I tried (deployed <span class="amp">&amp;</span> tested) few others myself, mainly <a href="https://github.com/kellegous/go">kellegous/go</a>, <a href="kutt.it">kutt.it</a> and <a href="https://github.com/adamyi/golinks">adamyi/golinks</a>. But, all of them &ldquo;too featureful&rdquo; for my&nbsp;needs. </p> <p><img alt="Simple, on-prem short-link server using Caddy webserver" src="https://i.imgur.com/4nZbnUE.png"></p> <p>What I wanted is to be able&nbsp;to:</p> <ul> <li>resolve only my custom shortlinks (hence, no need for&nbsp;url-shortener)</li> <li>not a public, internet-facing service (hence, any frontend, authentication, email verification etc. would be overkill&nbsp;)</li> <li>minimal setup (if possible, no webapp at&nbsp;all)</li> </ul> <p>Given my previous experience with <code>Caddy</code> webserver, which is an amazing one(<a href="https://caddyserver.com/docs/">why?</a>), I had a gut feeling that Caddy has something for me &ndash; under the sleeve &ndash; to meet my minimal set of requirements. Thankfully, I managed to find&nbsp;it.</p> <p>I believe <span class="caps">NGINX</span>, currently the most popular webserver, has some kind of similar mechanism as well. But, I&rsquo;m not an expert, and once I was genuinely intimidated by its config file syntax. <span class="caps">YMMV</span>.</p> <h2 id="what-you-gonna-need">What you gonna need?<a class="headerlink" href="#what-you-gonna-need" title="Permanent link">&para;</a></h2> <ul> <li>your own domain which will be the root of the shortlinks. While sub-domained <span class="caps">URL</span> like <code>go.yourname.com/*</code> is quite common, if you have some short domain, like you.co/*, only for this purpose, that&rsquo;s fine as&nbsp;well.</li> <li>A webhost server or public-facing instance with its own, <strong>public</strong> IPv4&nbsp;address.</li> <li>working knowledge of&nbsp;Linux</li> </ul> <h2 id="step-1-point-your-subdomain-to-the-right-place">Step-1: Point your subdomain to the right place<a class="headerlink" href="#step-1-point-your-subdomain-to-the-right-place" title="Permanent link">&para;</a></h2> <ul> <li>Find out what&rsquo;s the <strong>puplic IPv4 address</strong> of your instance that&rsquo;ll act as the webserver. It&rsquo;s usually on the cloud management dashboard.<ul> <li>make sure that, regardless of your cloud architecture (e.g. <span class="caps">VPC</span>, subnet, firewall etc.), the <span class="caps">SSL</span> port (<code>:443</code>) of the instance is reachable from the public&nbsp;internet.</li> </ul> </li> <li>now go to your domain name registrar (or, <span class="caps">DNS</span> management provider which in my case is Cloudflare). There, you need to point shortlink subdomain (<code>go.</code>)to the webserver&rsquo;s <span class="caps">IP</span>&nbsp;address.</li> </ul> <p>Actually, you can do this step at the last. But for some reason, I prefer it to do first. Because sometimes, <a href="https://blog.cloudflare.com/never-deal-with-dns-propagation-again/"><span class="caps">DNS</span> propagation</a> takes some time. But, once my webservice is up and running, I like to see the result instantaneously.&nbsp;;)</p> <h2 id="step-2-install-caddy-the-mighty-webserver">Step-2: Install Caddy, the mighty webserver<a class="headerlink" href="#step-2-install-caddy-the-mighty-webserver" title="Permanent link">&para;</a></h2> <p>Depending on your host <span class="caps">OS</span> (mine is Ubuntu 20.04 <span class="caps">LTS</span>), you need to install the <code>Caddy</code> webserver. While there are some hacky solutions to run, I think running Caddy as a background server is the simplest to manage. In fact, the documentation of Caddy is excellent, I&rsquo;d better leave that part to&nbsp;you. </p> <p>After running with the default config(<code>Caddyfile</code>) which is in Ubuntu&rsquo;s case located as <code>/etc/caddy/Caddyfile</code>, it should have a status somewhat like the below image. Please note that, in many cases, if running without <code>sudo</code> Caddy cannot attach itself with the <span class="caps">SSL</span> port (<code>:443</code>) which is necessary for serving <code>https://</code>. So, check for that error message in the &ldquo;status&rdquo;&nbsp;log.</p> <p><img alt="Caddy service on Ubuntu" src="https://i.imgur.com/cfS5nvZ.png?1"></p> <p><em><strong><span class="caps">P.S.</span></strong> By the way, want your console and command prompt to look 🚀 like mine? Here&rsquo;s the guide: <a href="https://blog.kmonsoor.com/pimp-up-my-terminal/">How do I pimp up my terminal on&nbsp;Linux</a></em></p> <h2 id="step-3-tell-caddy-your-short-links-to-redirect">Step-3: Tell Caddy your short-links to redirect<a class="headerlink" href="#step-3-tell-caddy-your-short-links-to-redirect" title="Permanent link">&para;</a></h2> <p>Now, it&rsquo;s time to configure Caddy to actually do the&nbsp;job.</p> <p>Caddy has its native <code>redir</code> <span class="dquo">&ldquo;</span>directive&rdquo; to redirect incoming web-request from one to another. While the <code>map</code> directive is relatively new, it makes the config file i.e. Caddyfile look elegant in case you have (or, will have in the long run) a long list of&nbsp;short-links.</p> <p>Here&rsquo;s mine which is working&nbsp;&hellip; </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span># /etc/caddy/Caddyfile <span class="linenos" data-linenos=" 2 "></span> <span class="linenos" data-linenos=" 3 "></span>go.kmonsoor.com { # replace it your web-url root <span class="linenos" data-linenos=" 4 "></span> <span class="linenos" data-linenos=" 5 "></span> map {path} {redirect-uri} { <span class="linenos" data-linenos=" 6 "></span> /blog https://blog.kmonsoor.com <span class="linenos" data-linenos=" 7 "></span> /photos https://photos.kmonsoor.com <span class="linenos" data-linenos=" 8 "></span> <span class="linenos" data-linenos=" 9 "></span> /resume https://drive.google.com/file/d/1nMS3i1ai6nsI70zZ7NFnNQ_XmvAa4GOl <span class="linenos" data-linenos="10 "></span> /resume-doc https://docs.google.com/document/d/1ECx1Yr8Jzz9I3S5VcoKnZQz56oIht2XaM5gSNetcWag <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span> /rickrolled https://www.youtube.com/watch?v=dQw4w9WgXcQ <span class="linenos" data-linenos="13 "></span> <span class="linenos" data-linenos="14 "></span> # will add new ones here like the above <span class="linenos" data-linenos="15 "></span> # ... <span class="linenos" data-linenos="16 "></span> } <span class="linenos" data-linenos="17 "></span> <span class="linenos" data-linenos="18 "></span> # this below code is required to actually make the above `map` work <span class="linenos" data-linenos="19 "></span> <span class="linenos" data-linenos="20 "></span> @hasRedir expression `{redirect-uri} != &quot;&quot;` <span class="linenos" data-linenos="21 "></span> redir @hasRedir {redirect-uri} <span class="linenos" data-linenos="22 "></span> <span class="linenos" data-linenos="23 "></span> # code below is to set the default response if the requested shortlink isn&#39;t here <span class="linenos" data-linenos="24 "></span> respond &quot;Thas&#39;s an unknown short URL ... :(&quot; <span class="linenos" data-linenos="25 "></span>} </code></pre></div> <p>Note: Don&rsquo;t forget to restart the <code>caddy</code> service to let the new config to take&nbsp;effect.</p> <h2 id="step-4-profit">Step-4: Profit !<a class="headerlink" href="#step-4-profit" title="Permanent link">&para;</a></h2> <p>Yeah, that&rsquo;s it. Now, add some own personal stuff with some cool short-links, and proudly share with the&nbsp;world.</p> <h2 id="whats-next">What&rsquo;s next ?<a class="headerlink" href="#whats-next" title="Permanent link">&para;</a></h2> <p>I&rsquo;m thinking that given the very low workload my shortlink resolver needs — unless I&rsquo;m becoming an overnight internet sensation — using a server instance only for this purpose is overkill. My next goal is to have the same service using some &ldquo;serverless&rdquo; function or using the &ldquo;<a href="https://developers.cloudflare.com/workers/examples/redirect">worker on the edge</a>&rdquo; thing from Cloudflare. Let&rsquo;s see&nbsp;;)</p>