I think ... - Techhttps://blog.kmonsoor.com/2021-06-06T00:00:00+06:00Create a free go-link server “on edge” using Cloudflare Worker KV2021-06-06T00:00:00+06:002021-06-06T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-06-06:/golink-server-using-cloudflare-worker-kv/<p>Among quite a few ways to implement a go-link server (i.e., url-forwarder, short-url server, etc.), I will show how to use free-tier Cloudflare Worker (&amp; <span class="caps">KV</span>) to create an in-house, on-edge, <strong>no-webserver</strong> go-link&nbsp;server.</p><p>Among quite a few ways to implement a go-link server (i.e., url-forwarder, short-url server, etc.), I&rsquo;m going to show you how to use free-tier Cloudflare Worker (&amp; <span class="caps">KV</span>) to create an in-house, on-edge, <strong>no-webserver</strong> go-link&nbsp;server.</p> <p>For example, the short-link for this article is <a href="https://go.kmonsoor.com/golink-kv">go.kmonsoor.com/golink-kv</a> </p> <p><img alt="overall structure" src="https://i.imgur.com/MjIS5gD.png"></p> <ul> <li><code>/latest</code> (by which I mean <code>go.yourdomain.co/latest</code>) may point to <code>https://www.yourcompany.com/about/news</code> which is a public&nbsp;page</li> <li><code>/hr-help</code> may point to <code>https://www.company-internal.com/long-link/hr/contact.html</code>, which is company&rsquo;s internal human-resources help&nbsp;portal</li> <li><code>/cnypromo</code> may point to <code>https://shop.yourcompany.com/sales/promotions/?marketing-promo=2021-cny</code> which is a temporary sales promotions page targeting the shoppers during the Chinese new year of&nbsp;2021.</li> </ul> <p>Please note that using the setup and the code below, it&rsquo;ll be possible to resolve short-links via a <strong>single</strong> sub-domain, e.g., <code>go.your-domain.co</code>. However, it&rsquo;s possible (with some modification of the code) to resolve/redirect via <em>any number of domains</em> (your own, of course) towards any other public or private <span class="caps">URL</span>, and all sorts of novelties. However, for brevity&rsquo;s sake, I will discuss the first one, a single sub-domain&nbsp;usecase.</p> <p>To set up a go-link server or short-<span class="caps">URL</span> resolver via a proper <span class="caps">KV</span>+Worker combination, we&rsquo;ll go through these&nbsp;steps:</p> <div class="toc"> <ul> <li><a href="#pre-requisites">Pre-requisites</a></li> <li><a href="#create-the-short-link-map-as-a-kv">Create the short-link map as a <span class="caps">KV</span></a></li> <li><a href="#mapping-a-kv-to-a-worker-variable">Mapping a <span class="caps">KV</span> to a Worker&nbsp;variable</a></li> <li><a href="#handling-a-route-with-webworker">Handling a route with&nbsp;webworker</a></li> <li><a href="#create-the-worker">Create the&nbsp;Worker</a></li> <li><a href="#pointing-a-dns-record-to-the-worker">Pointing a <span class="caps">DNS</span> record to the&nbsp;Worker</a></li> <li><a href="#next-step">Next&nbsp;step</a></li> <li><a href="#related">Related</a></li> </ul> </div> <h1 id="pre-requisites">Pre-requisites<a class="headerlink" href="#pre-requisites" title="Permanent link">&para;</a></h1> <ul> <li>The <span class="caps">DNS</span> resolver for the <strong>root</strong> domain (in the example below, <em><code>kmonsoor.com</code></em>) needs to be Cloudflare. Because the core of the solution, the &ldquo;worker&rdquo;, runs on the nearest (from the user) edge of Cloudflare using a standard <span class="caps">KV</span> (&ldquo;key, value&rdquo;)&nbsp;list.</li> <li>Write permission to the <span class="caps">DNS</span> configuration as you&rsquo;d need to add a new <span class="caps">AAAA</span> <span class="caps">DNS</span>&nbsp;record.</li> <li>Some knowledge of Javascript(<code>ES6</code>), as we are going to write the &ldquo;worker&rdquo; in that&nbsp;language.</li> </ul> <h1 id="create-the-short-link-map-as-a-kv">Create the short-link map as a <span class="caps">KV</span><a class="headerlink" href="#create-the-short-link-map-as-a-kv" title="Permanent link">&para;</a></h1> <p>We&rsquo;ll start the setup by creating the short-link map, the list between the short-link segments that you (or someone in your org) define, and the actual URLs they need to point&nbsp;to.</p> <p>Find the <span class="caps">KV</span> stuff in the <code>Workers</code> section. From the screenshot, please ignore the &ldquo;Route&rdquo; section for&nbsp;now. </p> <p><img alt="Find the KV stuff in the Workers section" src="https://i.imgur.com/b2Rk45u.png"></p> <ul> <li>you&rsquo;d need to create a Worker <span class="caps">KV</span> &ldquo;Namespace&rdquo;. Name the namespace as you seem fit. I named it <code>REDIRECTS</code> (in all caps just as a convention, not&nbsp;required). </li> <li>List the short links <span class="amp">&amp;</span> their respective target URLs. From the examples in the intro, the keys <code>latest</code>, <code>hr-help</code>, <code>cnypromo</code> etc. would be in as the &ldquo;key&rdquo;, and the target full links as the respective&nbsp;&ldquo;value&rdquo;.</li> <li>Remember <span class="caps">NOT</span> to start the short part with &lsquo;/&rsquo;. It&rsquo;ll be taken care of in the&nbsp;code.</li> </ul> <p><img alt="Create the short-link map as a KV" src="https://i.imgur.com/jkC8bSr.png"></p> <p>Once you&rsquo;ve listed all your desired (short-link, target-link) combinations, now we have a <span class="caps">KV</span> on Cloudflare. However, it&rsquo;s not referencable from your Worker code, not yet. Hence the next&nbsp;step.</p> <h1 id="mapping-a-kv-to-a-worker-variable">Mapping a <span class="caps">KV</span> to a Worker variable<a class="headerlink" href="#mapping-a-kv-to-a-worker-variable" title="Permanent link">&para;</a></h1> <p>Now, we will map the previously created <span class="caps">KV</span> to a variable that can be referenced from our Worker code. Please note that though I used different names, it can be the same as well. Also, note that multiple Workers can access a single <span class="caps">KV</span>, and vice versa is also true; a single Worker can reference multiple&nbsp;KVs.</p> <p><img alt="Mapping a KV to a Worker variable" src="https://i.imgur.com/lb7G9si.png"></p> <h1 id="handling-a-route-with-webworker">Handling a route with webworker<a class="headerlink" href="#handling-a-route-with-webworker" title="Permanent link">&para;</a></h1> <p><img alt="Handling a route with webworker" src="https://i.imgur.com/KohHRfR.png"></p> <h1 id="create-the-worker">Create the Worker<a class="headerlink" href="#create-the-worker" title="Permanent link">&para;</a></h1> <p>Now, we will write Worker-code that runs on <code>V8</code> runtime on the nearest (from the requesting user) &ldquo;edge&rdquo; location of Cloudflare, to execute the code and deliver the result(s) to the user. In this case, that would be to redirect user-requested address to the mapped one (by you, in the <span class="caps">KV</span> namespace&nbsp;above).</p> <p><img alt="Creating a worker" src="https://i.imgur.com/eNfZNyN.png"></p> <p>The code editor looks like&nbsp;this: </p> <p><img alt="The code editor for Cloudflare worker" src="https://i.imgur.com/pb9AE9v.png"></p> <p>If you rather prefer to copy-paste, please feel free to do it from the below GitHub&nbsp;Gist.</p> <div class="gist"> <script src="https://gist.github.com/kmonsoor/dc9f96660423c96471f8574ba018d867.js"></script> </div> <p>Once done, it should look like &hellip; <img alt="created webworker" src="https://i.imgur.com/XSdKB56.png"></p> <h1 id="pointing-a-dns-record-to-the-worker">Pointing a <span class="caps">DNS</span> record to the Worker<a class="headerlink" href="#pointing-a-dns-record-to-the-worker" title="Permanent link">&para;</a></h1> <p>Finally, we need to point a <span class="caps">DNS</span> record that&rsquo;ll redirect all requests to your re-soutign sub-domain (e.g. <code>go.your-domain.com</code>) to the Cloudflare Worker that we just&nbsp;created.</p> <p>According to the Cloudflare docs, the <span class="caps">DNNS</span> record must be an <span class="caps">AAAA</span> record, pointing to the IPv6 address <code>100::</code>. The &ldquo;Name&rdquo; here is the &ldquo;sub-domain&rdquo; part of your choice, which is better be short, to rightfully serve our goal&nbsp;here. </p> <p><img alt="Pointing a DNS record to it" src="https://i.imgur.com/62bk7pe.png"></p> <p>Voila ! Now, test some of the short-urls that you&rsquo;ve mapped via the <span class="caps">KV</span>. Enjoy ! Watch out for the target usage though. <a href="https://developers.cloudflare.com/workers/platform/limits#worker-limits">Here&rsquo;s the limit</a>. </p> <p>I think you&rsquo;ll be fine, unless you&rsquo;re some celebrity&nbsp;;)</p> <h1 id="next-step">Next step<a class="headerlink" href="#next-step" title="Permanent link">&para;</a></h1> <p>As the next step, I&rsquo;m thinking to create a generic <code>Go/Link</code> resolver browser extension. Then, someone can set their own default domain or company domain of choice as short-domain host. In that case, entering just <code>go/hr-help</code> on the browser will take to <code>https://www.company-internal.com/.../hr/contact.html</code> that we have discussed at the beginning (remember the example case of an internal human resources help&nbsp;portal?).</p> <h1 id="related">Related<a class="headerlink" href="#related" title="Permanent link">&para;</a></h1> <p>If you want to do this url-direction <strong>on your server, but only using webserver</strong>, try this: <a href="https://go.kmonsoor.com/golink-caddy">Personal short-link server using only&nbsp;Caddyserver</a></p> <hr> <p>If you find this post helpful, you can show your support <a href="https://www.patreon.com/kmonsoor">through Patreon</a> or by <a href="https://ko-fi.com/kmonsoor">buying me a coffee</a>. <em>Thanks!</em></p>TL;DR what cloud provider to use in 20212021-05-22T00:00:00+06:002021-05-22T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-05-22:/TLDR-what-cloud-to-use-2021/<p>Among the thousands of combinations a company can take to choose from the cloud providers and their products, this is my <span class="caps">TL</span>;<span class="caps">DR</span>&nbsp;suggestion</p><p>The sheer number of combinations a company can choose from the cloud providers and their product suites is mind-boggling. Hence, I decided to break it down in a concise form for the busy C-suite&nbsp;executives.</p> <p>According to my little experiences and humble opinion, I suggest&nbsp;&hellip;</p> <p>➤ If your company is a small SaaS shop with 10-ish engineers, stick with DigitalOcean, Linode, <span class="caps">OVH</span>, etc., which are best known as cloud &ldquo;instance&rdquo; providers.<br> Think McDonald&rsquo;s; reliable, cheapest, fast, but you won&rsquo;t take your date there. <br> <strong>Budget</strong>:&nbsp;💰</p> <p>➤ If you want a whole cloud experience (e.g., <span class="caps">VPC</span>, firewall, <span class="caps">WAF</span>, etc., on the menu), start with Google Cloud, then try <span class="caps">AWS</span> later.<br> Google Cloud would be the quickest to grasp the cloud concepts and get going. The <span class="caps">UI</span> of the <span class="caps">AWS</span> console is a bit messy compared to <span class="caps">GCP</span>; it just takes more time to get a proper grip.<br> Imagine them as full-course, Michelin-star restaurants. However, the product names are so abstract that they need a full-sized chart for that. ;) <strong>Budget</strong>:&nbsp;💰💰💰</p> <p>➤ Are you planning to set up a million-dollar infra for a billion-dollar company? Go for some <span class="caps">GCP</span>+<span class="caps">AWS</span> multi-cloud setup. You gonna get rebates from both on the scale of hundreds of thousands of dollars. And Microsoft Azure gonna offer you some million-$ free-tier, hoping to get the company hooked on Azure. :D <strong>Budget</strong>:&nbsp;💰💰</p> <p>➤ On the other hand, if you run a govt agency or a company where wearing suits is the mainstream, Microsoft Azure is your best bet.<br> A bunch of consultancy companies to choose from; you need to just approve the budget, you get the things to get up <span class="amp">&amp;</span> running but miss the deadline by months, if not years. But there&rsquo;d be no need for hiring more smarter ppl than that you already have. <strong>Budget</strong>:&nbsp;💰💰💸</p> <p>Need an even more comprehensive guide? Gotcha, fam&nbsp;&hellip;</p> <blockquote class="twitter-tweet"><p lang="en" dir="ltr"><span class="caps">CTO</span>: we&#39;re having hard time choosing a cloud provider<br>&#8230;<br>&#8220;say no more, fam, I gotcha &#8230;&#8221; <a href="https://t.co/hR3rMruWWi">pic.twitter.com/hR3rMruWWi</a></p>&mdash; Khaled Monsoor ✨ (@kmonsoor) <a href="https://twitter.com/kmonsoor/status/1395959443376857088?ref_src=twsrc%5Etfw">May 22, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <p><em><span class="caps">PS</span></em> This post is inspired by a LinkedIn post of mine where I shared about my short experience with the Microsoft Azure <strong>DevOps</strong>&nbsp;suite</p>Deploying a short-link aka go-link server using only Caddyserver2021-04-16T00:00:00+06:002021-04-16T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-04-16:/deploying-golink-server-using-Caddy/<p>Yeah, there are tons of open-source, full-fledged link-shorteners. But, none were exactly what I wanted. Hence, the minimal approach only by utilizing an amazing webserver, <code>Caddy</code>. Here we go&nbsp;&hellip;</p><p>Before I go into any details, please note that I&rsquo;ve used the term &ldquo;shortlink-server&rdquo; instead of &ldquo;url-shortener&rdquo; because the difference is significant for this post. A url-shortener takes a long url, and gives a short url, then redirects any requests for the shortened link to its longer counterpart. On the contrary, shortlink-server takes both the long and short url as inputs, then only does the redirect-ion&nbsp;part.</p> <h2 id="backstory">Backstory<a class="headerlink" href="#backstory" title="Permanent link">&para;</a></h2> <p>If I wanted a fantastic, personal url-shortener or &ldquo;go link&rdquo; server, there are many excellent solutions out there, which are not only free or open-source but full-fledged as well for personal or public usage. Instead, I wanted some &ldquo;service&rdquo; that would &ldquo;resolve&rdquo; my personal, short links. I have been using bit.ly for a long time for its customizable &ldquo;short-half&rdquo; part, but the problem with bit.ly is &ndash; for some God-forsaken reason &ndash; blocked by the Bangladeshi govt. So, I needed a replacement to be appropriately &ldquo;glocal&rdquo;. There are many free (unbranded) and commercial (branded) options as well, but I wanted something that would be resolved via my hosted service (and personal domain) and as cheap as possible. So, basically solution for a poor nerd&nbsp;:D</p> <p>Before jumped into this solution, I tried (deployed <span class="amp">&amp;</span> tested) few others myself, mainly <a href="https://github.com/kellegous/go">kellegous/go</a>, <a href="kutt.it">kutt.it</a> and <a href="https://github.com/adamyi/golinks">adamyi/golinks</a>. But, all of them &ldquo;too featureful&rdquo; for my&nbsp;needs.</p> <p><img alt="Simple, on-prem short-link server using Caddy webserver" src="https://i.imgur.com/4nZbnUE.png"></p> <p>What I wanted is to be able&nbsp;to:</p> <ul> <li>resolve only my custom shortlinks (hence, no need for&nbsp;url-shortener)</li> <li>not a public, internet-facing service (hence, any frontend, authentication, email verification etc. would be overkill&nbsp;)</li> <li>minimal setup (if possible, no webapp at&nbsp;all)</li> </ul> <p>Given my previous experience with <code>Caddy</code> webserver, which is an amazing one(<a href="https://caddyserver.com/docs/">why?</a>), I had a gut feeling that Caddy has something for me &ndash; under the sleeve &ndash; to meet my minimal set of requirements. Thankfully, I managed to find&nbsp;it.</p> <p>I believe <span class="caps">NGINX</span>, currently the most popular webserver, has some kind of similar mechanism as well. But, I&rsquo;m not an expert, and once I was genuinely intimidated by its config file syntax. <span class="caps">YMMV</span>.</p> <h2 id="what-you-gonna-need">What you gonna need?<a class="headerlink" href="#what-you-gonna-need" title="Permanent link">&para;</a></h2> <ul> <li>your own domain which will be the root of the shortlinks. While sub-domained <span class="caps">URL</span> like <code>go.company-name.com/*</code> is quite common, if you have some short domain, like you.co/*, only for this purpose, that&rsquo;s fine as&nbsp;well.</li> <li>A web-host server or public-facing instance with its own, <strong>public IPv4 address</strong>.</li> <li>working knowledge of&nbsp;Linux</li> </ul> <h2 id="step-1-point-your-subdomain-to-the-right-place">Step-1: Point your subdomain to the right place<a class="headerlink" href="#step-1-point-your-subdomain-to-the-right-place" title="Permanent link">&para;</a></h2> <ul> <li>Find out what&rsquo;s the <strong>puplic IPv4 address</strong> of your instance that&rsquo;ll act as the webserver. It&rsquo;s usually on the cloud management&nbsp;dashboard.</li> <li>make sure that, regardless of your cloud architecture (e.g. <span class="caps">VPC</span>, subnet, firewall etc.), the <span class="caps">SSL</span> port (<code>:443</code>) of the instance is reachable from the public&nbsp;internet.</li> <li>now go to your domain name registrar (or, <span class="caps">DNS</span> management provider which in my case is Cloudflare). There, you need to point shortlink subdomain (<code>go.</code>)to the webserver&rsquo;s <span class="caps">I.P.</span> address. In <span class="caps">DNS</span> terms, you&rsquo;ll be creating a <span class="caps">CNAME</span> entry on the domain&rsquo;s nameserver&nbsp;table.</li> </ul> <p>You can do this step as the last one. But for some reason, I prefer it to do first. Because sometimes, <a href="https://blog.cloudflare.com/never-deal-with-dns-propagation-again/"><span class="caps">DNS</span> propagation</a> takes some time. But, once my web service is up and running, I like to see the result instantaneously.&nbsp;;)</p> <h2 id="step-2-install-caddy-a-mighty-webserver">Step-2: Install Caddy, a mighty webserver<a class="headerlink" href="#step-2-install-caddy-a-mighty-webserver" title="Permanent link">&para;</a></h2> <p>Depending on your host <span class="caps">OS</span> (Ubuntu 20.04 <span class="caps">LTS</span> in my case), you need to <a href="https://caddyserver.com/docs/install">install the <code>Caddy</code> webserver</a>. While there are some hacky solutions to run, I think running <code>Caddy</code> as a background service is the simplest to manage. In fact, the documentation of Caddy is excellent, so I&rsquo;d better leave that part to&nbsp;you.</p> <p>After running with the default config(<code>Caddyfile</code>), (in Ubuntu&rsquo;s case, located as <code>/etc/caddy/Caddyfile</code>), it should show a status somewhat like the below image. Please note that, in many cases, if running without <code>sudo</code>, Caddy cannot attach itself with the <span class="caps">SSL</span> port (<code>:443</code>), which is necessary for serving <code>https://</code>. So, check for that error message in the &ldquo;status&rdquo;&nbsp;log.</p> <p><img alt="Caddy service on Ubuntu" src="https://i.imgur.com/cfS5nvZ.png?1"></p> <p><em><strong><span class="caps">PS</span>:</strong> By the way, want your console and command prompt to look 🚀 like mine? Here&rsquo;s the guide: <a href="https://blog.kmonsoor.com/pimp-up-my-terminal/">How do I pimp up my terminal on&nbsp;Linux</a></em></p> <h2 id="step-3-tell-caddy-your-short-links-to-redirect">Step-3: Tell Caddy your short-links to redirect<a class="headerlink" href="#step-3-tell-caddy-your-short-links-to-redirect" title="Permanent link">&para;</a></h2> <p>Now, it&rsquo;s time to configure Caddy to actually do the&nbsp;job.</p> <p>Caddy has its native <code>redir</code> <span class="dquo">&ldquo;</span>directive&rdquo; to redirect incoming web-request from one to another. While the <code>map</code> directive is relatively new, it makes the config file, i.e., Caddyfile, look elegant in case you have (or will have in the long run) a long list of&nbsp;short-links.</p> <p>Here&rsquo;s mine, which is working nicely&nbsp;&hellip; </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span># /etc/caddy/Caddyfile <span class="linenos" data-linenos=" 2 "></span> <span class="linenos" data-linenos=" 3 "></span>go.kmonsoor.com { # replace it your web-url root <span class="linenos" data-linenos=" 4 "></span> <span class="linenos" data-linenos=" 5 "></span> map {path} {redirect-uri} { <span class="linenos" data-linenos=" 6 "></span> /blog https://blog.kmonsoor.com <span class="linenos" data-linenos=" 7 "></span> /photos https://photos.kmonsoor.com <span class="linenos" data-linenos=" 8 "></span> <span class="linenos" data-linenos=" 9 "></span> /resume https://drive.google.com/file/d/1nMS3i1ai6nsI70zZ7NFnNQ_XmvAa4GOl <span class="linenos" data-linenos="10 "></span> /resume-doc https://docs.google.com/document/d/1ECx1Yr8Jzz9I3S5VcoKnZQz56oIht2XaM5gSNetcWag <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span> /rickrolled https://www.youtube.com/watch?v=dQw4w9WgXcQ <span class="linenos" data-linenos="13 "></span> <span class="linenos" data-linenos="14 "></span> # will add new ones here like the above <span class="linenos" data-linenos="15 "></span> # ... <span class="linenos" data-linenos="16 "></span> } <span class="linenos" data-linenos="17 "></span> <span class="linenos" data-linenos="18 "></span> # this below code is required to actually make the above `map` work <span class="linenos" data-linenos="19 "></span> <span class="linenos" data-linenos="20 "></span> @hasRedir expression `{redirect-uri} != &quot;&quot;` <span class="linenos" data-linenos="21 "></span> redir @hasRedir {redirect-uri} <span class="linenos" data-linenos="22 "></span> <span class="linenos" data-linenos="23 "></span> # code below is to set the default response if the requested shortlink isn&#39;t here <span class="linenos" data-linenos="24 "></span> respond &quot;Thas&#39;s an unknown short URL ... :(&quot; <span class="linenos" data-linenos="25 "></span>} </code></pre></div> <p>Note: Don&rsquo;t forget to restart the <code>caddy</code> service to let the new config to take&nbsp;effect.</p> <h2 id="step-4-profit">Step-4: Profit<a class="headerlink" href="#step-4-profit" title="Permanent link">&para;</a></h2> <p>Yeah, that&rsquo;s it. Now, add some own personal stuff with some cool short-links, and proudly share with the&nbsp;world.</p> <h2 id="whats-next">What&rsquo;s next ?<a class="headerlink" href="#whats-next" title="Permanent link">&para;</a></h2> <p>I&rsquo;m thinking that given the very low workload my shortlink resolver needs — unless I&rsquo;m becoming an overnight internet sensation — using a server instance only for this purpose is overkill. My next goal is to have the same service using some &ldquo;serverless&rdquo; function or using the &ldquo;<a href="https://developers.cloudflare.com/workers/examples/redirect">worker on the edge</a>&rdquo; thing from Cloudflare. Let&rsquo;s see&nbsp;;)</p> <p><strong>Update</strong> Now, actually done it. Here is the link: <a href="https://blog.kmonsoor.com/golink-server-using-cloudflare-worker-kv/">Free short-link server &ldquo;on edge&rdquo; using Cloudflare Worker <span class="caps">K.V.</span></a></p> <hr> <p>If you find this post helpful, you can show your support <a href="https://www.patreon.com/kmonsoor">through Patreon</a> or <a href="https://paypal.me/KhaledMonsoor/">Paypal</a> or by <a href="https://ko-fi.com/kmonsoor">buying me a coffee</a>. <em>Thanks!</em></p>Pimping up My Linux Terminal2021-03-31T00:00:00+06:002021-03-31T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-03-31:/pimp-up-my-terminal/<p>How do I pimp up my Linux terminal? A quick trip through Zsh, Oh-my-zsh, and other power tools to make the command-line-based workflow smooth and&nbsp;cool.</p><p>The purpose of this post is to be my quick, copy-paste source of the commands that I use to set up my terminal on a new *nix system. However, if someone else finds it useful, that&rsquo;d be some cherries on&nbsp;top.</p> <p>This command prompt in the below image is the end&nbsp;goal.</p> <p><img alt="The end goal of this post" src="https://i.imgur.com/oZahIog.png"></p> <p>Assuming, I&rsquo;m on a standard pc/server with Ubuntu Linux and I have <span class="caps">CLI</span> access with <code>sudo</code>. For other Linux distros or <em>MacOS</em>, some commands might be slightly&nbsp;different.</p> <h2 id="step-1-confirm-that-zsh-is-up-to-date">Step-1: Confirm that Zsh is up-to-date<a class="headerlink" href="#step-1-confirm-that-zsh-is-up-to-date" title="Permanent link">&para;</a></h2> <p>While on most of the Linux systems Zsh is present by default, on others that&rsquo;s not the case. So, let&rsquo;s make sure about&nbsp;it.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo apt install zsh </code></pre></div> <p>Confirm the version. <code>Oh-my-zsh</code> recommends Zsh to be <code>5.0.8</code> or&nbsp;higher.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>zsh --version <span class="linenos" data-linenos="2 "></span><span class="go">zsh 5.8 (x86_64-ubuntu-linux-gnu)</span> </code></pre></div> <p>Also, you gotta make sure that <code>git</code> (recommended v2.4.11 or higher) is also installed on the&nbsp;system.</p> <h2 id="step-2-install-oh-my-zsh-the-fun-configuration-framework">Step-2: Install Oh-my-zsh, the fun &ldquo;configuration&rdquo; framework<a class="headerlink" href="#step-2-install-oh-my-zsh-the-fun-configuration-framework" title="Permanent link">&para;</a></h2> <p>Install directly from the&nbsp;source.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sh -c <span class="s2">&quot;</span><span class="k">$(</span>curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh<span class="k">)</span><span class="s2">&quot;</span> </code></pre></div> <p>In the last step of this installation, it will ask to set Zsh as <span class="caps">THE</span> shell. Go&nbsp;ahead.</p> <p>Now we have the default prompt from <code>Oh-my-zsh</code>. </p> <p><img alt="After successful installation of Oh-my-zsh" src="https://i.imgur.com/HOVqqvi.png"></p> <p>Now, let&rsquo;s pimp up the prompt. Shall&nbsp;we?</p> <h2 id="step-3-install-powerlevel10k-a-powerful-prompt-theme">Step-3: Install <code>powerlevel10k</code>, a powerful prompt theme<a class="headerlink" href="#step-3-install-powerlevel10k-a-powerful-prompt-theme" title="Permanent link">&para;</a></h2> <p>I love the powerful Zsh theme <code>powerlevel10k</code>. More on <a href="https://github.com/romkatv/powerlevel10k#features">why this theme</a> is&nbsp;awesome.</p> <p>Let&rsquo;s install it on top of <code>oh-my-zsh</code>.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>git clone --depth<span class="o">=</span><span class="m">1</span> <span class="se">\</span> <span class="linenos" data-linenos="2 "></span> https://github.com/romkatv/powerlevel10k.git <span class="se">\</span> <span class="linenos" data-linenos="3 "></span><span class="gp"> $</span><span class="o">{</span>ZSH_CUSTOM:-<span class="nv">$HOME</span>/.oh-my-zsh/custom<span class="o">}</span>/themes/powerlevel10k </code></pre></div> <p>Now, gotta set <code>ZSH_THEME="powerlevel10k/powerlevel10k"</code> in <code>~/.zshrc</code> by adding that manually in the&nbsp;file.</p> <h2 id="step-4-make-sure-the-prompt-looks-like-as-you-want">Step-4: Make sure the prompt looks like as you want<a class="headerlink" href="#step-4-make-sure-the-prompt-looks-like-as-you-want" title="Permanent link">&para;</a></h2> <p>In this step, I&rsquo;m gonna bring in my already open-sourced Zsh config file aka <code>.zshrc</code>. </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp"># </span>deleting the current one <span class="p">&amp;</span> get my personal one from GitHub <span class="linenos" data-linenos="2 "></span><span class="gp">$ </span>rm .zshrc <span class="linenos" data-linenos="3 "></span><span class="gp">$</span> <span class="linenos" data-linenos="4 "></span><span class="gp">$ </span>wget https://raw.githubusercontent.com/kmonsoor/dot-files/master/.zshrc </code></pre></div> <p>I kept the powerlevel10k configs as comments so that Zsh doesn&rsquo;t complain if I use the config file early. Have to set <code>ZSH_THEME="powerlevel10k/powerlevel10k"</code> in the <code>~/.zshrc</code> as&nbsp;well.</p> <p>Otherwise, once the <code>powerlevel10k</code> theme will run for the first time by Zsh, a very friendly step-by-step prompt will run you through towards a desirable prompt for you. Also, whenever you want, you can invoke the config-wizard by executing <code>p10k configure</code> on the&nbsp;shell.</p> <p>Now is the time to enable the changes by restarting Zsh and enjoy the new config and the powerful&nbsp;prompt.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span><span class="nb">exec</span> zsh </code></pre></div> <h2 id="optional">Optional<a class="headerlink" href="#optional" title="Permanent link">&para;</a></h2> <p>Also, I usually install this very useful, but external plugin <code>zsh-syntax-highlighting</code> for&nbsp;oh-my-zsh.</p> <p><div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>git clone https://github.com/zsh-users/zsh-syntax-highlighting.git <span class="si">${</span><span class="nv">ZSH_CUSTOM</span><span class="k">:-</span><span class="p">~/.oh-my-zsh/custom</span><span class="si">}</span>/plugins/zsh-syntax-highlighting </code></pre></div> Don&rsquo;t forget to activate the plugin by including it in ~/.zshrc. For that, add <code>zsh-syntax-highlighting</code> inside the list of other&nbsp;plugins.</p> <p>plugins=( plugin_a plugin_b&nbsp;zsh-syntax-highlighting)</p>Install the latest Zsh on CentOS2020-10-20T00:00:00+06:002020-10-20T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2020-10-20:/install-latest-Zsh-on-CentOS/<p>If you&rsquo;re trying to install the latest version of Zsh instead of the default old one, here you&nbsp;go.</p><p>As the <strong>default</strong> Zsh on CentOS is usually an older version, many cool things are not possible on this version of Zsh, like installing <a href="https://ohmyz.sh/">oh-my-zsh</a> or using awesome <a href="https://github.com/romkatv/powerlevel10k">powerlevel10k</a> prompt system, it&rsquo;s understandable if you&rsquo;d like to have the latest Zsh on your system. Easy peasy&nbsp;!! </p> <p><strong>Note</strong>: Please remember to remove the <code>sudo</code> from the commands if you are already in &ldquo;root&rdquo; or sudo-er&nbsp;mode </p> <p>We&rsquo;ll be following these&nbsp;steps:</p> <div class="toc"> <ul> <li><a href="#install-the-pre-requisites">Install the&nbsp;pre-requisites</a></li> <li><a href="#download-the-latest-source">Download the latest&nbsp;source</a></li> <li><a href="#build-install">Build <span class="amp">&amp;</span>&nbsp;Install</a></li> <li><a href="#final-steps">Final&nbsp;steps</a></li> <li><a href="#related">Related</a></li> </ul> </div> <h1 id="install-the-pre-requisites">Install the pre-requisites<a class="headerlink" href="#install-the-pre-requisites" title="Permanent link">&para;</a></h1> <p>We need <span class="caps">GCC</span> (C++ compiler) and other related stuffs for building Zsh from the source&nbsp;code. </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo yum groupinstall <span class="s2">&quot;Development tools&quot;</span> <span class="linenos" data-linenos="2 "></span><span class="gp">$ </span>sudo yum install ncurses-devel </code></pre></div> <p>Now, check if <span class="caps">GCC</span> is installed properly, by<br> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>gcc -v </code></pre></div></p> <h1 id="download-the-latest-source">Download the latest source<a class="headerlink" href="#download-the-latest-source" title="Permanent link">&para;</a></h1> <p>Now, we gonna get the latest code of Zsh.<br> Please update the link (in the shown command) with the latest by checking <a href="https://www.zsh.org/pub/">this web-folder</a>.<br> Don&rsquo;t forget to update the filename as well, if&nbsp;needed.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span><span class="nb">cd</span> /usr/local/src <span class="linenos" data-linenos="2 "></span><span class="gp">$ </span>sudo curl -L https://www.zsh.org/pub/zsh-5.8.tar.xz <span class="se">\</span> <span class="linenos" data-linenos="3 "></span>-o zsh-5.8.tar.xz </code></pre></div> <h1 id="build-install">Build <span class="amp">&amp;</span> Install<a class="headerlink" href="#build-install" title="Permanent link">&para;</a></h1> <p>Unzip the file, &ldquo;dig in&rdquo; to the folder, and build <span class="amp">&amp;</span> install from the&nbsp;source. </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo tar -xf zsh-5.8.tar.xz <span class="c1"># the actual version of the downloaded file might be different</span> <span class="linenos" data-linenos="2 "></span><span class="gp">$ </span><span class="nb">cd</span> zsh-5.8 <span class="linenos" data-linenos="3 "></span><span class="gp">$ </span>sudo ./configure <span class="o">&amp;&amp;</span> sudo make <span class="o">&amp;&amp;</span> sudo make install </code></pre></div> <h1 id="final-steps">Final steps<a class="headerlink" href="#final-steps" title="Permanent link">&para;</a></h1> <p>Add Zsh to the login shells by adding &lsquo;/usr/local/bin/zsh&rsquo; on the last line of the config file, <code>/etc/shells</code> </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo -e /etc/shells <span class="linenos" data-linenos="2 "></span><span class="gp">$ </span>sudo chsh <span class="nv">$USER</span> </code></pre></div> <p>Update the system&rsquo;s default symlink to the new Zsh version.<br> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo ln -sf /usr/local/bin/zsh /bin/zsh <span class="linenos" data-linenos="2 "></span><span class="gp">$ </span>zsh --version </code></pre></div></p> <p>It&rsquo;s always a good habit to clean up after doing stuffs. ;) <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo make clean </code></pre></div></p> <p>That&rsquo;s it&nbsp;!</p> <p><img alt="voila" class="noZoom" src="https://i.imgur.com/BEFIOXfm.jpg"></p> <h1 id="related">Related<a class="headerlink" href="#related" title="Permanent link">&para;</a></h1> <p>Want to have a super, cool-looking command shell? Gotcha, fam. Check out my blog on <strong><a href="https://blog.kmonsoor.com/pimp-up-my-terminal/">Pimping up My Linux Terminal</a></strong>.</p> <hr> <p>If you find this post helpful, you can show your support <a href="https://www.patreon.com/kmonsoor">through Patreon</a> or <a href="https://paypal.me/KhaledMonsoor/">Paypal</a> or by <a href="https://ko-fi.com/kmonsoor">buying me a coffee</a>. <em>Thanks!</em></p>What I learned managing weekly release of a 50M+ users’ app2020-08-22T12:12:00+06:002020-08-22T12:12:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2020-08-22:/some-tips-on-managing-apps-release/<p>In any war, game, or enterprise resource management, the last thing you want is to underestimate your opponent(s). In release management, the opponent is “chaos”. The last thing a release manager should expect is that everything, from bug-free features to timely completion of production builds will be done as per the cadence&nbsp;schedule.</p><p><em>Disclaimer: Everything here are my personal observations, experiences, commentary, or “wisdom”. My past, present, or future employers shouldn’t be held responsible for it. Also, take in or apply the stuff below “with a grain of&nbsp;salt”.</em></p> <blockquote> <p>In any war, game, or enterprise resource management, the last thing you want is to underestimate your opponent(s). In release management, the opponent is&nbsp;“chaos”.</p> </blockquote> <p><img alt="releases" class="noZoom" src="https://i.imgur.com/BE87UxPl.jpg"></p> <p>The last thing a release manager should expect is that everything, from bug-free features to timely completion of production builds will be done as per the cadence&nbsp;schedule.</p> <p>Here is something that I had to learn over time. It might help someone,&nbsp;somewhere.</p> <h3 id="is-the-release-cadence-effective">Is the “release cadence” effective?<a class="headerlink" href="#is-the-release-cadence-effective" title="Permanent link">&para;</a></h3> <p>Not all apps are created equal, neither for the exact same consumer base nor with the same feature set. So, <span class="caps">THE</span> perfect release cadence or frequency doesn’t exist. Every app needs to find it’s own pace. The frequency might be feature-based, the target market base, fixed calendar-based, and/or mix of&nbsp;those.</p> <p>Though I can’t go in detail for each of them, I emphasize going through the process of identifying the right frequency. While for the passenger app of Grab, due to the sheer number of features (or change of features) and bug fixes, it makes sense to roll out a new version every week, it might make no sense for your app that gets new features once in a month or two that has a limited number of enterprise customers and your user-support team has to train them in&nbsp;sessions.</p> <p>Ask the questions across the company who are involved or have a vested interest in the release&nbsp;process.</p> <ul> <li>Who are the primary users? And, who can be considered&nbsp;secondary?</li> <li>Are the primary audience tech-savvy enough to be excited to navigate the new changes, or will they be pissed about the frequent&nbsp;changes?</li> <li>How frequently may the product teams push out new&nbsp;features?</li> <li>How resilient the quality assurance team is to exhaustively verify the new features and look out for new bugs that are going to break the old, stable&nbsp;features?</li> <li>How much automation is in place to support a more frequent build, release, and&nbsp;rollout?</li> </ul> <p>Particularly, if your <span class="caps">QA</span>/testing team isn’t well-distributed as per the app scopes, to provide the final sign-off on that specific scope, avoid intense weekly releases. Having a shiny process, just for the sake of having it, is an easy recipe for&nbsp;disaster.</p> <h3 id="is-the-process-well-documented-communicated">Is the process well-documented <span class="amp">&amp;</span> communicated?<a class="headerlink" href="#is-the-process-well-documented-communicated" title="Permanent link">&para;</a></h3> <p>Well-accepted process documents should work as the org-wide “release” manifesto, well discussed among the contributing teams and agreed by the&nbsp;stakeholders. </p> <p>Once the process and agreed upon process timeline is there, in case of confusion or conflict between the teams, the release team will have two clear paths to move&nbsp;forward.</p> <ul> <li>It’s already in the doc, point to&nbsp;it.</li> <li>If it’s not in the doc or not clear enough, an opportunity to&nbsp;improve</li> </ul> <p>If there is an exception, foreseeable disruption, or any upcoming change in the process, <em>communicate early</em>, and <em>communicate frequently</em>.</p> <h3 id="is-some-automation-possible">Is some automation possible?<a class="headerlink" href="#is-some-automation-possible" title="Permanent link">&para;</a></h3> <p>Identify the opportunities to automate. As a release manager, if you need something to do more than twice a day, start there. Have you been asked the same question twice? Include the answer in a <span class="caps">FAQ</span>&nbsp;doc.</p> <p>Document the need, identify the tool required. It can be as simple as a <span class="caps">SQL</span> query, a Python script, or a much more complex&nbsp;system. </p> <p>Before jumping headfirst into the thing, identify the matrix to monitor the improvement, and get approval (at least, verbally) from the&nbsp;stakeholders.</p> <p>Some example automation tool I&nbsp;use:</p> <ul> <li><a href="https://www.google.com/script/start/">Google script</a> to monitor changes on Google&nbsp;Sheet</li> <li><a href="https://slack.com/help/articles/202026038-An-introduction-to-Slackbot">Slack bot response</a> for selected&nbsp;keywords</li> <li>Python scripts for compiling custom <span class="caps">JIRA</span> reports. <a href="https://gist.github.com/kmonsoor/62ed1bc6cd3084648245073744182227">An&nbsp;example</a></li> <li><a href="https://slack.com/intl/en-sg/help/articles/206819278-Send-emails-to-Slack">Email forwarding to Slack <span class="caps">DM</span></a></li> </ul> <p>Automation paves the path to sustainable scalability. The release manager might be able to ping (aka “pull info or update”) three persons, but definitely shouldn’t try for thirty. So, establish some “push notification” mechanism as much as possible. E.g., comment on the Google <em>Sheet</em> or more frequent status update on the <span class="caps">JIRA</span> ticket,&nbsp;etc.</p> <h3 id="is-proper-divide-conquer-in-place">Is proper “Divide <span class="amp">&amp;</span> Conquer” in place?<a class="headerlink" href="#is-proper-divide-conquer-in-place" title="Permanent link">&para;</a></h3> <p>This <span class="caps">DC</span> method/mechanism/scheme (as a fan of the Marvel universe, I don’t mind the name mixup though :P ) is a controversial term. However, the “division of labor” is a well-established example. Regardless of the name and the flavor, it’s essential for the successful execution of any high-frequency release&nbsp;cycle. </p> <ul> <li>Design the process and its breakdown based on the roles, not the actual persons or their&nbsp;skillset</li> <li>Minimize switching between roles for individual persons in a single release cycle. e.g., if someone, in the <span class="caps">QA</span> team, is writing test cases, let them do that for at least the period. If someone is testing individual features at the beginning of the week, let her continue it until the end of the release. \ <em>Well-designed repetition is key to reaching&nbsp;excellence.</em></li> <li>Keep only integrated test (staging, then production) dependency for release rollout. For individual features, the developing team(s) should take care of it, before “signing off” it ready for shipping. The “not ready to be shipped” features must be isolated from the production scope, excluding it from the ongoing release&nbsp;cycle.</li> <li>Imagine the shipment process of a newer version as a concept like a train and different teams as different product pipelines. Products come to the train station only when it’s packed and ready to be shipped. I believe it&rsquo;s an agile concept, as&nbsp;well.</li> <li>Like a public train, the release “train” should depart as per schedule. If any feature is delayed due to unforeseen reasons or a recently discovered severe bug, the release-train should not wait for it. Instead, once the feature is stabilized later, it should wait for the next <em>train</em>.</li> <li>Adopting the concept of a train and its scheduled departure and well-documented checkpoints (i.e., phases) enables a large organization to expect and likewise communicate what to expect&nbsp;when.</li> </ul> <h3 id="are-you-deploying-on-a-wrong-moon-phase">Are you deploying on a wrong moon-phase?<a class="headerlink" href="#are-you-deploying-on-a-wrong-moon-phase" title="Permanent link">&para;</a></h3> <p>Probably, it should be a piece of common knowledge. Be it a backend deployment or app rollout on app stores; please avoid changing anything with the “production” tag, before a weekend or a multi-day&nbsp;holiday. </p> <p>Like the <a href="https://twitter.com/System32Comics/status/1266100094853476352/photo/1">printer meme</a>, the universe likes to throw unique, unforeseen issues after weekend deployments. It’s a mystery of the world, I&nbsp;guess.</p> <h3 id="is-there-a-provision-for-backup">Is there a provision for backup?<a class="headerlink" href="#is-there-a-provision-for-backup" title="Permanent link">&para;</a></h3> <p>Plan capacity with some&nbsp;slack. </p> <p>I’d suggest going for the 80/20 rule, meaning the weekly release should go out, even in the unavailability of 20% human&nbsp;resources. </p> <p>Unlike machines, people will be sick, go out for vacation, emergency travel plans, unexpectedly resign, etc. anything can happen. So, like any other robust system design, plan for&nbsp;“disaster”. </p> <h3 id="is-proper-continuous-integrationdeployment-cicd-in-place">Is proper Continuous Integration/Deployment (<span class="caps">CI</span>/<span class="caps">CD</span>) in place?<a class="headerlink" href="#is-proper-continuous-integrationdeployment-cicd-in-place" title="Permanent link">&para;</a></h3> <p>To have an efficient release process, automated and well-documented <span class="caps">CI</span>/<span class="caps">CD</span> pipeline is almost a&nbsp;prerequisite.</p> <ul> <li>Developers across the board should be able to build, test, merge their code, and build the binary from the latest stable and/or release&nbsp;checkpoints.</li> <li>Once their change, be it a feature or a bugfix, is ready to be merged with the release branch, a complete suite of automated tests should run on the resultant binary to detect any regression bug due to the change; with no dependency on the release&nbsp;team.</li> <li><span class="caps">QA</span>, while testing for a particular release, should be able to receive regression or production build without any manual effort from the release&nbsp;team.</li> </ul> <h3 id="is-there-continuous-feedback-established">Is there continuous feedback established?<a class="headerlink" href="#is-there-continuous-feedback-established" title="Permanent link">&para;</a></h3> <p>An efficient release process is an intensive process. And, no process can be perfect from the&nbsp;beginning. </p> <p>Every organization has, as its fingerprint, has different priorities, deliverables, audience, resources, and of course, budget. So, while the release cycle can start with an agile blueprint, down the line it needs to be well fit on its organizational&nbsp;body. </p> <p>Here comes the necessity for continuous feedback from the participants, stakeholders, and even the release&nbsp;team-members.</p> <h3 id="stay-on-the-good-book-of-the-app-stores">Stay on the “good book” of the app stores ;)<a class="headerlink" href="#stay-on-the-good-book-of-the-app-stores" title="Permanent link">&para;</a></h3> <p>If the app is in the center of revenue of your company, keep a close eye on the regular comm from the app stores, i.e., Google Play and Apple Appstore. It’s very easy to miss these communications, among other day-to-day&nbsp;stuff.</p> <p>Regardless of how ridiculous your company feels about some policies on the app stores, please read between the lines. In case of confusion, read again. Especially if you’re one of the big fishes.<br> If a particular app store weren’t strictly monitoring or enforcing some policies last month, don’t even think they aren’t going to enforce tomorrow and reject your&nbsp;app. </p> <p>The release manager or the team, being the interface with the app stores, should frequently communicate with the stakeholders and the product team to carefully go through relevant app store policies long before they invest company time and resources into a feature or a roadmap. Trust me; it happens more than anyone likes to admit that after month-long design and development, the app update with the feature gets rejected on the app stores, solely due to this new&nbsp;feature.</p> <p>If possible, try to get hold of and build a working relationship with the business development or dev support teams of Google and Apple. It might help in case of any misunderstanding and/or some negotiations about the compliance timeline.<br> I can’t go into any more detail though&nbsp;;)</p> <h3 id="notes">Notes:<a class="headerlink" href="#notes" title="Permanent link">&para;</a></h3> <ul> <li>Please don’t take the number 50M+ literally. The actual number is much higher but should be considered as company secrets ;).&nbsp;Right?</li> <li>This article is <span class="caps">NOT</span> a definitive process guide, and I’m not an expert on Agile methodologies. I’m just trying to share my learning from my release experience of 1.75 years. Some stuff might help any other release manager on the other side of the planet. That’s the goal&nbsp;here.</li> </ul> <p>Thanks,&nbsp;y’all.</p>Install latest Python 3 on Linux CentOS 72018-07-07T00:00:00+06:002018-07-07T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2018-07-07:/install-latest-python3-on-centos-7/<p>Install the latest and greatest Python 3 on CentOS 7&nbsp;systems</p><h2 id="why">Why<a class="headerlink" href="#why" title="Permanent link">&para;</a></h2> <p>Not all distro created equal. <br> Some are created to join the space race, some are to hold unto the leagcy, some are cutting-edge, some are cutting edge. Some are born to boot-up IoT devices some are to push out heavy&nbsp;graphics.</p> <p>That&rsquo;s the fun (albeit, power) of&nbsp;Linux.</p> <p><img alt="CentOS 7 logo" class="noZoom" src="https://i.imgur.com/6ZFCdoM.jpg"></p> <p>CentOS 7 is a powerful and stable distro that runs on thousands (probably, millions) production-grade servers.<br> In the matter of stability, it&rsquo;s a beast. However, it doesn&rsquo;t ship with Python 3, by default. You can install it via <span class="caps">EPEL</span> repository, or the below simple&nbsp;steps.</p> <p>Also, take a note. The Python <strong>2</strong> comes with the system, which is probably 2.7.5, do <strong><span class="caps">NOT</span></strong> mess with it. Many system components rely on that specific version. If you need the latest versions of 2, use <code>virtualenv</code> or <code>pipenv</code>.</p> <h3 id="prepare-your-system">Prepare your system<a class="headerlink" href="#prepare-your-system" title="Permanent link">&para;</a></h3> <p>Start with installing pre-requisite utilities for compilation and development&nbsp;support.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo yum update <span class="o">&amp;&amp;</span> sudo yum groupinstall -y <span class="s2">&quot;development tools&quot;</span> <span class="linenos" data-linenos="2 "></span>$ sudo yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel <span class="se">\</span> <span class="linenos" data-linenos="3 "></span> sqlite-devel readline-devel tk-devel gdbm-devel <span class="se">\</span> <span class="linenos" data-linenos="4 "></span> db4-devel libpcap-devel xz-devel expat-devel </code></pre></div> <h3 id="download-latest-python-source-code-from-pythonorg">Download latest Python source code from Python.org<a class="headerlink" href="#download-latest-python-source-code-from-pythonorg" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz <span class="linenos" data-linenos="2 "></span>$ tar xf Python-3.6.6.tar.xz <span class="linenos" data-linenos="3 "></span>$ <span class="nb">cd</span> Python-3.6.6 </code></pre></div> <h3 id="enable-performance-optimizations-optional-but-highly-recommended">Enable performance optimizations (optional, but highly recommended)<a class="headerlink" href="#enable-performance-optimizations-optional-but-highly-recommended" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ ./configure --prefix<span class="o">=</span>/usr/local --enable-shared <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">&quot;-Wl,-rpath /usr/local/lib&quot;</span> <span class="linenos" data-linenos="2 "></span>$ ./configure --enable-optimizations </code></pre></div> <h3 id="build-and-install">Build and install<a class="headerlink" href="#build-and-install" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ make <span class="linenos" data-linenos="2 "></span>$ sudo make altinstall </code></pre></div> <p>Now, Python 3.6.6 is ready to be used in your system; located in <code>/usr/local/bin/python3.6</code> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ which python3.6 <span class="linenos" data-linenos="2 "></span>/usr/local/bin/python3.6 <span class="linenos" data-linenos="3 "></span> <span class="linenos" data-linenos="4 "></span>$ python3.6 <span class="linenos" data-linenos="5 "></span>Python <span class="m">3</span>.6.6 <span class="o">(</span>default, Jul <span class="m">10</span> <span class="m">2018</span>, <span class="m">14</span>:04:26<span class="o">)</span> <span class="linenos" data-linenos="6 "></span><span class="o">[</span>GCC <span class="m">4</span>.8.5 <span class="m">20150623</span> <span class="o">(</span>Red Hat <span class="m">4</span>.8.5-28<span class="o">)]</span> on linux <span class="linenos" data-linenos="7 "></span>Type <span class="s2">&quot;help&quot;</span>, <span class="s2">&quot;copyright&quot;</span>, <span class="s2">&quot;credits&quot;</span> or <span class="s2">&quot;license&quot;</span> <span class="k">for</span> more information. <span class="linenos" data-linenos="8 "></span>&gt;&gt;&gt; </code></pre></div></p> <p>For convenience, you can create a symbolic-link with a shorter name. If you had system-installed Python3 (unlikely), <strong>don&rsquo;t</strong> do this, as some system-components may depend on that specific older version of Python 3. <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo ln -s /usr/local/bin/python3.6 /usr/local/bin/python3 <span class="linenos" data-linenos="2 "></span>$ python3 <span class="linenos" data-linenos="3 "></span>Python <span class="m">3</span>.6.6 <span class="o">(</span>default, Jul <span class="m">10</span> <span class="m">2018</span>, <span class="m">14</span>:04:26<span class="o">)</span> <span class="linenos" data-linenos="4 "></span><span class="o">[</span>GCC <span class="m">4</span>.8.5 <span class="m">20150623</span> <span class="o">(</span>Red Hat <span class="m">4</span>.8.5-28<span class="o">)]</span> on linux <span class="linenos" data-linenos="5 "></span>Type <span class="s2">&quot;help&quot;</span>, <span class="s2">&quot;copyright&quot;</span>, <span class="s2">&quot;credits&quot;</span> or <span class="s2">&quot;license&quot;</span> <span class="k">for</span> more information. <span class="linenos" data-linenos="6 "></span>&gt;&gt;&gt; </code></pre></div></p> <h2 id="install-wheel-and-pip"><del>Install wheel and pip</del><a class="headerlink" href="#install-wheel-and-pip" title="Permanent link">&para;</a></h2> <p>You don&rsquo;t need to, because <code>Python 3.6.6</code> includes these necessary tools included. <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ pip3.6 -V <span class="linenos" data-linenos="2 "></span>pip <span class="m">10</span>.0.1 from /usr/local/lib/python3.6/site-packages/pip <span class="o">(</span>python <span class="m">3</span>.6<span class="o">)</span> <span class="linenos" data-linenos="3 "></span>$ wheel version <span class="linenos" data-linenos="4 "></span>wheel <span class="m">0</span>.29.0 </code></pre></div></p>HA(High-Availability) Setup for InfluxDB2018-01-18T00:00:00+06:002018-01-18T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2018-01-18:/ha-setup-for-influxdb/<p>Create a robust, highly-available, time-series InfluxDB cluster with the community(free) version of&nbsp;it</p><p><strong><span class="caps">NOTE</span></strong> <em>Since I have written this article, all the components used in this below architecture have gone through many updates and releases. While the general premise involving <code>influxdb-relay</code> and the multiplexing might still hold, please sync up with the latest release docs before jumping into some serious system&nbsp;design.</em></p> <hr> <p>Currently, from version 0.9, you cannot create an InfluxDB cluster from the open-sourced free edition. Only commercially available InfluxDB Enterprise can do that for now. That stirred up the early-adopter enthusiast users, especially for their usage in professional setups. They complained that InfluxData, the company behind InfluxDB, is trying to milk the <span class="caps">OSS</span> solution for&nbsp;profit.</p> <p><img alt="Archiving isn't easy ... tobias-fischer-PkbZahEG2Ng" src="https://i.imgur.com/0IdYOYnl.jpg"></p> <p>I can&rsquo;t blame the InfluxData guys much, as they got to pay their bills too. So far, we — the users of open-source systems — couldn&rsquo;t show much promise about the financial realities of the projects. Continuing development of <span class="caps">OSS</span> products, by only depending on donations, patrons, or enterprise sponsorship, is far too rare and unpredictable, even for the projects that many successful organizations heavily rely&nbsp;on.</p> <p>Anyways, InfluxDB then promised and later introduced <code>Influx Relay</code> as a complimentary consolation for missing <span class="caps">HA</span> parts of InfluxDB. You can get the details here and here about&nbsp;that. </p> <h2 id="premise">Premise<a class="headerlink" href="#premise" title="Permanent link">&para;</a></h2> <p>For my needs, I have to try to create a reliable <span class="caps">HA</span>(High-Availability) setup from available free options, hence InfluxDB and the relay. It&rsquo;s quite a bit far from an InfluxDB-cluster in terms of robustness or ease of setup, but it&rsquo;s got the job done, at least for&nbsp;me.</p> <p>I needed a setup to receive system-stats from at least 500+ instances and to store them for a while, but without breaking the bank in bills from <span class="caps">AWS</span>. Meaning, I could ask for and could use only couple of instances for my&nbsp;solution.</p> <p>Here were my&nbsp;trade-offs.</p> <ul> <li>Not too many instances for this purpose. Neither, any of the heavyweight lifters e.g. <span class="caps">AWS</span>&rsquo; m3-xlarge etc. To use only what&rsquo;s&nbsp;necessary. </li> <li>To satisfy the budget, hence avoiding pay-per-use solutions as far as it is&nbsp;possible.</li> <li>Solutions must not be crazy complex, so that handover to the DevOps team be&nbsp;smooth.</li> <li>Reading the data would be too rarely w.r.t. writing. The related Grafana dashboards will be only used to investigate issues by a handful of&nbsp;people.</li> </ul> <h2 id="overall-design">Overall Design<a class="headerlink" href="#overall-design" title="Permanent link">&para;</a></h2> <h3 id="write">Write<a class="headerlink" href="#write" title="Permanent link">&para;</a></h3> <p>From a birds&rsquo; eye view, I decided to use two server instances to run parallelly, hosting InfluxDB on them independently and then sending the same data over to them for storing. This scheme mostly looks like <a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1"><span class="caps">RAID</span>-1 systems</a>.</p> <p><img alt="Overall architecture" src="https://i.imgur.com/ZKYIyOd.png"></p> <p>That brings up a couple of&nbsp;challenges.</p> <ul> <li> <p>None of the agents I used on the sender side could multiplex output. That means, they were able to send data to a single destination, not multiple. On the Windows front, I&rsquo;ve used <code>Telegraf</code> which is able randomly to switch between pre-listed destinations, but <span class="caps">NOT</span> multiple at-once.<br> In the case of Linux hosts, I used <code>Netdata</code> which is excellent in its own right, but unable to send stats to multiple destinations.<br> Here comes <code>Influx-relay</code>. It can receive time-series data-stream from hosts on a <span class="caps">TCP</span> or <span class="caps">UDP</span> port, buffer for a while, and then re-send those received and buffered data to multiple receive ends which can either be an InfluxDB instance or another listening Influx-relay instances.<br> This chaining can broaden the relaying scheme even further. However, for my purpose, this relay-chaining was not necessary. Rather, from the relay, I am sending data to the separate InfluxDB instances, running on two separate&nbsp;instances. </p> </li> <li> <p>Now that I partially multiplexed the output, my hosts (senders) still are able to send to one destination. So, I need a proxy as well as a load-balancer. For a while, I was torn between <span class="caps">NGINX</span> and HAProxy. Both were new to&nbsp;me. </p> </li> </ul> <p>However, for a couple of reasons, I went for HAProxy. Firstly, I don&rsquo;t need <span class="caps">HTTP</span> session management. Secondly, as I wanted to keep my <span class="caps">UDP</span> for later, HAProxy was perfectly capable of that.<br> <span class="caps">NGINX</span> has the support recently, but the maturity was a concern. Also, configuring <span class="caps">NGINX</span> seems a little intimidating (which I know might not be so true). Last but not least, and for what it&rsquo;s worth, out-of-the-box, HAProxy&rsquo;s stat page carries much more in-depth information than that of free-version of <span class="caps">NGINX</span>.<br> Upon receiving the stats stream, HAProxy was supposed to send that to different Influx-relays in a load-balanced&nbsp;fashion.</p> <p>So, here&rsquo;s my rough&nbsp;plan. </p> <p>collector-agent &rarr; HAProxy &rarr; (50/50 load-balanced) &rarr; Influx-relay &rarr; (multiplexed) &rarr; 2 InfluxDB&nbsp;instances</p> <p>Now, each one of the received data is to go to both of the InfluxDB instances, or at least to one in case of failure (or, overload per se) of any the relays or Influx instances.   Also, I have chosen to keep Influx-relays deployed as Dockerized and kept HAProxy and InfluxDB instances running as native services. Of course, you can Dockerize HAProxy and InfluxDB,&nbsp;too. </p> <h3 id="read">Read<a class="headerlink" href="#read" title="Permanent link">&para;</a></h3> <p>As I&rsquo;ve already noted in the section that reading the data, meaning to fetch data to visualize on Grafana end, will happen rarely and sporadically; only to investigate alarms or any other client-side performance&nbsp;issues. </p> <p>So, the read requests, reaching the HAProxy end, needed not much routing, other than directly to InfluxDB itself. Still, to better distribute the load I decided to load-balance it 50/50&nbsp;basis.</p> <h3 id="ports">Ports<a class="headerlink" href="#ports" title="Permanent link">&para;</a></h3> <ul> <li>As all the <span class="caps">READ</span> requests are routed through <code>HAProxy</code> running on each of the instances, to the external world only HAProxy&rsquo;s port should be opened for this&nbsp;purpose. </li> <li>On the other hand, for <span class="caps">WRITE</span> requests, InfluxDBs are receiving data from relays, one of its own instance and another one on other instance, so InfluxDB should listen on its own port for <span class="caps">WRITE</span> requests only. But, this must be accessible only from own <span class="caps">VPS</span> zone, but not open to the outside&nbsp;world.</li> <li>In case of HAProxy as well as InfluxDB, you can use the default ports, obviously, which is 8086 <span class="amp">&amp;</span> 8088 respectively. Or, you can choose to go for other ports (security through obfuscation). Your call. In this writing, I&rsquo;ll go with the&nbsp;defaults.</li> </ul> <h3 id="authentication-ssl">Authentication, <span class="caps">SSL</span><a class="headerlink" href="#authentication-ssl" title="Permanent link">&para;</a></h3> <p>You can configure <span class="caps">SSL</span> with your own server certificates through the HAProxy configs. You can even go for <span class="caps">SSL</span> from the relays to InfluxDB writes. If your sender hosts are connecting to your HAProxy through public internet, you should at least go for password-based authentication, better to utilize <span class="caps">SSL</span>. However, for brevity&rsquo;s sake, I&rsquo;ll skip them in this&nbsp;post.</p> <p>**Note: * Please bear in mind, this is an &ldquo;in-progress&rdquo; post; prematurely published to force me to work on it. I have the plan to add all the necessary configurations <span class="amp">&amp;</span> commands, that I used,&nbsp;here.</p>Pelican Static sites - SEO Optimization2017-01-07T00:00:00+06:002017-01-07T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2017-01-07:/pelican-how-to-make-seo-friendly/<p>Usually <code>Pelican</code> static-site generator is not very concerned about <span class="caps">SEO</span> of the generated site. Related themes and their templates also don&rsquo;t take it much seriously. But you shouldn&rsquo;t loose <span class="caps">SEO</span>,&nbsp;right?</p><p>Writing is a hard job, especially it is your hobby beside of your day job. But what&rsquo;s the benefit if nobody reads it just because they couldn&rsquo;t find&nbsp;it.</p> <p>Usually <code>Pelican</code> <a href="https://github.com/getpelican/pelican">static-site generator</a> is not very concerned about <span class="caps">SEO</span> of the generated site mainly because it&rsquo;s not that focused on commercial usage. That&rsquo;s what I felt. Related themes and their templates also don&rsquo;t take it much seriously. But you shouldn&rsquo;t loose <span class="caps">SEO</span> just because you migrated from Wordpress or whatever you were using previously,&nbsp;right?</p> <p>Often times theme-authors focus more on look-n-feel of the theme, but not so much on the <span class="caps">SEO</span>&nbsp;concerns.</p> <p>That&rsquo;s this blog about; let&rsquo;s fix&nbsp;that.</p> <h4 id="note-1-about-other-static-site-generators">Note 1: About other static-site generators<a class="headerlink" href="#note-1-about-other-static-site-generators" title="Permanent link">&para;</a></h4> <p>Although the following discussions <span class="amp">&amp;</span> codes mostly are specific to <a href="https://github.com/getpelican/pelican">Pelican</a> templates which uses <a href="http://jinja.pocoo.org/">jinja2</a> templating language, the concepts and concerns here are applicable to most static-site generators and their&nbsp;themes.</p> <h4 id="note-2-im-in-no-way-an-seo-expert">Note 2: I&rsquo;m, in no way, an <span class="caps">SEO</span> expert<a class="headerlink" href="#note-2-im-in-no-way-an-seo-expert" title="Permanent link">&para;</a></h4> <p>This writeup is just a collection of my findings while correcting my blog&rsquo;s <span class="caps">SEO</span> course; fixing the stupid mistakes. Also, this isn&rsquo;t a commercial site. There are lot more and in-depth aspects of <span class="caps">SEO</span> optimization other than the following, that can be very important for commercial&nbsp;projects.</p> <h2 id="getting-started">Getting started<a class="headerlink" href="#getting-started" title="Permanent link">&para;</a></h2> <h3 id="lookup-for-missing-pieces">Lookup for missing pieces<a class="headerlink" href="#lookup-for-missing-pieces" title="Permanent link">&para;</a></h3> <p>Make sure all the linked resources(links, images, <span class="caps">CSS</span> <span class="amp">&amp;</span> <span class="caps">JS</span> files) that you&rsquo;ve used in your pages are valid. For that check browser&rsquo;s <code>console</code> in <code>Developers tools</code> for any errors e.g. unavailable urls, faulty html/css&nbsp;etc.</p> <h3 id="know-the-critical-spots">Know the critical spots<a class="headerlink" href="#know-the-critical-spots" title="Permanent link">&para;</a></h3> <ul> <li><strong>pelicanconf.py</strong> - It&rsquo;s usually in your root folder of the&nbsp;site.</li> <li><strong>base.html</strong> - It&rsquo;s in the <code>templates</code> folder of the theme folder that you are using. Changes here will impact all <span class="caps">HTML</span> pages generated by&nbsp;Pelican.</li> <li><strong>article.html</strong> - In same folder as <code>base.html</code>. Changes here will impact only articles&rsquo;&nbsp;pages.</li> </ul> <h2 id="avoiding-duplication">Avoiding Duplication<a class="headerlink" href="#avoiding-duplication" title="Permanent link">&para;</a></h2> <h3 id="avoid-your-source-getting-indexed-by-google">Avoid your source getting indexed by Google<a class="headerlink" href="#avoid-your-source-getting-indexed-by-google" title="Permanent link">&para;</a></h3> <p>If the source of your blog is not sourced-open, meaning the content is not in a open-sourced repo, this isn&rsquo;t your concern. But if it is, it should be a&nbsp;concern. </p> <p>The reason is that <code>Github.com</code>(or where your repo is hosted on) is stronger domain than yours, so Google &ldquo;sees&rdquo; that your site-contents (even though it&rsquo;s yours) also is on Github. So, there&rsquo;s a high chance that it&rsquo;ll mark your contents as &ldquo;duplicate&rdquo;. And, duplicate contents gets heavy hammers from Google&rsquo;s <span class="caps">SEO</span> point-of-view. Searching any writeup from yours, even it&rsquo;s very unique on Internet, Google search will show both your site and the repo as well, possibly links from your site will be on the lower&nbsp;side.</p> <p>It begs the question how to avoid that. It&rsquo;s not&nbsp;complicated.</p> <p>For example, Github.com&rsquo;s <a href="https://github.com/robots.txt">robot.txt</a> allows Google (or any search engine for that matter) to index only the <code>master</code> branch of any open-source repo. So, if your repo don&rsquo;t have any branch named <code>master</code>, it won&rsquo;t be indexed. That&rsquo;s it. Rename your &ldquo;master&rdquo; as &ldquo;live&rdquo;, &ldquo;main&rdquo;, &ldquo;production&rdquo; or whatever you feel&nbsp;like.</p> <ol> <li>Create a new&nbsp;&ldquo;master&rdquo;</li> </ol> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ git checkout -b new-master <span class="linenos" data-linenos="2 "></span>$ git push -u origin new-master </code></pre></div> <ol start="2"> <li>Tell Github about new&nbsp;move</li> </ol> <p><img alt="change repo-settings on Github" src="http://i.imgur.com/wjf6zwul.png"></p> <ol start="3"> <li>Now, delete <code>master</code> branch from your&nbsp;repo.</li> </ol> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ git branch -d master <span class="linenos" data-linenos="2 "></span>$ git push origin :master </code></pre></div> <h3 id="utilize-relcanonical-link">Utilize <code>rel="canonical"</code> link<a class="headerlink" href="#utilize-relcanonical-link" title="Permanent link">&para;</a></h3> <p>This is kind of a must-do for avoiding being marked as duplicate content. This will also defend you against automatic content-scraping schemes by always having a pointer to your original&nbsp;source. </p> <p>To utilize it on Pelican, add or make sure that a <code>&lt;link&gt;</code> element with the attribute <code>rel="canonical"</code> to the <code>&lt;head&gt;</code> section of your base template named <code>base.html</code>.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>{% if article %} <span class="linenos" data-linenos="2 "></span>&lt;link rel=&quot;canonical&quot; href=&quot;{{ SITEURL }}/{{ article.url }}&quot;/&gt; <span class="linenos" data-linenos="3 "></span>{% endif%} </code></pre></div> <p>For example, if you see this page&rsquo;s source (pressing <span class="caps">CTRL</span>+u), you should see something&nbsp;like</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>&lt;link rel=&quot;canonical&quot; href=&quot;https://blog.kmonsoor.com/pelican-how-to-make-seo-friendly/&quot;/&gt; </code></pre></div> <h3 id="proper-title-of-each-page">Proper <strong>&lt; title &gt;</strong> of each page<a class="headerlink" href="#proper-title-of-each-page" title="Permanent link">&para;</a></h3> <p>Every page on your site should have a proper title. For search engines, it represents the page. It should concisely reflect a page&rsquo;s content. But try to keep it less 60 characters or search-engines may choose to truncate it. Use each characters&nbsp;wisely.</p> <p>It may look like this in your <code>base.html</code>.</p> <p><div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>{% if article %} <span class="linenos" data-linenos="2 "></span>&lt;title&gt;{{ article.title }} -- {{ TAGLINE }}&lt;/title&gt; <span class="linenos" data-linenos="3 "></span>{% else %} <span class="linenos" data-linenos="4 "></span>&lt;title&gt;{{ TAGLINE }}&lt;/title&gt; <span class="linenos" data-linenos="5 "></span>{% endif%} </code></pre></div> The <strong>else</strong> clause here is to ensure that non-article pages also get a title, even it&rsquo;s just your <strong>tagline</strong> defined in <code>pelicanconf.py</code>.</p> <h3 id="meta-descriptions">Meta descriptions<a class="headerlink" href="#meta-descriptions" title="Permanent link">&para;</a></h3> <p>Include a meta-description to be added on each page of your site. Though it may don&rsquo;t directly hit <span class="caps">SEO</span> ranking, but it appears as a snippet on the search page. So, user should get a proper glimpse of what your page gonna talk about. Make sure your theme uses Pelican&rsquo;s <code>summary</code>-tagged text for this purpose. Else, ensure it yourself by editing <code>base.html</code>.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>{% if article and article.summary %} <span class="linenos" data-linenos="2 "></span>&lt;meta name=&quot;description&quot; content=&quot;{{ article.summary|striptags }}&quot;/&gt; <span class="linenos" data-linenos="3 "></span>{% else %} <span class="linenos" data-linenos="4 "></span>&lt;meta name=&quot;description&quot; content=&quot;{{ SITE_SUMMARY }}&quot;/&gt; <span class="linenos" data-linenos="5 "></span>{% endif%} </code></pre></div> <h3 id="use-search-console-extensively">Use <code>search-console</code> extensively<a class="headerlink" href="#use-search-console-extensively" title="Permanent link">&para;</a></h3> <p>Google&rsquo;s <a href="https://www.google.com/webmasters/tools/home">Search-console, previously known as webmaster-tools</a> is your friend. Utilize it as far as you can&nbsp;go.</p> <h4 id="extensively-use-pagespeed-insights">Extensively use <a href="https://developers.google.com/speed/pagespeed/insights/">PageSpeed Insights</a><a class="headerlink" href="#extensively-use-pagespeed-insights" title="Permanent link">&para;</a></h4> <p>To understand where are current bottlenecks of your site, this tools gives quite a lot insights. Address those&nbsp;one-by-one.</p> <h4 id="set-preferred-version-of-your-site">Set preferred version of your site<a class="headerlink" href="#set-preferred-version-of-your-site" title="Permanent link">&para;</a></h4> <p>If you have <code>www</code>, <code>http</code> and <code>https</code> versions of your site, tell Google here which one is preferred. It&rsquo;s only applicable to your domain-root. Once applied and Google re-indexed your site, all the search-results from your site will show that preferred version of your&nbsp;site.</p> <p><img alt="setting preference for www or non-www version" src="http://i.imgur.com/51JY1oel.png"></p> <p>You have to add and do it same both for <code>http</code> and <code>https</code> version of your site, if you have&nbsp;both.</p> <p>If you have both, <strong>either</strong> you can use a javascript-code snippet in the <code>&lt;head&gt;</code> of <code>base.html</code> to redirect any <code>http</code> page to its <code>https</code> counterpart.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>&lt;script type=&quot;text/javascript&quot;&gt; <span class="linenos" data-linenos="2 "></span> var host = &quot;your-site.com&quot;; <span class="linenos" data-linenos="3 "></span> if ((host == window.location.host) &amp;&amp; (window.location.protocol != &quot;https:&quot;)) <span class="linenos" data-linenos="4 "></span> window.location.protocol = &quot;https&quot;; <span class="linenos" data-linenos="5 "></span>&lt;/script&gt; </code></pre></div> <p><strong>or</strong>, if your site is served through <span class="caps">NGINX</span> or Apache, you can do it through site&rsquo;s <code>.htaccess</code> file, by adding the&nbsp;following.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>RewriteEngine On <span class="linenos" data-linenos="2 "></span>RewriteCond %{HTTPS} !on <span class="linenos" data-linenos="3 "></span>RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} </code></pre></div> <p><strong>Or</strong>, if you are using CloudFlare <span class="caps">CDN</span>, you can create a page-rule for that as I have shown&nbsp;below.</p> <p><img alt="always-https by CloudFlare page-rules" src="http://i.imgur.com/9ISFbtvm.png"></p> <h4 id="check-index-status">Check index-status<a class="headerlink" href="#check-index-status" title="Permanent link">&para;</a></h4> <p>Once in a while Check for Google&rsquo;s index status of your site on the <code>search-console</code>. Look for error messages or&nbsp;suggestions.</p> <p>After every major change in your site&rsquo;s structure, make sure Google bots &ldquo;know&rdquo; about it. You can somewhat expedite the process by <a href="https://www.google.com/webmasters/tools/submit-url">manually submitting</a> your&nbsp;site.</p> <h3 id="include-opengraph-data">Include <code>OpenGraph</code> data<a class="headerlink" href="#include-opengraph-data" title="Permanent link">&para;</a></h3> <p>Make sure each of your pages is including proper <a href="http://ogp.me/">OpenGraph</a> tags, e.g. <code>og:title</code>, <code>og:content</code> etc., in your&nbsp;template.</p> <p>Though, OpenGraph originated from facebook Inc., these tags are now widely used by other social engines, even by Google+. In absence of Twitter tags, Twitter also uses these <code>og</code> tags. Try to include both <code>og:</code> and <code>twitter:</code> tags. Proper data in these tags makes your article cleanly-sharable in these social&nbsp;sites.</p> <p>The below snippet that <a href="https://github.com/kmonsoor/blog.kmonsoor.com/blob/pelican-how-to-make-seo-friendly/plumage/templates/base.html">I use myself</a> can serve as a starting&nbsp;point.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span>&lt;!-- OpenGraph protocol tags: http://ogp.me/ --&gt; <span class="linenos" data-linenos=" 2 "></span>&lt;!-- originally adopted to be used for: https://blog.kmonsoor.com --&gt; <span class="linenos" data-linenos=" 3 "></span>&lt;meta property=&quot;og:site_name&quot; content=&quot;{{ SITENAME }}&quot; /&gt; <span class="linenos" data-linenos=" 4 "></span>&lt;meta property=&quot;og:type&quot; content=&quot;article&quot; /&gt; <span class="linenos" data-linenos=" 5 "></span>{% if article and article.title %} <span class="linenos" data-linenos=" 6 "></span>&lt;meta property=&quot;og:title&quot; content=&quot;{{ article.title }} -- {{ TAGLINE }}&quot; /&gt; <span class="linenos" data-linenos=" 7 "></span>&lt;meta property=&quot;og:url&quot; content=&quot;{{ SITEURL }}/{{ article.url }}&quot; /&gt; <span class="linenos" data-linenos=" 8 "></span>{% endif%} <span class="linenos" data-linenos=" 9 "></span>{% if article and article.summary %} <span class="linenos" data-linenos="10 "></span>&lt;meta property=&quot;og:description&quot; content=&quot;{{ article.summary|striptags }}&quot; /&gt; <span class="linenos" data-linenos="11 "></span>{% else %} <span class="linenos" data-linenos="12 "></span>&lt;meta name=&quot;og:description&quot; content=&quot;{{ SITE_SUMMARY }}&quot;/&gt; <span class="linenos" data-linenos="13 "></span>{% endif%} <span class="linenos" data-linenos="14 "></span>{% if article and article.date %} <span class="linenos" data-linenos="15 "></span>&lt;meta property=&quot;article:published_time&quot; content=&quot;{{ article.date }}&quot; /&gt; <span class="linenos" data-linenos="16 "></span>{% endif%} <span class="linenos" data-linenos="17 "></span>{% if article and article.modified %} <span class="linenos" data-linenos="18 "></span>&lt;meta property=&quot;article:modified_time&quot; content=&quot;{{ article.modified }}&quot; /&gt; <span class="linenos" data-linenos="19 "></span>{% endif%} <span class="linenos" data-linenos="20 "></span>&lt;!-- End of OpenGraph protocol tags --&gt; <span class="linenos" data-linenos="21 "></span> <span class="linenos" data-linenos="22 "></span>{% if TWITTER_USERNAME %} <span class="linenos" data-linenos="23 "></span>&lt;meta name=&quot;twitter:site&quot; content=&quot;@{{ TWITTER_USERNAME }}&quot; /&gt; <span class="linenos" data-linenos="24 "></span>&lt;meta name=&quot;twitter:creator&quot; content=&quot;@{{ TWITTER_USERNAME }}&quot; /&gt; <span class="linenos" data-linenos="25 "></span>{% endif%} <span class="linenos" data-linenos="26 "></span>&lt;meta name=&quot;twitter:image&quot; content=&quot;INSERT-YOUR-TWITTER-IMAGE-LINK&quot; /&gt; <span class="linenos" data-linenos="27 "></span>{% if article and article.summary %} <span class="linenos" data-linenos="28 "></span>&lt;meta name=&quot;twitter:card&quot; content=&quot;{{ article.summary|striptags }}&quot; /&gt; <span class="linenos" data-linenos="29 "></span>{% else %} <span class="linenos" data-linenos="30 "></span>&lt;meta name=&quot;twitter:card&quot; content=&quot;{{ SITE_SUMMARY }}&quot;/&gt; <span class="linenos" data-linenos="31 "></span>{% endif%} </code></pre></div> <h4 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">&para;</a></h4> <ul> <li>Grab your own Twitter&rsquo;s avatar link do the&nbsp;following:</li> <li>Go to your Twitter profile&nbsp;page</li> <li>Right-click on your profile&nbsp;picture</li> <li> <p>Select &ldquo;Copy image address&rdquo; / &ldquo;Copy image&nbsp;link&rdquo;</p> </li> <li> <p>For <code>OpenGraph</code> tags you may also consider to use <a href="https://github.com/whiskyechobravo/pelican-open_graph/tree/master">pelican-opengraph</a>&nbsp;plugin.</p> </li> <li> <p>For all these to work properly, make sure <code>SITEURL</code>, <code>TAGLINE</code>, <code>SITE_SUMMARY</code>, <code>TWITTER_USERNAME</code> are properly defined in your <code>pelicanconf.py</code> alongwith in <code>publishconf.py</code> files. Please remember that definitions in <code>publishconf.py</code> only apply when you using <code>make publish</code> command.</p> </li> </ul> <h3 id="loading-performance">Loading Performance<a class="headerlink" href="#loading-performance" title="Permanent link">&para;</a></h3> <h4 id="compress-everything">Compress everything<a class="headerlink" href="#compress-everything" title="Permanent link">&para;</a></h4> <ul> <li>PageSpeed impacts <span class="caps">SEO</span> directly. Google punishes slow-site especially when search is made on a mobile device. Mobile-optimized sites will definitely rank higher on searches from&nbsp;mobile-devices.</li> </ul> <p>So, make sure all static files are compressed. If not, compress your themes theme&rsquo;s <span class="caps">JS</span>, <span class="caps">CSS</span> files yourself to a <em>.min.</em> version and then replace those in the template files of the&nbsp;theme.</p> <p>Or, better to use <a href="https://github.com/getpelican/pelican-plugins/tree/master/gzip_cache">gzip_cache</a> for gzipping all the <span class="caps">HTML</span> files statically, also and <a href="https://github.com/getpelican/pelican-plugins/tree/master/yuicompressor">yuicompressor</a> plugin for compressing <span class="caps">JS</span> <span class="amp">&amp;</span> <span class="caps">CSS</span> files for Pelican. Those will make sure that, upon build, everything is&nbsp;compressed.</p> <h4 id="utilize-cdn-if-you-can">Utilize <span class="caps">CDN</span> if you can<a class="headerlink" href="#utilize-cdn-if-you-can" title="Permanent link">&para;</a></h4> <ul> <li> <p>Use <span class="caps">CDN</span>-ed versions of common libraries(e.g. jQuery, Bootstrap etc.) rather than hosting your own copy, unless your theme actively modified it. Look it up on <a href="cdnjs.cloudflare.com">CloudFlare cdnjs</a>, <a href="https://cdnjs.com">cdnjs</a>, or on <a href="https://www.jsdelivr.com">jsdelivr</a> etc. and use those&nbsp;links.</p> </li> <li> <p>Try to use a <span class="caps">CDN</span> for edge-distribution of your site. I only know of CloudFlare that provide this service for free for a single site. There might be others. <span class="caps">CF</span> also make managing <span class="caps">DNS</span> configuration little&nbsp;breezy.</p> </li> </ul> <h3 id="engage-commenting">Engage commenting<a class="headerlink" href="#engage-commenting" title="Permanent link">&para;</a></h3> <p>While serving a static site, integrating a commenting-system looks a little far-fetched. However, blogs without proper commenting system feels kinda lame sometimes. Of course, <span class="caps">YMMV</span>.</p> <p>But, it&rsquo;s not difficult; easily can be done by systems like <a href="https://disqus.com/">Disqus</a> etc. I&rsquo;m not affiliated with them, by the&nbsp;way.</p> <h3 id="host-images-separately">Host images separately<a class="headerlink" href="#host-images-separately" title="Permanent link">&para;</a></h3> <p>Host all the images separately that you&rsquo;ve used in your articles. Use image-specific hosting e.g. imgur.com, imgpile.com, UltraIMG.com, postimage.org&nbsp;etc.</p> <p>But, why? Because, these services provide couple of benefits besides being&nbsp;free.</p> <ul> <li>Firstly, while loading the page, browser can parallelize loading from this hosts rather than your original blog&nbsp;hosts.</li> <li>More often than not, these services use own <span class="caps">CDN</span>.</li> <li>Often these services resize your uploaded images automatically to be used in different contexts, which enables you to choose the best-fit size on the fly but without doing it by&nbsp;hand.</li> </ul> <h2 id="other-tips">Other tips<a class="headerlink" href="#other-tips" title="Permanent link">&para;</a></h2> <h3 id="name-your-images-properly">Name your images properly<a class="headerlink" href="#name-your-images-properly" title="Permanent link">&para;</a></h3> <p>Because search-engines index images too. With proper names, images becomes relevant with the topic, hence potential to draw&nbsp;traffic.</p> <h3 id="use-google-keyword-planner">Use Google Keyword planner<a class="headerlink" href="#use-google-keyword-planner" title="Permanent link">&para;</a></h3> <p>Even if you are not willing to blow money on ads, it will immensely help you to find out more searched for&nbsp;keywords.</p> <h3 id="page-headers-controversial">Page-headers (controversial)<a class="headerlink" href="#page-headers-controversial" title="Permanent link">&para;</a></h3> <ul> <li>Don&rsquo;t use multiple 1<sup>st</sup>-level headers <code>H1</code> style. The main title link probably already have used it once. Look it up. So, avoid it anymore, meaning avoid underlined-style(<code>=======</code>) or hash-style(single &lsquo;#&rsquo;) headers in your markdown&nbsp;files.</li> </ul> <p>However, 2<sup>nd</sup>-level <code>H2</code> tags can be (read &lsquo;should be&rsquo;) used multiple times. In case of markdown files, that&rsquo;s <code>---------</code> underlines, or line starting with&nbsp;double-hash(&lsquo;##&rsquo;).</p> <h3 id="and-theres-lot-more">And, there&rsquo;s lot more &hellip; ;)<a class="headerlink" href="#and-theres-lot-more" title="Permanent link">&para;</a></h3> <p>As I&rsquo;ll gain more insights, I hope to grow this post. For now, this is a work-in-progress.<br> Thanks for reading down so far. Adios&nbsp;!</p>Ubuntu - Update APT-GET Server to Use Fastest Mirror2016-10-10T00:00:00+06:002016-10-10T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2016-10-10:/ubuntu-update-apt-get-server-to-use-fastest-mirror/<p>Updating Ubuntu from it&rsquo;s default server takes too damn long. Give it a 10x&nbsp;boost.</p><h2 id="prelude">Prelude<a class="headerlink" href="#prelude" title="Permanent link">&para;</a></h2> <p>By default, Ubuntu sets the update server pointing to its own (<a href="http://archive.ubuntu.com">http://archive.ubuntu.com</a>). It is the safest bet for Ubuntu. But, that&rsquo;s not the case for users, especially who are outside <span class="caps">USA</span>. To make it smoother as well as distribute the load, Ubuntu also provides a list of mirror sites. You can find the official mirror list <a href="https://launchpad.net/ubuntu/+archivemirrors">here</a>. </p> <p>However, it needs some&nbsp;configuration.</p> <h2 id="on-gui">On <span class="caps">GUI</span><a class="headerlink" href="#on-gui" title="Permanent link">&para;</a></h2> <p>Yes, you can do the selection on Ubuntu using its <span class="caps">GUI</span> tool. But, the problem is it don&rsquo;t always work as you&nbsp;want.</p> <blockquote> <p>It works on geolocation, giving me the local server, which is waaaayy slower where I am. The network temporal distance is the important factor here, not spatial distance (<a href="http://askubuntu.com/a/9035/113604">http://askubuntu.com/a/9035/113604</a>)</p> </blockquote> <p><img alt="Ubuntu-select-update-server-GUI" src="http://i.imgur.com/sCWr0zrl.png"></p> <h2 id="so-what-else-do-we-have">so, what else do we have ?<a class="headerlink" href="#so-what-else-do-we-have" title="Permanent link">&para;</a></h2> <p>I use a handy tool for this purpose to point me to the fastest server <span class="caps">FOR</span> <span class="caps">ME</span>. It&rsquo;s <a href="https://github.com/jblakeman/apt-select">apt-select</a>.</p> <p>To install it, i found this method most hassle-free. You may also try <code>pip install</code> method described on the repo&rsquo;s <span class="caps">README</span>&nbsp;file.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>git clone https://github.com/jblakeman/apt-select.git </code></pre></div> <p>That&rsquo;s it. Now to execute it, just run the &ldquo;main&rdquo; Python&nbsp;file.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span><span class="nb">cd</span> apt-select <span class="hll"><span class="linenos" data-linenos="2 "></span><span class="gp">$ </span>./apt-select.py -c -t <span class="m">3</span> -m one-week-behind </span></code></pre></div> <p>We are choosing here the best 3 mirrors(due to <code>-t 3</code>) which are at most <code>one week behind</code> from the main Ubuntu server. For general purposes, that&rsquo;s good&nbsp;enough.</p> <p>It will choose one using the latency <span class="amp">&amp;</span> ping time, and also show servers&rsquo; bandwidth. Then the tool asks you to select new mirror from the <code>3</code> options came up. Usually, stick to the top option, hence <code>1</code>. For example in my case, it&nbsp;shows:</p> <div class="highlight"><pre><span></span><code><span class="hll"><span class="linenos" data-linenos=" 1 "></span><span class="gp">[khaled:~] $ </span>apt-select -c -t <span class="m">3</span> -m one-week-behind </span><span class="linenos" data-linenos=" 2 "></span><span class="go">Getting list of mirrors...done.</span> <span class="linenos" data-linenos=" 3 "></span><span class="go">Testing latency to mirror(s)</span> <span class="linenos" data-linenos=" 4 "></span><span class="go">[3/3] 100%</span> <span class="linenos" data-linenos=" 5 "></span><span class="go">Getting list of launchpad URLs...done.</span> <span class="linenos" data-linenos=" 6 "></span><span class="go">Looking up 3 status(es)</span> <span class="linenos" data-linenos=" 7 "></span><span class="go">[3/3] 100%</span> <span class="linenos" data-linenos=" 8 "></span><span class="go">1. mirror.dhakacom.com (current)</span> <span class="linenos" data-linenos=" 9 "></span><span class="go"> Latency: 1.89 ms</span> <span class="linenos" data-linenos="10 "></span><span class="go"> Org: dhakaCom Limited</span> <span class="linenos" data-linenos="11 "></span><span class="go"> Status: Up to date</span> <span class="linenos" data-linenos="12 "></span><span class="go"> Speed: 1 Gbps</span> <span class="linenos" data-linenos="13 "></span><span class="go">2. mirror.dhakacom.com</span> <span class="linenos" data-linenos="14 "></span><span class="go"> Latency: 1.96 ms</span> <span class="linenos" data-linenos="15 "></span><span class="go"> Org: dhakaCom Limited</span> <span class="linenos" data-linenos="16 "></span><span class="go"> Status: Up to date</span> <span class="linenos" data-linenos="17 "></span><span class="go"> Speed: 1 Gbps</span> <span class="linenos" data-linenos="18 "></span><span class="go">3. archive.ubuntu.com</span> <span class="linenos" data-linenos="19 "></span><span class="go"> Latency: 205.23 ms</span> <span class="linenos" data-linenos="20 "></span><span class="go"> Org: Canonical Ltd.</span> <span class="linenos" data-linenos="21 "></span><span class="go"> Status: Up to date</span> <span class="linenos" data-linenos="22 "></span><span class="go"> Speed: 100 Mbps</span> <span class="linenos" data-linenos="23 "></span><span class="go">Choose a mirror (1 - 3)</span> <span class="linenos" data-linenos="24 "></span><span class="go">&#39;q&#39; to quit 1</span> <span class="linenos" data-linenos="25 "></span><span class="go">New config file saved to /home/khaled/sources.list</span> </code></pre></div> <p>Then, the tool prepares a new <code>sources.list</code> file to replace the system&rsquo;s one&nbsp;with.</p> <p>Now, replace the file with the new one which will be now used by the <code>apt-get *</code> commands. As it overwrites a system config file, you need to do with <code>sudo</code>.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo mv /etc/apt/sources.list /etc/apt/sources.list.backup <span class="o">&amp;&amp;</span> sudo mv sources.list /etc/apt/ </code></pre></div> <p>That&rsquo;s it. Now, all you update/upgrade will be fetched from the newly selected&nbsp;mirror.</p> <p>However, I highly recommend to run <code>update</code> once right&nbsp;away.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo apt-get update </code></pre></div> <p>Then, only if you want, run <code>upgrade</code>:</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span><span class="gp">$ </span>sudo apt-get upgrade </code></pre></div> <hr> <h2 id="you-may-also-like">You may also like<a class="headerlink" href="#you-may-also-like" title="Permanent link">&para;</a></h2> <ul> <li><a href="https://blog.kmonsoor.com/open-source-as-if-you-gonna-die-tonight/?utm_source=related_footer&amp;utm_keyword=coding">Open Source as-if You Gonna Die&nbsp;Tonight</a></li> <li><a href="https://blog.kmonsoor.com/pelican-how-to-make-seo-friendly/?utm_source=related_footer&amp;utm_keyword=python">Pelican Static sites - <span class="caps">SEO</span>&nbsp;Optimization</a></li> <li><a href="https://blog.kmonsoor.com/generate-er-diagram-from-sql-database/?utm_source=related_footer&amp;utm_keyword=coding">Generate <span class="caps">ER</span> diagram from a <span class="caps">SQL</span>-based&nbsp;database</a></li> </ul>