I think ...https://blog.kmonsoor.com/2021-06-06T00:00:00+06:00Create a free go-link server “on edge” using Cloudflare Worker KV2021-06-06T00:00:00+06:002021-06-06T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-06-06:/golink-server-using-cloudflare-worker-kv/<p>Among quite a few ways to implement a go-link server (i.e. url-forwarder, short-url server etc.), here I&rsquo;m going to show you how to use free-tier Cloudflare Worker (&amp; <span class="caps">KV</span>) to create an in-house, on-edge, <strong>no-webserver</strong> go-link&nbsp;server.</p><p>Among quite a few ways to implement a go-link server (i.e. url-forwarder, short-url server etc.), here I&rsquo;m going to show you how to use free-tier Cloudflare Worker (&amp; <span class="caps">KV</span>) to create an in-house, on-edge, <strong>no-webserver</strong> go-link&nbsp;server.</p> <p>For example, the short-link for this article is <a href="https://go.kmonsoor.com/golink-kv">go.kmonsoor.com/golink-kv</a> </p> <p><img alt="overall structure" src="https://i.imgur.com/MjIS5gD.png"></p> <ul> <li><code>/latest</code> (by which I mean <code>go.yourdomain.co/latest</code>) may point to <code>https://www.yourcompany.com/about/news</code> which is a public&nbsp;page</li> <li><code>/hr-help</code> may point to <code>https://www.company-internal.com/long-link/hr/contact.html</code> which is company&rsquo;s internal human-resources help&nbsp;portal</li> <li><code>/cnypromo</code> may point to <code>https://shop.yourcompany.com/sales/promotions/?marketing-promo=2021-cny</code> which is a temporary sales promotions page targeting the shoppers during the Chinese new year of&nbsp;2021.</li> </ul> <p>Please note that using the setup and the code below, it&rsquo;ll be possible resolve shortlinks via a <strong>single</strong> sub-domain, e.g. <code>go.your-domain.co</code>. However, it&rsquo;s totally possible (with some modification of the code) to resolve/re-direct via <em>any number of domains</em> (your own, of course) towards any other public or private <span class="caps">URL</span>, and all sorts of novelties. However, for brevity&rsquo;s sake, I&rsquo;m going to discuss the first one, single sub-domain&nbsp;usecase.</p> <p>To setup a go-link server or short-<span class="caps">URL</span> resolver via a proper <span class="caps">KV</span>+Worker combination, we&rsquo;ll go through these&nbsp;steps:</p> <div class="toc"> <ul> <li><a href="#pre-requisites">Pre-requisites</a></li> <li><a href="#create-the-short-link-map-as-a-kv">Create the short-link map as a <span class="caps">KV</span></a></li> <li><a href="#mapping-a-kv-to-a-worker-variable">Mapping a <span class="caps">KV</span> to a Worker&nbsp;variable</a></li> <li><a href="#handling-a-route-with-webworker">Handling a route with&nbsp;webworker</a></li> <li><a href="#create-the-worker">Create the&nbsp;Worker</a></li> <li><a href="#pointing-a-dns-record-to-the-worker">Pointing a <span class="caps">DNS</span> record to the&nbsp;Worker</a></li> <li><a href="#next-step">Next&nbsp;step</a></li> <li><a href="#related">Related</a></li> </ul> </div> <h1 id="pre-requisites">Pre-requisites<a class="headerlink" href="#pre-requisites" title="Permanent link">&para;</a></h1> <ul> <li>The <span class="caps">DNS</span> resolver for the <strong>root</strong> domain (in the example below, <em><code>kmonsoor.com</code></em>) needs to be Cloudflare. Because the core of the solution, the &ldquo;worker&rdquo;, runs on the nearest (from the user) edge of Cloudflare using a common <span class="caps">KV</span> (&ldquo;key, value&rdquo;)&nbsp;list.</li> <li>Write permission to the <span class="caps">DNS</span> configuration as you&rsquo;d need to add a new <span class="caps">AAAA</span> <span class="caps">DNS</span>&nbsp;record.</li> <li>Some knowledge of Javascript(<span class="caps">ES6</span>), as we gonna write the worker in that&nbsp;language.</li> </ul> <h1 id="create-the-short-link-map-as-a-kv">Create the short-link map as a <span class="caps">KV</span><a class="headerlink" href="#create-the-short-link-map-as-a-kv" title="Permanent link">&para;</a></h1> <p>We&rsquo;ll start the setup by creating the short-link map, the list between the short-link segments that you (or, someone in your org) define and the actual URLs they needed to point&nbsp;to.</p> <p>Find the <span class="caps">KV</span> stuff in the <code>Workers</code> section. From the screenshot, please ignore the &ldquo;Route&rdquo; section for&nbsp;now. </p> <p><img alt="Find the KV stuff in the Workers section" src="https://i.imgur.com/b2Rk45u.png"></p> <ul> <li>you&rsquo;d need to create a Worker <span class="caps">KV</span> &ldquo;Namespace&rdquo;. Name the namespace as you seem fit. I named it <code>REDIRECTS</code> (in all caps just as convention, not&nbsp;required). </li> <li>List the short-links <span class="amp">&amp;</span> their respective target URLs. From the examples in the intro, the keys <code>latest</code>, <code>hr-help</code>, <code>cnypromo</code> etc. would be in as the &ldquo;key&rdquo;, and the target full links as the respective&nbsp;&ldquo;value&rdquo;.</li> <li>Remember <span class="caps">NOT</span> to start the short part with &lsquo;/&rsquo;. In the code, it&rsquo;ll be taken care&nbsp;of.</li> </ul> <p><img alt="Create the short-link map as a KV" src="https://i.imgur.com/jkC8bSr.png"></p> <p>Once you&rsquo;ve listed all your desired (short-link, target-link) combinations, now we have a <span class="caps">KV</span> on CLoudflare. However, it&rsquo;s not referencable from your Worker code, not yet. Hence the next&nbsp;step.</p> <h1 id="mapping-a-kv-to-a-worker-variable">Mapping a <span class="caps">KV</span> to a Worker variable<a class="headerlink" href="#mapping-a-kv-to-a-worker-variable" title="Permanent link">&para;</a></h1> <p>Now, we gonna map the previously created <span class="caps">KV</span> to a variable that can be referenced from our Workeer code. Please note that though I used diffrent names, but it can be same as well. Also note that a single <span class="caps">KV</span> can be accessed by multiple workers, and vice versa, a single Worker can reference multiple&nbsp;KVs.</p> <p><img alt="Mapping a KV to a Worker variable" src="https://i.imgur.com/lb7G9si.png"></p> <h1 id="handling-a-route-with-webworker">Handling a route with webworker<a class="headerlink" href="#handling-a-route-with-webworker" title="Permanent link">&para;</a></h1> <p><img alt="Handling a route with webworker" src="https://i.imgur.com/KohHRfR.png"></p> <h1 id="create-the-worker">Create the Worker<a class="headerlink" href="#create-the-worker" title="Permanent link">&para;</a></h1> <p>Now, we going to write Workercode that runs on V8-runtime on an nearest(from the requesting user) edge location of Cloudflare to complete its functionality and deliver it to the user. Tn this case, that would be to redirect user&rsquo;s requested address to the mapped one by you, in the <span class="caps">KV</span> namespace&nbsp;above.</p> <p><img alt="Creating a worker" src="https://i.imgur.com/eNfZNyN.png"></p> <p>The code editor looks like&nbsp;this: </p> <p><img alt="The code editor for Cloudflare worker" src="https://i.imgur.com/pb9AE9v.png"></p> <p>If you rather prefer to copy-paste, please feel free to do it from the below GitHub&nbsp;Gist.</p> <div class="gist"> <script src="https://gist.github.com/kmonsoor/dc9f96660423c96471f8574ba018d867.js"></script> </div> <p>Once done, it should look like &hellip; <img alt="created webworker" src="https://i.imgur.com/XSdKB56.png"></p> <h1 id="pointing-a-dns-record-to-the-worker">Pointing a <span class="caps">DNS</span> record to the Worker<a class="headerlink" href="#pointing-a-dns-record-to-the-worker" title="Permanent link">&para;</a></h1> <p>Finally, we need to point a <span class="caps">DNS</span> record that&rsquo;ll redirect all requests to your re-soutign sub-domain (e.g. <code>go.your-domain.com</code>) to the Cloudflare Worker that we just&nbsp;created.</p> <p>According to the Cloudflare docs, the <span class="caps">DNNS</span> record must be an <span class="caps">AAAA</span> record, pointing to the IPv6 address <code>100:</code>. The &ldquo;Name&rdquo; here is the &ldquo;sub-domain&rdquo; part of your choice which is better be short to serve our goal&nbsp;here. </p> <p><img alt="Pointing a DNS record to it" src="https://i.imgur.com/62bk7pe.png"></p> <p>Voila ! Now, test some of the short-urls that you&rsquo;ve mapped via the <span class="caps">KV</span>. Enjoy ! Watch out for the target usage though <a href="https://developers.cloudflare.com/workers/platform/limits#worker-limits">against the limit</a>. </p> <p>I think you&rsquo;ll be fine, unnless you&rsquo;re some kind of celebrity&nbsp;;)</p> <h1 id="next-step">Next step<a class="headerlink" href="#next-step" title="Permanent link">&para;</a></h1> <p>As the next next step, I&rsquo;m thinking to create a generic <code>Go/Link</code> resolver browser extension. Then, someone can set their own default domain or company domain of choice as short-domain host. In that case, entering just <code>go/hr-help</code> on the browser will take to <code>https://www.company-internal.com/.../hr/contact.html</code> that we have discussed at the beginning (remember the example case of an internal human resources help&nbsp;portal?).</p> <h1 id="related">Related<a class="headerlink" href="#related" title="Permanent link">&para;</a></h1> <ul> <li>If you want to do this url-direction <strong>on your own webserver, but only using webserver</strong>, try this: <a href="https://blog.kmonsoor.com/personal-shortlink-server-using-Caddy/">Personal short-link server using only&nbsp;Caddyserver</a></li> </ul>TL;DR what cloud provider to use in 20212021-05-22T00:00:00+06:002021-05-22T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-05-22:/TLDR-what-cloud-to-use-2021/<p>Among the thousands of combinations a company can take to choose from the cloud providers and their products, this is my <span class="caps">TL</span>;<span class="caps">DR</span>&nbsp;suggestion</p><p>The sheer number of combinations a company can choose from the cloud providers and their product suites is mind-boggling. Hence, I decided to break it down in a concise form for the busy C-suite&nbsp;executives.</p> <p>According to my little experiences and humble opinion, I suggest&nbsp;&hellip;</p> <p>➤ If your company is a small SaaS shop with 10-ish engineers, stick with DigitalOcean, Linode, <span class="caps">OVH</span>, etc., which are best known as cloud &ldquo;instance&rdquo; providers.<br> Think McDonald&rsquo;s; reliable, cheapest, fast, but you won&rsquo;t take your date there. <br> <strong>Budget</strong>:&nbsp;💰</p> <p>➤ If you want a whole cloud experience (e.g., <span class="caps">VPC</span>, firewall, <span class="caps">WAF</span>, etc., on the menu), start with Google Cloud, then try <span class="caps">AWS</span> later.<br> Google Cloud would be the quickest to grasp the cloud concepts and get going. The <span class="caps">UI</span> of the <span class="caps">AWS</span> console is a bit messy compared to <span class="caps">GCP</span>; it just takes more time to get a proper grip.<br> Imagine them as full-course, Michelin-star restaurants. However, the product names are so abstract that they need a full-sized chart for that. ;) <strong>Budget</strong>:&nbsp;💰💰💰</p> <p>➤ Are you planning to set up a million-dollar infra for a billion-dollar company? Go for some <span class="caps">GCP</span>+<span class="caps">AWS</span> multi-cloud setup. You gonna get rebates from both on the scale of hundreds of thousands of dollars. And Microsoft Azure gonna offer you some million-$ free-tier, hoping to get the company hooked on Azure. :D <strong>Budget</strong>:&nbsp;💰💰</p> <p>➤ On the other hand, if you run a govt agency or a company where wearing suits is the mainstream, Microsoft Azure is your best bet.<br> A bunch of consultancy companies to choose from; you need to just approve the budget, you get the things to get up <span class="amp">&amp;</span> running but miss the deadline by months, if not years. But there&rsquo;d be no need for hiring more smarter ppl than that you already have. <strong>Budget</strong>:&nbsp;💰💰💸</p> <p>Need an even more comprehensive guide? Gotcha, fam&nbsp;&hellip;</p> <blockquote class="twitter-tweet"><p lang="en" dir="ltr"><span class="caps">CTO</span>: we&#39;re having hard time choosing a cloud provider<br>&#8230;<br>&#8220;say no more, fam, I gotcha &#8230;&#8221; <a href="https://t.co/hR3rMruWWi">pic.twitter.com/hR3rMruWWi</a></p>&mdash; Khaled Monsoor ✨ (@kmonsoor) <a href="https://twitter.com/kmonsoor/status/1395959443376857088?ref_src=twsrc%5Etfw">May 22, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <p><em><span class="caps">PS</span></em> This post is inspired by a LinkedIn post of mine where I shared about my short experience with the Microsoft Azure <strong>DevOps</strong>&nbsp;suite</p>Deploying a short-link aka go-link server using only Caddyserver2021-04-16T00:00:00+06:002021-04-16T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-04-16:/deploying-golink-server-using-Caddy/<p>Yeah, there are tons of open-source, full-fledged link-shorteners. But, none were exactly what I wanted. Hence, the minimal approach only utilizing the amazing webserver, <code>Caddy</code>. Here we go&nbsp;&hellip;</p><p>Before I go into any details, please note that I&rsquo;ve used the term &ldquo;shortlink-server&rdquo; instead of &ldquo;url-shortener&rdquo; because the difference is significant for this post. A url-shortener takes a long url, and gives a short url, then redirects any requests for the shortened link to its long counterpart. On the contrary, shortlink-server takes both the long and short url, then only does the redirect-ion&nbsp;part.</p> <h2 id="backstory">Backstory<a class="headerlink" href="#backstory" title="Permanent link">&para;</a></h2> <p>If I wanted an amazing, personal url-shortener or &ldquo;go link&rdquo; server, there are many amazing solutions out there, which are not only free or open-source but full-fledged as well for personal or public usage. Rather, I wanted some &ldquo;service&rdquo; that will &ldquo;resolve&rdquo; my personal, short links. I have been using bit.ly for a long time for its customizable &ldquo;short-half&rdquo; part, but the problem with bit.ly is &ndash; for some God-forsaken reason &ndash; blocked by the Bangladeshi govt. So, I needed a replacement to be properly &ldquo;glocal&rdquo;. There are many free (unbranded) and commercial (branded) options as well, but I wanted something that will be resolved via my hosted service (and personal domain) and as cheap as possible. So, basically solution for a poor nerd&nbsp;:D</p> <p>Before jumped into this solution, I tried (deployed <span class="amp">&amp;</span> tested) few others myself, mainly <a href="https://github.com/kellegous/go">kellegous/go</a>, <a href="kutt.it">kutt.it</a> and <a href="https://github.com/adamyi/golinks">adamyi/golinks</a>. But, all of them &ldquo;too featureful&rdquo; for my&nbsp;needs. </p> <p><img alt="Simple, on-prem short-link server using Caddy webserver" src="https://i.imgur.com/4nZbnUE.png"></p> <p>What I wanted is to be able&nbsp;to:</p> <ul> <li>resolve only my custom shortlinks (hence, no need for&nbsp;url-shortener)</li> <li>not a public, internet-facing service (hence, any frontend, authentication, email verification etc. would be overkill&nbsp;)</li> <li>minimal setup (if possible, no webapp at&nbsp;all)</li> </ul> <p>Given my previous experience with <code>Caddy</code> webserver, which is an amazing one(<a href="https://caddyserver.com/docs/">why?</a>), I had a gut feeling that Caddy has something for me &ndash; under the sleeve &ndash; to meet my minimal set of requirements. Thankfully, I managed to find&nbsp;it.</p> <p>I believe <span class="caps">NGINX</span>, currently the most popular webserver, has some kind of similar mechanism as well. But, I&rsquo;m not an expert, and once I was genuinely intimidated by its config file syntax. <span class="caps">YMMV</span>.</p> <h2 id="what-you-gonna-need">What you gonna need?<a class="headerlink" href="#what-you-gonna-need" title="Permanent link">&para;</a></h2> <ul> <li>your own domain which will be the root of the shortlinks. While sub-domained <span class="caps">URL</span> like <code>go.yourname.com/*</code> is quite common, if you have some short domain, like you.co/*, only for this purpose, that&rsquo;s fine as&nbsp;well.</li> <li>A webhost server or public-facing instance with its own, <strong>public</strong> IPv4&nbsp;address.</li> <li>working knowledge of&nbsp;Linux</li> </ul> <h2 id="step-1-point-your-subdomain-to-the-right-place">Step-1: Point your subdomain to the right place<a class="headerlink" href="#step-1-point-your-subdomain-to-the-right-place" title="Permanent link">&para;</a></h2> <ul> <li>Find out what&rsquo;s the <strong>puplic IPv4 address</strong> of your instance that&rsquo;ll act as the webserver. It&rsquo;s usually on the cloud management dashboard.<ul> <li>make sure that, regardless of your cloud architecture (e.g. <span class="caps">VPC</span>, subnet, firewall etc.), the <span class="caps">SSL</span> port (<code>:443</code>) of the instance is reachable from the public&nbsp;internet.</li> </ul> </li> <li>now go to your domain name registrar (or, <span class="caps">DNS</span> management provider which in my case is Cloudflare). There, you need to point shortlink subdomain (<code>go.</code>)to the webserver&rsquo;s <span class="caps">IP</span>&nbsp;address.</li> </ul> <p>Actually, you can do this step at the last. But for some reason, I prefer it to do first. Because sometimes, <a href="https://blog.cloudflare.com/never-deal-with-dns-propagation-again/"><span class="caps">DNS</span> propagation</a> takes some time. But, once my webservice is up and running, I like to see the result instantaneously.&nbsp;;)</p> <h2 id="step-2-install-caddy-the-mighty-webserver">Step-2: Install Caddy, the mighty webserver<a class="headerlink" href="#step-2-install-caddy-the-mighty-webserver" title="Permanent link">&para;</a></h2> <p>Depending on your host <span class="caps">OS</span> (mine is Ubuntu 20.04 <span class="caps">LTS</span>), you need to install the <code>Caddy</code> webserver. While there are some hacky solutions to run, I think running Caddy as a background server is the simplest to manage. In fact, the documentation of Caddy is excellent, I&rsquo;d better leave that part to&nbsp;you. </p> <p>After running with the default config(<code>Caddyfile</code>) which is in Ubuntu&rsquo;s case located as <code>/etc/caddy/Caddyfile</code>, it should have a status somewhat like the below image. Please note that, in many cases, if running without <code>sudo</code> Caddy cannot attach itself with the <span class="caps">SSL</span> port (<code>:443</code>) which is necessary for serving <code>https://</code>. So, check for that error message in the &ldquo;status&rdquo;&nbsp;log.</p> <p><img alt="Caddy service on Ubuntu" src="https://i.imgur.com/cfS5nvZ.png?1"></p> <p><em><strong><span class="caps">P.S.</span></strong> By the way, want your console and command prompt to look 🚀 like mine? Here&rsquo;s the guide: <a href="https://blog.kmonsoor.com/pimp-up-my-terminal/">How do I pimp up my terminal on&nbsp;Linux</a></em></p> <h2 id="step-3-tell-caddy-your-short-links-to-redirect">Step-3: Tell Caddy your short-links to redirect<a class="headerlink" href="#step-3-tell-caddy-your-short-links-to-redirect" title="Permanent link">&para;</a></h2> <p>Now, it&rsquo;s time to configure Caddy to actually do the&nbsp;job.</p> <p>Caddy has its native <code>redir</code> <span class="dquo">&ldquo;</span>directive&rdquo; to redirect incoming web-request from one to another. While the <code>map</code> directive is relatively new, it makes the config file i.e. Caddyfile look elegant in case you have (or, will have in the long run) a long list of&nbsp;short-links.</p> <p>Here&rsquo;s mine which is working&nbsp;&hellip; </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos=" 1 "></span># /etc/caddy/Caddyfile <span class="linenos" data-linenos=" 2 "></span> <span class="linenos" data-linenos=" 3 "></span>go.kmonsoor.com { # replace it your web-url root <span class="linenos" data-linenos=" 4 "></span> <span class="linenos" data-linenos=" 5 "></span> map {path} {redirect-uri} { <span class="linenos" data-linenos=" 6 "></span> /blog https://blog.kmonsoor.com <span class="linenos" data-linenos=" 7 "></span> /photos https://photos.kmonsoor.com <span class="linenos" data-linenos=" 8 "></span> <span class="linenos" data-linenos=" 9 "></span> /resume https://drive.google.com/file/d/1nMS3i1ai6nsI70zZ7NFnNQ_XmvAa4GOl <span class="linenos" data-linenos="10 "></span> /resume-doc https://docs.google.com/document/d/1ECx1Yr8Jzz9I3S5VcoKnZQz56oIht2XaM5gSNetcWag <span class="linenos" data-linenos="11 "></span> <span class="linenos" data-linenos="12 "></span> /rickrolled https://www.youtube.com/watch?v=dQw4w9WgXcQ <span class="linenos" data-linenos="13 "></span> <span class="linenos" data-linenos="14 "></span> # will add new ones here like the above <span class="linenos" data-linenos="15 "></span> # ... <span class="linenos" data-linenos="16 "></span> } <span class="linenos" data-linenos="17 "></span> <span class="linenos" data-linenos="18 "></span> # this below code is required to actually make the above `map` work <span class="linenos" data-linenos="19 "></span> <span class="linenos" data-linenos="20 "></span> @hasRedir expression `{redirect-uri} != &quot;&quot;` <span class="linenos" data-linenos="21 "></span> redir @hasRedir {redirect-uri} <span class="linenos" data-linenos="22 "></span> <span class="linenos" data-linenos="23 "></span> # code below is to set the default response if the requested shortlink isn&#39;t here <span class="linenos" data-linenos="24 "></span> respond &quot;Thas&#39;s an unknown short URL ... :(&quot; <span class="linenos" data-linenos="25 "></span>} </code></pre></div> <p>Note: Don&rsquo;t forget to restart the <code>caddy</code> service to let the new config to take&nbsp;effect.</p> <h2 id="step-4-profit">Step-4: Profit !<a class="headerlink" href="#step-4-profit" title="Permanent link">&para;</a></h2> <p>Yeah, that&rsquo;s it. Now, add some own personal stuff with some cool short-links, and proudly share with the&nbsp;world.</p> <h2 id="whats-next">What&rsquo;s next ?<a class="headerlink" href="#whats-next" title="Permanent link">&para;</a></h2> <p>I&rsquo;m thinking that given the very low workload my shortlink resolver needs — unless I&rsquo;m becoming an overnight internet sensation — using a server instance only for this purpose is overkill. My next goal is to have the same service using some &ldquo;serverless&rdquo; function or using the &ldquo;<a href="https://developers.cloudflare.com/workers/examples/redirect">worker on the edge</a>&rdquo; thing from Cloudflare. Let&rsquo;s see&nbsp;;)</p>How do I Pimp up My Terminal on Linux2021-03-31T00:00:00+06:002021-03-31T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2021-03-31:/pimp-up-my-terminal/<p>How do I pimp up my Linux terminal? A quick trip through Zsh, Oh-my-zsh, and other power tools to make the command-line-based workflow smooth and&nbsp;cool.</p><p>The purpose of this post is to be my quick, copy-paste source of the commands that I use to set up my terminal on a new *nix system. However, if someone else finds it useful, that&rsquo;d be some cherries on&nbsp;top.</p> <p>This command prompt in the below image is the end&nbsp;goal.</p> <p><img alt="The end goal of this post" src="https://i.imgur.com/YqnBifw.png"></p> <p>Assuming I&rsquo;m on a standard Linux machine with Ubuntu and I have <span class="caps">CLI</span> access. For other Linux distros or <em>MacOS</em>, some commands might be slightly&nbsp;different.</p> <h2 id="step-1-confirm-that-zsh-is-up-to-date">Step-1: Confirm that Zsh is up-to-date<a class="headerlink" href="#step-1-confirm-that-zsh-is-up-to-date" title="Permanent link">&para;</a></h2> <p>While on some Linux systems, Zsh is present by default, on some it&rsquo;s not the case. So, let&rsquo;s make sure about&nbsp;it.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo apt install zsh </code></pre></div> <p>Confirm the version. Oh-my-zsh recommends Zsh to be <code>5.0.8</code> or&nbsp;higher.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ zsh --version <span class="linenos" data-linenos="2 "></span>zsh 5.8 (x86_64-ubuntu-linux-gnu) </code></pre></div> <p>Also, you gotta make sure that <code>git</code> (recommended v2.4.11 or higher) is also installed on the&nbsp;system.</p> <h2 id="step-2-install-oh-my-zsh-the-fun-configuration-framework">Step-2: Install Oh-my-zsh, the fun &ldquo;configuration&rdquo; framework<a class="headerlink" href="#step-2-install-oh-my-zsh-the-fun-configuration-framework" title="Permanent link">&para;</a></h2> <p>Install directly from the&nbsp;source.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sh -c &quot;$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)&quot; </code></pre></div> <p>In the last step of this installation, it will ask to set Zsh as <span class="caps">THE</span> shell. Go&nbsp;ahead.</p> <p>Now we have the default prompt from <code>Oh-my-zsh</code>. </p> <p><img alt="After successful installation of Oh-my-zsh" src="https://i.imgur.com/HOVqqvi.png"></p> <p>Now, let&rsquo;s pimp up the prompt. Shall&nbsp;we?</p> <h2 id="step-3-install-powerlevel10k-a-powerful-prompt-theme">Step-3: Install <code>powerlevel10k</code>, a powerful prompt theme<a class="headerlink" href="#step-3-install-powerlevel10k-a-powerful-prompt-theme" title="Permanent link">&para;</a></h2> <p>I love the powerful Zsh theme <code>powerlevel10k</code>. More on <a href="https://github.com/romkatv/powerlevel10k#features">why this theme</a> is&nbsp;awesome.</p> <p>Let&rsquo;s install it on top of <code>oh-my-zsh</code>.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ git clone --depth=1 \ <span class="linenos" data-linenos="2 "></span> https://github.com/romkatv/powerlevel10k.git \ <span class="linenos" data-linenos="3 "></span> ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k </code></pre></div> <p>Now, gotta set <code>ZSH_THEME="powerlevel10k/powerlevel10k"</code> in <code>~/.zshrc</code> by adding that manually in the&nbsp;file.</p> <h2 id="step-4-make-sure-the-prompt-looks-like-as-you-want">Step-4: Make sure the prompt looks like as you want<a class="headerlink" href="#step-4-make-sure-the-prompt-looks-like-as-you-want" title="Permanent link">&para;</a></h2> <p>In this step, I&rsquo;m gonna bring in my already open-sourced Zsh config file aka <code>.zshrc</code>. </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span># deleting the current one &amp; get my personal one from GitHub <span class="linenos" data-linenos="2 "></span>$ rm .zshrc <span class="linenos" data-linenos="3 "></span>$ <span class="linenos" data-linenos="4 "></span>$ wget https://raw.githubusercontent.com/kmonsoor/dot-files/master/.zshrc </code></pre></div> <p>I kept the powerlevel10k configs as comments so that Zsh doesn&rsquo;t complain if I use the config file early. Have to set <code>ZSH_THEME="powerlevel10k/powerlevel10k"</code> in the <code>~/.zshrc</code> as&nbsp;well.</p> <p>Otherwise, once the <code>powerlevel10k</code> theme will run for the first time by Zsh, a very friendly step-by-step prompt will run you through towards a desirable prompt for you. Also, whenever you want, you can invoke the config-wizard by executing <code>p10k configure</code> on the&nbsp;shell.</p> <p>Now is the time to enable the changes by restarting Zsh and enjoy the new config and the powerful&nbsp;prompt.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ exec zsh </code></pre></div> <h2 id="optional">Optional<a class="headerlink" href="#optional" title="Permanent link">&para;</a></h2> <p>Also, I usually install this very useful, but external plugin <code>fast-syntax-highlighting</code> for&nbsp;oh-my-zsh.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ git clone https://github.com/zdharma/fast-syntax-highlighting.git \ <span class="linenos" data-linenos="2 "></span> ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting </code></pre></div>Install the latest Zsh on CentOS2020-10-20T00:00:00+06:002020-10-20T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2020-10-20:/install-latest-Zsh-on-CentOS/<p>If you&rsquo;re looking forward to install the latest version of Zsh instead of the default old one, here you&nbsp;go.</p><p>As the <strong>default</strong> Zsh on CentOS is usually a older version, many cool stuffs are not possible on this Zsh, like installing <a href="https://ohmyz.sh/">oh-my-zsh</a> or using <a href="https://github.com/romkatv/powerlevel10k">powerlevel10k</a> cool prompt system, it&rsquo;s understandable if you&rsquo;d like to have the latest Zsh on board. <br> Easy peasy&nbsp;!! </p> <p><strong>Note</strong>: Please remember to remove the &ldquo;sudo&rdquo; from the commands if you are already in &ldquo;root&rdquo; or sudo-er&nbsp;mode </p> <p>We&rsquo;ll be following these&nbsp;steps:</p> <div class="toc"> <ul> <li><a href="#install-the-pre-requisites">Install the&nbsp;pre-requisites</a></li> <li><a href="#download-the-latest-source">Download the latest&nbsp;source</a></li> <li><a href="#build-install">Build <span class="amp">&amp;</span>&nbsp;Install</a></li> <li><a href="#final-steps">Final&nbsp;steps</a></li> <li><a href="#related">Related</a></li> </ul> </div> <h1 id="install-the-pre-requisites">Install the pre-requisites<a class="headerlink" href="#install-the-pre-requisites" title="Permanent link">&para;</a></h1> <p>We need <span class="caps">GCC</span> (C++ compiler) and other related stuffs for building Zsh from the source&nbsp;code. </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo yum groupinstall <span class="s2">&quot;Development tools&quot;</span> <span class="linenos" data-linenos="2 "></span>$ sudo yum install ncurses-devel </code></pre></div> <p>Now, check if <span class="caps">GCC</span> is installed properly, by<br> <code>$ gcc -v</code> </p> <h1 id="download-the-latest-source">Download the latest source<a class="headerlink" href="#download-the-latest-source" title="Permanent link">&para;</a></h1> <p>Now, we gonna get the latest code of Zsh.<br> Please update the link (in the shown command) with the latest by checking <a href="https://www.zsh.org/pub/">this web-folder</a>.<br> Don&rsquo;t forget to update the filename as well, if needed. <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ <span class="nb">cd</span> /usr/local/src <span class="linenos" data-linenos="2 "></span>$ sudo curl -L https://www.zsh.org/pub/zsh-5.8.tar.xz <span class="se">\</span> <span class="linenos" data-linenos="3 "></span>-o zsh-5.8.tar.xz </code></pre></div></p> <h1 id="build-install">Build <span class="amp">&amp;</span> Install<a class="headerlink" href="#build-install" title="Permanent link">&para;</a></h1> <p>Unzip the file, &ldquo;dig in&rdquo; to the folder, and build <span class="amp">&amp;</span> install from the&nbsp;source. </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo tar -xf zsh-5.8.tar.xz <span class="linenos" data-linenos="2 "></span>$ <span class="nb">cd</span> zsh-5.8 <span class="linenos" data-linenos="3 "></span>$ sudo ./configure <span class="o">&amp;&amp;</span> sudo make <span class="o">&amp;&amp;</span> sudo make install </code></pre></div> <h1 id="final-steps">Final steps<a class="headerlink" href="#final-steps" title="Permanent link">&para;</a></h1> <p>Add Zsh to the login shells by adding &lsquo;/usr/local/bin/zsh&rsquo; on the last line of the config file, <code>/etc/shells</code> </p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo -e /etc/shells <span class="linenos" data-linenos="2 "></span><span class="c1"># Please don&#39;t forget to replace &#39;kmonsoor&#39; with your username</span> <span class="linenos" data-linenos="3 "></span>$ sudo chsh kmonsoor </code></pre></div> <p>Update the system&rsquo;s default symlink to the new Zsh version.<br> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo ln -sf /usr/local/bin/zsh /bin/zsh <span class="linenos" data-linenos="2 "></span>$ zsh --version </code></pre></div></p> <p>It&rsquo;s always a good habit to clean up after doing stuffs. ;) <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo make clean </code></pre></div></p> <p><img alt="voila" src="https://i.imgur.com/BEFIOXf.jpg"></p> <h1 id="related">Related<a class="headerlink" href="#related" title="Permanent link">&para;</a></h1> <p>Want to have a super, cool-looking command shell? Gotcha, fam. Check out my blog on <strong><a href="https://blog.kmonsoor.com/pimp-up-my-terminal/">How do I Pimp up My Terminal on Linux</a></strong>.</p>What I learned managing weekly release of a 50M+ users’ app2020-08-22T12:12:00+06:002020-08-22T12:12:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2020-08-22:/some-tips-on-managing-apps-release/<p>In any war, game, or enterprise resource management, the last thing you want is to underestimate your opponent(s). In release management, the opponent is “chaos”. The last thing a release manager should expect is that everything, from bug-free features to timely completion of production builds will be done as per the cadence&nbsp;schedule.</p><p><em>Disclaimer: Everything here are my personal observations, experiences, commentary, or “wisdom”. My past, present, or future employers shouldn’t be held responsible for it. Also, take in or apply the stuff below “with a grain of&nbsp;salt”.</em></p> <blockquote> <p>In any war, game, or enterprise resource management, the last thing you want is to underestimate your opponent(s). In release management, the opponent is&nbsp;“chaos”.</p> </blockquote> <p><img alt="releases" class="noZoom" src="https://i.imgur.com/BE87UxPl.jpg"></p> <p>The last thing a release manager should expect is that everything, from bug-free features to timely completion of production builds will be done as per the cadence&nbsp;schedule.</p> <p>Here is something that I had to learn over time. It might help someone,&nbsp;somewhere.</p> <h3 id="is-the-release-cadence-effective">Is the “release cadence” effective?<a class="headerlink" href="#is-the-release-cadence-effective" title="Permanent link">&para;</a></h3> <p>Not all apps are created equal, neither for the exact same consumer base nor with the same feature set. So, <span class="caps">THE</span> perfect release cadence or frequency doesn’t exist. Every app needs to find it’s own pace. The frequency might be feature-based, the target market base, fixed calendar-based, and/or mix of&nbsp;those.</p> <p>Though I can’t go in detail for each of them, I emphasize going through the process of identifying the right frequency. While for the passenger app of Grab, due to the sheer number of features (or change of features) and bug fixes, it makes sense to roll out a new version every week, it might make no sense for your app that gets new features once in a month or two that has a limited number of enterprise customers and your user-support team has to train them in&nbsp;sessions.</p> <p>Ask the questions across the company who are involved or have a vested interest in the release&nbsp;process.</p> <ul> <li>Who are the primary users? And, who can be considered&nbsp;secondary?</li> <li>Are the primary audience tech-savvy enough to be excited to navigate the new changes, or will they be pissed about the frequent&nbsp;changes?</li> <li>How frequently may the product teams push out new&nbsp;features?</li> <li>How resilient the quality assurance team is to exhaustively verify the new features and look out for new bugs that are going to break the old, stable&nbsp;features?</li> <li>How much automation is in place to support a more frequent build, release, and&nbsp;rollout?</li> </ul> <p>Particularly, if your <span class="caps">QA</span>/testing team isn’t well-distributed as per the app scopes, to provide the final sign-off on that specific scope, avoid intense weekly releases. Having a shiny process, just for the sake of having it, is an easy recipe for&nbsp;disaster.</p> <h3 id="is-the-process-well-documented-communicated">Is the process well-documented <span class="amp">&amp;</span> communicated?<a class="headerlink" href="#is-the-process-well-documented-communicated" title="Permanent link">&para;</a></h3> <p>Well-accepted process documents should work as the org-wide “release” manifesto, well discussed among the contributing teams and agreed by the&nbsp;stakeholders. </p> <p>Once the process and agreed upon process timeline is there, in case of confusion or conflict between the teams, the release team will have two clear paths to move&nbsp;forward.</p> <ul> <li>It’s already in the doc, point to&nbsp;it.</li> <li>If it’s not in the doc or not clear enough, an opportunity to&nbsp;improve</li> </ul> <p>If there is an exception, foreseeable disruption, or any upcoming change in the process, <em>communicate early</em>, and <em>communicate frequently</em>.</p> <h3 id="is-some-automation-possible">Is some automation possible?<a class="headerlink" href="#is-some-automation-possible" title="Permanent link">&para;</a></h3> <p>Identify the opportunities to automate. As a release manager, if you need something to do more than twice a day, start there. Have you been asked the same question twice? Include the answer in a <span class="caps">FAQ</span>&nbsp;doc.</p> <p>Document the need, identify the tool required. It can be as simple as a <span class="caps">SQL</span> query, a Python script, or a much more complex&nbsp;system. </p> <p>Before jumping headfirst into the thing, identify the matrix to monitor the improvement, and get approval (at least, verbally) from the&nbsp;stakeholders.</p> <p>Some example automation tool I&nbsp;use:</p> <ul> <li><a href="https://www.google.com/script/start/">Google script</a> to monitor changes on Google&nbsp;Sheet</li> <li><a href="https://slack.com/help/articles/202026038-An-introduction-to-Slackbot">Slack bot response</a> for selected&nbsp;keywords</li> <li>Python scripts for compiling custom <span class="caps">JIRA</span> reports. <a href="https://gist.github.com/kmonsoor/62ed1bc6cd3084648245073744182227">An&nbsp;example</a></li> <li><a href="https://slack.com/intl/en-sg/help/articles/206819278-Send-emails-to-Slack">Email forwarding to Slack <span class="caps">DM</span></a></li> </ul> <p>Automation paves the path to sustainable scalability. The release manager might be able to ping (aka “pull info or update”) three persons, but definitely shouldn’t try for thirty. So, establish some “push notification” mechanism as much as possible. E.g., comment on the Google <em>Sheet</em> or more frequent status update on the <span class="caps">JIRA</span> ticket,&nbsp;etc.</p> <h3 id="is-proper-divide-conquer-in-place">Is proper “Divide <span class="amp">&amp;</span> Conquer” in place?<a class="headerlink" href="#is-proper-divide-conquer-in-place" title="Permanent link">&para;</a></h3> <p>This <span class="caps">DC</span> method/mechanism/scheme (as a fan of the Marvel universe, I don’t mind the name mixup though :P ) is a controversial term. However, the “division of labor” is a well-established example. Regardless of the name and the flavor, it’s essential for the successful execution of any high-frequency release&nbsp;cycle. </p> <ul> <li>Design the process and its breakdown based on the roles, not the actual persons or their&nbsp;skillset</li> <li>Minimize switching between roles for individual persons in a single release cycle. e.g., if someone, in the <span class="caps">QA</span> team, is writing test cases, let them do that for at least the period. If someone is testing individual features at the beginning of the week, let her continue it until the end of the release. \ <em>Well-designed repetition is key to reaching&nbsp;excellence.</em></li> <li>Keep only integrated test (staging, then production) dependency for release rollout. For individual features, the developing team(s) should take care of it, before “signing off” it ready for shipping. The “not ready to be shipped” features must be isolated from the production scope, excluding it from the ongoing release&nbsp;cycle.</li> <li>Imagine the shipment process of a newer version as a concept like a train and different teams as different product pipelines. Products come to the train station only when it’s packed and ready to be shipped. I believe it&rsquo;s an agile concept, as&nbsp;well.</li> <li>Like a public train, the release “train” should depart as per schedule. If any feature is delayed due to unforeseen reasons or a recently discovered severe bug, the release-train should not wait for it. Instead, once the feature is stabilized later, it should wait for the next <em>train</em>.</li> <li>Adopting the concept of a train and its scheduled departure and well-documented checkpoints (i.e., phases) enables a large organization to expect and likewise communicate what to expect&nbsp;when.</li> </ul> <h3 id="are-you-deploying-on-a-wrong-moon-phase">Are you deploying on a wrong moon-phase?<a class="headerlink" href="#are-you-deploying-on-a-wrong-moon-phase" title="Permanent link">&para;</a></h3> <p>Probably, it should be a piece of common knowledge. Be it a backend deployment or app rollout on app stores; please avoid changing anything with the “production” tag, before a weekend or a multi-day&nbsp;holiday. </p> <p>Like the <a href="https://twitter.com/System32Comics/status/1266100094853476352/photo/1">printer meme</a>, the universe likes to throw unique, unforeseen issues after weekend deployments. It’s a mystery of the world, I&nbsp;guess.</p> <h3 id="is-there-a-provision-for-backup">Is there a provision for backup?<a class="headerlink" href="#is-there-a-provision-for-backup" title="Permanent link">&para;</a></h3> <p>Plan capacity with some&nbsp;slack. </p> <p>I’d suggest going for the 80/20 rule, meaning the weekly release should go out, even in the unavailability of 20% human&nbsp;resources. </p> <p>Unlike machines, people will be sick, go out for vacation, emergency travel plans, unexpectedly resign, etc. anything can happen. So, like any other robust system design, plan for&nbsp;“disaster”. </p> <h3 id="is-proper-continuous-integrationdeployment-cicd-in-place">Is proper Continuous Integration/Deployment (<span class="caps">CI</span>/<span class="caps">CD</span>) in place?<a class="headerlink" href="#is-proper-continuous-integrationdeployment-cicd-in-place" title="Permanent link">&para;</a></h3> <p>To have an efficient release process, automated and well-documented <span class="caps">CI</span>/<span class="caps">CD</span> pipeline is almost a&nbsp;prerequisite.</p> <ul> <li>Developers across the board should be able to build, test, merge their code, and build the binary from the latest stable and/or release&nbsp;checkpoints.</li> <li>Once their change, be it a feature or a bugfix, is ready to be merged with the release branch, a complete suite of automated tests should run on the resultant binary to detect any regression bug due to the change; with no dependency on the release&nbsp;team.</li> <li><span class="caps">QA</span>, while testing for a particular release, should be able to receive regression or production build without any manual effort from the release&nbsp;team.</li> </ul> <h3 id="is-there-continuous-feedback-established">Is there continuous feedback established?<a class="headerlink" href="#is-there-continuous-feedback-established" title="Permanent link">&para;</a></h3> <p>An efficient release process is an intensive process. And, no process can be perfect from the&nbsp;beginning. </p> <p>Every organization has, as its fingerprint, has different priorities, deliverables, audience, resources, and of course, budget. So, while the release cycle can start with an agile blueprint, down the line it needs to be well fit on its organizational&nbsp;body. </p> <p>Here comes the necessity for continuous feedback from the participants, stakeholders, and even the release&nbsp;team-members.</p> <h3 id="stay-on-the-good-book-of-the-app-stores">Stay on the “good book” of the app stores ;)<a class="headerlink" href="#stay-on-the-good-book-of-the-app-stores" title="Permanent link">&para;</a></h3> <p>If the app is in the center of revenue of your company, keep a close eye on the regular comm from the app stores, i.e., Google Play and Apple Appstore. It’s very easy to miss these communications, among other day-to-day&nbsp;stuff.</p> <p>Regardless of how ridiculous your company feels about some policies on the app stores, please read between the lines. In case of confusion, read again. Especially if you’re one of the big fishes.<br> If a particular app store weren’t strictly monitoring or enforcing some policies last month, don’t even think they aren’t going to enforce tomorrow and reject your&nbsp;app. </p> <p>The release manager or the team, being the interface with the app stores, should frequently communicate with the stakeholders and the product team to carefully go through relevant app store policies long before they invest company time and resources into a feature or a roadmap. Trust me; it happens more than anyone likes to admit that after month-long design and development, the app update with the feature gets rejected on the app stores, solely due to this new&nbsp;feature.</p> <p>If possible, try to get hold of and build a working relationship with the business development or dev support teams of Google and Apple. It might help in case of any misunderstanding and/or some negotiations about the compliance timeline.<br> I can’t go into any more detail though&nbsp;;)</p> <h3 id="notes">Notes:<a class="headerlink" href="#notes" title="Permanent link">&para;</a></h3> <ul> <li>Please don’t take the number 50M+ literally. The actual number is much higher but should be considered as company secrets ;).&nbsp;Right?</li> <li>This article is <span class="caps">NOT</span> a definitive process guide, and I’m not an expert on Agile methodologies. I’m just trying to share my learning from my release experience of 1.75 years. Some stuff might help any other release manager on the other side of the planet. That’s the goal&nbsp;here.</li> </ul> <p>Thanks,&nbsp;y’all.</p>Porqué hacerlo, en vez de cómo hacerlo2020-05-26T00:00:00+06:002020-05-26T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2020-05-26:/Porqué-hacerlo-en-vez-de-cómo-hacerlo/<p>Para establecer un proceso, a menudo nos vemos envueltos en los detalles del &ldquo;cómo&rdquo; mucho antes de lo que debería ser. Eso, a su vez, arruina el objetivo del proceso en&nbsp;sí.</p><p><em>[The <a href="https://blog.kmonsoor.com/process-emphasize-on-why-not-how/">original article</a> is in English. The article below is a professional&nbsp;translation]</em></p> <p>Para que la gente adopte cualquier proceso, la parte más importante de la &ldquo;difusión&rdquo; o &ldquo;venta&rdquo; es el porqué hacerlo, no el cómo hacerlo. Cuando la audiencia está lo suficientemente inspirada, estará dispuesta a aprender el proceso, a optimizarlo para que se ajuste al objetivo y, lo que es más importante, a hacer un esfuerzo adicional (más allá de lo que está escrito explícitamente) para lograr el objetivo o maximizar el&nbsp;impacto.</p> <p>Por eso la guerra de guerrillas funciona; incluso frente a un ejército formal masivamente desproporcionado. Porque un ejército formal dispone de conjuntos meticulosos de procesos, pero suele carecer de la parte de inspiración en los lugares&nbsp;adecuados.</p> <p>Como propietario de cualquier proceso, lo primero de todo es estar sólidamente inspirado. Si eso falta, todo lo demás se desmoronará; independientemente de cuántos consultores de gestión hayan trabajado en ello. Para asegurarse de ello, hágase la pregunta del porqué. Profundice lo suficiente como para contentarse con la respuesta. Entonces estarás listo para hacer y entregar la mejor versión de la parte del&nbsp;cómo.</p> <p>Si vas a delegar un proceso en otra persona para que se haga cargo de él, asegúrate de que esté satisfecha con la respuesta del porqué; no sólo que haya aceptado hacerse cargo. ¿Cómo lo sabe? Ella indagará en la parte del por qué y le preguntará al respecto. Estará lo suficientemente ansiosa por aprender la parte del cómo y ejecutarla en&nbsp;consecuencia.</p>Why to do it, rather than how to do it2020-04-26T00:00:00+06:002020-04-26T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2020-04-26:/process-emphasize-on-why-not-how/<p>To establish a process, we often get wrapped into the &ldquo;how&rdquo; details much earlier than it should be. That, in turn, ruins the goal of the process&nbsp;itself.</p><p><em>[ This article is also available in <a href="https://blog.kmonsoor.com/Porqu%C3%A9-hacerlo-en-vez-de-c%C3%B3mo-hacerlo/">Español</a>&nbsp;]</em></p> <p>To make people adopt any process, the most important part of the “preach” or “sell” is why to do it, not how to do it. When the audience is inspired enough, they&rsquo;ll be willing to learn the process, optimize the process to fit the goal and most importantly go the extra mile (beyond what&rsquo;s written explicitly) to achieve the objective(s) or maximize the&nbsp;impact.</p> <p>That&rsquo;s why guerrilla warfare works; even in the face of massively disproportionate formal army. Because a formal army has meticulous sets of processes in place, but usually lacks the inspiration part in the right&nbsp;places.</p> <p>As the owner of any process, first of all, you should be solidly inspired. If that’s missing, everything else will fall apart; regardless of how many management consultants worked on it. To ensure that ask yourself the why question. Dig deep enough to content yourself with the answer. Then you’re ready to make and deliver the best version of the how&nbsp;part.</p> <p>If you’re delegating a process to someone else to own, make sure she is satisfied with the answer of why; not just agreed to take over. How do you know? She will dig into the why part and question you about it. She will be eager enough to learn how part and execute&nbsp;accordingly.</p>Install latest Python 3 on Linux CentOS 72018-07-07T00:00:00+06:002018-07-07T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2018-07-07:/install-latest-python3-on-centos-7/<p>Install the latest and greatest Python 3 on CentOS 7&nbsp;systems</p><h2 id="why">Why<a class="headerlink" href="#why" title="Permanent link">&para;</a></h2> <p>Not all distro created equal. <br> Some are created to join the space race, some are to hold unto the leagcy, some are cutting-edge, some are cutting edge. Some are born to boot-up IoT devices some are to push out heavy&nbsp;graphics.</p> <p>That&rsquo;s the fun (albeit, power) of&nbsp;Linux.</p> <p><img alt="CentOS 7 logo" class="noZoom" src="https://i.imgur.com/6ZFCdoM.jpg"></p> <p>CentOS 7 is a powerful and stable distro that runs on thousands (probably, millions) production-grade servers.<br> In the matter of stability, it&rsquo;s a beast. However, it doesn&rsquo;t ship with Python 3, by default. You can install it via <span class="caps">EPEL</span> repository, or the below simple&nbsp;steps.</p> <p>Also, take a note. The Python <strong>2</strong> comes with the system, which is probably 2.7.5, do <strong><span class="caps">NOT</span></strong> mess with it. Many system components rely on that specific version. If you need the latest versions of 2, use <code>virtualenv</code> or <code>pipenv</code>.</p> <h3 id="prepare-your-system">Prepare your system<a class="headerlink" href="#prepare-your-system" title="Permanent link">&para;</a></h3> <p>Start with installing pre-requisite utilities for compilation and development&nbsp;support.</p> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo yum update <span class="o">&amp;&amp;</span> sudo yum groupinstall -y <span class="s2">&quot;development tools&quot;</span> <span class="linenos" data-linenos="2 "></span>$ sudo yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel <span class="se">\</span> <span class="linenos" data-linenos="3 "></span> sqlite-devel readline-devel tk-devel gdbm-devel <span class="se">\</span> <span class="linenos" data-linenos="4 "></span> db4-devel libpcap-devel xz-devel expat-devel </code></pre></div> <h3 id="download-latest-python-source-code-from-pythonorg">Download latest Python source code from Python.org<a class="headerlink" href="#download-latest-python-source-code-from-pythonorg" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz <span class="linenos" data-linenos="2 "></span>$ tar xf Python-3.6.6.tar.xz <span class="linenos" data-linenos="3 "></span>$ <span class="nb">cd</span> Python-3.6.6 </code></pre></div> <h3 id="enable-performance-optimizations-optional-but-highly-recommended">Enable performance optimizations (optional, but highly recommended)<a class="headerlink" href="#enable-performance-optimizations-optional-but-highly-recommended" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ ./configure --prefix<span class="o">=</span>/usr/local --enable-shared <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">&quot;-Wl,-rpath /usr/local/lib&quot;</span> <span class="linenos" data-linenos="2 "></span>$ ./configure --enable-optimizations </code></pre></div> <h3 id="build-and-install">Build and install<a class="headerlink" href="#build-and-install" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ make <span class="linenos" data-linenos="2 "></span>$ sudo make altinstall </code></pre></div> <p>Now, Python 3.6.6 is ready to be used in your system; located in <code>/usr/local/bin/python3.6</code> <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ which python3.6 <span class="linenos" data-linenos="2 "></span>/usr/local/bin/python3.6 <span class="linenos" data-linenos="3 "></span> <span class="linenos" data-linenos="4 "></span>$ python3.6 <span class="linenos" data-linenos="5 "></span>Python <span class="m">3</span>.6.6 <span class="o">(</span>default, Jul <span class="m">10</span> <span class="m">2018</span>, <span class="m">14</span>:04:26<span class="o">)</span> <span class="linenos" data-linenos="6 "></span><span class="o">[</span>GCC <span class="m">4</span>.8.5 <span class="m">20150623</span> <span class="o">(</span>Red Hat <span class="m">4</span>.8.5-28<span class="o">)]</span> on linux <span class="linenos" data-linenos="7 "></span>Type <span class="s2">&quot;help&quot;</span>, <span class="s2">&quot;copyright&quot;</span>, <span class="s2">&quot;credits&quot;</span> or <span class="s2">&quot;license&quot;</span> <span class="k">for</span> more information. <span class="linenos" data-linenos="8 "></span>&gt;&gt;&gt; </code></pre></div></p> <p>For convenience, you can create a symbolic-link with a shorter name. If you had system-installed Python3 (unlikely), <strong>don&rsquo;t</strong> do this, as some system-components may depend on that specific older version of Python 3. <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ sudo ln -s /usr/local/bin/python3.6 /usr/local/bin/python3 <span class="linenos" data-linenos="2 "></span>$ python3 <span class="linenos" data-linenos="3 "></span>Python <span class="m">3</span>.6.6 <span class="o">(</span>default, Jul <span class="m">10</span> <span class="m">2018</span>, <span class="m">14</span>:04:26<span class="o">)</span> <span class="linenos" data-linenos="4 "></span><span class="o">[</span>GCC <span class="m">4</span>.8.5 <span class="m">20150623</span> <span class="o">(</span>Red Hat <span class="m">4</span>.8.5-28<span class="o">)]</span> on linux <span class="linenos" data-linenos="5 "></span>Type <span class="s2">&quot;help&quot;</span>, <span class="s2">&quot;copyright&quot;</span>, <span class="s2">&quot;credits&quot;</span> or <span class="s2">&quot;license&quot;</span> <span class="k">for</span> more information. <span class="linenos" data-linenos="6 "></span>&gt;&gt;&gt; </code></pre></div></p> <h2 id="install-wheel-and-pip"><del>Install wheel and pip</del><a class="headerlink" href="#install-wheel-and-pip" title="Permanent link">&para;</a></h2> <p>You don&rsquo;t need to, because <code>Python 3.6.6</code> includes these necessary tools included. <div class="highlight"><pre><span></span><code><span class="linenos" data-linenos="1 "></span>$ pip3.6 -V <span class="linenos" data-linenos="2 "></span>pip <span class="m">10</span>.0.1 from /usr/local/lib/python3.6/site-packages/pip <span class="o">(</span>python <span class="m">3</span>.6<span class="o">)</span> <span class="linenos" data-linenos="3 "></span>$ wheel version <span class="linenos" data-linenos="4 "></span>wheel <span class="m">0</span>.29.0 </code></pre></div></p>HA(High-Availability) Setup for InfluxDB2018-01-18T00:00:00+06:002018-01-18T00:00:00+06:00Khaled Monsoortag:blog.kmonsoor.com,2018-01-18:/ha-setup-for-influxdb/<p>Create a robust, highly-available, time-series InfluxDB cluster with the community(free) version of&nbsp;it</p><p><strong><span class="caps">NOTE</span></strong> <em>Since I have written this article, all the components used in this below architecture have gone through many updates and releases. While the general premise involving <code>influxdb-relay</code> and the multiplexing might still hold, please sync up with the latest release docs before jumping into some serious system&nbsp;design.</em></p> <hr> <p>Currently, from version 0.9, you cannot create an InfluxDB cluster from the open-sourced free edition. Only commercially available InfluxDB Enterprise can do that for now. That stirred up the early-adopter enthusiast users, especially for their usage in professional setups. They complained that InfluxData, the company behind InfluxDB, is trying to milk the <span class="caps">OSS</span> solution for&nbsp;profit.</p> <p>I can&rsquo;t blame the InfluxData guys much, as they gotta pay their bills too. So far, we — the users of open-source systems — couldn&rsquo;t show much promise about the financial realities of the projects. Continuing development of <span class="caps">OSS</span> future by only depending on donations, patrons, or enterprise sponsorship is far too rare and unpredictable, even for the projects that many successful organizations heavily rely&nbsp;on.</p> <p>Anyways, InfluxDB then promised and later introduced <code>Influx Relay</code> as a complimentary consolation for missing <span class="caps">HA</span> parts of InfluxDB. You can get the details here and here about&nbsp;that. </p> <h2 id="premise">Premise<a class="headerlink" href="#premise" title="Permanent link">&para;</a></h2> <p>For my needs, I have to try to create a reliable <span class="caps">HA</span>(High-Availability) setup from available free options, hence InfluxDB and the relay. It&rsquo;s quite a bit far from an InfluxDB-cluster in terms of robustness or ease of setup, but it&rsquo;s got the job done, at least for&nbsp;me.</p> <p>I needed a setup to receive system-stats from at least 500+ instances and to store them for a while, but without breaking the bank in bills from <span class="caps">AWS</span>. Meaning, I could ask for and could use only couple of instances for my&nbsp;solution.</p> <p>Here were my&nbsp;trade-offs.</p> <ul> <li>Not too many instances for this purpose. Neither, any of the heavyweight lifters e.g. <span class="caps">AWS</span>&rsquo; m3-xlarge etc. To use only what&rsquo;s&nbsp;necessary. </li> <li>To satisfy the budget, hence avoiding pay-per-use solutions as far as it is&nbsp;possible.</li> <li>Solutions must not be crazy complex, so that handover to the DevOps team be&nbsp;smooth.</li> <li>Reading the data would be too rarely w.r.t. writing. The related Grafana dashboards will be only used to investigate issues by a handful of&nbsp;people.</li> </ul> <h2 id="overall-design">Overall Design<a class="headerlink" href="#overall-design" title="Permanent link">&para;</a></h2> <h3 id="write">Write<a class="headerlink" href="#write" title="Permanent link">&para;</a></h3> <p>From a birds&rsquo; eye view, I decided to use two instances to run parallelly, hosting InfluxDB on them independently and then send exactly same data over to them for storing. This scheme mostly looks like <a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1"><span class="caps">RAID</span>-1 systems</a>.</p> <p><img alt="Overall architecture" src="https://i.imgur.com/ZKYIyOd.png"></p> <p>That brings up a couple of&nbsp;challenges.</p> <ul> <li> <p>None of the agents I used on the sender side could multiplex output. Meaning, they were able to send data to a single destination, not multiple. On the Windows front, I&rsquo;ve used <code>Telegraf</code> which is able randomly to switch between pre-listed destinations, but <span class="caps">NOT</span> multiple at-once.<br> In the case of Linux hosts, I used <code>Netdata</code> which is excellent in its own right, but unable to send stats to multiple destinations.<br> Here comes <code>Influx-relay</code>. It can receive time-series data-stream from hosts on a <span class="caps">TCP</span> or <span class="caps">UDP</span> port, buffer for a while, and then re-send those received and buffered data to multiple receive ends which can either be an InfluxDB instance or another listening Influx-relay instances.<br> This chaining can broaden the relaying scheme even further. However, for my purpose, this relay-chaining was not necessary. Rather, from the relay, I am sending data to the separate InfluxDB instances, running on two separate&nbsp;instances. </p> </li> <li> <p>Now that I partially multiplexed the output, my hosts (senders) still are able to send to one destination. So, I need a proxy as well as a load-balancer. For a while, I was torn between <span class="caps">NGINX</span> and HAProxy. Both were new to&nbsp;me. </p> </li> </ul> <p>However, for a couple of reasons, I went for HAProxy. Firstly, I have no need for <span class="caps">HTTP</span> session management. Secondly, as I wanted to keep my <span class="caps">UDP</span> for later, HAProxy was perfectly capable of that.<br> <span class="caps">NGINX</span> has the support recently, maturity was a concern. Also, configuring <span class="caps">NGINX</span> seems little intimating (which I know not so true). Last but not least, and for what its worth, out-of-the-box, HAProxy&rsquo;s stat page carries much more in-depth information than that of free-version of <span class="caps">NGINX</span>.<br> Upon receiving the stats stream, HAProxy was supposed to send that to different Influx-relays in a load-balanced&nbsp;fashion.</p> <p>So, here&rsquo;s my rough&nbsp;plan. </p> <p>collector-agent &rarr; HAProxy &rarr; (50/50 load-balanced) &rarr; Influx-relay &rarr; (multiplexed) &rarr; 2 InfluxDB&nbsp;instances</p> <p>Now, each of the received data is to go to both the InfluxDB, or at least one in case of failure (or, overload) of any the relays or Influx instances.   Also, I have chosen to keep Influx-relays deployed as Dockerized and kept HAProxy and InfluxDB instances running as native services. Of course, you can Dockerize HAProxy and InfluxDB,&nbsp;too. </p> <h3 id="read">Read<a class="headerlink" href="#read" title="Permanent link">&para;</a></h3> <p>As I&rsquo;ve already noted in the section that reading the data, meaning to fetch data to visualize on Grafana end, will happen rarely and sporadically; only to investigate alarms or any other client-side performance&nbsp;issues. </p> <p>So, the read requests, reaching the HAProxy end, needed not much routing, other than directly to InfluxDB itself. Still, to better distribute the load I decided to load-balance it 50/50&nbsp;basis.</p> <h3 id="ports">Ports<a class="headerlink" href="#ports" title="Permanent link">&para;</a></h3> <ul> <li>As all the <span class="caps">READ</span> requests are routed through <code>HAProxy</code> running on each of the instances, to the external world only HAProxy&rsquo;s port should be opened for this&nbsp;purpose. </li> <li>On the other hand, for <span class="caps">WRITE</span> requests, InfluxDBs are receiving data from relays, one of its own instance and another one on other instance, so InfluxDB should listen on its own port for <span class="caps">WRITE</span> requests only. But, this must be accessible only from own <span class="caps">VPS</span> zone, but not open to the outside&nbsp;world.</li> <li>In case of HAProxy as well as InfluxDB, you can use the default ports, obviously, which is 8086 <span class="amp">&amp;</span> 8088 respectively. Or, you can choose to go for other ports (security through obfuscation). Your call. In this writing, I&rsquo;ll go with the&nbsp;defaults.</li> </ul> <h3 id="authentication-ssl">Authentication, <span class="caps">SSL</span><a class="headerlink" href="#authentication-ssl" title="Permanent link">&para;</a></h3> <p>You can configure <span class="caps">SSL</span> with your own server certificates through the HAProxy configs. You can even go for <span class="caps">SSL</span> from the relays to InfluxDB writes. If your sender hosts are connecting to your HAProxy through public internet, you should at least go for password-based authentication, better to utilize <span class="caps">SSL</span>. However, for brevity&rsquo;s sake, I&rsquo;ll skip them in this&nbsp;post.</p> <p>**Note: * Please bear in mind, this is an &ldquo;in-progress&rdquo; post; prematurely published to force me to work on it. I have the plan to add all the necessary configurations <span class="amp">&amp;</span> commands, that I used,&nbsp;here.</p>