I think ... - Tech

Create a free go-link server “on edge” using Cloudflare Worker KV

2021-06-06T00:00:00+06:00

Among quite a few ways to implement a go-link server (i.e., url-forwarder, short-url server, etc.), I’m going to show you how to use free-tier Cloudflare Worker (& KV) to create an in-house, on-edge, no-webserver go-link server.

For example, the short-link for this article is go.kmonsoor.com/golink-kv

/latest (by which I mean go.yourdomain.co/latest) may point to https://www.yourcompany.com/about/news which is a public page
/hr-help may point to https://www.company-internal.com/long-link/hr/contact.html, which is company’s internal human-resources help portal
/cnypromo may point to https://shop.yourcompany.com/sales/promotions/?marketing-promo=2021-cny which is a temporary sales promotions page targeting the shoppers during the Chinese new year of 2021.

Please note that using the setup and the code below, it’ll be possible to resolve short-links via a single sub-domain, e.g., go.your-domain.co. However, it’s possible (with some modification of the code) to resolve/redirect via any number of domains (your own, of course) towards any other public or private URL, and all sorts of novelties. However, for brevity’s sake, I will discuss the first one, a single sub-domain usecase.

To set up a go-link server or short-URL resolver via a proper KV+Worker combination, we’ll go through these steps:

Pre-requisites
Create the short-link map as a KV
Mapping a KV to a Worker variable
Handling a route with webworker
Create the Worker
Pointing a DNS record to the Worker
Next step
Related

Pre-requisites¶

The DNS resolver for the root domain (in the example below, kmonsoor.com) needs to be Cloudflare. Because the core of the solution, the “worker”, runs on the nearest (from the user) edge of Cloudflare using a standard KV (“key, value”) list.
Write permission to the DNS configuration as you’d need to add a new AAAA DNS record.
Some knowledge of Javascript(ES6), as we are going to write the “worker” in that language.

Create the short-link map as a KV¶

We’ll start the setup by creating the short-link map, the list between the short-link segments that you (or someone in your org) define, and the actual URLs they need to point to.

Find the KV stuff in the Workers section. From the screenshot, please ignore the “Route” section for now.

you’d need to create a Worker KV “Namespace”. Name the namespace as you seem fit. I named it REDIRECTS (in all caps just as a convention, not required).
List the short links & their respective target URLs. From the examples in the intro, the keys latest, hr-help, cnypromo etc. would be in as the “key”, and the target full links as the respective “value”.
Remember NOT to start the short part with ‘/’. It’ll be taken care of in the code.

Once you’ve listed all your desired (short-link, target-link) combinations, now we have a KV on Cloudflare. However, it’s not referencable from your Worker code, not yet. Hence the next step.

Mapping a KV to a Worker variable¶

Now, we will map the previously created KV to a variable that can be referenced from our Worker code. Please note that though I used different names, it can be the same as well. Also, note that multiple Workers can access a single KV, and vice versa is also true; a single Worker can reference multiple KVs.

Handling a route with webworker¶

Create the Worker¶

Now, we will write Worker-code that runs on V8 runtime on the nearest (from the requesting user) “edge” location of Cloudflare, to execute the code and deliver the result(s) to the user. In this case, that would be to redirect user-requested address to the mapped one (by you, in the KV namespace above).

The code editor looks like this:

If you rather prefer to copy-paste, please feel free to do it from the below GitHub Gist.

Once done, it should look like …

Pointing a DNS record to the Worker¶

Finally, we need to point a DNS record that’ll redirect all requests to your re-soutign sub-domain (e.g. go.your-domain.com) to the Cloudflare Worker that we just created.

According to the Cloudflare docs, the DNNS record must be an AAAA record, pointing to the IPv6 address 100::. The “Name” here is the “sub-domain” part of your choice, which is better be short, to rightfully serve our goal here.

Voila ! Now, test some of the short-urls that you’ve mapped via the KV. Enjoy ! Watch out for the target usage though. Here’s the limit.

I think you’ll be fine, unless you’re some celebrity ;)

Next step¶

As the next step, I’m thinking to create a generic Go/Link resolver browser extension. Then, someone can set their own default domain or company domain of choice as short-domain host. In that case, entering just go/hr-help on the browser will take to https://www.company-internal.com/.../hr/contact.html that we have discussed at the beginning (remember the example case of an internal human resources help portal?).

If you want to do this url-direction on your server, but only using webserver, try this: Personal short-link server using only Caddyserver

If you find this post helpful, you can show your support through Patreon or by buying me a coffee. Thanks!

TL;DR what cloud provider to use in 2021

2021-05-22T00:00:00+06:00

The sheer number of combinations a company can choose from the cloud providers and their product suites is mind-boggling. Hence, I decided to break it down in a concise form for the busy C-suite executives.

According to my little experiences and humble opinion, I suggest …

➤ If your company is a small SaaS shop with 10-ish engineers, stick with DigitalOcean, Linode, OVH, etc., which are best known as cloud “instance” providers.
Think McDonald’s; reliable, cheapest, fast, but you won’t take your date there.
Budget: 💰

➤ If you want a whole cloud experience (e.g., VPC, firewall, WAF, etc., on the menu), start with Google Cloud, then try AWS later.
Google Cloud would be the quickest to grasp the cloud concepts and get going. The UI of the AWS console is a bit messy compared to GCP; it just takes more time to get a proper grip.
Imagine them as full-course, Michelin-star restaurants. However, the product names are so abstract that they need a full-sized chart for that. ;) Budget: 💰💰💰

➤ Are you planning to set up a million-dollar infra for a billion-dollar company? Go for some GCP+AWS multi-cloud setup. You gonna get rebates from both on the scale of hundreds of thousands of dollars. And Microsoft Azure gonna offer you some million-$ free-tier, hoping to get the company hooked on Azure. :D Budget: 💰💰

➤ On the other hand, if you run a govt agency or a company where wearing suits is the mainstream, Microsoft Azure is your best bet.
A bunch of consultancy companies to choose from; you need to just approve the budget, you get the things to get up & running but miss the deadline by months, if not years. But there’d be no need for hiring more smarter ppl than that you already have. Budget: 💰💰💸

Need an even more comprehensive guide? Gotcha, fam …

CTO: we're having hard time choosing a cloud provider
…
“say no more, fam, I gotcha …” pic.twitter.com/hR3rMruWWi
— Khaled Monsoor ✨ (@kmonsoor) May 22, 2021

PS This post is inspired by a LinkedIn post of mine where I shared about my short experience with the Microsoft Azure DevOps suite

Deploying a short-link aka go-link server using only Caddyserver

2021-04-16T00:00:00+06:00

Before I go into any details, please note that I’ve used the term “shortlink-server” instead of “url-shortener” because the difference is significant for this post. A url-shortener takes a long url, and gives a short url, then redirects any requests for the shortened link to its longer counterpart. On the contrary, shortlink-server takes both the long and short url as inputs, then only does the redirect-ion part.

Backstory¶

If I wanted a fantastic, personal url-shortener or “go link” server, there are many excellent solutions out there, which are not only free or open-source but full-fledged as well for personal or public usage. Instead, I wanted some “service” that would “resolve” my personal, short links. I have been using bit.ly for a long time for its customizable “short-half” part, but the problem with bit.ly is – for some God-forsaken reason – blocked by the Bangladeshi govt. So, I needed a replacement to be appropriately “glocal”. There are many free (unbranded) and commercial (branded) options as well, but I wanted something that would be resolved via my hosted service (and personal domain) and as cheap as possible. So, basically solution for a poor nerd :D

Before jumped into this solution, I tried (deployed & tested) few others myself, mainly kellegous/go, kutt.it and adamyi/golinks. But, all of them “too featureful” for my needs.

What I wanted is to be able to:

resolve only my custom shortlinks (hence, no need for url-shortener)
not a public, internet-facing service (hence, any frontend, authentication, email verification etc. would be overkill )
minimal setup (if possible, no webapp at all)

Given my previous experience with Caddy webserver, which is an amazing one(why?), I had a gut feeling that Caddy has something for me – under the sleeve – to meet my minimal set of requirements. Thankfully, I managed to find it.

I believe NGINX, currently the most popular webserver, has some kind of similar mechanism as well. But, I’m not an expert, and once I was genuinely intimidated by its config file syntax. YMMV.

What you gonna need?¶

your own domain which will be the root of the shortlinks. While sub-domained URL like go.company-name.com/* is quite common, if you have some short domain, like you.co/*, only for this purpose, that’s fine as well.
A web-host server or public-facing instance with its own, public IPv4 address.
working knowledge of Linux

Step-1: Point your subdomain to the right place¶

Find out what’s the puplic IPv4 address of your instance that’ll act as the webserver. It’s usually on the cloud management dashboard.
make sure that, regardless of your cloud architecture (e.g. VPC, subnet, firewall etc.), the SSL port (:443) of the instance is reachable from the public internet.
now go to your domain name registrar (or, DNS management provider which in my case is Cloudflare). There, you need to point shortlink subdomain (go.)to the webserver’s I.P. address. In DNS terms, you’ll be creating a CNAME entry on the domain’s nameserver table.

You can do this step as the last one. But for some reason, I prefer it to do first. Because sometimes, DNS propagation takes some time. But, once my web service is up and running, I like to see the result instantaneously. ;)

Step-2: Install Caddy, a mighty webserver¶

Depending on your host OS (Ubuntu 20.04 LTS in my case), you need to install the Caddy webserver. While there are some hacky solutions to run, I think running Caddy as a background service is the simplest to manage. In fact, the documentation of Caddy is excellent, so I’d better leave that part to you.

After running with the default config(Caddyfile), (in Ubuntu’s case, located as /etc/caddy/Caddyfile), it should show a status somewhat like the below image. Please note that, in many cases, if running without sudo, Caddy cannot attach itself with the SSL port (:443), which is necessary for serving https://. So, check for that error message in the “status” log.

PS: By the way, want your console and command prompt to look 🚀 like mine? Here’s the guide: How do I pimp up my terminal on Linux

Step-3: Tell Caddy your short-links to redirect¶

Now, it’s time to configure Caddy to actually do the job.

Caddy has its native redir “directive” to redirect incoming web-request from one to another. While the map directive is relatively new, it makes the config file, i.e., Caddyfile, look elegant in case you have (or will have in the long run) a long list of short-links.

Here’s mine, which is working nicely …

# /etc/caddy/Caddyfile

go.kmonsoor.com {   # replace it your web-url root

    map {path} {redirect-uri} {
        /blog    https://blog.kmonsoor.com
        /photos  https://photos.kmonsoor.com

        /resume     https://drive.google.com/file/d/1nMS3i1ai6nsI70zZ7NFnNQ_XmvAa4GOl
        /resume-doc https://docs.google.com/document/d/1ECx1Yr8Jzz9I3S5VcoKnZQz56oIht2XaM5gSNetcWag

        /rickrolled    https://www.youtube.com/watch?v=dQw4w9WgXcQ

        # will add new ones here like the above
        # ...
    }

    # this below code is required to actually make the above `map` work

    @hasRedir expression `{redirect-uri} != ""`
    redir @hasRedir {redirect-uri}

    # code below is to set the default response if the requested shortlink isn't here
    respond "Thas's an unknown short URL ... :("  
}

Note: Don’t forget to restart the caddy service to let the new config to take effect.

Step-4: Profit¶

Yeah, that’s it. Now, add some own personal stuff with some cool short-links, and proudly share with the world.

What’s next ?¶

I’m thinking that given the very low workload my shortlink resolver needs — unless I’m becoming an overnight internet sensation — using a server instance only for this purpose is overkill. My next goal is to have the same service using some “serverless” function or using the “worker on the edge” thing from Cloudflare. Let’s see ;)

Update Now, actually done it. Here is the link: Free short-link server “on edge” using Cloudflare Worker K.V.

If you find this post helpful, you can show your support through Patreon or Paypal or by buying me a coffee. Thanks!

Pimping up My Linux Terminal

2021-03-31T00:00:00+06:00

The purpose of this post is to be my quick, copy-paste source of the commands that I use to set up my terminal on a new *nix system. However, if someone else finds it useful, that’d be some cherries on top.

This command prompt in the below image is the end goal.

Assuming, I’m on a standard pc/server with Ubuntu Linux and I have CLI access with sudo. For other Linux distros or MacOS, some commands might be slightly different.

Step-1: Confirm that Zsh is up-to-date¶

While on most of the Linux systems Zsh is present by default, on others that’s not the case. So, let’s make sure about it.

$ sudo apt install zsh

Confirm the version. Oh-my-zsh recommends Zsh to be 5.0.8 or higher.

$ zsh --version
zsh 5.8 (x86_64-ubuntu-linux-gnu)

Also, you gotta make sure that git (recommended v2.4.11 or higher) is also installed on the system.

Step-2: Install Oh-my-zsh, the fun “configuration” framework¶

Install directly from the source.

$ sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

In the last step of this installation, it will ask to set Zsh as THE shell. Go ahead.

Now we have the default prompt from Oh-my-zsh.

Now, let’s pimp up the prompt. Shall we?

Step-3: Install `powerlevel10k`, a powerful prompt theme¶

I love the powerful Zsh theme powerlevel10k. More on why this theme is awesome.

Let’s install it on top of oh-my-zsh.

$ git clone --depth=1 \
  https://github.com/romkatv/powerlevel10k.git \
  ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

Now, gotta set ZSH_THEME="powerlevel10k/powerlevel10k" in ~/.zshrc by adding that manually in the file.

Step-4: Make sure the prompt looks like as you want¶

In this step, I’m gonna bring in my already open-sourced Zsh config file aka .zshrc.

# deleting the current one & get my personal one from GitHub
$ rm .zshrc
$
$ wget https://raw.githubusercontent.com/kmonsoor/dot-files/master/.zshrc

I kept the powerlevel10k configs as comments so that Zsh doesn’t complain if I use the config file early. Have to set ZSH_THEME="powerlevel10k/powerlevel10k" in the ~/.zshrc as well.

Otherwise, once the powerlevel10k theme will run for the first time by Zsh, a very friendly step-by-step prompt will run you through towards a desirable prompt for you. Also, whenever you want, you can invoke the config-wizard by executing p10k configure on the shell.

Now is the time to enable the changes by restarting Zsh and enjoy the new config and the powerful prompt.

$ exec zsh

Optional¶

Also, I usually install this very useful, but external plugin zsh-syntax-highlighting for oh-my-zsh.

$ git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

Don’t forget to activate the plugin by including it in ~/.zshrc. For that, add zsh-syntax-highlighting inside the list of other plugins.

plugins=( plugin_a plugin_b zsh-syntax-highlighting)

Install the latest Zsh on CentOS

2020-10-20T00:00:00+06:00

As the default Zsh on CentOS is usually an older version, many cool things are not possible on this version of Zsh, like installing oh-my-zsh or using awesome powerlevel10k prompt system, it’s understandable if you’d like to have the latest Zsh on your system. Easy peasy !!

Note: Please remember to remove the sudo from the commands if you are already in “root” or sudo-er mode

We’ll be following these steps:

Install the pre-requisites
Download the latest source
Build & Install
Final steps
Related

Install the pre-requisites¶

We need GCC (C++ compiler) and other related stuffs for building Zsh from the source code.

$ sudo yum groupinstall "Development tools"  
$ sudo yum install ncurses-devel

Now, check if GCC is installed properly, by

$ gcc -v

Download the latest source¶

Now, we gonna get the latest code of Zsh.
Please update the link (in the shown command) with the latest by checking this web-folder.
Don’t forget to update the filename as well, if needed.

$ cd /usr/local/src
$ sudo curl -L https://www.zsh.org/pub/zsh-5.8.tar.xz \
-o zsh-5.8.tar.xz

Build & Install¶

Unzip the file, “dig in” to the folder, and build & install from the source.

$ sudo tar -xf zsh-5.8.tar.xz  # the actual version of the downloaded file might be different
$ cd zsh-5.8
$ sudo ./configure && sudo make && sudo make install

Final steps¶

Add Zsh to the login shells by adding ‘/usr/local/bin/zsh’ on the last line of the config file, /etc/shells

$ sudo -e /etc/shells
$ sudo chsh $USER

Update the system’s default symlink to the new Zsh version.

$ sudo ln -sf /usr/local/bin/zsh /bin/zsh
$ zsh --version

It’s always a good habit to clean up after doing stuffs. ;)

$ sudo make clean

That’s it !

Want to have a super, cool-looking command shell? Gotcha, fam. Check out my blog on Pimping up My Linux Terminal.

If you find this post helpful, you can show your support through Patreon or Paypal or by buying me a coffee. Thanks!

What I learned managing weekly release of a 50M+ users’ app

2020-08-22T12:12:00+06:00

Disclaimer: Everything here are my personal observations, experiences, commentary, or “wisdom”. My past, present, or future employers shouldn’t be held responsible for it. Also, take in or apply the stuff below “with a grain of salt”.

In any war, game, or enterprise resource management, the last thing you want is to underestimate your opponent(s). In release management, the opponent is “chaos”.

The last thing a release manager should expect is that everything, from bug-free features to timely completion of production builds will be done as per the cadence schedule.

Here is something that I had to learn over time. It might help someone, somewhere.

Is the “release cadence” effective?¶

Not all apps are created equal, neither for the exact same consumer base nor with the same feature set. So, THE perfect release cadence or frequency doesn’t exist. Every app needs to find it’s own pace. The frequency might be feature-based, the target market base, fixed calendar-based, and/or mix of those.

Though I can’t go in detail for each of them, I emphasize going through the process of identifying the right frequency. While for the passenger app of Grab, due to the sheer number of features (or change of features) and bug fixes, it makes sense to roll out a new version every week, it might make no sense for your app that gets new features once in a month or two that has a limited number of enterprise customers and your user-support team has to train them in sessions.

Ask the questions across the company who are involved or have a vested interest in the release process.

Who are the primary users? And, who can be considered secondary?
Are the primary audience tech-savvy enough to be excited to navigate the new changes, or will they be pissed about the frequent changes?
How frequently may the product teams push out new features?
How resilient the quality assurance team is to exhaustively verify the new features and look out for new bugs that are going to break the old, stable features?
How much automation is in place to support a more frequent build, release, and rollout?

Particularly, if your QA/testing team isn’t well-distributed as per the app scopes, to provide the final sign-off on that specific scope, avoid intense weekly releases. Having a shiny process, just for the sake of having it, is an easy recipe for disaster.

Is the process well-documented & communicated?¶

Well-accepted process documents should work as the org-wide “release” manifesto, well discussed among the contributing teams and agreed by the stakeholders.

Once the process and agreed upon process timeline is there, in case of confusion or conflict between the teams, the release team will have two clear paths to move forward.

It’s already in the doc, point to it.
If it’s not in the doc or not clear enough, an opportunity to improve

If there is an exception, foreseeable disruption, or any upcoming change in the process, communicate early, and communicate frequently.

Is some automation possible?¶

Identify the opportunities to automate. As a release manager, if you need something to do more than twice a day, start there. Have you been asked the same question twice? Include the answer in a FAQ doc.

Document the need, identify the tool required. It can be as simple as a SQL query, a Python script, or a much more complex system.

Before jumping headfirst into the thing, identify the matrix to monitor the improvement, and get approval (at least, verbally) from the stakeholders.

Some example automation tool I use:

Google script to monitor changes on Google Sheet
Slack bot response for selected keywords
Python scripts for compiling custom JIRA reports. An example
Email forwarding to Slack DM

Automation paves the path to sustainable scalability. The release manager might be able to ping (aka “pull info or update”) three persons, but definitely shouldn’t try for thirty. So, establish some “push notification” mechanism as much as possible. E.g., comment on the Google Sheet or more frequent status update on the JIRA ticket, etc.

Is proper “Divide & Conquer” in place?¶

This DC method/mechanism/scheme (as a fan of the Marvel universe, I don’t mind the name mixup though :P ) is a controversial term. However, the “division of labor” is a well-established example. Regardless of the name and the flavor, it’s essential for the successful execution of any high-frequency release cycle.

Design the process and its breakdown based on the roles, not the actual persons or their skillset
Minimize switching between roles for individual persons in a single release cycle. e.g., if someone, in the QA team, is writing test cases, let them do that for at least the period. If someone is testing individual features at the beginning of the week, let her continue it until the end of the release. \ Well-designed repetition is key to reaching excellence.
Keep only integrated test (staging, then production) dependency for release rollout. For individual features, the developing team(s) should take care of it, before “signing off” it ready for shipping. The “not ready to be shipped” features must be isolated from the production scope, excluding it from the ongoing release cycle.
Imagine the shipment process of a newer version as a concept like a train and different teams as different product pipelines. Products come to the train station only when it’s packed and ready to be shipped. I believe it’s an agile concept, as well.
Like a public train, the release “train” should depart as per schedule. If any feature is delayed due to unforeseen reasons or a recently discovered severe bug, the release-train should not wait for it. Instead, once the feature is stabilized later, it should wait for the next train.
Adopting the concept of a train and its scheduled departure and well-documented checkpoints (i.e., phases) enables a large organization to expect and likewise communicate what to expect when.

Are you deploying on a wrong moon-phase?¶

Probably, it should be a piece of common knowledge. Be it a backend deployment or app rollout on app stores; please avoid changing anything with the “production” tag, before a weekend or a multi-day holiday.

Like the printer meme, the universe likes to throw unique, unforeseen issues after weekend deployments. It’s a mystery of the world, I guess.

Is there a provision for backup?¶

Plan capacity with some slack.

I’d suggest going for the 80/20 rule, meaning the weekly release should go out, even in the unavailability of 20% human resources.

Unlike machines, people will be sick, go out for vacation, emergency travel plans, unexpectedly resign, etc. anything can happen. So, like any other robust system design, plan for “disaster”.

Is proper Continuous Integration/Deployment (CI/CD) in place?¶

To have an efficient release process, automated and well-documented CI/CD pipeline is almost a prerequisite.

Developers across the board should be able to build, test, merge their code, and build the binary from the latest stable and/or release checkpoints.
Once their change, be it a feature or a bugfix, is ready to be merged with the release branch, a complete suite of automated tests should run on the resultant binary to detect any regression bug due to the change; with no dependency on the release team.
QA, while testing for a particular release, should be able to receive regression or production build without any manual effort from the release team.

Is there continuous feedback established?¶

An efficient release process is an intensive process. And, no process can be perfect from the beginning.

Every organization has, as its fingerprint, has different priorities, deliverables, audience, resources, and of course, budget. So, while the release cycle can start with an agile blueprint, down the line it needs to be well fit on its organizational body.

Here comes the necessity for continuous feedback from the participants, stakeholders, and even the release team-members.

Stay on the “good book” of the app stores ;)¶

If the app is in the center of revenue of your company, keep a close eye on the regular comm from the app stores, i.e., Google Play and Apple Appstore. It’s very easy to miss these communications, among other day-to-day stuff.

Regardless of how ridiculous your company feels about some policies on the app stores, please read between the lines. In case of confusion, read again. Especially if you’re one of the big fishes.
If a particular app store weren’t strictly monitoring or enforcing some policies last month, don’t even think they aren’t going to enforce tomorrow and reject your app.

The release manager or the team, being the interface with the app stores, should frequently communicate with the stakeholders and the product team to carefully go through relevant app store policies long before they invest company time and resources into a feature or a roadmap. Trust me; it happens more than anyone likes to admit that after month-long design and development, the app update with the feature gets rejected on the app stores, solely due to this new feature.

If possible, try to get hold of and build a working relationship with the business development or dev support teams of Google and Apple. It might help in case of any misunderstanding and/or some negotiations about the compliance timeline.
I can’t go into any more detail though ;)

Notes:¶

Please don’t take the number 50M+ literally. The actual number is much higher but should be considered as company secrets ;). Right?
This article is NOT a definitive process guide, and I’m not an expert on Agile methodologies. I’m just trying to share my learning from my release experience of 1.75 years. Some stuff might help any other release manager on the other side of the planet. That’s the goal here.

Thanks, y’all.

Install latest Python 3 on Linux CentOS 7

2018-07-07T00:00:00+06:00

Why¶

Not all distro created equal.
Some are created to join the space race, some are to hold unto the leagcy, some are cutting-edge, some are cutting edge. Some are born to boot-up IoT devices some are to push out heavy graphics.

That’s the fun (albeit, power) of Linux.

CentOS 7 is a powerful and stable distro that runs on thousands (probably, millions) production-grade servers.
In the matter of stability, it’s a beast. However, it doesn’t ship with Python 3, by default. You can install it via EPEL repository, or the below simple steps.

Also, take a note. The Python 2 comes with the system, which is probably 2.7.5, do NOT mess with it. Many system components rely on that specific version. If you need the latest versions of 2, use virtualenv or pipenv.

Prepare your system¶

Start with installing pre-requisite utilities for compilation and development support.

$ sudo yum update && sudo yum groupinstall -y "development tools"  
$ sudo yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel \
       sqlite-devel readline-devel tk-devel gdbm-devel \
       db4-devel libpcap-devel xz-devel expat-devel

Download latest Python source code from Python.org¶

$ wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz
$ tar xf Python-3.6.6.tar.xz
$ cd Python-3.6.6

Enable performance optimizations (optional, but highly recommended)¶

$ ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"  
$ ./configure --enable-optimizations

Build and install¶

$ make
$ sudo make altinstall

Now, Python 3.6.6 is ready to be used in your system; located in /usr/local/bin/python3.6

$ which python3.6
/usr/local/bin/python3.6

$ python3.6
Python 3.6.6 (default, Jul 10 2018, 14:04:26)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

For convenience, you can create a symbolic-link with a shorter name. If you had system-installed Python3 (unlikely), don’t do this, as some system-components may depend on that specific older version of Python 3.

$ sudo ln -s /usr/local/bin/python3.6 /usr/local/bin/python3
$ python3
Python 3.6.6 (default, Jul 10 2018, 14:04:26)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Install wheel and pip¶

You don’t need to, because Python 3.6.6 includes these necessary tools included.

$ pip3.6 -V
pip 10.0.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
$ wheel version
wheel 0.29.0

HA(High-Availability) Setup for InfluxDB

2018-01-18T00:00:00+06:00

NOTE Since I have written this article, all the components used in this below architecture have gone through many updates and releases. While the general premise involving influxdb-relay and the multiplexing might still hold, please sync up with the latest release docs before jumping into some serious system design.

Currently, from version 0.9, you cannot create an InfluxDB cluster from the open-sourced free edition. Only commercially available InfluxDB Enterprise can do that for now. That stirred up the early-adopter enthusiast users, especially for their usage in professional setups. They complained that InfluxData, the company behind InfluxDB, is trying to milk the OSS solution for profit.

I can’t blame the InfluxData guys much, as they got to pay their bills too. So far, we — the users of open-source systems — couldn’t show much promise about the financial realities of the projects. Continuing development of OSS products, by only depending on donations, patrons, or enterprise sponsorship, is far too rare and unpredictable, even for the projects that many successful organizations heavily rely on.

Anyways, InfluxDB then promised and later introduced Influx Relay as a complimentary consolation for missing HA parts of InfluxDB. You can get the details here and here about that.

Premise¶

For my needs, I have to try to create a reliable HA(High-Availability) setup from available free options, hence InfluxDB and the relay. It’s quite a bit far from an InfluxDB-cluster in terms of robustness or ease of setup, but it’s got the job done, at least for me.

I needed a setup to receive system-stats from at least 500+ instances and to store them for a while, but without breaking the bank in bills from AWS. Meaning, I could ask for and could use only couple of instances for my solution.

Here were my trade-offs.

Not too many instances for this purpose. Neither, any of the heavyweight lifters e.g. AWS’ m3-xlarge etc. To use only what’s necessary.
To satisfy the budget, hence avoiding pay-per-use solutions as far as it is possible.
Solutions must not be crazy complex, so that handover to the DevOps team be smooth.
Reading the data would be too rarely w.r.t. writing. The related Grafana dashboards will be only used to investigate issues by a handful of people.

Overall Design¶

Write¶

From a birds’ eye view, I decided to use two server instances to run parallelly, hosting InfluxDB on them independently and then sending the same data over to them for storing. This scheme mostly looks like RAID-1 systems.

That brings up a couple of challenges.

None of the agents I used on the sender side could multiplex output. That means, they were able to send data to a single destination, not multiple. On the Windows front, I’ve used Telegraf which is able randomly to switch between pre-listed destinations, but NOT multiple at-once.
In the case of Linux hosts, I used Netdata which is excellent in its own right, but unable to send stats to multiple destinations.
Here comes Influx-relay. It can receive time-series data-stream from hosts on a TCP or UDP port, buffer for a while, and then re-send those received and buffered data to multiple receive ends which can either be an InfluxDB instance or another listening Influx-relay instances.
This chaining can broaden the relaying scheme even further. However, for my purpose, this relay-chaining was not necessary. Rather, from the relay, I am sending data to the separate InfluxDB instances, running on two separate instances.
Now that I partially multiplexed the output, my hosts (senders) still are able to send to one destination. So, I need a proxy as well as a load-balancer. For a while, I was torn between NGINX and HAProxy. Both were new to me.

However, for a couple of reasons, I went for HAProxy. Firstly, I don’t need HTTP session management. Secondly, as I wanted to keep my UDP for later, HAProxy was perfectly capable of that.
NGINX has the support recently, but the maturity was a concern. Also, configuring NGINX seems a little intimidating (which I know might not be so true). Last but not least, and for what it’s worth, out-of-the-box, HAProxy’s stat page carries much more in-depth information than that of free-version of NGINX.
Upon receiving the stats stream, HAProxy was supposed to send that to different Influx-relays in a load-balanced fashion.

So, here’s my rough plan.

collector-agent → HAProxy → (50/50 load-balanced) → Influx-relay → (multiplexed) → 2 InfluxDB instances

Now, each one of the received data is to go to both of the InfluxDB instances, or at least to one in case of failure (or, overload per se) of any the relays or Influx instances. Also, I have chosen to keep Influx-relays deployed as Dockerized and kept HAProxy and InfluxDB instances running as native services. Of course, you can Dockerize HAProxy and InfluxDB, too.

Read¶

As I’ve already noted in the section that reading the data, meaning to fetch data to visualize on Grafana end, will happen rarely and sporadically; only to investigate alarms or any other client-side performance issues.

So, the read requests, reaching the HAProxy end, needed not much routing, other than directly to InfluxDB itself. Still, to better distribute the load I decided to load-balance it 50/50 basis.

Ports¶

As all the READ requests are routed through HAProxy running on each of the instances, to the external world only HAProxy’s port should be opened for this purpose.
On the other hand, for WRITE requests, InfluxDBs are receiving data from relays, one of its own instance and another one on other instance, so InfluxDB should listen on its own port for WRITE requests only. But, this must be accessible only from own VPS zone, but not open to the outside world.
In case of HAProxy as well as InfluxDB, you can use the default ports, obviously, which is 8086 & 8088 respectively. Or, you can choose to go for other ports (security through obfuscation). Your call. In this writing, I’ll go with the defaults.

Authentication, SSL¶

You can configure SSL with your own server certificates through the HAProxy configs. You can even go for SSL from the relays to InfluxDB writes. If your sender hosts are connecting to your HAProxy through public internet, you should at least go for password-based authentication, better to utilize SSL. However, for brevity’s sake, I’ll skip them in this post.

**Note: * Please bear in mind, this is an “in-progress” post; prematurely published to force me to work on it. I have the plan to add all the necessary configurations & commands, that I used, here.

Pelican Static sites - SEO Optimization

2017-01-07T00:00:00+06:00

Writing is a hard job, especially it is your hobby beside of your day job. But what’s the benefit if nobody reads it just because they couldn’t find it.

Usually Pelican static-site generator is not very concerned about SEO of the generated site mainly because it’s not that focused on commercial usage. That’s what I felt. Related themes and their templates also don’t take it much seriously. But you shouldn’t loose SEO just because you migrated from Wordpress or whatever you were using previously, right?

Often times theme-authors focus more on look-n-feel of the theme, but not so much on the SEO concerns.

That’s this blog about; let’s fix that.

Note 1: About other static-site generators¶

Although the following discussions & codes mostly are specific to Pelican templates which uses jinja2 templating language, the concepts and concerns here are applicable to most static-site generators and their themes.

Note 2: I’m, in no way, an SEO expert¶

This writeup is just a collection of my findings while correcting my blog’s SEO course; fixing the stupid mistakes. Also, this isn’t a commercial site. There are lot more and in-depth aspects of SEO optimization other than the following, that can be very important for commercial projects.

Getting started¶

Lookup for missing pieces¶

Make sure all the linked resources(links, images, CSS & JS files) that you’ve used in your pages are valid. For that check browser’s console in Developers tools for any errors e.g. unavailable urls, faulty html/css etc.

Know the critical spots¶

pelicanconf.py - It’s usually in your root folder of the site.
base.html - It’s in the templates folder of the theme folder that you are using. Changes here will impact all HTML pages generated by Pelican.
article.html - In same folder as base.html. Changes here will impact only articles’ pages.

Avoiding Duplication¶

Avoid your source getting indexed by Google¶

If the source of your blog is not sourced-open, meaning the content is not in a open-sourced repo, this isn’t your concern. But if it is, it should be a concern.

The reason is that Github.com(or where your repo is hosted on) is stronger domain than yours, so Google “sees” that your site-contents (even though it’s yours) also is on Github. So, there’s a high chance that it’ll mark your contents as “duplicate”. And, duplicate contents gets heavy hammers from Google’s SEO point-of-view. Searching any writeup from yours, even it’s very unique on Internet, Google search will show both your site and the repo as well, possibly links from your site will be on the lower side.

It begs the question how to avoid that. It’s not complicated.

For example, Github.com’s robot.txt allows Google (or any search engine for that matter) to index only the master branch of any open-source repo. So, if your repo don’t have any branch named master, it won’t be indexed. That’s it. Rename your “master” as “live”, “main”, “production” or whatever you feel like.

Create a new “master”

$ git checkout -b new-master
$ git push -u origin new-master

Tell Github about new move

Now, delete master branch from your repo.

$ git branch -d master
$ git push origin :master

Utilize `rel="canonical"` link¶

This is kind of a must-do for avoiding being marked as duplicate content. This will also defend you against automatic content-scraping schemes by always having a pointer to your original source.

To utilize it on Pelican, add or make sure that a <link> element with the attribute rel="canonical" to the <head> section of your base template named base.html.

{% if article %}
<link rel="canonical" href="{{ SITEURL }}/{{ article.url }}"/>
{% endif%}

For example, if you see this page’s source (pressing CTRL+u), you should see something like

<link rel="canonical" href="https://blog.kmonsoor.com/pelican-how-to-make-seo-friendly/"/>

Proper < title > of each page¶

Every page on your site should have a proper title. For search engines, it represents the page. It should concisely reflect a page’s content. But try to keep it less 60 characters or search-engines may choose to truncate it. Use each characters wisely.

It may look like this in your base.html.

{% if article %}
<title>{{ article.title }} -- {{ TAGLINE }}</title>
{% else %}
<title>{{ TAGLINE }}</title>
{% endif%}

The else clause here is to ensure that non-article pages also get a title, even it’s just your tagline defined in pelicanconf.py.

Meta descriptions¶

Include a meta-description to be added on each page of your site. Though it may don’t directly hit SEO ranking, but it appears as a snippet on the search page. So, user should get a proper glimpse of what your page gonna talk about. Make sure your theme uses Pelican’s summary-tagged text for this purpose. Else, ensure it yourself by editing base.html.

{% if article and article.summary %}
<meta name="description" content="{{ article.summary|striptags }}"/>
{% else %}
<meta name="description" content="{{ SITE_SUMMARY }}"/>
{% endif%}

Use `search-console` extensively¶

Google’s Search-console, previously known as webmaster-tools is your friend. Utilize it as far as you can go.

Extensively use PageSpeed Insights ¶

To understand where are current bottlenecks of your site, this tools gives quite a lot insights. Address those one-by-one.

Set preferred version of your site¶

If you have www, http and https versions of your site, tell Google here which one is preferred. It’s only applicable to your domain-root. Once applied and Google re-indexed your site, all the search-results from your site will show that preferred version of your site.

You have to add and do it same both for http and https version of your site, if you have both.

If you have both, either you can use a javascript-code snippet in the <head> of base.html to redirect any http page to its https counterpart.

<script type="text/javascript">
  var host = "your-site.com";
  if ((host == window.location.host) && (window.location.protocol != "https:"))
    window.location.protocol = "https";
</script>

or, if your site is served through NGINX or Apache, you can do it through site’s .htaccess file, by adding the following.

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

Or, if you are using CloudFlare CDN, you can create a page-rule for that as I have shown below.

Check index-status¶

Once in a while Check for Google’s index status of your site on the search-console. Look for error messages or suggestions.

After every major change in your site’s structure, make sure Google bots “know” about it. You can somewhat expedite the process by manually submitting your site.

Include `OpenGraph` data¶

Make sure each of your pages is including proper OpenGraph tags, e.g. og:title, og:content etc., in your template.

Though, OpenGraph originated from facebook Inc., these tags are now widely used by other social engines, even by Google+. In absence of Twitter tags, Twitter also uses these og tags. Try to include both og: and twitter: tags. Proper data in these tags makes your article cleanly-sharable in these social sites.

The below snippet that I use myself can serve as a starting point.

<!-- OpenGraph protocol tags: http://ogp.me/ -->
<!-- originally adopted to be used for: https://blog.kmonsoor.com -->
<meta property="og:site_name" content="{{ SITENAME }}" />
<meta property="og:type" content="article" />
{% if article and article.title %}    
<meta property="og:title" content="{{ article.title }} -- {{ TAGLINE }}" />
<meta property="og:url" content="{{ SITEURL }}/{{ article.url }}" />
{% endif%}
{% if article and article.summary %}
<meta property="og:description" content="{{ article.summary|striptags }}" />
{% else %}
<meta name="og:description" content="{{ SITE_SUMMARY }}"/>
{% endif%}
{% if article and article.date %}
<meta property="article:published_time" content="{{ article.date }}" />
{% endif%}
{% if article and article.modified %}
<meta property="article:modified_time" content="{{ article.modified }}" />
{% endif%}
<!-- End of OpenGraph protocol tags -->

{% if TWITTER_USERNAME %}
<meta name="twitter:site" content="@{{ TWITTER_USERNAME }}" />
<meta name="twitter:creator" content="@{{ TWITTER_USERNAME }}" />
{% endif%}
<meta name="twitter:image" content="INSERT-YOUR-TWITTER-IMAGE-LINK" />
{% if article and article.summary %}
<meta name="twitter:card" content="{{ article.summary|striptags }}" />
{% else %}
<meta name="twitter:card" content="{{ SITE_SUMMARY }}"/>
{% endif%}

Notes¶

Grab your own Twitter’s avatar link do the following:
Go to your Twitter profile page
Right-click on your profile picture
Select “Copy image address” / “Copy image link”
For OpenGraph tags you may also consider to use pelican-opengraph plugin.
For all these to work properly, make sure SITEURL, TAGLINE, SITE_SUMMARY, TWITTER_USERNAME are properly defined in your pelicanconf.py alongwith in publishconf.py files. Please remember that definitions in publishconf.py only apply when you using make publish command.

Loading Performance¶

Compress everything¶

PageSpeed impacts SEO directly. Google punishes slow-site especially when search is made on a mobile device. Mobile-optimized sites will definitely rank higher on searches from mobile-devices.

So, make sure all static files are compressed. If not, compress your themes theme’s JS, CSS files yourself to a .min. version and then replace those in the template files of the theme.

Or, better to use gzip_cache for gzipping all the HTML files statically, also and yuicompressor plugin for compressing JS & CSS files for Pelican. Those will make sure that, upon build, everything is compressed.

Utilize CDN if you can¶

Use CDN-ed versions of common libraries(e.g. jQuery, Bootstrap etc.) rather than hosting your own copy, unless your theme actively modified it. Look it up on CloudFlare cdnjs, cdnjs, or on jsdelivr etc. and use those links.
Try to use a CDN for edge-distribution of your site. I only know of CloudFlare that provide this service for free for a single site. There might be others. CF also make managing DNS configuration little breezy.

Engage commenting¶

While serving a static site, integrating a commenting-system looks a little far-fetched. However, blogs without proper commenting system feels kinda lame sometimes. Of course, YMMV.

But, it’s not difficult; easily can be done by systems like Disqus etc. I’m not affiliated with them, by the way.

Host images separately¶

Host all the images separately that you’ve used in your articles. Use image-specific hosting e.g. imgur.com, imgpile.com, UltraIMG.com, postimage.org etc.

But, why? Because, these services provide couple of benefits besides being free.

Firstly, while loading the page, browser can parallelize loading from this hosts rather than your original blog hosts.
More often than not, these services use own CDN.
Often these services resize your uploaded images automatically to be used in different contexts, which enables you to choose the best-fit size on the fly but without doing it by hand.

Other tips¶

Name your images properly¶

Because search-engines index images too. With proper names, images becomes relevant with the topic, hence potential to draw traffic.

Use Google Keyword planner¶

Even if you are not willing to blow money on ads, it will immensely help you to find out more searched for keywords.

Page-headers (controversial)¶

Don’t use multiple 1^st-level headers H1 style. The main title link probably already have used it once. Look it up. So, avoid it anymore, meaning avoid underlined-style(=======) or hash-style(single ‘#’) headers in your markdown files.

However, 2^nd-level H2 tags can be (read ‘should be’) used multiple times. In case of markdown files, that’s --------- underlines, or line starting with double-hash(‘##’).

And, there’s lot more … ;)¶

As I’ll gain more insights, I hope to grow this post. For now, this is a work-in-progress.
Thanks for reading down so far. Adios !

Ubuntu - Update APT-GET Server to Use Fastest Mirror

2016-10-10T00:00:00+06:00

Prelude¶

By default, Ubuntu sets the update server pointing to its own (http://archive.ubuntu.com). It is the safest bet for Ubuntu. But, that’s not the case for users, especially who are outside USA. To make it smoother as well as distribute the load, Ubuntu also provides a list of mirror sites. You can find the official mirror list here.

However, it needs some configuration.

On GUI¶

Yes, you can do the selection on Ubuntu using its GUI tool. But, the problem is it don’t always work as you want.

It works on geolocation, giving me the local server, which is waaaayy slower where I am. The network temporal distance is the important factor here, not spatial distance (http://askubuntu.com/a/9035/113604)

so, what else do we have ?¶

I use a handy tool for this purpose to point me to the fastest server FOR ME. It’s apt-select.

To install it, i found this method most hassle-free. You may also try pip install method described on the repo’s README file.

$ git clone https://github.com/jblakeman/apt-select.git

That’s it. Now to execute it, just run the “main” Python file.

$ cd apt-select
$ ./apt-select.py -c -t 3 -m one-week-behind

We are choosing here the best 3 mirrors(due to -t 3) which are at most one week behind from the main Ubuntu server. For general purposes, that’s good enough.

It will choose one using the latency & ping time, and also show servers’ bandwidth. Then the tool asks you to select new mirror from the 3 options came up. Usually, stick to the top option, hence 1. For example in my case, it shows:

[khaled:~] $ apt-select -c -t 3 -m one-week-behind
Getting list of mirrors...done.
Testing latency to mirror(s)
[3/3] 100%
Getting list of launchpad URLs...done.
Looking up 3 status(es)
[3/3] 100%
1. mirror.dhakacom.com (current)
    Latency: 1.89 ms
    Org:     dhakaCom Limited
    Status:  Up to date
    Speed:   1 Gbps
2. mirror.dhakacom.com
    Latency: 1.96 ms
    Org:     dhakaCom Limited
    Status:  Up to date
    Speed:   1 Gbps
3. archive.ubuntu.com
    Latency: 205.23 ms
    Org:     Canonical Ltd.
    Status:  Up to date
    Speed:   100 Mbps
Choose a mirror (1 - 3)
'q' to quit 1
New config file saved to /home/khaled/sources.list

Then, the tool prepares a new sources.list file to replace the system’s one with.

Now, replace the file with the new one which will be now used by the apt-get * commands. As it overwrites a system config file, you need to do with sudo.

$ sudo mv /etc/apt/sources.list /etc/apt/sources.list.backup && sudo mv sources.list /etc/apt/

That’s it. Now, all you update/upgrade will be fetched from the newly selected mirror.

However, I highly recommend to run update once right away.

$ sudo apt-get update

Then, only if you want, run upgrade:

$ sudo apt-get upgrade