Tag Cloud
golang telegram bot python cli urwid elasticsearch aws ecs apache spark scala AWS EMR hadoop webhooks ssl nginx digital-ocean emr apache pig datapipeline found.io elastic-cloud rails try capybara docker devops capistrano heka bigquery kafka protobuf vim iterm javascript emberjs git scripting dnsmasq bem frontend meteorjs meteorite heroku

Page 1 of 4
Older Posts

Golang Telegram Bot - Migrations, Cronjobs & Refactors
Written by Ho Man Aug 20, 2017 7 minutes read

This post is kind of like a continuation from the previous posts of my Golang Telegram Bot, so if you haven’t seen that yet, it’s probably better to start with those first: part 1 and part 2. I basically wanted my telegram bot to be able to remember dated / timed reminders and send messages to notify me when that time comes (like a calendar). Furthermore...

Read more →

Building a Python CLI Stock Ticker with Urwid
Written by Ho Man Jun 28, 2017 11 minutes read

A bit of context - I do some investing in equities on the side and I’ve always wanted to build a simple stock ticker in the form of a CLI app that runs in my terminal setup. There were a few out there but none that would show just the information I needed, in a minimalistic fashion. And I thought it would be a fun project for me since I don’t have...

Read more →

Upgrading to ElasticSearch 5.2.2 on Amazon ECS
Written by Ho Man Apr 03, 2017 5 minutes read

In one of my previous post, I talked about how I set up Elasticsearch 2.3.5 on ECS. I got a comment in that post that prompted me to update the setup for Elasticsearch 5. It’s been awhile, but better late than never right? So I gave it a go! In this post I’ll like to share what I found in the process.

There were a couple of other configuration changes...

Read more →

Tuning My Apache Spark Data Processing Cluster on Amazon EMR
Written by Ho Man Mar 10, 2017 14 minutes read

Lately, I had the chance to work on some data integration at Pocketmath where I wrote a bunch of Spark scripts in Scala to run some transformations on a data set of about 250GB that will run on a monthly basis. In this post, I talk about some of the problems I encountered, and some considerations while setting up the cluster and also how I improved...

Read more →

How I Deployed My Golang Telegram Bot
Written by Ho Man Jan 21, 2017 13 minutes read

Continuing where I left off in the first part of my Golang Telegram Bot, in this post I go through all the steps I took to get to a one command deployment of my Telegram bot into a Digital Ocean Ubuntu 16.04 instance. A number of components were involved: dockerizing the app, setting up a self-signed SSL cert, get the Nginx to work as a reverse proxy...

Read more →

How I Built a Simple Telegram Bot in Go
Written by Ho Man Dec 25, 2016 20 minutes read

This is the first part of the Golang Telegram Bot series. In this series, I’ll show you, with code samples, how I built a Golang Telegram Bot for my own use. It would listen in and respond in real-time to certain text cues. Finally I’ll also show you how to get a self-signed SSL cert working with Nginx and deploying the application in a Docker...

Read more →

Preventing Duplicates: Amazon EMR Pig to Elasticsearch
Written by Ho Man Dec 24, 2016 7 minutes read

In the previous post, I went through some steps I took to set up the Amazon EMR Hadoop cluster to run Apache Pig scripts for indexing data to Elasticsearch. In today’s series, I walk through some of the problems I encountered when I set the system up and some of the configuration tweaks to both Elasticsearch and the EMR cluster that I feel you should...

Read more →

How to set up Amazon EMR Hadoop with Pig to index to Elasticsearch
Written by Ho Man Dec 14, 2016 10 minutes read

In one of my recent posts, I briefly talked about using Apache Pig, to index an Elasticsearch cluster. In this post, I do a walkthrough of the DevOps configurations and steps I took, along with the code that was required to get it work at the start (barring some issues that I’ll talk about in the next post).

Production Setup

The process starts...

Read more →

How to set up Elasticsearch Cluster in Amazon ECS
Written by Ho Man Dec 05, 2016 9 minutes read

At Pocketmath, we heavily utilize the EC2 container service (ECS) to host a significant portion of our applications. It provides us with an easily scalable, zero-downtime infrastructure. Recently, I upgraded the Elasticsearch to 2.3.5 for our clusters, so I thought it was a good idea just to jot down some of the things I had to do or was already...

Read more →

AWS Elasticsearch, Elastic Cloud vs Self-managed
Written by Ho Man Nov 22, 2016 5 minutes read

From past experience, I found the maintenance and tuning of a Elastisearch cluster to be a little troublesome overtime. So it isn’t surprising to see hosted Elasticsearch services pop up one after another. Ok, to be fair, there are hosted services for nearly everything nowadays, from Kafka to Wordpress lol. There is really no shortage of them. Most...

Read more →