Tag Cloud
elasticsearch aws ecs apache spark scala AWS EMR hadoop golang telegram bot webhooks ssl nginx digital-ocean emr apache pig datapipeline found.io elastic-cloud rails try capybara docker devops capistrano heka bigquery kafka protobuf vim iterm javascript emberjs git scripting dnsmasq bem frontend meteorjs meteorite heroku

Page 1 of 3
Older Posts

Upgrading to ElasticSearch 5.2.2 on Amazon ECS
Written by Ho Man Apr 03, 2017 5 minutes read

In one of my previous post, I talked about how I set up Elasticsearch 2.3.5 on ECS. I got a comment in that post that prompted me to update the setup for Elasticsearch 5. It’s been awhile, but better late than never right? So I gave it a go! In this post I’ll like to share what I found in the process.

There were a couple of other configuration changes...

Read more →

Tuning My Apache Spark Data Processing Cluster on Amazon EMR
Written by Ho Man Mar 10, 2017 14 minutes read

Lately, I had the chance to work on some data integration at Pocketmath where I wrote a bunch of Spark scripts in Scala to run some transformations on a data set of about 250GB that will run on a monthly basis. In this post, I talk about some of the problems I encountered, and some considerations while setting up the cluster and also how I improved...

Read more →

How I Deployed My Golang Telegram Bot
Written by Ho Man Jan 21, 2017 13 minutes read

Continuing where I left off in the first part of my Golang Telegram Bot, in this post I go through all the steps I took to get to a one command deployment of my Telegram bot into a Digital Ocean Ubuntu 16.04 instance. A number of components were involved: dockerizing the app, setting up a self-signed SSL cert, get the Nginx to work as a reverse proxy...

Read more →

How I Built a Simple Telegram Bot in Go
Written by Ho Man Dec 25, 2016 20 minutes read

This is the first part of the Golang Telegram Bot series. In this series, I’ll show you, with code samples, how I built a Golang Telegram Bot for my own use. It would listen in and respond in real-time to certain text cues. Finally I’ll also show you how to get a self-signed SSL cert working with Nginx and deploying the application in a Docker...

Read more →

Preventing Duplicates: Amazon EMR Pig to Elasticsearch
Written by Ho Man Dec 24, 2016 7 minutes read

In the previous post, I went through some steps I took to set up the Amazon EMR Hadoop cluster to run Apache Pig scripts for indexing data to Elasticsearch. In today’s series, I walk through some of the problems I encountered when I set the system up and some of the configuration tweaks to both Elasticsearch and the EMR cluster that I feel you should...

Read more →

How to set up Amazon EMR Hadoop with Pig to index to Elasticsearch
Written by Ho Man Dec 14, 2016 10 minutes read

In one of my recent posts, I briefly talked about using Apache Pig, to index an Elasticsearch cluster. In this post, I do a walkthrough of the DevOps configurations and steps I took, along with the code that was required to get it work at the start (barring some issues that I’ll talk about in the next post).

Production Setup

The process starts...

Read more →

How to set up Elasticsearch Cluster in Amazon ECS
Written by Ho Man Dec 05, 2016 9 minutes read

At Pocketmath, we heavily utilize the EC2 container service (ECS) to host a significant portion of our applications. It provides us with an easily scalable, zero-downtime infrastructure. Recently, I upgraded the Elasticsearch to 2.3.5 for our clusters, so I thought it was a good idea just to jot down some of the things I had to do or was already...

Read more →

AWS Elasticsearch, Elastic Cloud vs Self-managed
Written by Ho Man Nov 22, 2016 5 minutes read

From past experience, I found the maintenance and tuning of a Elastisearch cluster to be a little troublesome overtime. So it isn’t surprising to see hosted Elasticsearch services pop up one after another. Ok, to be fair, there are hosted services for nearly everything nowadays, from Kafka to Wordpress lol. There is really no shortage of them. Most...

Read more →

Nil, Try & The Lonely Operator
Written by Ho Man Jul 28, 2016 3 minutes read

Recently, I left a comment on one of my colleague’s PR and we had a discussion with him about the use of try vs the lonely operator &. and it led to a number of conclusions personally.

I used to use lots of .try. I’ve also come across codebases littered with it, be it in the presentation layer or in the models. From personal experience, I’ll say...

Read more →

Capybara & Waiting
Written by Ho Man Jul 27, 2016 4 minutes read

All of us do TDD or at least some form of automated testing, I hope! If you’re writing tests in Rails, you’re likely to be doing feature tests with Capybara as well.

Some of these slipped my mind while adding feature specs at work at pocketmath and I spent extra time that I shouldn’t have! So I hope this post can be a reminder to myself in future...

Read more →