Tag Cloud
Concourse CI/CD devops prometheus kubernetes monitoring modbus kepware c-programming IoT golang telegram bot python cli urwid elasticsearch aws ecs apache spark scala AWS EMR hadoop webhooks ssl nginx digital-ocean emr apache pig datapipeline found.io elastic-cloud rails try capybara docker capistrano heka bigquery kafka protobuf vim iterm javascript emberjs git scripting dnsmasq bem frontend meteorjs meteorite heroku

Page 2 of 4
Newer Posts Older Posts

How to set up Amazon EMR Hadoop with Pig to index to Elasticsearch
Written by Ho Man Dec 14, 2016 10 minutes read

In one of my recent posts, I briefly talked about using Apache Pig, to index an Elasticsearch cluster. In this post, I do a walkthrough of the DevOps configurations and steps I took, along with the code that was required to get it work at the start (barring some issues that I’ll talk about in the next post).

Production Setup

The process starts...

Read more →

How to set up Elasticsearch Cluster in Amazon ECS
Written by Ho Man Dec 05, 2016 9 minutes read

At Pocketmath, we heavily utilize the EC2 container service (ECS) to host a significant portion of our applications. It provides us with an easily scalable, zero-downtime infrastructure. Recently, I upgraded the Elasticsearch to 2.3.5 for our clusters, so I thought it was a good idea just to jot down some of the things I had to do or was already...

Read more →

AWS Elasticsearch, Elastic Cloud vs Self-managed
Written by Ho Man Nov 22, 2016 5 minutes read

From past experience, I found the maintenance and tuning of a Elastisearch cluster to be a little troublesome overtime. So it isn’t surprising to see hosted Elasticsearch services pop up one after another. Ok, to be fair, there are hosted services for nearly everything nowadays, from Kafka to Wordpress lol. There is really no shortage of them. Most...

Read more →

Nil, Try & The Lonely Operator
Written by Ho Man Jul 28, 2016 3 minutes read

Recently, I left a comment on one of my colleague’s PR and we had a discussion with him about the use of try vs the lonely operator &. and it led to a number of conclusions personally.

I used to use lots of .try. I’ve also come across codebases littered with it, be it in the presentation layer or in the models. From personal experience, I’ll say...

Read more →

Capybara & Waiting
Written by Ho Man Jul 27, 2016 4 minutes read

All of us do TDD or at least some form of automated testing, I hope! If you’re writing tests in Rails, you’re likely to be doing feature tests with Capybara as well.

Some of these slipped my mind while adding feature specs at work at pocketmath and I spent extra time that I shouldn’t have! So I hope this post can be a reminder to myself in future...

Read more →

Setting up Dockerized Golang + Postgres on Digital Ocean
Written by Ho Man Apr 27, 2016 5 minutes read

My previous deploy on the Rails stack was a little more involved so I chose to just deploy it in the conventional capistrano way after setting the server up.

But with the rise in popularity of Docker recently, I’ve been wanting to deploy something into production with Docker but never found the right app for it until this one.

It was a simple Golang...

Read more →

Capistrano, Postgres, Rails, Nginx & Puma on DigitalOcean
Written by Ho Man Jan 22, 2016 10 minutes read

Recently, I’ve been working on my squash club, UCSC’s new site. And of course, being slightly short of time, I kinda just fell back on Rails to quickly get something up for the club.

Before Heroku decided to put a 7 USD price on their free tier, it was an easy default for hosting any mini prototypes or projects. Ok I admit, I’ve historically used...

Read more →

Kafka + Heka to Bigquery Setup for Realtime Logs
Written by Ho Man Jul 04, 2015 4 minutes read

This is part 2 of the 3 part series of a quick intro of the realtime logging system in Wego. In Part 1 of this series, I talked about how we generated Go packages for Protobuf (protocol buffers) in Wego.

The packages allows us to process 2 different modes of communication protocol with just one set of schema defined in proto files:

  • json package...
Read more →

Golang + Protobuf
Written by Ho Man May 05, 2015 6 minutes read

(This is part 1 of the 3 part series where I discuss how we handled one of our new API projects and the usage of Protobuf + Heka + Kafka + BigQuery to view realtime logs.)

Recently, I’ve had the chance to work with Golang in Wego to build our analytics API that will log our visits into flat files + upload them into Kafka for further processing,...

Read more →

Predefined Structs vs Type Assertion in Go
Written by Ho Man May 21, 2014 5 minutes read

Suppose you have a JSON of this format:

    "collection": [
        "name": { first_name: "Ho Man", last_name: "Boa"
        "phone": "123456"
        "name": { first_name: "John", last_name: "Lee"
        "phone": "2345"


There are essentially 2 methods to do this, using pre...

Read more →