browse by category or date


Three weeks ago, I was given a challenge to develop simple web application using Go during weekend. I excitedly accepted it. I mean what could go wrong? I have readskimmed the docs and Go have a somewhat familiar syntax with C#/JavaScript.

As it turned out, I failed spectacularly. I badly overestimate how much I understand Go. So here I am, retracing back the Go path. I managed to borrow this book from NLB, “The Go Programming Language” as my main reference.



The first question that came to my mind is how to get Go installed on this blog server. It’s really easy.

wget https://dl.google.com/go/go1.13.linux-amd64.tar.gz

Then extract the archive to /usr/local:

tar -C /usr/local -xzf go1.13.linux-amd64.tar.gz

Then add the bin folder to $PATH

export PATH=$PATH:/usr/local/go/bin

Make it permanent by adding above line to your bash profile

echo "export PATH=\$PATH:/usr/local/go/bin" >> .bashrc

Reload the bash profile

source ~/.bashrc

Test that Go is running

go version

Now that Go is installed and running properly, I need to find a way to showcase whatever Go is doing in this blog. Luckily, it’s super easy to create a simple web server in Go.

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", handler) // each request calls handler	
	log.Fatal(http.ListenAndServe("localhost:21000", nil))
}

// handler echoes the Path component of the requested URL.
func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "URL.Path = %q\n", r.URL.Path)
}

Next, we link this Go webserver with NGINX by adding this lines to NGINX’s sites configuration

location /cgo-bin/ {
        valid_referers none blocked sodeve.net;

        if ($invalid_referer){
            return 403;
        }   
        proxy_pass         http://127.0.0.1:21000/;
        proxy_redirect     off;
    }

For any address under /cgo-bin/ 😀, NGINX will relay the request to Go’s server above.

Now we reached the part where we are generating the Lissajous curve. Below code is straight copy-pasted from the reference book above.

package main

import (
	"errors"
	"fmt"
	"image"
	"image/color"
	"image/gif"
	"log"
	"math"
	"math/rand"
	"net/http"
)

func main() {	
	http.HandleFunc("/lissajous", handleLissajous)	
	log.Fatal(http.ListenAndServe("localhost:21000", nil))
}

func handleLissajous(w http.ResponseWriter, r *http.Request) {
	var palette = []color.Color{color.White, color.Black}

	const (
		whiteIndex = 0     // first color in palette
		blackIndex = 1     // next color in palette
		cycles     = 5     // number of complete x oscillator revolutions
		res        = 0.001 // angular resolution
		size       = 100   // image canvas covers [-size .. + size]
		nframes    = 64    // number of animation frames
		delay      = 8     //delay between frames in 10ms units
	)
	freq := rand.Float64() * 3.0 // relative frequency of y oscillator
	anim := gif.GIF{LoopCount: nframes}

	phase := 0.0 //phase difference
	for i := 0; i < nframes; i++ {
		rect := image.Rect(0, 0, 2*size+1, 2*size+1)
		img := image.NewPaletted(rect, palette)
		clr := uint8(1)
		for t := 0.0; t < cycles*2*math.Pi; t += res {
			x := math.Sin(t)
			y := math.Sin(t*freq + phase)
			img.SetColorIndex(size+int(x*size+0.5), size+int(y*size+0.5), clr)
		}
		phase += 0.1
		anim.Delay = append(anim.Delay, delay)
		anim.Image = append(anim.Image, img)
	}
	gif.EncodeAll(w, &anim) //ignore error

}

The resulting Lissajous curve image is below (if no image shown below, it means my Go server is down 😛 ):

From here, we can play with the color:

And we can even use a user-defined gradient color:

Color From:
Color To:

Stay tune for my future adventure in Go!

About Hardono

Hi, I'm Hardono. I am working as a Software Developer. I am working mostly in Windows, dealing with .NET, conversing in C#. But I know a bit of Linux, mainly because I need to keep this blog operational. I've been working in Logistics/Transport industry for more than 9 years.

I am planning to get a new laptop. And its RAM must be at least 32 GB. So I visited some prominent e-commerce websites which are based in Singapore. To my frustration, none of these websites allows me to filter the laptops based on the RAM size.

My first immediate instinct was telling me to scrape the data from these websites, compile then share it here using JSON Table Editor. But I have a small worry that this kind of activity is deemed illegal in Singapore. So I googled around, and found this enlightening quora answer by Bernard Chung. Bernard says that there are four laws that could apply to web scraping activity:

  1. Contract Law
  2. Tort Law
  3. Intellectual Property Law
  4. the Computer Misuse and Cybersecurity Act

So, which of the law above that could expose me to legal problem should I carry on with web scraping? I’m not planning to use any illegal means to access their website (e.g. exploit vulnerabilities, SQL-Injection, hacking/cracking), so law no 4 should not be relevant to this. What about Intellectual Property Law? Since I am only after the data, not other things that might classify as their Intellectual Property, law no 3 should not be relevant. What about Tort Law? Since I will access their website just like a normal human do (albeit much much faster), I should not be considered as trespassing.

So it’s only left to Contract Law. If these websites put out a Terms and Conditions (TOC) or Term of Service (TOS) which specifically forbid automated data extraction, my action will now be considered as illegal.

I guess now I need to find out which of these websites are specifically forbid automated data extraction. Here’s is my result:

  1. COURTS
    Allow Filter by RAM Size: NO
    Filter Laptops by: Price, Brand, Customer Rating, Laptop Type, Screen Size, Processor and Color
    Allow Data Scraping: NO

    You may not extract and/or re-use the Brand Names or parts of the content of COURTS Online without our express written consent. In particular, you may not use any data mining, robots, or similar data gathering and extraction tools to extract (whether once or many times) for re-use of any substantial parts of the content of COURTS Online, without our express written consent. You may also not create and/or publish your own database that features substantial parts of COURTS Online (e.g. prices and product listings) without our express written consent.

    COURTS CONDITIONS OF USE AND BUYER AGREEMENT

  2. HARVEY NORMAN
    Allow Filter by RAM Size: NO
    Filter Laptops by: Price, Brand, Processor and Type
    Allow Data Scraping: No explicit prohibition regarding automated data extraction (Term Of Use)

  3. LAZADA
    Allow Filter by RAM Size: NO
    Filter Laptops by: Price, Brand, Customer Rating, Laptop Type, Screen Size, Service and Location
    Allow Data Scraping: No explicit prohibition regarding automated data extraction (T&C)

  4. BEST DENKI
    Allow Filter by RAM Size: NO
    Filter Laptops by: Price, and Brand
    Allow Data Scraping: No explicit prohibition regarding automated data extraction (T&C)

Now that I’ve got the legal foundation covered, my next step is to actually build the web-scraper code. See you again soon!

About Hardono

Hi, I'm Hardono. I am working as a Software Developer. I am working mostly in Windows, dealing with .NET, conversing in C#. But I know a bit of Linux, mainly because I need to keep this blog operational. I've been working in Logistics/Transport industry for more than 9 years.

Recently I found this data laying around in my Projects folder. I believe I scrapped Wikipedia to have this data. Clearly there is no much use of this data at the moment. Maybe just to showcase how JSON Table Editor handles unicode characters.

Anyway, enjoy this historical data.

About Hardono

Hi, I'm Hardono. I am working as a Software Developer. I am working mostly in Windows, dealing with .NET, conversing in C#. But I know a bit of Linux, mainly because I need to keep this blog operational. I've been working in Logistics/Transport industry for more than 9 years.